us.codecraft.webmagic.scheduler

Class RedisPriorityScheduler

java.lang.Object

us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

us.codecraft.webmagic.scheduler.RedisScheduler

us.codecraft.webmagic.scheduler.RedisPriorityScheduler

All Implemented Interfaces:

DuplicateRemover, MonitorableScheduler, Scheduler

public class RedisPriorityScheduler
extends RedisScheduler

the redis scheduler with priority

Author:

sai Created by sai on 16-5-27.

Field Summary

Fields inherited from class us.codecraft.webmagic.scheduler.RedisScheduler

pool

Fields inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

logger

Constructor Summary

Constructors

Constructor and Description
RedisPriorityScheduler(redis.clients.jedis.JedisPool pool)
RedisPriorityScheduler(String host)

Method Summary

Modifier and Type	Method and Description
Request	poll(Task task) get an url to crawl
protected void	pushWhenNoDuplicate(Request request, Task task)
void	resetDuplicateCheck(Task task) Reset duplicate check.

Methods inherited from class us.codecraft.webmagic.scheduler.RedisScheduler

getItemKey, getLeftRequestsCount, getQueueKey, getSetKey, getTotalRequestsCount, isDuplicate

Methods inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface us.codecraft.webmagic.scheduler.Scheduler

push

Constructor Detail

RedisPriorityScheduler

public RedisPriorityScheduler(String host)

RedisPriorityScheduler

public RedisPriorityScheduler(redis.clients.jedis.JedisPool pool)

Method Detail

pushWhenNoDuplicate

protected void pushWhenNoDuplicate(Request request,
                                   Task task)

Overrides:

pushWhenNoDuplicate in class RedisScheduler

poll

public Request poll(Task task)

Description copied from interface: Scheduler

get an url to crawl

Specified by:

poll in interface Scheduler

Overrides:

poll in class RedisScheduler

Parameters:

task - the task of spider

Returns:

the url to crawl

resetDuplicateCheck

public void resetDuplicateCheck(Task task)

Description copied from interface: DuplicateRemover

Reset duplicate check.

Specified by:

resetDuplicateCheck in interface DuplicateRemover

Overrides:

resetDuplicateCheck in class RedisScheduler

Parameters:

task - task