us.codecraft.webmagic.scheduler

Class RedisScheduler

java.lang.Object

us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

us.codecraft.webmagic.scheduler.RedisScheduler

All Implemented Interfaces:

DuplicateRemover, MonitorableScheduler, Scheduler

Direct Known Subclasses:

RedisPriorityScheduler

public class RedisScheduler
extends DuplicateRemovedScheduler
implements MonitorableScheduler, DuplicateRemover

Use Redis as url scheduler for distributed crawlers.

Since:

0.2.0

Author:

code4crafter@gmail.com

Field Summary

Fields

Modifier and Type	Field and Description
protected redis.clients.jedis.JedisPool	pool

Fields inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

logger

Constructor Summary

Constructors

Constructor and Description
RedisScheduler(redis.clients.jedis.JedisPool pool)
RedisScheduler(String host)

Method Summary

Modifier and Type	Method and Description
protected String	getItemKey(Task task)
int	getLeftRequestsCount(Task task)
protected String	getQueueKey(Task task)
protected String	getSetKey(Task task)
int	getTotalRequestsCount(Task task) Get TotalRequestsCount for monitor.
boolean	isDuplicate(Request request, Task task) Check whether the request is duplicate.
Request	poll(Task task) get an url to crawl
protected void	pushWhenNoDuplicate(Request request, Task task)
void	resetDuplicateCheck(Task task) Reset duplicate check.

Methods inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface us.codecraft.webmagic.scheduler.Scheduler

push

Field Detail

pool

protected redis.clients.jedis.JedisPool pool

Constructor Detail

RedisScheduler

public RedisScheduler(String host)

RedisScheduler

public RedisScheduler(redis.clients.jedis.JedisPool pool)

Method Detail

resetDuplicateCheck

public void resetDuplicateCheck(Task task)

Description copied from interface: DuplicateRemover

Reset duplicate check.

Specified by:

resetDuplicateCheck in interface DuplicateRemover

Parameters:

task - task

isDuplicate

public boolean isDuplicate(Request request,
                           Task task)

Description copied from interface: DuplicateRemover

Check whether the request is duplicate.

Specified by:

isDuplicate in interface DuplicateRemover

Parameters:

request - request

task - task

Returns:

true if is duplicate

pushWhenNoDuplicate

protected void pushWhenNoDuplicate(Request request,
                                   Task task)

Overrides:

pushWhenNoDuplicate in class DuplicateRemovedScheduler

poll

public Request poll(Task task)

Description copied from interface: Scheduler

get an url to crawl

Specified by:

poll in interface Scheduler

Parameters:

task - the task of spider

Returns:

the url to crawl

getSetKey

protected String getSetKey(Task task)

getQueueKey

protected String getQueueKey(Task task)

getItemKey

protected String getItemKey(Task task)

getLeftRequestsCount

public int getLeftRequestsCount(Task task)

Specified by:

getLeftRequestsCount in interface MonitorableScheduler

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

Description copied from interface: DuplicateRemover

Get TotalRequestsCount for monitor.

Specified by:

getTotalRequestsCount in interface DuplicateRemover

Specified by:

getTotalRequestsCount in interface MonitorableScheduler

Parameters:

task - task

Returns:

number of total request