us.codecraft.webmagic.scheduler

Class PriorityScheduler

java.lang.Object

us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

us.codecraft.webmagic.scheduler.PriorityScheduler

All Implemented Interfaces:

MonitorableScheduler, Scheduler

Direct Known Subclasses:

DelayQueueScheduler, LevelLimitScheduler

@ThreadSafe
public class PriorityScheduler
extends DuplicateRemovedScheduler
implements MonitorableScheduler

Priority scheduler. Request with higher priority will poll earlier.

Since:

0.2.1

Author:

code4crafter@gmail.com

Field Summary

Fields

Modifier and Type	Field and Description
static int	INITIAL_CAPACITY

Fields inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

logger

Constructor Summary

Constructors

Constructor and Description
PriorityScheduler()

Method Summary

Modifier and Type	Method and Description
int	getLeftRequestsCount(Task task)
int	getTotalRequestsCount(Task task)
Request	poll(Task task) get an url to crawl
void	pushWhenNoDuplicate(Request request, Task task)

Methods inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface us.codecraft.webmagic.scheduler.Scheduler

push

Field Detail

INITIAL_CAPACITY

public static final int INITIAL_CAPACITY

See Also:

Constant Field Values

Constructor Detail

PriorityScheduler

public PriorityScheduler()

Method Detail

pushWhenNoDuplicate

public void pushWhenNoDuplicate(Request request,
                                Task task)

Overrides:

pushWhenNoDuplicate in class DuplicateRemovedScheduler

poll

public Request poll(Task task)

Description copied from interface: Scheduler

get an url to crawl

Specified by:

poll in interface Scheduler

Parameters:

task - the task of spider

Returns:

the url to crawl

getLeftRequestsCount

public int getLeftRequestsCount(Task task)

Specified by:

getLeftRequestsCount in interface MonitorableScheduler

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

Specified by:

getTotalRequestsCount in interface MonitorableScheduler