us.codecraft.webmagic.scheduler

Interface Scheduler

All Known Subinterfaces:

MonitorableScheduler

All Known Implementing Classes:

DelayQueueScheduler, DuplicateRemovedScheduler, FileCacheQueueScheduler, LevelLimitScheduler, PriorityScheduler, QueueScheduler, RedisPriorityScheduler, RedisScheduler

public interface Scheduler

Scheduler is the part of url management.
You can implement interface Scheduler to do: manage urls to fetch remove duplicate urls

Since:

0.1.0

Author:

code4crafter@gmail.com

Method Summary

Modifier and Type	Method and Description
Request	poll(Task task) get an url to crawl
void	push(Request request, Task task) add a url to fetch

Method Detail

push

void push(Request request,
          Task task)

add a url to fetch

Parameters:

request - request

task - task

poll

Request poll(Task task)

get an url to crawl

Parameters:

task - the task of spider

Returns:

the url to crawl