us.codecraft.webmagic.scheduler

Class FileCacheQueueScheduler

java.lang.Object

us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

us.codecraft.webmagic.scheduler.FileCacheQueueScheduler

All Implemented Interfaces:

Closeable, AutoCloseable, MonitorableScheduler, Scheduler

public class FileCacheQueueScheduler
extends DuplicateRemovedScheduler
implements MonitorableScheduler, Closeable

Store urls and cursor in files so that a Spider can resume the status when shutdown.

Since:

0.2.0

Author:

code4crafter@gmail.com

Field Summary

Fields inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

logger

Constructor Summary

Constructors

Constructor and Description
FileCacheQueueScheduler(String filePath)

Method Summary

Modifier and Type	Method and Description
void	close()
int	getLeftRequestsCount(Task task)
int	getTotalRequestsCount(Task task)
Request	poll(Task task) get an url to crawl
protected void	pushWhenNoDuplicate(Request request, Task task)

Methods inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface us.codecraft.webmagic.scheduler.Scheduler

push

Constructor Detail

FileCacheQueueScheduler

public FileCacheQueueScheduler(String filePath)

Method Detail

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

pushWhenNoDuplicate

protected void pushWhenNoDuplicate(Request request,
                                   Task task)

Overrides:

pushWhenNoDuplicate in class DuplicateRemovedScheduler

poll

public Request poll(Task task)

Description copied from interface: Scheduler

get an url to crawl

Specified by:

poll in interface Scheduler

Parameters:

task - the task of spider

Returns:

the url to crawl

getLeftRequestsCount

public int getLeftRequestsCount(Task task)

Specified by:

getLeftRequestsCount in interface MonitorableScheduler

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

Specified by:

getTotalRequestsCount in interface MonitorableScheduler