us.codecraft.webmagic.scheduler

Class BloomFilterDuplicateRemover

java.lang.Object

us.codecraft.webmagic.scheduler.BloomFilterDuplicateRemover

All Implemented Interfaces:

DuplicateRemover

public class BloomFilterDuplicateRemover
extends Object
implements DuplicateRemover

BloomFilterDuplicateRemover for huge number of urls.

Since:

0.5.1

Author:

code4crafer@gmail.com

Constructor Summary

Constructors

Constructor and Description
BloomFilterDuplicateRemover(int expectedInsertions)
BloomFilterDuplicateRemover(int expectedInsertions, double fpp)

Method Summary

Modifier and Type	Method and Description
int	getTotalRequestsCount(Task task) Get TotalRequestsCount for monitor.
protected String	getUrl(Request request)
boolean	isDuplicate(Request request, Task task) Check whether the request is duplicate.
protected com.google.common.hash.BloomFilter<CharSequence>	rebuildBloomFilter()
void	resetDuplicateCheck(Task task) Reset duplicate check.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions)

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions,
                                   double fpp)

Parameters:

expectedInsertions - the number of expected insertions to the constructed

fpp - the desired false positive probability (must be positive and less than 1.0)

Method Detail

rebuildBloomFilter

protected com.google.common.hash.BloomFilter<CharSequence> rebuildBloomFilter()

isDuplicate

public boolean isDuplicate(Request request,
                           Task task)

Description copied from interface: DuplicateRemover

Check whether the request is duplicate.

Specified by:

isDuplicate in interface DuplicateRemover

Parameters:

request - request

task - task

Returns:

true if is duplicate

getUrl

protected String getUrl(Request request)

resetDuplicateCheck

public void resetDuplicateCheck(Task task)

Description copied from interface: DuplicateRemover

Reset duplicate check.

Specified by:

resetDuplicateCheck in interface DuplicateRemover

Parameters:

task - task

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

Description copied from interface: DuplicateRemover

Get TotalRequestsCount for monitor.

Specified by:

getTotalRequestsCount in interface DuplicateRemover

Parameters:

task - task

Returns:

number of total request