com.yishuifengxiao.common.crawler.scheduler.remover

Class SimpleDuplicateRemover

java.lang.Object

com.yishuifengxiao.common.crawler.scheduler.remover.SimpleDuplicateRemover

All Implemented Interfaces:

DuplicateRemover

public class SimpleDuplicateRemover
extends Object
implements DuplicateRemover

全路径去重器
简单实现的请求去重器

Version:

1.0.0

Author:

qingteng

Constructor Summary

Constructors

Constructor and Description
SimpleDuplicateRemover()

Method Summary

Modifier and Type	Method and Description
void	doWhenNoDuplicate(Task task, RequestCache requestCache, Request request) 当前请求没有重复时需要进行的操作，一般来说，只需将该请求存入请求任务缓存器即可
boolean	noDuplicate(Task task, RequestCache requestCache, Request request) 判断当前请求是否重复

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleDuplicateRemover

public SimpleDuplicateRemover()

Method Detail

noDuplicate

public boolean noDuplicate(Task task,
                           RequestCache requestCache,
                           Request request)

Description copied from interface: DuplicateRemover

判断当前请求是否重复

Specified by:

noDuplicate in interface DuplicateRemover

Parameters:

task - 当前任务信息

requestCache - 请求任务缓存器

request - 当前需要判断的请求

Returns:

true表示当前请求没有存在过，需要调度器调度，以便后期进行该请求,false表示该请求已经进行过，调度器会抛弃该请求

doWhenNoDuplicate

public void doWhenNoDuplicate(Task task,
                              RequestCache requestCache,
                              Request request)

Description copied from interface: DuplicateRemover

当前请求没有重复时需要进行的操作，一般来说，只需将该请求存入请求任务缓存器即可

Specified by:

doWhenNoDuplicate in interface DuplicateRemover

Parameters:

task - 当前任务信息

requestCache - 请求任务缓存器

request - 当前需要判断的请求