com.yishuifengxiao.common.crawler.scheduler.remover

Class HostDuplicateRemover

java.lang.Object

com.yishuifengxiao.common.crawler.scheduler.remover.HostDuplicateRemover

All Implemented Interfaces:

DuplicateRemover

public class HostDuplicateRemover
extends Object
implements DuplicateRemover

无查询参数去重器
去除URL上所有的查询参数的简单去重器

Version:

1.0.0

Author:

qingteng

Constructor Summary

Constructors

Constructor and Description
HostDuplicateRemover()

Method Summary

Modifier and Type	Method and Description
void	doWhenNoDuplicate(Task task, RequestCache requestCache, Request request) 当前请求没有重复时需要进行的操作，一般来说，只需将该请求存入请求任务缓存器即可
boolean	noDuplicate(Task task, RequestCache requestCache, Request request) 判断当前请求是否重复

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HostDuplicateRemover

public HostDuplicateRemover()

Method Detail

noDuplicate

public boolean noDuplicate(Task task,
                           RequestCache requestCache,
                           Request request)

Description copied from interface: DuplicateRemover

判断当前请求是否重复

Specified by:

noDuplicate in interface DuplicateRemover

Parameters:

task - 当前任务信息

requestCache - 请求任务缓存器

request - 当前需要判断的请求

Returns:

true表示当前请求没有存在过，需要调度器调度，以便后期进行该请求,false表示该请求已经进行过，调度器会抛弃该请求

doWhenNoDuplicate

public void doWhenNoDuplicate(Task task,
                              RequestCache requestCache,
                              Request request)

Description copied from interface: DuplicateRemover

当前请求没有重复时需要进行的操作，一般来说，只需将该请求存入请求任务缓存器即可

Specified by:

doWhenNoDuplicate in interface DuplicateRemover

Parameters:

task - 当前任务信息

requestCache - 请求任务缓存器

request - 当前需要判断的请求