us.codecraft.webmagic.downloader

Class HttpClientDownloader

java.lang.Object

us.codecraft.webmagic.downloader.AbstractDownloader

us.codecraft.webmagic.downloader.HttpClientDownloader

All Implemented Interfaces:

Downloader

@ThreadSafe
public class HttpClientDownloader
extends AbstractDownloader

The http downloader based on HttpClient.

Since:

0.1.0

Author:

code4crafter@gmail.com

Constructor Summary

Constructors

Constructor and Description
HttpClientDownloader()

Method Summary

Modifier and Type	Method and Description
Page	download(Request request, Task task) Downloads web pages and store in Page object.
protected Page	handleResponse(Request request, String charset, org.apache.http.HttpResponse httpResponse, Task task)
void	setHttpUriRequestConverter(HttpUriRequestConverter httpUriRequestConverter)
void	setProxyProvider(ProxyProvider proxyProvider)
void	setThread(int thread) Tell the downloader how many threads the spider used.

Methods inherited from class us.codecraft.webmagic.downloader.AbstractDownloader

download, download, onError, onSuccess

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HttpClientDownloader

public HttpClientDownloader()

Method Detail

setHttpUriRequestConverter

public void setHttpUriRequestConverter(HttpUriRequestConverter httpUriRequestConverter)

setProxyProvider

public void setProxyProvider(ProxyProvider proxyProvider)

download

public Page download(Request request,
                     Task task)

Description copied from interface: Downloader

Downloads web pages and store in Page object.

Parameters:

request - request

task - task

Returns:

page

setThread

public void setThread(int thread)

Description copied from interface: Downloader

Tell the downloader how many threads the spider used.

Parameters:

thread - number of threads

handleResponse

protected Page handleResponse(Request request,
                              String charset,
                              org.apache.http.HttpResponse httpResponse,
                              Task task)
                       throws IOException

Throws:

IOException