Package | Description |
---|---|
us.codecraft.webmagic |
Main class "Spider" and models.
|
us.codecraft.webmagic.configurable | |
us.codecraft.webmagic.downloader |
Downloader is the part that downloads web pages and store in Page object.
|
us.codecraft.webmagic.downloader.selenium | |
us.codecraft.webmagic.example | |
us.codecraft.webmagic.handler | |
us.codecraft.webmagic.model |
Page model and annotations used to customize a crawler.
|
us.codecraft.webmagic.model.samples | |
us.codecraft.webmagic.processor |
PageProcessor custom part of a crawler for specific site.
|
us.codecraft.webmagic.processor.example | |
us.codecraft.webmagic.proxy | |
us.codecraft.webmagic.samples | |
us.codecraft.webmagic.samples.scheduler | |
us.codecraft.webmagic.scripts |
Modifier and Type | Method and Description |
---|---|
static Page |
Page.fail() |
Page |
SimpleHttpClient.get(Request request) |
Page |
SimpleHttpClient.get(String url) |
Page |
Page.setRawText(String rawText) |
Page |
Page.setSkip(boolean skip) |
Modifier and Type | Method and Description |
---|---|
protected void |
Spider.extractAndAddRequests(Page page,
boolean spawnUrl) |
Modifier and Type | Method and Description |
---|---|
void |
ConfigurablePageProcessor.process(Page page) |
Modifier and Type | Method and Description |
---|---|
Page |
PhantomJSDownloader.download(Request request,
Task task) |
Page |
HttpClientDownloader.download(Request request,
Task task) |
Page |
Downloader.download(Request request,
Task task)
Downloads web pages and store in Page object.
|
protected Page |
HttpClientDownloader.handleResponse(Request request,
String charset,
org.apache.http.HttpResponse httpResponse,
Task task) |
Modifier and Type | Method and Description |
---|---|
Page |
SeleniumDownloader.download(Request request,
Task task) |
Modifier and Type | Method and Description |
---|---|
void |
GithubRepoPageMapper.process(Page page) |
Modifier and Type | Method and Description |
---|---|
void |
CompositePageProcessor.process(Page page) |
RequestMatcher.MatchOther |
SubPageProcessor.processPage(Page page)
process the page, extract urls to fetch, extract the data and store
|
Modifier and Type | Method and Description |
---|---|
void |
AfterExtractor.afterProcess(Page page) |
T |
PageMapper.get(Page page) |
List<T> |
PageMapper.getAll(Page page) |
Modifier and Type | Method and Description |
---|---|
void |
OschinaAnswer.afterProcess(Page page) |
void |
DianpingFtlDataScanner.afterProcess(Page page) |
Modifier and Type | Method and Description |
---|---|
void |
SimplePageProcessor.process(Page page) |
void |
PageProcessor.process(Page page)
process the page, extract urls to fetch, extract the data and store
|
Modifier and Type | Method and Description |
---|---|
void |
ZhihuPageProcessor.process(Page page) |
void |
GithubRepoPageProcessor.process(Page page) |
void |
BaiduBaikePageProcessor.process(Page page) |
Modifier and Type | Method and Description |
---|---|
void |
SimpleProxyProvider.returnProxy(Proxy proxy,
Page page,
Task task) |
void |
ProxyProvider.returnProxy(Proxy proxy,
Page page,
Task task)
Return proxy to Provider when complete a download.
|
Modifier and Type | Method and Description |
---|---|
void |
ZhihuPageProcessor.process(Page page) |
void |
TianyaPageProcesser.process(Page page) |
void |
SinaBlogProcessor.process(Page page) |
void |
QzoneBlogProcessor.process(Page page) |
void |
PhantomJSPageProcessor.process(Page page) |
void |
NjuBBSProcessor.process(Page page) |
void |
MeicanProcessor.process(Page page) |
void |
MamacnPageProcessor.process(Page page) |
void |
KaichibaProcessor.process(Page page) |
void |
IteyeBlogProcessor.process(Page page) |
void |
InfoQMiniBookProcessor.process(Page page) |
void |
HuxiuProcessor.process(Page page) |
void |
GithubRepoPageProcessor.process(Page page) |
void |
F58PageProcesser.process(Page page) |
void |
DiaoyuwengProcessor.process(Page page) |
void |
DiandianBlogProcessor.process(Page page) |
void |
AngularJSProcessor.process(Page page) |
void |
AmanzonPageProcessor.process(Page page) |
void |
AlexanderMcqueenGoodsProcessor.process(Page page) |
Modifier and Type | Method and Description |
---|---|
void |
ZipCodePageProcessor.process(Page page) |
Modifier and Type | Method and Description |
---|---|
void |
ScriptProcessor.process(Page page) |
Copyright © 2017. All rights reserved.