In the chapter One, we reference the components of WebMagic. WebMagic the biggest advantage is that you can customize the function of components flexible to achieve the function you want to have.
In the Spider class,
Pipeline are the fields of Spider class. Beside the
PageProcessor have been assigned when the Spider create, the other three components can be changed by the setter method.
|setScheduler()||Change the Scheduler||spipder.setScheduler(new FileCacheQueueScheduler("D:\data\webmagic"))|
|setDownloader()||Change the Downloader||spipder.setDownloader(new SeleniumDownloader()))|
|addDownloader()||Change the Pipeline，one Spider can have a few of pipeline||spipder.addPipeline(new FilePipeline())|