상세한 내용: https://blog.michaelyin.info/web-scraping-framework-review-scrapy-vs-selenium/
Conclusion
So which one is better web scraping framwork? There is no solid answer, the answer depends heavily on the actual situation. Below is a quick reference table.
Framework | Selenium | Scrapy |
|---|---|---|
Javascript Support | Support javascript very well | It is time consuming to inspect and develop spider to simulate ajax/pjax requests |
| Data Size | Good option for small data set | Work fine on big data set |
| Ecosystem | Not many related projects or plugins | Many related projects, plugins on open source websites such as Github, and many discussions on StackOverflow can help you fix the potential issue. |
| Extensibility | Not very easy to extend the project | You can easily develop custom middleware or pipeline to add custom function, easy to maintain. |