Crawling Articles from Newspapers

• get_rss_providers method from DatbaseHandler class

1. enter list of RSS sources in DB
2. Crawl RSS feeds of given resources
3. persist a <uri (PK), title, source> tuple in DB
4. [do a prefiltering]
5. crawl URIs and fetch articles
6. extract articles
7. persist article body in DB

do it all somehow paralell

• Project interpreter will be python3
• try to maintain PEP8 style convention
• make sure your ide uses the .editorconfig

install with pip3 -r requirements.txt

Git: https://github.com/codelucas/newspaper Walkthrough: Newspaper Crawling.ipynb (Jupyter/iPython Notebook) Adding a new source: https://github.com/codelucas/newspaper/blob/master/docs/user_guide/advanced.rst