Project Task: Manga Crawler and Manga Viewer This is project, we will build a simple system for manga reading. The system contains two separated modules: a Manga Crawler and a Manga Viewer. Programming languages requirements: Python. You can use TCP, UDP for communication

For this module, you can choose one or many of the following sites to get data: truyentranh.net, or truyentranhtuan.com. You may suggest any other manga site. The crawl module needs to get info of at least 100 manga For each manga, the following need to be stored:

• Manga name
• Author
• Description
• Categories
• Last update
• Chapter name and list of pages The info must be stored in JSON file manga_info.json. A sample file is provided

Site to get data: truyentranh.net 128 manga crawled and stored in manga_info.json

Instructions:

Programming language: Python 3

Firstly, I declare Spider class and create a spider to perform the crawl and extract structured data from the site. A spider should have behaviours such as initialize, crawl, gather_urls, …

The manga_info.json instruction:

After trying and failing, I found the most effective method to crawl more than 100 manga from this site.

The crawlers can be divided into 3 modules:

• Preprocessing: from http://truyentranh.net/danh-sach.tall.html, the spider gather manga urls in alphabetical order, with 5 manga urls whose title start with each letter.
• Manga crawling: the spider visits each manga url gathered in above step, gathers manga informations (name, thumbnail, author, description, categories, last update) and all the chapter urls on that page.
• Chapter crawling: the spider visits each chapter url gathered in above step, gather chapter name and all the contents (images) on that page.

Each step splited into 3 step:

• Extract: HTTP request, fetch the HTML (and resolve the domain).
• Transform: take features out of the HTML (title, images, content…) + run NLP algorithms.
• Store: save to the JSON.

Build a simple webserver to feed data for web browser to read manga base on the info constructed in the above module The webserver read info from manga_info.json and construct the html:

• /allmanga.html: list of all manga, name, author and author of each manga are shown
• /manga_info.html?id=<manga_id>: show detailed info of a manga with manga_id include manga name, description, author, categories, chapter list….
• /chapter.html?id=<chapter_id>: view all the page of a chapter with chapter_id