• Requests
• GRequests

• BeautifulSoup, doc
• lxml
• html5lib
• html.parser (standard module)
• ElementTree (standard module)
• xml.dom
• xml.dom.minidom

• Selenium Python

• PhantomJS

• RoboBrowser

• mechanize

• MechanicalSoup

• ghost.py - webkit web client

• windmill

• Scrapy

• Grab

• Xvfb- virtual framebuffer for headless browsing

https://realpython.com/blog/python/headless-selenium-testing-with-python-and-phantomjs/

https://github.com/dhamaniasad/HeadlessBrowsers

http://twill.idyll.org/

https://code.google.com/archive/p/pywebkitgtk/

https://github.com/teddziuba/stanislaw

http://n1k0.github.io/casperjs/

https://github.com/makinacorpus/spynner

https://www.gnu.org/software/pythonwebkit/

http://webdriver.io/

https://www.blog.pythonlibrary.org/2012/05/06/website-automation-with-python-firefox-and-selenium/

• webscraper.io

http://mydataprovider.com/2016/06/28/compare-web-scraping-services-by-price/

https://www.palantir.com/

https://data-miner.io/

http://www.fminer.com/

https://quickcode.io/

https://www.import.io/builder/

• Scraper - Chrome plugin

• Scrapely
• Portia
• ScrapingHub

https://scrapinghub.com/crawlera/

https://scrapinghub.com/scrapy-cloud/

https://blog.scrapinghub.com/2016/09/28/how-to-run-python-scripts-in-scrapy-cloud/

http://stackoverflow.com/questions/2861/options-for-html-scraping

http://htql.net/

http://arshaw.com/scrapemark/

https://pythonhosted.org/pyquery/

See also

• beautifulsoup
• requests
• scrapy
• selenium
• __scraping__