Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Python3爬虫系列的理论验证,首先研究I/O模型,分别用Python实现了blocking I/O、nonblocking I/O、I/O multiplexing各模型下的TCP服务端和客户端。然后,研究同步I/O操作(依序下载、多进程并发、多线程并发)和异步I/O(asyncio)之间的效率差别

Notifications You must be signed in to change notification settings

wangy8961/python3-concurrency

Open more actions menu

Repository files navigation

Python aiohttp BeautifulSoup4 requests pymongo progressbar2

1. 爬虫系列

2. 使用方法

2.1 Server

为防止DDoS攻击,本次测试需要在本地搭建一个HTTP测试服务器,具体方法参考 Python3爬虫系列03 (实验) - 同步阻塞下载

2.2 Client

(1) 下载代码

[root@CentOS ~]# git clone https://github.com/wangy8961/python3-concurrency.git
[root@CentOS ~]# cd python3-concurrency/

(2) 准备虚拟环境

如果你的操作系统是Linux:

[root@CentOS python3-concurrency]# python3 -m venv venv3
[root@CentOS python3-concurrency]# source venv3/bin/activate

Windows激活虚拟环境的命令是: venv3\Scripts\activate

(3) 安装依赖包

如果你的操作系统是Linux:

(venv3) [root@CentOS python3-concurrency]# pip install -r requirements-linux.txt

如果你的操作系统是Windows(不会使用uvloop):

(venv3) C:\Users\wangy> pip install -r requirements-win32.txt

(4) 测试

依序下载:

(venv3) [root@CentOS python3-concurrency]# python sequential.py

多进程下载:

(venv3) [root@CentOS python3-concurrency]# python processpool.py

多线程下载:

(venv3) [root@CentOS python3-concurrency]# python threadpool.py

异步下载:

(venv3) [root@CentOS python3-concurrency]# python asynchronous.py

About

Python3爬虫系列的理论验证,首先研究I/O模型,分别用Python实现了blocking I/O、nonblocking I/O、I/O multiplexing各模型下的TCP服务端和客户端。然后,研究同步I/O操作(依序下载、多进程并发、多线程并发)和异步I/O(asyncio)之间的效率差别

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

Morty Proxy This is a proxified and sanitized view of the page, visit original site.