Commits

Breadcrumbs

History for

onmaster

User selector

All users

All time

Commit history

Commits on Sep 25, 2024

• 修复response 判空问题

  Boris-code

  committed
  df1d2d3

  Copy full SHA for df1d2d3

Commits on Mar 19, 2024

• 支持python3.12

  Boris-code

  committed
  210c4a3

  Copy full SHA for 210c4a3

Commits on Mar 18, 2024

• 钉钉报警支持指定用户id，爬虫代码支持提示

  Boris-code

  committed
  033feba

  Copy full SHA for 033feba

Commits on Sep 5, 2023

• 修复 taskspider 不回调start_callback的bug

  Boris-code

  committed
  821094e

  Copy full SHA for 821094e

Commits on Aug 30, 2023

• 修复重新导入失败item的bug

  Boris-code

  committed
  5f06a5c

  Copy full SHA for 5f06a5c

Commits on Aug 18, 2023

• 优化报警中的包名名

  Boris-code

  committed
  486f94a

  Copy full SHA for 486f94a

Commits on Jul 10, 2023

• fixed spider end log

  Boris-code

  committed
  66ea30d

  Copy full SHA for 66ea30d

• fixed spider end log

  Boris-code

  committed
  1f393cd

  Copy full SHA for 1f393cd

• fixed spider end log

  Boris-code

  committed
  cc31fdd

  Copy full SHA for cc31fdd

Commits on Jun 28, 2023

• 完善代理池，支持删除代理

  Boris-code

  committed
  a1aa394

  Copy full SHA for a1aa394

Commits on Mar 29, 2023

• 修复exception_request及failed_request参数适配问题

  Boris-code

  committed
  a937378

  Copy full SHA for a937378

Commits on Mar 17, 2023

• task spider 可选择是否使用mysql

  Boris-code

  committed
  d495f7c

  Copy full SHA for d495f7c

Commits on Mar 7, 2023

• version 1.8.6b2, fix _stop property conflict with thread

  Boris-code

  committed
  90b1741

  Copy full SHA for 90b1741

• version 1.8.6b1, support use stop_spider method to stop spider

  Boris-code

  committed
  51e22ef

  Copy full SHA for 51e22ef

Commits on Nov 29, 2022

• fix airspider task queue max size bug

  Boris-code

  committed
  7ff9c79

  Copy full SHA for 7ff9c79

• response为空时不经过校验函数

  Boris-code

  committed
  6f97e5b

  Copy full SHA for 6f97e5b

Commits on Nov 25, 2022

• 修复任务爬虫依赖其他爬虫的bug

  Boris-code

  committed
  f74f5d9

  Copy full SHA for f74f5d9

Commits on Nov 22, 2022

• 修复有依赖爬虫时，依赖爬虫不结束，新批次开启不了的bug

  Boris-code

  committed
  910a06d

  Copy full SHA for 910a06d

Commits on Nov 20, 2022

• Task spider 支持检查依赖爬虫的状态，依赖的爬虫做完才可结束自己

  Boris-code

  committed
  b7a983c

  Copy full SHA for b7a983c

Commits on Nov 18, 2022

• 命令行工具支持retry，可重试失败的请求或者item

  Boris-code

  committed
  009b819

  Copy full SHA for 009b819

• 支持重新导入失败的item

  Boris-code

  committed
  13b751f

  Copy full SHA for 13b751f

• 批次爬虫支持设置不自动启动下一批次

  Boris-code

  committed
  0992fde

  Copy full SHA for 0992fde

Commits on Nov 2, 2022

• 修复下载中间件中自定义返回response时，response.browser属性不存在导致异常的bug

  Boris-code

  committed
  d539ce6

  Copy full SHA for d539ce6

Commits on Oct 28, 2022

• 优化批次爬虫采集速度计算逻辑

  Boris-code

  committed
  ef4164f

  Copy full SHA for ef4164f

Commits on Oct 27, 2022

• 判断爬虫是否停滞时，增加判断redis中是否有任务的条件

  Boris-code

  committed
  3527aa1

  Copy full SHA for 3527aa1

• 1. 批次超时报警后，若后续批次完成，则发个批次完成的报警，提醒已恢复正常 2. 精简代码

  Boris-code

  committed
  771862d

  Copy full SHA for 771862d

Commits on Oct 26, 2022

• 修复爬虫心跳bug

  Boris-code

  committed
  f4e9e6d

  Copy full SHA for f4e9e6d

Commits on Oct 25, 2022

• air spider 支持去重

  Boris-code

  committed
  051c2ae

  Copy full SHA for 051c2ae

Commits on Oct 24, 2022

• 修复批次报警误报的bug

  Boris-code

  committed
  6ed6da5

  Copy full SHA for 6ed6da5

Commits on Oct 20, 2022

• 批次表的时间类型统一为datetime

  Boris-code

  committed
  e6d34a6

  Copy full SHA for e6d34a6

• spider的debug模式支持保存数据到数据库

  Boris-code

  committed
  1ce1758

  Copy full SHA for 1ce1758

Commits on Sep 29, 2022

• 下载器使用自定义配置生效

  Boris-code

  committed
  49d2eb7

  Copy full SHA for 49d2eb7

Commits on Sep 28, 2022

• exception_request及failed_request透传异常参数e

  Boris-code

  committed
  9b2d1a7

  Copy full SHA for 9b2d1a7

Commits on Sep 22, 2022

• 封装BatchParser

  Boris-code

  committed
  49949f8

  Copy full SHA for 49949f8

• 优化代码

  Boris-code

  committed
  6570d3f

  Copy full SHA for 6570d3f