Commits

Commits on Nov 19, 2024

重新打包了单一的库文件
 gnuos
committed
f2f15bd
Copy full SHA for f2f15bd
把缓存目录加入了忽略规则里，重新生成了打包的库文件
 gnuos
committed
9d87949
Copy full SHA for 9d87949
把处理下载链接的逻辑和处理内容的逻辑分离了，添加了文档和注释，增加了Referer请求参数
 gnuos
committed
31fad95
Copy full SHA for 31fad95
优化了Parser函数和Error类的结构，优化了模块的导入位置，增加了设置初始化种子链接的便捷方法
 gnuos
committed
6070e24
Copy full SHA for 6070e24

Commits on Nov 15, 2024

优化了依赖包版本，把request库换成了got，代码语法改成了ESM格式
 gnuos
committed
941c1fb
Copy full SHA for 941c1fb

Commits on Nov 24, 2019

Version 2.0.0
brendonboshell
committed
202819c
Copy full SHA for 202819c
Update README.md
brendonboshell
committed
2216cd4
Copy full SHA for 2216cd4
Merge branch 'useragent'
brendonboshell
committed
8f1cae3
Copy full SHA for 8f1cae3
Update README.md
brendonboshell
committed
9841aaf
Copy full SHA for 9841aaf
Merge branch 'filter-sitemaps-2'
brendonboshell
committed
be8ddb3
Copy full SHA for be8ddb3
Bump sequelize from 5.15.0 to 5.15.1
Show description for 5da46cc

dependabot[bot]
authored and
brendonboshell
committed
5da46cc
Copy full SHA for 5da46cc
Update README
brendonboshell
committed
cdb1cd2
Copy full SHA for cdb1cd2

Commits on Nov 11, 2019

Allow userAgent option to be a function
Show description for 8ce53bb
hjr3
committed
8ce53bb
Copy full SHA for 8ce53bb

Commits on Nov 8, 2019

Apply urlFilter to sitemap urls within sitemap
hjr3
committed
059e55e
Copy full SHA for 059e55e

Commits on Oct 1, 2019

Add error message to 'crawledurl' event
Show description for bfc8ff7
hjr3
committed
bfc8ff7
Copy full SHA for bfc8ff7

Commits on Sep 28, 2019

Version 1.7.2
brendonboshell
committed
6f85107
Copy full SHA for 6f85107

Commits on Sep 4, 2019

Update DbUrlList to use symbol operators
Show description for deda1a0
hjr3
committed
deda1a0
Copy full SHA for deda1a0

Commits on Aug 15, 2019

Version 1.7.1
brendonboshell
committed
4f5260a
Copy full SHA for 4f5260a
Rollback sinon (so tests pass)
brendonboshell
committed
4b3d174
Copy full SHA for 4b3d174

Commits on May 9, 2019

dependencies update
MrRefactoring
committed
98c1e95
Copy full SHA for 98c1e95

Commits on Apr 19, 2019

Merge remote-tracking branch 'taina0407/patch-2'
brendonboshell
committed
1bffda9
Copy full SHA for 1bffda9
Merge remote-tracking branch 'adversinc/feature/custom-opts'
brendonboshell
committed
0353c85
Copy full SHA for 0353c85

Commits on Mar 13, 2019

Add callback function for htmlLinkParser handler
Dino Fabrello
committed
65f675c
Copy full SHA for 65f675c
Allow resetting db storage on run
Dino Fabrello
committed
93cb211
Copy full SHA for 93cb211
Allow setting successful pages recrawl period
Dino Fabrello
committed
3ece606
Copy full SHA for 3ece606
Allow custom db table name
Dino Fabrello
committed
36545ac
Copy full SHA for 36545ac

Commits on Feb 19, 2019

Merge remote-tracking branch 'cbess/robotstxt-disable'
brendonboshell
committed
b0cf4e5
Copy full SHA for b0cf4e5
update readme
cbess
committed
e70c227
Copy full SHA for e70c227

Commits on Feb 8, 2019

provide option to disable robotstxt check
cbess
committed
c2d9ea4
Copy full SHA for c2d9ea4

Commits on Jan 26, 2019

Merge remote-tracking branch 'cbess/robotstxt-status'
brendonboshell
committed
e731a53
Copy full SHA for e731a53
Include robotstxt error message
cbess
committed
e4ddbc5
Copy full SHA for e4ddbc5
Add robotsIgnoreServerError option
Show description for 96447d7
cbess
committed
96447d7
Copy full SHA for 96447d7
Update robotstxt test
Show description for a85f9d7
cbess
committed
a85f9d7
Copy full SHA for a85f9d7
Handle 500 for robots txt
Show description for a54b75e
cbess
committed
a54b75e
Copy full SHA for a54b75e

Commits on Jan 24, 2019

Merge pull request #23 from cbess/master
Show description for 43f0b53
brendonboshell
authored
43f0b53
Copy full SHA for 43f0b53