Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

master-cloner/cornerstone

Open more actions menu

Repository files navigation

Cornerstone

克隆大师基类, 基于 symfony/panther(https://github.com/symfony/panther)

Action

用于 MasterCloner clone 项目页面资源的基础类库

推荐使用于 Ubuntu 或 Mac 环境下且以安装 Chrome 浏览器
PHP 版本 7.1 以上

效果如下

1.配置待克隆项目

// 配置基础 Uri (必须)
$base_uri = 'https://www.bilibili.com'

// 待克隆 urls
$config['wait_capture_urls']  => [
        'index' => 'https://www.bilibili.com',  // (必须)
        //'list'   => 'https://www.bilibili.com/v/dance/',
        //'detail' => 'https://www.bilibili.com/video/av50530804/'
    ]
// 启用深克隆,进行本地化资源
$config['is_deep_clone'] = true 
// 启用 CDN 克隆,进行 CDN 资源本地化(TODO)
$config['is_cdn_clone'] = true
// 本地化资源类型(图片,js,css)
$config['deep_clone_resource_type'] =  [
        'images',
        'js',
        'css',
    ],
    
// 声明资源类型为 laravel,进行模版内容替换(TODO)
$config['is_laravel_resource'] = true

2.运行

cli 运行方式

php index.php

// 超链接 - 运行后根目录生成 links.txt 文件
php links.php

fpm 运行方式

php -S localhost:8000
chrome open localhost:8000

3.在运行结束后,复制 response 至 Laravel 或其它项目中即可


Tips:

如果进程卡死,请释放端口 9515

lsof -i:9515
kill PID

Todo

拆分爬取类型

  • 模版类型
  • 资源类型
    • 存储类型

提取配置抽象成类

CDN 等资源本地化

About

Crawl the same/asynchronous rendering of web pages for localized resource storage

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages

Morty Proxy This is a proxified and sanitized view of the page, visit original site.