Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Beta: Execute ghe-restore tasks in parallel #601
Conversation
5a5b659
to
bde8ae9
775123b
to
21c22e3
2c89a01
to
0883e2a
|
LGTM |
| ghe-ssh "$GHE_HOSTNAME" -- /bin/bash 1>&3 | ||
| done | ||
|
|
||
| ghe-ssh "$GHE_HOSTNAME" -- "sudo sh -c 'rm $GHE_REMOTE_DATA_USER_DIR/elasticsearch-restore/*.gz'" 1>&3 |
ryansimmen
Apr 7, 2020
Author
Member
When both ghe-restore-es-audit-log and ghe-restore-es-hookshot run at the same time during parallel restore, there is a race condition whereupon ghe-ssh "$GHE_HOSTNAME" -- "sudo sh -c 'rm $GHE_REMOTE_DATA_USER_DIR/elasticsearch-restore/*.gz'" 1>&3 from one script deletes the *.gz files that the other script had put down. I moved these files to separate directories so this wouldn't happen and my fix for the single-vm or configured edge case is to copy all files to the elasticsearch-restore folder so elasticsearch-post-start will process them accordingly.
When both ghe-restore-es-audit-log and ghe-restore-es-hookshot run at the same time during parallel restore, there is a race condition whereupon ghe-ssh "$GHE_HOSTNAME" -- "sudo sh -c 'rm $GHE_REMOTE_DATA_USER_DIR/elasticsearch-restore/*.gz'" 1>&3 from one script deletes the *.gz files that the other script had put down. I moved these files to separate directories so this wouldn't happen and my fix for the single-vm or configured edge case is to copy all files to the elasticsearch-restore folder so elasticsearch-post-start will process them accordingly.
|
Looks good |
Execute long running restore tasks in parallel by leveraging moreutils parallel.
This resolves github/ghes-infrastructure#407
Most of this work is based upon work done in #597