Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
Discussion options

Hello, this is a free form general discussion, feel free to say Hi or ask any questions you might have about my projects.

You must be logged in to vote

Replies: 1 comment · 3 replies

Comment options

hi, your blocklist tools isn't working anymore. Would you be able to share the code or the method you used to create the search api against all blocklists ?

You must be logged in to vote
3 replies
@lightswitch05
Comment options

lightswitch05 Jan 22, 2025
Maintainer Author

Yeah, I had some infrastructure costs I wasn't too happy about and eventually gave it up, but all the code is out there. Now, this was a personal for-fun project, so fair warning about the code quality and documentation, but I'll discuss the broad overview here. Primary reasons to stop the project:

  1. It was using a TON of storage. I think there is a lot of optimization on the table here, but I just don't have the interest anymore.
  2. Bots were abusing the service, I think they may have been spammers trying to get an early warning when one of their domains were added to a list, but it could have been anyone trying to automate some process. I just don't have the interest in trying to block them by adding some 3rd party captcha, and I certainly don't have the interest in providing bandwidth to them.

Anyways, those are my reasons for shutting it down. Here is the overview of the project:

  1. The Blocklist API, this was the main source of the data and the search you are looking for. It uses a Postgresql DB to store all the data. Here is the schema. In retrospect, there are a LOT of improvements that could be made here to reduce storage needs, but I just don't have the free time or frankly interest in it any more to do those improvements. There are two primary types of endpoints: Public endpoints to drive the website and search, private endpoints to ingest data from new blocklists.
  2. The list-loader. This is the main worker to get new lists and load them up in the API. I think I had it on a cron schedule to run once a day. There is a LOT of logic in here. Off the top of my head (its been awhile, so take this with a grain of salt), it loads up the latest version of a list from the internet. Then it loads the latest version of a list from the blocklist-api. It does a diff of the lists and then uploads the results to the blocklist API. This is what drives the history of the blocklist API and is a significant resource hog.
  3. The UI. Just the website frontend, uses React, makes API calls to the blocklist API. Pretty straightforward.
  4. Github history generator. Anytime I added a new list, I tried to include all its history (assuming it was on github). This tool steps through the repo generating URLs to each version. I think I then fed that output to the list-loader to go through each list version and load it up into the API using the same logic as the daily cron. This is a very manual process only for new lists that happened to be on Github already (also supported gitlab).
  5. blockist-tools. A java package used in both the API and the list-loader for parsing blocklists and generating diffs.
@paulverbeke
Comment options

Thanks a lot for taking the time to explain your service. I'm not really surprised that it was a cost heavy endeavor, but I'll be honest I didn't foresee the abuse aspect of this.
Thanks for making this public and enjoy your next projects !

@lightswitch05
Comment options

lightswitch05 Jan 23, 2025
Maintainer Author

Thanks for the question! It was good motivation to have at least a little documentation for the overall project and my reasons for shutting it down.

I'll add that my grand vision for this tool was never achieved. I wanted something that could also parse Adblock-style lists, but that was a bit of complexity I never attempted. As useful as hosts-style lists are, the real benefit is adblock lists.

Of course, I'm sure spammers would also love a way to detect when Adblock lists start blocking them, so maybe it's better such a tool does not exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.