The web scraper is used to extract data from webpages using Python's BeautifulSoup and natural language processing. We wish to use DeFi exploit specific APIs to autonmously funnel data into REAL, but the DeFi industry is still relatively new.

To expand the data-driven functionality of Rektify AI we are building our own API with all the DeFi attack data we've curated, aggregated, and wrangled to be the one-stop shop for the DeFi ecosystem. In the meanwhile, Python's BeautifulSoup allows us to extract qualitative and quantative attack data from HTML/XML text inside of webpages.

The following websites are used in Rektify web scraper:

• rekt.news - #1 resource for DeFi hacks across the ecosystem
• bitcoinexchangeguide
• CertiK
• Token Insight
• coingeek
• coinintelligence - Risk scores on CEXs
• cryptobriefing hack archive
• crypto.sec
• crypto slam
• flipside crypto
• defiprime.com
• defiyieldapp-rektdatabase
• etherscan
• hedgewithcrypto
• idex
• rugdoc projects
• slowmist.io
• squanch
• quadrigainitiative
• watchpug
• wowisme
• DeFi Hacks Analysis - Root Cause
• Forta Data
• Crypto Scam DB
• Web3 Rekt
• Crypto Rank

• TVL data: DeFi Llama
• Protocol behavior (real time alerts): Twitter, Discord
• Overall DeFi market trends: DeFi Pulse
• NFT API: QuickNode, CryptoSlam
• Social behavior of DeFi users/protocols: LunarCrush, PyPi page

The preceding are a few APIs Sena will be using to curate protocol specific data. DeFi Llama's API is a source to retrieve total value locked at the time of the exploit, a registery of DeFi protocols, and on-chain data. DeFi Pulse's API gives access to ETH gas prices as well as an on-slot of data providers for future needs. Twitter's API allows us to monitor protocol activity. All data curated from these APIs will be curated under columns in the SEAL dataset.

Each data vector needed to be cleaned: protocol names, dates, and amount lost in attack. The most intricate vector that needed prime attention were the dates. Dates and amounts with concatenated together with excessive spaces and a vertical bar(|).

Here's an example of how the data is formatted within the HTML text:

Input: '$55,000,000 | 11/05/2021'

The amount lost and dates are concatenated together which makes the task of curating each item difficult. But with BeautifulSoup the task is much easier.

Here's a few lines of code used to extract the dates from the unstructured html text data:

clean_dates = [] # Empty list to store cleaned dates
for item in dates:
    i = item.lstrip(',000,000 | ') # Removes the leading characters
    
    clean_dates.append(i)

Trial and error was used to find the proper parameters for the .lstrip method used to remove leading chars in Strings.

Output: 11/05/2021

Redirect link: Follow the logic and results in the web scraper directory.