Scraper untuk mengambil data jurnal dari SINTA Kemdiktisaintek.
- Parallel requests dengan ThreadPoolExecutor
- Retry mechanism (3x dengan exponential backoff)
- Error logging ke file JSON (terpisah untuk halaman dan end year)
- Recovery mode untuk halaman gagal
- Recovery mode untuk end year certificate error
- Append mode - skip duplicate berdasarkan Profile ID
- Resume capability - lanjut dari data yang sudah ada
- Colored terminal output - output berwarna untuk memudahkan monitoring
| Kolom | Deskripsi |
|---|---|
| Profile ID | ID unik jurnal (dari URL) |
| URL | URL profil jurnal |
| Nama | Nama jurnal |
| Tipe Jurnal | Pengabdian/Penelitian |
| Lokasi | Lokasi institusi |
| P-ISSN | Print ISSN |
| E-ISSN | Electronic ISSN |
| Subject Area | Bidang ilmu |
| Google Scholar | URL Google Scholar |
| Website | URL website jurnal |
| Editor URL | URL editor |
| Certificate | Status akreditasi (S1-S6) |
| End Year Certificate | Tahun akhir sertifikasi |
| Garuda URL | URL Garuda |
pip install -r requirements.txt# Test 1 halaman (default)
python main.py --test
# Test 5 halaman
python main.py --test 5
# Test 10 halaman
python main.py --test 10# Scrape semua halaman (1-1354) - append mode, skip duplicate
python main.py
# Mulai dari halaman tertentu (misal page 90)
python main.py --page 90
# Dengan custom workers
python main.py --workers 10
# Mulai dari awal (overwrite existing data)
python main.py --fresh# Retry halaman yang gagal - append ke existing data
python main.py --fails-only
# Re-scrape end year certificate yang error
python main.py --get-year-error
# Re-scrape end year certificate yang kosong (termasuk yang tidak ada data)
python main.py --get-year-certificate| Option | Fungsi |
|---|---|
--test [N] |
Test scrape N halaman (default 1) |
--page N |
Mulai scrape dari halaman N sampai akhir |
--workers N |
Set jumlah parallel workers (default 5) |
--fresh |
Mulai fresh, overwrite existing CSV |
--fails-only |
Retry hanya halaman yang gagal |
--get-year-error |
Retry end year certificate yang error |
--get-year-certificate |
Retry end year certificate yang kosong |
journals.csv- Data jurnal hasil scrapingfailed_pages.json- Log halaman yang gagalend_year_errors.json- Log end year certificate yang errorscraped_pages.json- Log halaman yang berhasil di-scrape (sorted)
Scraper menggunakan append mode dengan skip duplicate:
- Saat mulai, load semua Profile ID yang sudah ada di CSV
- Saat scrape, skip jurnal dengan Profile ID yang sudah ada
- Hanya tambahkan jurnal baru ke CSV
Keuntungan:
- Bisa dihentikan dan dilanjutkan kapan saja
- Tidak ada data duplikat
- Aman untuk di-run berkali-kali
Contoh:
# First run - scrape page 1-100
python main.py --test 100
# Result: 1400 journals saved, pages 1-100 logged
# Stop (Ctrl+C), then continue later
python main.py --page 101
# Result: Start from page 101 to end
# Atau langsung tanpa --page (pakai append mode)
python main.py
# Result: Skip journals yang sudah ada (by Profile ID), continue
# Jika terputus di tengah jalan
python main.py --page 90
# Result: Mulai dari page 90 sampai 1354
failed_pages.json (halaman error):
{
"15": {
"error": "Timeout",
"timestamp": "2026-02-18 10:30:45",
"attempts": 3
}
}end_year_errors.json (end year certificate error):
{
"https://sinta.kemdiktisaintek.go.id/journals/profile/671": {
"nama": "Jurnal Pendidikan Indonesia",
"error": "Timeout",
"timestamp": "2026-02-18 10:31:02",
"attempts": 3
}
}scraped_pages.json (halaman yang berhasil):
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, ...]Note: Hanya halaman yang berhasil di-scrape yang masuk ke log ini. Halaman yang error/timeout tidak dicatat.
Edit konstanta di main.py:
TIMEOUT = 60 # Request timeout (detik)
DELAY_MIN = 1 # Delay minimum per halaman
DELAY_MAX = 4 # Delay maksimum per halaman
BATCH_DELAY_MIN = 1 # Delay minimum antar batch
BATCH_DELAY_MAX = 15 # Delay maksimum antar batch
BATCH_SIZE = 10 # Jumlah halaman per batch
MAX_RETRIES = 3 # Maksimal retry per request
MAX_WORKERS = 10 # Parallel workers untuk end year certificateBatch 1:
├─ Scrape 10 halaman parallel (delay 1-4s per halaman)
├─ Fetch end year certificate parallel (10 workers)
├─ Skip duplicate berdasarkan Profile ID
└─ Delay 1-15s sebelum batch berikutnya
Batch 2:
├─ Scrape 10 halaman berikutnya parallel
├─ Fetch end year certificate parallel
└─ Delay 1-15s
...dst
- Pengabdian: Jika nama mengandung "abdi" atau " PKM " (dengan spasi)
- Penelitian: Semua lainnya
| Opsi | Fungsi |
|---|---|
--get-year-error |
Retry hanya jurnal yang error saat fetch end year (dari log) |
--get-year-certificate |
Retry semua jurnal dengan end year kosong (dari CSV) |
Kapan menggunakan yang mana?
--get-year-error: Jika ingin retry hanya yang error (timeout, HTTP error, dll)--get-year-certificate: Jika ingin retry semua yang kosong, termasuk yang memang tidak ada data accreditation
Found 1400 existing journals in CSV (will skip duplicates)
Previously scraped pages: 100
Last scraped page: 100
=== Starting scrape from page 1 to 1354 ===
Page workers: 5, End year workers: 10
Batch size: 10, Batch delay: 1-15s
Page delay: 1-4s, Timeout: 60s
Retry: 3 attempts with backoff [1, 2, 4]
Output: journals.csv (append mode, skip duplicates)
Failed pages log: failed_pages.json
Scraped pages log: scraped_pages.json
--- Batch 1 (pages 1-10) ---
Page 1: Waiting 2 seconds before request...
Page 1: Found 14 journals (Bright Green)
Fetching end year certificates for 14 journals (Page 1)... (Cyan)
Progress: 5/14 end year certificates fetched (Page 1) (Sky Blue)
Progress: 10/14 end year certificates fetched (Page 1) (Sky Blue)
Skipped 14 duplicate journals (Page 1) (Magenta)
Progress: Page 1 done. Total journals: 0 (Mint Green)
...
=== Scrape complete! ===
New journals added: 18900
Total in CSV: 20300
Total scraped pages: 1349
Failed pages: 5
Data appended to journals.csv
Failed pages logged to failed_pages.json
Run with --fails-only to retry failed pages
| Pesan | Warna |
|---|---|
| Page found X journals | 🟢 Bright Green |
| Fetching end year certificates | 🔵 Cyan |
| Progress updates | 🔵 Sky Blue |
| Skipped duplicates | 🟣 Magenta |
| Page done | 🟢 Mint Green |
| Errors | 🔴 Bright Red |
| Retry warnings | 🟡 Amber/Yellow |