prototype code for splitting files (which can consist of multiple documents concatenated together) into distinct documents and extracting key metadata.

dc is the main processing code:

dc
├── classify.py
├── doctext.py
├── llm.py
├── prompts.py
└── toc.py

classify takes a file and classifies each page as a new document or a continuation of the previous document.

toc takes the page classifications and bundles them into distinct documents with extracted document-level metadata, producing a table-of-contents for a file.

The tasks directory contains code for exploring and experimenting with the current codebase. case-eda has code to process a sample of 25 cases, and also hand-labeled page-classification data for those cases. tasks/case-eda/src/summarize-case-from-metadata.py is a demonstration for how to extract things like "investigated officer" from just the extracted metadata from case files, although a more robust approach might be to use the metadata to filter for key documents to highlight or to pass to an extraction model.

tasks/case-eda/src/case-document-summary.py formats the table-of-contents data into a chronological summary of case documents, and spins up the webserver found here