Fetch path
HTTP → browser fallback
Fast requests handle ordinary pages; Playwright takes over when rendering is required.
Back to work
Released v0.1 · open source
Developer infrastructure
CrawlTrove
A self-hosted scraper and crawler that turns pages, documents, and images into clean Markdown with corpus-ready evidence.
- Role
- Product · Engineering
- Platform
- Self-hosted web · JSON API · Docker
- Stack
- Python 3.11 · FastAPI · Playwright · Postgres
- Status
- Released v0.1 · open source
Problem
Scraping a page is easy. Building a corpus you can trust is not.
Useful source material arrives as static HTML, JavaScript applications, PDFs, ebooks, and images. Extraction alone is not enough when the result also needs provenance, quality signals, and a repeatable path back to its source.
CrawlTrove treats scraping as the first stage of corpus preparation rather than the final output.
Engineering decisions
Take the cheap path first, then preserve evidence through every fallback.
The service starts with an HTTP fetch and escalates to Playwright only when a page needs a real browser. PDF, EPUB, and image inputs enter the same pipeline through parsing and OCR.
Public-target validation, explicit authentication options, bounded jobs, restart-safe checkpoints, and corpus metadata make the service usable beyond a one-off scraping script.
Released outcome
A public self-hosted release with the product and its operating path exposed.
The v0.1.0 release includes the dashboard, JSON API, Docker runtime, Postgres support, and 60 tracked test modules. The source, setup, security guidance, and release history are public and inspectable.
Verified proof
What the current build demonstrates.
Inputs
HTML · PDF · EPUB · images
Document parsing and OCR bring non-HTML sources through the same extraction path.
Corpus signals
Quality · license · dedup
Each result carries language, provenance, quality, and duplicate-detection metadata.
Product capture
Authentic interface, captured from the product.
