Compass Data
A visual pipeline builder for searching, scraping, and enriching data at scale.
Why we built this
We built this for a client project that needed data cleaning. Actually, more than cleaning. Data finding and intent data that didn’t exist anywhere. No off-the-shelf tools fitted the bill, so we built our own. Real prototype stuff.
Having used it across a few projects, we refined and developed it into something much more capable. We just used it as part of our SAP partner data enrichment work , turning a thousand stale records into sales-ready data.
How we built it
The first version was a hacky Streamlit app that fell over a lot. But we built it the best way: on a live project, adding features as we found what the project really needed. Then when we got serious, we rebuilt the whole thing into a proper standalone app.
React frontend with drag-drop pipeline builder. FastAPI backend connects ScrapingBee for web scraping, OpenAI for enrichment, and stores everything in Firebase. Each node in the pipeline is a composable operation: search, extract, transform, enrich, save.
