Meridian

From a new data drop to a deployed site

Seven steps, in this order. Two of them did not exist as code until recently — the Excel conversion and the Lanxess catalog were run by hand, which meant lanxess-catalog.json was committed with no way to rebuild it. Both are now in the repository, and both were verified to reproduce the committed payloads byte-for-byte.

Drop .xlsx files into input/excelfiles/. Existing CSVs are skipped unless --force is passed.

python3 -m pipeline.excel_to_csv

~30s for 26 files. pandas writes dates as ISO YYYY-MM-DD while older hand-made CSVs are day-first; both parse only because "%Y-%m-%d" is in date_formats. A source with a third date format will parse to NaT silently — the review queue in step 3 is where you would notice.

python3 manage_knowledge_base.py build --input-dir input --knowledge-base knowledge_base

Generates observed country aliases, entity candidates, and the observed HS hierarchy. Candidates are not used by processing. If new companies arrived in this drop, review knowledge_base/entity_alias_candidates.csv and promote what is correct into entity_aliases.csv before step 3 — afterwards is too late for this run.

python3 process_data.py --all

The long step — it writes output/<file>/ per source plus the concatenated output/trade_clean_all.csv, currently ~540 MB. Then read the review queues before trusting anything downstream: a queue that suddenly jumps from 2% to 100% of rows means a format changed, not that the data got worse.

python3 -m scripts.opportunity.build     # must be first
python3 -m scripts.prospect.index        # reads dashboard-data.json
python3 -m scripts.analytics.cube
python3 -m scripts.analytics.entities

The last three are independent of each other; only the first two are ordered.

Only needed when input/Lanxess Products.xlsx changes. It does not read the customs pipeline at all — it is the one payload built from a customer spreadsheet.

python3 -m scripts.lanxess.catalog        # ~0.5s — writes output/ and public/
python3 -m scripts.lanxess.opportunities  # ~0.3s — the verification oracle

The second is not needed by the app. It recomputes the Lanxess figures in Python, by a separate path from the TypeScript that renders them, and agreement between the two is what makes those numbers trustworthy. It currently reports 803 clean opportunities worth $911,279,665 across 24 accounts — which is what /lanxess and /opportunities both display.

npx tsc --noEmit
npm run build
git add public/ && git commit

npm test is an alias for npm run build. There are no unit tests in this repository — the build and the oracle in step 5 are the whole safety net, which is worth knowing before relying on it.

Push. Vercel builds from the repository root — framework preset Next.js, Node 22.x, no environment variables, no database. Root Directory must stay at the repository root; the Python package is named pipeline/ rather than src/ precisely because Next.js would treat src/app as a source root and take over the build from app/.

The unmeasured rows are the heavy ones, and they are blank rather than estimated because running them rewrites tracked payloads — timing them was not worth dirtying the working tree. Fill them in the next time a real drop is processed.