world_law_pipeline

Code that builds the endomorphosis/world_municipal_law and endomorphosis/world_national_law Hugging Face datasets. Scrapers write to a staging tree, and the normalizer/uploader validates the rows, holds back rows whose source policy forbids publishing, then writes sharded ZSTD parquet, metadata, a manifest and the dataset card.

path	what
run_loop.py	normalizer + uploader: discovery, batch ordering, holds, doc overrides, streaming/sharded writes, Hub commit plan (incl. stale-shard deletes). Always try --once --dry-run first
normalize.py, schema.py, SCHEMA.md	schema v1.1.0 (34 columns incl. optional provenance), staging contract, hold policy (§2.1)
card.py	dataset cards (municipal + national; cross-linked) incl. provenance/archived-share table
tools/split_national.py	one-off 2026-10-11 move of iso_* places from world_municipal_law to world_national_law (upload + verify sha256/rows, then a separate delete commit)
tools/prune_processed.py	verified pruning of processed/ sources + state/confirmed_batches/<bot>.jsonl (called by run_loop.py after each upload)
world/	refactor of the US american_municipal_law hub code (CID function byte-identical; see world/CID.md)
tests/	test_us_cid.py (US CID parity; needs the original ipfs_datasets_py checkout), test_hold_policy.py, test_routing.py, test_disk_prune.py (synthetic, /tmp only), make_synthetic.py
codes/	build_place_additions.py (places missing from the candidate list, incl. former-type false positives restored by audit_former_types.py), build_place_codes.py: place_id ↔ national code crosswalk (CBS, TERYT, RÚIAN, KSH, GKZ, INE, ISTAT, INSEE, AGS, BFS, dantai/JIS, IBGE, INEGI, UBIGEO, CUT, BPS/Kemendagri, LAU, NUTS); cached inputs are not bundled. Run it to regenerate
archive_fetcher/	Wayback/Common Crawl fetcher (core.py, ratelimit.py, staging.py, cli.py) + vendored legal_scrapers helpers
scrapers/eu/	NL CVDR, PL e-dzienniki
scrapers/national/	national/subnational collectors (shared/collect_*.py, ~230 jurisdictions), archive emitters, run scripts
scrapers/asia_latam/	JP reiki (gyosei, d1-law, joureikun, legalsquare), ES BOP, BR, PE, MX, CL, ID, ZA, AU, archive jobs (arch/)
scrapers/web/	DE Ortsrecht crawler + robots audit
state/doc_overrides.json	per-doc quarantine list applied by run_loop.py (misfiled / non-law docs)
extras/staging_scripts/	ad-hoc probe scripts found under staging/national/_archive_work/ (kept for provenance; not part of the pipeline)

Excluded on purpose: virtualenvs, __pycache__, caches (archive/_cache, _ratelimit), raw downloads (raw/, _sitemaps/, data/), *.bak* backups and world/*_draft.py drafts, and all staged/output data.

Environment: Python 3.13, pyarrow, huggingface_hub (upload only), requests; openpyxl for the code builder's xlsx inputs; tesseract for the OCR collectors.

Paths / environment variables

variable	default	used by
WORLD_LAW_ROOT	/workspace/world_law	run_loop.py (staging/, out/, processed/, quarantine/, held/, state/), scrapers (staging + pipeline paths), codes/
GLOBAL_MUNICIPALITIES_ROOT	/workspace/global_municipalities	scrapers that read global_municipalities_candidate.csv
GLOBAL_MUNICIPALITIES_CSV	/workspace/global_municipalities/global_municipalities_candidate.csv	codes/build_place_codes.py
TESSDATA_PREFIX	/home/box/tessdata (set via os.environ.setdefault, so an existing value wins)	OCR collectors in scrapers/national/shared/ (collect_sr.py and collect_ht.py also probe /home/box/tessdata/*.traineddata for the language choice)

Still hard-coded (documented, not env-ified): /workspace/ipfs_datasets_py[_latest] (the upstream ipfs_datasets_py checkout used by some collectors, cvdr.py, es_sede.py, de_ortsrecht_crawler.py and tests/test_us_cid.py), /workspace/municipal_law/_verify_tmp (US CID test fixtures), the sys.path hint in schema.py's docstring, and a few f-string paths in one-off scripts (e.g. scrapers/national/archive/peek.py). run_loop.py command-line flags (--staging --out --state --held …) override everything.

Uploader usage: python run_loop.py --once (hourly-safe: flock lock in state/run_loop.lock; a sha256 ledger in state/ledger.jsonl means nothing is processed twice; staged files move to processed/ only after every commit succeeds; 429/5xx commits are retried with backoff). Always try --once --dry-run first.

Repo routing (since 2026-10-11)

Places whose place_key starts with iso_ (place_id iso:*: admin_level country, adm1, supranational) are written to endomorphosis/world_national_law (local state out_national/, --national-repo, --out-national); everything else (municipalities, counties, water boards, ...) to endomorphosis/world_municipal_law (out/). Both repos have the same layout (world_law/data, metadata, manifests, places.parquet) and their own card. Each repo is committed separately (plan files planned_commit.jsonl / planned_commit_national.jsonl); staged files stay in staging only for the repo whose commit failed. The ledger records the target repo per file.

Disk guard and verified pruning

At the start of each run run_loop.py checks free space on /: below --disk-warn-gb (15) it logs, writes state/disk_alert.json and prints a DISK_LOW free=... line; below --disk-stop-gb (8) it builds no new places (and stops building mid-run if it crosses the threshold; already-built places still commit). After the upload it calls tools/prune_processed.py, which deletes a processed/<bot>/ file (plain or .gz) only when: the bot is in state/prune_config.json (default ["eu", "national"]); its content sha256 matches a ledger entry >= 2h old; the commit that carried its place is in the Hub repo history; and every parquet of its place in out//out_national/ matches the Hub file's sha256 and size. Each delete is first appended to state/pruned_manifest.jsonl. The ledger is never changed, so a re-staged identical file is still skipped. --no-prune disables it.

Secret scan (2026-10-10): no tokens, keys, cookies files, .env or credentials-in-URL in this bundle. See the report given to the maintainer for details. Collectors read HF_TOKEN / ~/.cache/huggingface/token from the environment only. collect_td.py fetches a public Supabase anon key at runtime (not stored). Several OCR collectors hard-code /home/box/tessdata.