Skip to content

CLI Reference

hkex-scraper [options]

Options

Option Type Default Description
--full-history flag off Scrape all filings from April 1999 to today.
--from-date DD/MM/YYYY ~2 months ago Start of the scrape range.
--to-date DD/MM/YYYY today End of the scrape range.
--limit N int 0 (unlimited) Process at most N filings.
--metadata-only flag off Phase 1 only — no document downloads.
--backfill-docs flag off Phase 2 only — process documents for filings already in the database.
--link-only flag off Only create/refresh graph edges.
--dry-run flag off Fetch data but write nothing.
--database-target SINKS comma-separated sink ids from DATABASE_TARGET Override the sinks for this run. Valid: postgres, mysql, sqlite, mongodb, mariadb, neo4j, clickhouse, duckdb, surrealdb. Order sets read precedence.
--coverage-report flag off Print chunk coverage from the read source, then exit.
--parity-report flag off Print per-sink filing counts and the spread, then exit. Requires two or more sinks.
--verify flag off Compare every configured sink by filing id and document hash (document_sha256), then exit non-zero on any difference. Requires two or more sinks.
--version flag — Print the version and exit.

Behaviour

  • With no mode flag, a full run is performed: Phase 1 (metadata), then graph linking, then Phase 2 (documents) — unless --metadata-only.
  • --backfill-docs and --link-only run phases in isolation for incremental maintenance.
  • --coverage-report, --parity-report, and --verify are read-only reports that exit after printing.
  • A --limit also caps the number of documents Phase 2 processes.

Exit codes

Code Meaning
0 Success — every configured sink's writes succeeded.
1 A configured sink failed a write, --verify found a difference between sinks, or configuration is invalid/unusable (unset/unknown DATABASE_TARGET, missing driver or connection details).

Examples

# Backfill documents only, 50 at a time
hkex-scraper --backfill-docs --limit 50

# Rebuild graph edges from existing filings
hkex-scraper --link-only

# Write to SQLite only for this run
hkex-scraper --database-target postgres --limit 100

# Validate config without writing
hkex-scraper --dry-run --limit 10

# Coverage and parity
hkex-scraper --coverage-report
hkex-scraper --database-target postgres,sqlite --parity-report

# Cross-sink reconciliation (filing ids + document hashes)
hkex-scraper --database-target postgres,sqlite --verify

# Version
hkex-scraper --version

Logs

Each run writes a timestamped log to logs/hkex_filings_<YYYY-MM-DD_HH-MM>.log in the working directory, and failed SurrealDB statements are appended to logs/hkex_failed.sql. Credentials are never written to either file.