Skip to content

CLI Reference

hkex-scraper [options]

Options

Option Type Default Description
--full-history flag off Scrape all filings from April 1999 to today.
--from-date DD/MM/YYYY ~2 months ago Start of the scrape range.
--to-date DD/MM/YYYY today End of the scrape range.
--limit N int 0 (unlimited) Process at most N filings.
--metadata-only flag off Phase 1 only — no document downloads.
--backfill-docs flag off Phase 2 only — process documents for filings already in the database.
--link-only flag off Only create/refresh graph edges.
--no-link flag off Skip the post-ingest graph phase. Useful for a large metadata backfill, which does not need edges.
--dry-run flag off Fetch data but write nothing.
--database-target SINKS comma-separated sink ids from DATABASE_TARGET Override the sinks for this run. Valid: postgres, mysql, sqlite, mongodb, mariadb, neo4j, clickhouse, duckdb, surrealdb. Order sets read precedence.
--coverage-report flag off Print chunk coverage from the read source, then exit.
--parity-report flag off Print per-sink filing counts and the spread, then exit. Requires two or more sinks.
--verify flag off Compare every configured sink by filing id and document hash (document_sha256), then exit non-zero on any difference. Requires two or more sinks.
--version flag — Print the version and exit.

Behavior

  • With no mode flag, a full run is performed: Phase 1 (metadata), then graph linking, then Phase 2 (documents) — unless --metadata-only.
  • --backfill-docs and --link-only run phases in isolation for incremental maintenance.

Timeouts

Variable Default Meaning
API_READ_TIMEOUT 30 Read timeout for one HKEX API page, in seconds.
CHUNK_DEADLINE_SECONDS 300 Hard ceiling for one date chunk. A chunk that exceeds it is logged as skipped and the run continues.
LINK_TIMEOUT_SECONDS 300 Hard ceiling for the graph phase; the edges are rebuilt on the next --link-only run.
REQUEST_DELAY_SECONDS 0 Minimum gap between request starts, process-wide. Raise it to be politer to HKEX.

Coordination

One producer scrapes at a time. The run takes a PostgreSQL session-level advisory lock before the first chunk and releases it on exit; a second node logs the holder and exits without work, because the corpus is idempotent and the holder ingests the same filings. A missing driver or an unreachable database degrades to "unlocked" rather than stopping the scrape.

Each run also upserts one row in scraper_heartbeat (run_id, phase, detail, updated_at) at start, after metadata, before documents and at completion, so a hung run is visible to the watchdog, not just an absent one.

Variable Default Meaning
SCRAPER_LOCK 1 Set to 0 for a manual run that must not wait for the lock.
SCRAPE_LOCK_KEY 917231002 Advisory-lock key. Change only if another tool needs the same key.
  • --coverage-report, --parity-report, and --verify are read-only reports that exit after printing.
  • A --limit also caps the number of documents Phase 2 processes.

Exit codes

Code Meaning
0 Success — every configured sink's writes succeeded.
1 A configured sink failed a write, --verify found a difference between sinks, or configuration is invalid/unusable (unset/unknown DATABASE_TARGET, missing driver or connection details).

Examples

# Backfill documents only, 50 at a time
hkex-scraper --backfill-docs --limit 50

# Rebuild graph edges from existing filings
hkex-scraper --link-only

# Write to SQLite only for this run
hkex-scraper --database-target postgres --limit 100

# Validate config without writing
hkex-scraper --dry-run --limit 10

# Coverage and parity
hkex-scraper --coverage-report
hkex-scraper --database-target postgres,sqlite --parity-report

# Cross-sink reconciliation (filing ids + document hashes)
hkex-scraper --database-target postgres,sqlite --verify

# Version
hkex-scraper --version

Logs

Each run writes a timestamped log to logs/hkex_filings_<YYYY-MM-DD_HH-MM>.log in the working directory, and failed SurrealDB statements are appended to logs/hkex_failed.sql. Credentials are never written to either file.