Bounded data collection

Collect, clean, deduplicate, and export web research data with a time limit.

Arka can collect web data for a topic, clean the text, remove duplicate URLs, and write a machine-readable dataset. Collection is bounded by both time and record count so it finishes predictably.

arka data collect "renewable energy incentives" --for 5m --limit 50 --output data.jsonl
arka data collect "AI infrastructure" --for 30s --format csv --output data.csv --json

Each record includes the topic, title, source URL, cleaned text, and source type. The final report includes row count, elapsed time, output path, and whether the time or record bound was reached. Configure Serper, Tavily, or Brave through arka integration setup for stronger search coverage; the collector still has a DuckDuckGo fallback.

Natural language also works: “auto collect data about battery recycling for 10 minutes.”

Category catalogs

For requests such as “collect all Indian aeroplanes,” use the catalog mode:

arka data catalog "Indian aeroplanes" --for 10m --limit 500 --format json --output aircraft.json

Arka first issues bounded catalog/count searches, then paginates until the time or row limit is reached and deduplicates source URLs. The JSON report includes pages, rows, truncated, and reported_total. A total is left null unless an authoritative source explicitly reports one; Arka never guesses that it collected every item. Treat coverage: partial as a prompt to continue with a longer --for window or a more authoritative source.