How DRM3 Indexes 5,500+ News Feeds with Provenance

World News RAG is DRM3's news intelligence index. 5,500+ curated RSS feeds [2], 35+ languages, millions of articles per year. The index powers semantic search, entity extraction, topic classification, and downstream intelligence products like SignalForge. Every pipeline stage produces an Ed25519-signed provenance receipt.

Feed catalog

5,500+ feeds, hand-curated, categorized by region, language, and topic. All publicly offered RSS or Atom. Feeds that fail consistently are quarantined automatically.

Fetch controls

DRM3Bot identifies itself on every request with a contact URL. Per-domain rate limiting defaults to one second between requests, hard cap ten seconds. High-volume publishers get custom delays, timeouts, and headers.

The pipeline checks robots.txt (DRM3Bot-specific and wildcard), ai.txt (TDM Reservation Protocol, EU AI Act), TDM-Reservation headers, and X-Robots-Tag headers before every fetch. Opt-out signals trigger automatic quarantine. Crawl-delay directives override defaults when they specify a longer interval.

Publishers can request removal manually. Domain ownership confirmed, removal processed within 7 business days, cached content deleted.

Pipeline

Articles arrive via RSS. Body extraction uses Mozilla Readability. Extracted text feeds into entity extraction, topic classification, sentiment analysis, signal detection, and content scoring. Stored alongside analysis results because semantic search requires original content for embeddings and provenance receipts must reference the data they attest to.

The public interface is a news index: original headlines in the source language, English translations for non-English articles, source attribution, direct publisher links. 35+ languages including Chinese, Hindi, Russian, Spanish, Portuguese, Arabic, Japanese, Korean, German, French. The analytical layer is behind authentication.

Provenance chain

Every article carries a chain of provenance receipts. Fetch receipt: source URL, response status, content hash, timestamp. Extraction receipt: what text came out of the HTML. Analysis receipt: which model, which prompt, what the model returned. One article, three or four chained Ed25519 attestations. Independently verifiable.

Same signing infrastructure across every DRM3 product: DNS scans, third-party data feeds, AI inference, news analysis. One protocol.

EU AI Act enforcement begins August 2026. [3] Transparency requirements for AI systems processing copyrighted content apply here. Provenance receipts are the compliance mechanism.

The public interface shows original headlines, source attribution, and direct publisher links. The analytical layer behind it powers SignalForge, semantic search, and classification.

news.drm3.network · Verify provenance · Pipeline status · Publisher opt-out · Terms · Privacy