MetaLook
●@metalook_io
Breaking news curated and deduplicated in real time by the MetaLook semantic clustering pipeline (archived).
What We Shipped
Multi-source ingestion engine
Automated ingestion services polled structured RSS feeds and custom scrapers across more than 100 domestic and international publications.
Centralized data normalization layer
The pipeline transformed heterogeneous text formats, character encodings, and timestamps into a single structured schema.
Semantic deduplication engine
Vector clustering algorithms identified related articles covering the same breaking event, rendering only one authoritative story per topic.
Automated AI summarization pipeline
Large language models extracted key entities and generated concise, factual executive summaries for each unique story.
Bilingual content distribution
The system automatically translated and structured bilingual English and Vietnamese summaries with synchronized metadata.
Editorial management workspace
An administrative dashboard allowed editors to manage feed sources, adjust prioritization weights, and override automated classifications.
Problem & Context
MetaLook needed to monitor and curate thousands of daily articles across more than 100 media outlets and news feeds. Manual editorial workflows created significant latency and overwhelmed readers with repetitive coverage of the same events.
The core technical challenge was building an automated system that accurately clusters syndicated stories discussing the same underlying event regardless of wording variations, followed by instant bilingual AI summarization.
Engineering Challenges
Resilience against third-party source outages
External publishers frequently alter DOM structures or experience downtime. Isolated polling queues prevent single-source failures from stalling the ingestion pipeline.
High-precision semantic deduplication
Simple keyword matching fails when multiple outlets rewrite the same story with different vocabularies. We deployed vector embeddings to cluster articles based on semantic intent.
LLM cost control and hallucination prevention
Content caching, token hashing, and batch inference minimized LLM costs while strict factual verification filters prevented hallucinated outputs.
Operational outcomes
- 30-40
- stories processed per hour around the clock
- 90%
- distinct stories identified after deduplication
- 30-60 min
- average latency from source publish to feed
The pipeline processed ~800 articles daily across 100+ syndicated sources while live, identifying unique stories via vector embeddings.
Delivered on schedule in 3 months by a 5-engineer team from initial architecture to production deployment.
How this benefits client projects
High-throughput ETL pipelines, unstructured data normalization, and production LLM cost optimizations apply directly to client data platforms.