A national news aggregation and monitoring platform: collects articles from registered news sites in Turkmen, Russian and English, filters them by keywords and reports coverage.
My role
Backend developer. Built ingestion, source discovery, the APIs and reporting.
Stack
Go
Echo
GORM
PostgreSQL
RSS
Serper API
The problem
Analysts needed every publication on given topics across many sites in three languages — and new sources keep appearing, so a fixed list of sites misses coverage.
Architecture
A background loop polls RSS feeds on a configurable interval, decodes items and matches them against keywords, detects the language and upserts news into the corpus database. A discovery job queries Google News, extracts domains, drops known ones and queues the rest for moderation. Admin and public APIs (Echo) serve news, reports and Excel exports.
Key decisions
01Upserts make polling idempotent
Why
Feeds repeat items between polls. Inserting with ON CONFLICT means the loop can run as often as needed without creating duplicates.
02Discovered sources go through moderation
Why
Search results contain noise. New domains are stored as new, valid or invalid, and only approved ones join the polling list — coverage grows without polluting the corpus.
03Separate databases for platform data and the corpus
Why
The news corpus grows fast and is queried for reports. Keeping it apart from users and API keys lets each be backed up and migrated on its own.
Trade-off
No cross-database joins; the service manages two connections.
What I built
Scheduled RSS ingestion from registered sites with keyword filtering
Language detection for Turkmen, Russian and English content
Source discovery through Google News (Serper API) with a moderation queue for new sites
Two PostgreSQL databases: platform users and keys, and the news corpus
Idempotent upserts, SQL reports per site and language, Excel export