Media · E-commerce · In-house
Recommender systems at Seznam scale
Discovery and ranking across 50M+ requests a day: a balanced history-based recommender, clickbait and quality classifiers, agentic content crawlers, and the experimentation framework to prove each change.
Stack
- Python
- recommender models
- LLM classifiers (DSPy)
- A/B experimentation
- SQL analytics
Prior in-house work: scoped to my contribution.
Results
- 50M+requests/day at Seznam scale
- +5.5%clicks from the balanced recommender≈ 35–40M CZK/yr, A/B tested
- +18%CTR from agentic content crawlersclicks +12%
- −40%clickbait exposure, quality classifierlow-quality −50%, clicks −2%
Context
Seznam runs a national-scale newsfeed, and the recommender behind it handles discovery and ranking across more than 50 million requests a day. The feed mixes news, entertainment, and interest content, and the business runs on clicks and the ad revenue they carry. We held a long behavioural history for every user, but ranking mostly ignored it and scored items on the last handful of clicks instead.
I was the senior product manager for the recommender systems for about ten years. This case is scoped to the part I owned directly: the ranking logic, the content-quality classifiers, the crawlers that fed the feed, and the experiments that decided what shipped.
Problem
Three problems, spread across years. First, the feed was one-note: ranking only saw recent interactions, so a run of clicks on one topic crowded everything else out while a long history sat unused. Second, content quality dropped sharply once editorial stopped hand-checking every article: publishers exploited it and flooded the feed with clickbait and thin content. Third, the high-quality niche content that would have widened the feed was uneconomic to commission by hand.
Each of these threatened the same number: clicks, and the revenue attached to them.
Approach
The through-line was to measure before shipping and not reach for the expensive tool first. For the one-note feed, a balanced recommender weighted by interaction age: a weighting function, not a bigger model. For content classification, optimised LLM prompts with DSPy instead of training and maintaining a separate classifier per property. For the crawlers, a cheap regex pre-filter in front of the LLM so the model only ever saw promising text. And when the business wanted revenue-per-mille folded straight into ranking, an A/B test to settle it rather than an argument in a meeting.
What I built
Scoped to my contribution. I wrote the weighting function for the history-based recommender myself; it penalises click age and content type so the feed stays mixed. I specified the content-quality and clickbait classifiers: 95% precision on clickbait: and the asynchronous pipeline that runs them on article save rather than in the request path. I wrote the DSPy optimisation scripts behind the content-type, interest, and adult-content classifiers. I wrote the source-filtering and quality-scoring rules for the agentic crawlers. I designed the URL-decoration and session-tracking join that connected ad revenue to individual recommender zones, and negotiated it through an ad team wary of touching their scripts. And I owned the experiment branch that weighted RPM in ranking and produced the evidence to reject it.
Results
Every figure here was measured against a pre-change baseline or an A/B split. The balanced recommender lifted clicks 5.5% (roughly 35–40M CZK a year in extra revenue, at a running cost of about five servers) on an A/B test on live traffic. The quality classifier cut clickbait exposure 40% and low-quality content 50% for a 2% cost in total clicks; that −2% was the deliberate trade, measured on an A/B split, with the 95% precision figure from a labelled clickbait set. The agentic crawlers raised CTR 18% and clicks 12% against the pre-rollout baseline, adding thousands of vetted interest articles a day with no newsroom growth. The three DSPy classifiers went from decision to deployed in one week. Separately, working through the daily view statistics I caught a JavaScript bug that was closing the entire in-feed video platform instead of a single show; I sized the damage in SQL and handed engineers an exact fix, and it was resolved and the database cleaned in one man-day.
What I’d tell you if you’re facing this
Three things carry over.
Put a cheap filter in front of the expensive model. The regex pre-filter on the crawlers and the asynchronous classifier pipeline weren’t clever, but they were the reason the LLM economics worked at this volume. Design the cost path first.
When the business wants a change that trades user relevance for yield, run the A/B test instead of having the argument. The RPM-ranking proposal sounded plausible and I would not have talked anyone out of it; two weeks of data on about 1% of traffic showed it hurt CTR and relevance, and that ended it cleanly.
Most of “shipped and trusted” is un-glamorous. Sizing damage in SQL, cleaning corrupted preferences, building a whitelist so a fast-shipped classifier does not punish established authors: none of that demos well, but skipping it is how a good system loses users’ trust.
Selected decisions
Seven decisions from the recommender and content-platform work: the call I made, the result, and the trade-off.
+18% CTR
Agentic crawlers for niche interest content
Problem Niche-interest content is uneconomic to produce by hand. The content platform was short on high-quality, tightly-themed articles, which capped CTR and engagement.
Decision I decided we would not grow the newsroom: automated collection would replace that expansion. I wrote the rules for filtering sources and scoring their quality myself.
Solution Agentic crawlers that find, fetch, and categorise high-quality niche articles from the web and feed them into the platform.
Result Thousands of quality interest articles a day. CTR rose 18% and total clicks 12%. Measured: Online, against the pre-rollout baseline.
Trade-off LLM and API running cost was high at first. I added a regex pre-filter so only promising text reached the expensive model.
+5.5% clicks
A balanced recommender that uses long-term history
Problem The Seznam recommender only weighed a user's most recent interactions. The feed skewed one-note even though we held a long history per user.
Decision I designed an algorithm that accounts for the whole history, and wrote the weighting function myself: penalising click age and content type.
Solution An algorithm that mixes news, entertainment, and interest content in balance, weighted by interaction age.
Result +5.5% clicks: roughly 35–40M CZK a year in extra revenue, at a cost of about five servers. Measured: A/B test on real traffic.
Trade-off It added a little feed-assembly latency. We cached results for less-active users to hold the response under 100 ms.
1 week to deploy classifiers
Fast LLM classifiers with DSPy
Problem Content platforms need to recognise many content properties: type, interest, adult content. Training a traditional ML classifier for each was slow and compute-heavy.
Decision I decided to move from classical models to optimised LLM prompts with DSPy, and wrote the optimisation scripts myself.
Solution Stanford's DSPy to auto-optimise LLM prompts and build reliable classifiers fast.
Result Content-type, interest, and adult-content classifiers fully deployed in one week. Measured: On a labelled evaluation set per classifier.
Trade-off An LLM call is costlier and slower than a local BERT model. We moved classification to asynchronous background processing on article save.
−40% clickbait exposure
Clickbait and quality detection, under pressure
Problem Editors stopped hand-checking every newsfeed article. Publishers exploited that and flooded the feed with clickbait and low-quality content.
Decision I put a temporary manual hotfix on the worst-hit sites and, in parallel, ran the build and immediate rollout of an automated classifier.
Solution A content-quality and clickbait classifier at 95% precision on clickbait.
Result Clickbait exposure −40%, low-quality content −50%, total clicks −2%. Measured: A/B, plus precision on a labelled clickbait set.
Trade-off The fast-shipped model produced false positives. We added a quick feedback loop and a whitelist for established authors.
Joining ad and recommender data
Problem Better business decisions and feed optimisation needed revenue per zone and user type. The ad and recommender systems were data-siloed.
Decision I designed a solution around URL decoration and session tracking, and negotiated the implementation with the ad team, who were wary of touching their scripts.
Solution Recommender parameters carried into the ad scripts via URL decoration and a shared ad session.
Result A complete, real-time view of revenue per session for every zone and user type. Measured: Reconciled against the ad system's own revenue reporting.
Trade-off URL decoration could affect partners' analytics filters. We stripped the tracking params from the URL via the History API immediately after load.
Turning down RPM-based ranking to protect relevance
Problem The business wanted RPM (yield) put straight into the recommender, on the assumption that higher-RPM sites are higher quality. That risked degrading feed relevance for users.
Decision I refused a blanket rollout without evidence. I pushed for and designed an A/B test to check the business hypothesis on real traffic.
Solution An experiment branch weighting RPM in the recommender, evaluated for impact.
Result The A/B test showed a drop in CTR and overall feed relevance: which stopped a change that would have hurt the user experience. Measured: A/B test on real traffic.
Trade-off A temporary performance dip for the test group (about 1% of users for two weeks) was the necessary price of hard data to reject a bad decision.
1 day to fix and clean up
Fixing a bug that closed the Seznam.cz video platform in the feed
Problem Users were meant to be able to hide shows they weren't interested in. A JS bug closed the entire in-feed video platform instead of one show, dropping views and losing traffic.
Decision I caught the bug myself while reviewing daily view stats. I ran the SQL analysis of the data damage, located the problem, and handed engineers an exact fix procedure.
Solution Database analysis, then coordinating the developers on the fix rollout and cleaning the corrupted preferences.
Result The faulty logic fully fixed and the database cleaned in a single man-day. Measured: Daily view statistics, before and after.
Trade-off We had to delete the last 48 hours of user preferences, so some users had to re-hide unwanted shows: a small price for restoring the video section.