Challenge
The same listing lives on four marketplaces and in three messenger channels at once, each time with different wording, different photos and a different price. The hard part of an aggregator is not collecting listings. It is working out that four records are the same car.
Solution
The system has three layers: collection, processing, delivery.
Deduplication rests on three independent signals. The seller's phone number is a strong marker, a text fingerprint catches rewritten descriptions, and an image hash finds the same photos reordered and resized. No single signal is reliable alone, and together they hold up.
Attributes are extracted by a language model. Sellers write prose, and search needs fields: year, engine, transmission, condition. Regular expressions lose on the first unusual phrasing.
Architecture and implementation
How it is put together:
· collection from four external marketplaces and messenger channels, each source in its own queue
· normalisation against a shared reference catalogue
· deduplication by phone, text fingerprint and image hash
· attribute extraction by a language model
· a search index with filters over those attributes
· a freshness filter that drops listings that have gone stale
The front end is server-rendered for search engines. Stack: Python with Celery queues, PostgreSQL, the Meilisearch engine, object storage, Next.js, Docker, and a language model for parsing text.
Result
Three verticals in one search: cars, phones, property. The project is live and under active development.
It is also our working answer to what adopting AI means in practice. Not a chat widget, but a model inside a pipeline with a measurable job: turning a seller's free text into fields you can filter on.
