Challenge
Comparing rates looks simple right up to the moment there are more than three sources. Some publish an API, some only a page, some change their format without warning, and some go quiet. The reader, meanwhile, needs a table they can trust right now, and a row that went stale an hour ago is worse than a missing one.
Solution
The system was built as a collection pipeline rather than a folder of scripts.
Each source is polled on its own schedule in its own way, the answer is normalised, and only after a plausibility check does it reach the shared table. A source that has gone silent or started returning nonsense is flagged and does not spoil the comparison for the rest. Freshness is tracked per row, and data past its allowed age is not presented as current.
Architecture and implementation
How it works:
· polling dozens of independent sources on schedule
· two ways of fetching data: a programmatic interface and page parsing
· normalisation against a shared catalogue of directions
· a plausibility check that discards outliers
· flags on sources that go quiet or return errors
· a comparison table with sorting and filters
Every row carries its own freshness marker.
Result
The service launched in 2017 and was built the way modern aggregators are built: collect, normalise, verify, compare. The skill transfers whole: monitoring competitor prices, rolling up stock levels from several suppliers, comparing tariffs are the same problem in a different subject.
