One tick, five candles: aggregating a feed you can't trust to arrive once

Ticker's aggregator sits between a Redis Stream of price ticks and a TimescaleDB table of candles. Its job sounds like a one-liner — group ticks into time buckets, compute open/high/low/close/volume — and most of the interesting decisions are in the parts that sentence skips.

One tick becomes five candles

Every tick updates five bars at once: a configured finest interval, plus a fixed 5m/15m/1h/1d set. Bucketing is just truncation — a tick's timestamp rounded down to the interval gives the bar it belongs to, so no bucket boundaries need tracking and a tick that arrives slightly late still lands in the right bar.

State is a map keyed by symbol and interval, each entry holding one open bar. A tick either extends the current bar (high becomes a max, low a min, close the latest price, volume a running sum) or, if its bucket is later than the one being held, closes the old bar and seeds a fresh one from that tick. Open, high, low and close are each set-once, max, min, or last-write. Volume is the only field that accumulates, which turns out to matter more than it looks.

Writing on every tick, not on close

The obvious design writes a bar to the database when it closes. This one writes on every single tick.

That's more writes by a wide margin, and it buys one specific thing: the current, still-forming bar in the database is always accurate. A server-rendered page that queries the last hour of candles gets live data, not a bar frozen at the last boundary crossing. Without it, the newest candle on a freshly loaded chart would be stale by up to a full interval, and the fix would be a second code path reading the in-memory state — which only works while the reader and the aggregator are the same process.

Upserting full state rather than a delta is what makes this affordable. Each write says what the bar is, not what changed, so repeating one is harmless and a failed one is repaired by the next tick in that bucket.

Only the finest interval streams

When a bar updates, only the finest interval is published for live streaming. The coarser four are fetched over REST when someone switches tabs.

The reasoning is that only the default view needs to move in real time. Streaming all five would multiply the message rate by five to keep four charts fresh that nobody is looking at. A tab switch is a request boundary anyway, so the data is at most one request old at exactly the moment it becomes visible.

Bucket close does trigger one piece of work beyond the write: the cached indicator series for that symbol and interval is invalidated, because a closed bar changes every EMA and RSI value computed over that window.

Delivery semantics, honestly

The aggregator reads through a Redis Stream consumer group, which promises at-least-once delivery. Getting that guarantee is not the same as being handed it.

A consumer group only redelivers if something claims entries that were read but never acknowledged. Reading with > returns new messages only, so a tick whose processing fails is not retried by default — it sits in the pending list indefinitely, which is silent data loss wearing the costume of a durability guarantee. Making at-least-once real takes a periodic reclaim pass that claims entries idle beyond a threshold and pushes them back through the same handling path.

Which then creates the problem the design has to actually solve. Reprocessing a tick recomputes high, low and close to the same values, because max, min and last-write don't care how many times you apply them. Volume doesn't work that way. A re-applied tick adds its volume twice and the bar quietly inflates — and because the write is a full-state upsert, the database faithfully records the wrong number.

So processing is deduplicated by stream ID before anything accumulates. A tick whose ID has already been applied is acknowledged and skipped rather than re-run. The idempotent upsert everyone points at is real, but it protects the write, not the accumulation in front of it, and only the second of those was ever at risk.

A tick is also applied to all five intervals before any of them is written. Applying and writing interval by interval means a failure partway through leaves some bars updated and the rest permanently missing that tick — and since Go iterates maps in random order, which bars diverge changes every time.

Shutting down

On termination every open bar is flushed to the database. This is a safety net rather than a load-bearing batch write, since each tick already wrote synchronously; it exists so a deploy can't lose the few seconds between the last tick and the process exiting.

Where a different shape fits

• The database can do the bucketing. TimescaleDB's continuous aggregates compute rollups from raw rows, which removes the in-memory state machine entirely. That trades write volume and query-time cost for a much smaller application, and it's the better answer if you're storing every tick anyway.
• Exactly-once is genuinely required. Dedupe keys carried through to a transactional write get you there. It costs a write-path uniqueness check on every message, which is real money at tick rates.
• Late data matters. Truncation is processing-time-ish bucketing that happens to work because ticks carry their own timestamps and arrive in near-order. A feed that replays hours late needs event-time windows with an explicit lateness bound, and bars that can be revised after they close.

The aggregator is small, and nearly all of it is the same idea from different angles: write the whole truth every time, so that being handed the same tick twice, or dying halfway through, costs you nothing you can't recompute.