Once you decomposed the pipeline into stages, did observability become significantly easier? I imagine stage-level latency and queue depth give you a much clearer signal than profiling a single async service.
Exactly. Once each stage in the pipeline lived as its own job in the queue, observability got dramatically better. Stage-level latency, queue depth, and failure rates made bottlenecks obvious in a way that was hard to see inside a single async service. It also made it much easier to tune concurrency per stage instead of guessing globally.
Once you decomposed the pipeline into stages, did observability become significantly easier? I imagine stage-level latency and queue depth give you a much clearer signal than profiling a single async service.
Exactly. Once each stage in the pipeline lived as its own job in the queue, observability got dramatically better. Stage-level latency, queue depth, and failure rates made bottlenecks obvious in a way that was hard to see inside a single async service. It also made it much easier to tune concurrency per stage instead of guessing globally.