I like the article, but I think there is a clear misconception of what a SPOF is, and I've noticed this happening across many engineers and engineering teams lately.
A system, like Pub/Sub, being a critical dependency or hard dependency does not make it a SPOF. Otherwise any system is a SPOF, creating absurd architectures that end up with more complexity and downtime than the original one.
For example, I assume incident.io are yet to see if their added complexity of multiple event streams does not cause worse availability issues than what they had, by introducing complexity like problems with data consistency.
If the the underlying "system" components are not redundant and highly available, then you have a SPOF. eg. A single machine is what provides Pub/Sub.
In this case, what you have is a hard-dependancy with an avaialbility lower than what you want to achieve, but there are always shared hard-dependancies that they will have. There is no way around this.
What we should aim for is that any system that is a critical dependancy or a hard dependancy with such a big blast radious, to be redundant and robust in itself. Not just the system, but the practices around it.
This is one of those projects where the end goal is to build the scariest button possible. You cannot complete the project until you’ve pressed that button. You do all the work to build the button, then you have to press it.
Never not scary, never not eventually fun, but type 2 fun for sure.
But that's the company that build the thing, which has apparently been told off by the CNCF? Seems like for once the thing functioned as it was intended to function?
I like the article, but I think there is a clear misconception of what a SPOF is, and I've noticed this happening across many engineers and engineering teams lately.
A system, like Pub/Sub, being a critical dependency or hard dependency does not make it a SPOF. Otherwise any system is a SPOF, creating absurd architectures that end up with more complexity and downtime than the original one. For example, I assume incident.io are yet to see if their added complexity of multiple event streams does not cause worse availability issues than what they had, by introducing complexity like problems with data consistency.
If the the underlying "system" components are not redundant and highly available, then you have a SPOF. eg. A single machine is what provides Pub/Sub.
In this case, what you have is a hard-dependancy with an avaialbility lower than what you want to achieve, but there are always shared hard-dependancies that they will have. There is no way around this. What we should aim for is that any system that is a critical dependancy or a hard dependancy with such a big blast radious, to be redundant and robust in itself. Not just the system, but the practices around it.
Disclaimer: I work at the incident company
This is one of those projects where the end goal is to build the scariest button possible. You cannot complete the project until you’ve pressed that button. You do all the work to build the button, then you have to press it.
Never not scary, never not eventually fun, but type 2 fun for sure.
NATS is such a cool piece of technology that deserves more hype than it gets
Given their recentish argument with the CNCF[0] not sure how much I would trust it.
[0] https://www.cncf.io/blog/2025/05/01/protecting-nats-and-the-...
But that's the company that build the thing, which has apparently been told off by the CNCF? Seems like for once the thing functioned as it was intended to function?
I really like NATS and I agree, but im generally hesitant to use it in critical systems due to https://jepsen.io/analyses/nats-2.12.1
What is your take on this?
I still can't get over the demise of Memphis [0]. :'(
[0] https://github.com/superstreamlabs/memphis
Second for NATS.
I've deployed Core NATS as few times now at scale, including within financial trading. It's been a dream to operate.
I have limited experience with JetStream, but coming from Kafka and Rabbit MQ, it seemed nearly as featureful but also easier to maintain.
[dead]
[dead]