Very interesting. That would suggest to me that all traffic is passing through a stateful device vs. direct connectivity from a router. Routers can give simple bandwidth metrics. A stateful device or mirrored ports to a netflow analyzer would be required to create detailed connection logs though mirrored ports wont break state such as UDP virtual tables being full. Are they doing detailed logging of traffic?
There was no UDP connectivity from any of our servers for ~10 minutes, but TCP was working fine, I could SSH into them.
Even Hetzner's Robot interface was down.
Edit: https://status.hetzner.com/incident/5e901309-0873-42af-90c5-...
Very interesting. That would suggest to me that all traffic is passing through a stateful device vs. direct connectivity from a router. Routers can give simple bandwidth metrics. A stateful device or mirrored ports to a netflow analyzer would be required to create detailed connection logs though mirrored ports wont break state such as UDP virtual tables being full. Are they doing detailed logging of traffic?
> We are currently experiencing a fault with UDP forwarding.
Exactly right.
From our logs, the main outage went from 21:01 to 21:10 UTC.
EDIT: Also https://status.hetzner.com/incident/affecfea-fd04-41a1-a680-... (Fault report on Cloud API and Hetzner Console)
It seems to be back up. I can access https://console.hetzner.com/ now, and our VPS are back online.
Yeah first time in 6 years being with them that I've had an outage of this scale. Cloud VMs and Dedicated servers all down / intermittent
Everything went down but seems to have recovered now.
They sent out an email about raising their prices earlier today. Perhaps it was everyone running for the doors.
[dead]