It's always DNS, so why are we running our own?

(s2.dev)

38 points | by misker1 6 hours ago ago

28 comments

  • reticulates 3 hours ago ago

    I'm not familiar with your product, just the blog post + cell architecture documentation. Question: the documentation states that basins are assigned to a cell on creation and cannot move. Why choose to have explicit DNS records instead of wildcards with a subdomain per location? e.g: {basin}.us-east-1.s2.dev where *.us-east-1.s2.dev points to the cell? My best guess is that you prefer to have explicit architecture in case you need to move to a new cell for some reason, but maybe there was a different reason? The stuff around hitting the database on every query that necessitates a firewall seems like a huge risk vs. a wildcard CNAME which adds no additional burden.

    • reticulates 2 hours ago ago

      Also, unsolicited advice but having your website, integration endpoints, status page and email under the same domain is a bad idea, a single point of failure. Especially a domain that is due to expire in just a few months. You should have a separate domain for infrastructure and a separate status page domain, each renewed for 10 years in advance (the maximum). e.g: s2.dev for website, s2status.com for status page and s2stream(s).com for your Basin URLs.

      • misker1 2 hours ago ago

        Thank you for the advice, the status page move makes a lot of sense! I don't think we have the bandwidth to separate the basin subdomains just now, but very helpful in terms of SPoF.

        edit: oh, and the 10 year renewal, that's the top of the list.

    • misker1 2 hours ago ago

      The main two reasons: we have multiple cells per location (we want to keep cells decent in size to reduce failures) and a move between cells is something we have on the roadmap. I tried to find a wildcard solution at the beginning of building this, I just couldn't find a good one.

  • bajard 2 hours ago ago

    My ISP kept intercepting, editing and blocking my DNS queries. I had to encrypt, obfuscate, self-host, re-route and randomize them to get rid of this harassment.

    • roncesvalles 2 hours ago ago

      Just use DoH? It's generally understood that most ISPs fuck with port 53 DNS traffic now.

      • eddythompson80 2 hours ago ago

        Doh is not as cool to say encrypted, obfuscated, rerouted and randomized.

  • whinvik an hour ago ago

    I am very curious how they decided to have most infra in GCP but only DNS on AWS Route 53. I would expect that one could have everything on just GCP no?

    The other interesting thing that I am starting to think more about is that every part of the infra stack will get slowly rebuilt because of AI. More load, more granular permissions, more firewalls are some of the things I can think that will need to be built in.

    • justinc8687 29 minutes ago ago

      I for one, do the same with my company. I've thought of "worse case scenarios" and the one I think among the worst is my AWS account gets hacked and ransomed. Having my DNS hosted with another provider gives me a quick way to rebuild the stack from off-site backups, switch over DNS and tell the hackers to go f$%$ themselves. If my DNS is hosted with AWS or God forbid, my registrar, then I'm just boned.

    • saldoran 44 minutes ago ago

      [flagged]

  • MiscIdeaMaker99 an hour ago ago

    Back in the day, I remember running our own DNS servers because we had customers who wanted to white-label our product with their own domains, and _lots_ of marketing departments didn't want to deal with their IT people, so we offered to host DNS for them. Of course, that's assuming they had IT people at all.

    About half of the domains were registered by us on behalf of the customer, while the other half delegated a sub-domain to us. There was also a small minority who hosted all of the DNS themselves.

  • getfacl 3 hours ago ago

    Out of curiosity, was using PowerDNS with CloudSQL (fed from a spanner stream) considered?

    There are always trade offs of course but it would have the advantage of DNS spikes not impacting spanner.

    • InsidiousWombat 14 minutes ago ago

      We run PowerDNS with MariaDB replication on the backend

      Zero problems for last 15 years. Near-instant propagation.

      Scaling options are great, as the amount of changes in records per second is generally small even if you have hundreds of domains

  • TacticalCoder an hour ago ago

    DNS is incredibly complex IMO. But I still run unbound at home: it s of course not the same thing as running a publicly facing DNS.

    But being able to blacklist hundreds of thousands, heck probably millions, of domains that are homoglyph attacks on my usual banking and brokers website is just too good to pass upon.

    I started doing that after my broker sent me a warning by email about a phishing site that was a 100% perfect copy of the real thing. Basically:

        mycoolbroker.com
    
    vs:

        mycoo1broker.com
    
    Of course I use 2FA etc. But I still wouldn't want my credentials phished and then using those an attacker could try to reset the 2FA.

    So I wrote a little piece of code generating a shitload of potential homoglyphic attacks on my most important sites' names.

    I also blacklist a great many TLDs entirely. So many countries get null routed / NX_DOMAINed.

    And so many ads disappear. And sk many domains serving porn and malware.

    All going down the drain.

    I love it.

    As a side note I also reject trafic from a huge number of IP blocks (entire countries / entire ISPs / entire companies like Meta).

    It s amazing how much of the Web / Internet keeps working just fine after all that.

    It s also pretty much something you set up once then forget about it.

    And I did all that in the pre-LLM days. Now it s even easier.

    • Gareth321 an hour ago ago

      I also run unbound on OPNsense but for almost exactly the opposite reason: evading censorship. Here in Europe so many websites are blocked by DNS at the ISP level due to local or EU laws and edicts. Unbound ensures I can browse a free and open internet free from government and corporate censorship.

  • misker1 5 hours ago ago

    I wanted to share our journey of running our own DNS server (it's not as scary in the age of AI, but still...). If anyone has questions I can answer them.

    • koito17 3 hours ago ago

      If you're running a k8s cluster, you're already running a DNS server. It shouldn't be too scary IMO, but I've also wasted a lot of time writing DNS-related libraries and application in the past, so I'm probably biased.

      I find the following part of the article interesting.

        These [DNS] records would need strong consistency too. When a basin is created, its address should be instantly resolvable, otherwise clients would be waiting around for a cached NXDOMAIN to expire.
      
      DNS is, almost by design, an eventually consistent system. Strong consistency would mean authoritative servers for a zone have to coordinate updates and agree on the current state before answering queries. That would almost defeat the availability goals that date back to the very origin of DNS. To quote the very old RFC 1034:

        Access to information is more critical than instantaneous updates or guarantees of consistency
      
      I am guessing S2 has clients query a specific DNS server directly so that there is strong consistency in practice. Also, it's unclear to me whether it's intended for records to propagate to other name servers. It'd be nice if the article included an appendix for people who don't know what S2 is (or how clients typically access the service).
      • lbriner 3 hours ago ago

        > DNS is, almost by design, an eventually consistent system.

        I agree with this. Offering instant updates creates a large performance benchmark, especially if you experience massive growth. Unless this is mitigated with TTL to some extent? I would personally be looking at how to reduce the database load for misbehaving clients e.g. 5 second cache on all queries or allowing a customer to choose a "tier" where they won't need fast instant updates. If they choose e.g. 5 minutes TTL, you can cache these extensively, even just as disk files rather than RAM.

        • misker1 2 hours ago ago

          I agree with this too and to be really honest we are still tweaking all the caches between DNS Firewall our S2 DNS server and the CNAMEs we hand out. The instant part was more for new basins being created.

      • misker1 2 hours ago ago

        Ah, I wrote strong consistency but what I really meant was atomic, on the creation side. Before, our basin DNS entries were created async in Route53, and they could stay in that Creating status for a long time (DDoS, provider limits, etc.). Now, once the API says 200, the DNS entry is guaranteed to exist at our nameservers.

        Clients use regular resolvers, no special server, accessed via http1/2. The funny thing is that we actually had a client who was relying on DNS being strongly consistent. Since that isn't really feasible, we asked them to add a small delay after creating a basin, and they did.

    • xtiansimon 2 hours ago ago

      I manage my own DNS record at zoneedit (set it and forget it). What’s something else (tricky, instructive, whatever) about running your own DNS that I don’t experience?

      • InsidiousWombat 11 minutes ago ago

        It's not tricky. The problems are entirely due to one or more of

        * Not understanding what DNS is (eventual consistency etc.) * Round peg square hole problem ("oh we rely on the record updating instantly") * Using a Microsoft products to serve DNS

    • slyall 3 hours ago ago

      What is the scale on the graph?

      • misker1 2 hours ago ago

        Daily queries, over ~7 weeks.

        • InsidiousWombat 9 minutes ago ago

          The scale is per day ? So, like 25 queries per second in peak ?

          You could've just used PowerDNS + SQL backend. Near-instant updates, easily can handle far bigger traffic, easily scalable

  • andrewstuart 2 hours ago ago

    You are. Not me.

  • ehteshamaliii an hour ago ago

    [flagged]

  • dominojab an hour ago ago

    [dead]