Quad9? Not a techie

[1911-CV]

Thank You!

Bwoodcock:

Thank you for your work and for sharing your knowledge on this site.

[bwoodcock]

The DNSSEC for these Schwab hosts was suspected by your folks, even though no other DNS resolver including anycast providers like Google and OpenDNS had any problem with these hosts.

Ok, I've looked into it a little, and my understanding of the situation is that Schwab's DNSSEC was broken. Until very recently, Quad9 was the only recursive resolver operator that validated DNSSEC signatures, so yes, you'd have been able to get n answer from Google or OpenDNS, whether it led you to Schwab or to a hijacker. But because this was a recurring problem that Schwab wasn't fixing, we put in a negative trust anchor in May, after which you'd get answers for them, even though their DNSSEC signature was broken. Now that other recursive resolver operators are starting to enable DNSSEC validation, I imagine they've had to do the same thing.

You should get a follow-up from one of the techs with a more detailed update.

[SkyHawk]

Ok, I've looked into it a little, and my understanding of the situation is that Schwab's DNSSEC was broken. Until very recently, Quad9 was the only recursive resolver operator that validated DNSSEC signatures, so yes, you'd have been able to get n answer from Google or OpenDNS, whether it led you to Schwab or to a hijacker. But because this was a recurring problem that Schwab wasn't fixing, we put in a negative trust anchor in May, after which you'd get answers for them, even though their DNSSEC signature was broken. Now that other recursive resolver operators are starting to enable DNSSEC validation, I imagine they've had to do the same thing.

You should get a follow-up from one of the techs with a more detailed update.

Thank you for the update. It is strange though that I could query Quad9 from other networks across the US and get proper resolution for the same hostnames, just not from my Norcal network Maybe some of your servers or sites were not validating DNSSEC, that could explain it.

Anyhow I will be on the lookout in email for any additional info, thanks again!

[bwoodcock]

It is strange though that I could query Quad9 from other networks across the US and get proper resolution for the same hostnames.

There was a problem getting the negative trust anchor functioning on the San Francisco server cluster, so it was continuing to DNSSEC validate Schwab's domains while the other sites had stopped trying to do so.

Remember that "proper resolution" of a domain with an invalid signature is to not resolve it. The behavior you were looking for is to ignore the signature, and that's what we're now doing. But it's not a safe thing to do. If Schwab tells us that the data we're getting is wrong, we shouldn't give it to you, because the presumed reason they'd tell us it was wrong is because it was someone's man-in-the-middle hijack attempt. In fact, though, they're incorrectly asserting that the data they're serving is wrong. So we have to ignore the fact that they're telling us that the data is wrong and give it to you anyway, and that's a one-off special case. Which will have to be reversed back out again at some unknown point in the future when they figure out what they're doing and get their security in order. And if their security is in this bad shape, honestly I'm not wildly optimistic that they'll figure out who all they need to go back and ask to remove negative trust anchors, to get it working again. So. Bad situation, bad solution, technical debt incurred, messy clean-up for lots of people at some point in the future.

[SkyHawk]

Remember that "proper resolution" of a domain with an invalid signature is to not resolve it.

Sure, I get it.

Anyhow, right now those hosts/domains are passing DNSSEC tests when I run analysis with tools from Sandia and Verisign, presumably you don't need NTA anymore?

https://dnsviz.net/d/stream.schwab.com/dnssec/

https://dnsviz.net/d/streetsmart.schwab.com/dnssec/

https://dnssec-analyzer.verisignlabs.com/stream.schwab.com

https://dnssec-analyzer.verisignlabs.com/streetsmart.schwab.com

[SkyHawk]

PS for the OP, I just noticed that Quad9 does not do EDNS at 9.9.9.9

But they do it at 9.9.9.11

What does this mean for you? Using 9.9.9.9 could result in some performance issues with sites like Facebook/Youtube (videos), Apple (app downloads or updates) and plenty of others.

9.9.9.9 will give you the best anonymity if you care about such things, like you are a Chinese dissident or you buy your heroin on the darkweb or something. But 9.9.9.11 will give you the best routing to the sites you visit. You wont be sent to Europe or India to get your next IOS update for example, the difference between many hours versus some minutes in my experience.

I personally would never use an anycast DNS resolver without EDNS - but I also don't do shady stuff on the Internet and I don't lose sleep at night worrying about leaving breadcrumbs for the PRC or CIA.

[bwoodcock]

I personally would never use an anycast DNS resolver without EDNS.

Unlike other recursive resolvers, Quad9 allows you to choose whether or not you want us to pass your identity along to CDNs via EDNS CS... But I've never understood why anyone would want to. I mean, presumably you don't walk around with your name and address and phone number on the outside of your clothes, and written on the outside of your car... Why gratuitously give private information to people who haven't asked for it, particularly when that normalizes companies that shouldn't be asking for it penalizing people who choose not to give it to them to monetize?

Again, it's your choice, but the vast majority of people prefer not to label all their queries, and I'm curious why you'd choose to do so?

[SkyHawk]

Unlike other recursive resolvers, Quad9 allows you to choose whether or not you want us to pass your identity along to CDNs via EDNS CS... But I've never understood why anyone would want to. I mean, presumably you don't walk around with your name and address and phone number on the outside of your clothes, and written on the outside of your car... Why gratuitously give private information to people who haven't asked for it, particularly when that normalizes companies that shouldn't be asking for it penalizing people who choose not to give it to them to monetize?

Again, it's your choice, but the vast majority of people prefer not to label all their queries, and I'm curious why you'd choose to do so?

I love that you give a choice! Very slick. My real world experience with anycast DNS and sub-optimal routing before EDNS/ECS was widely in use, for any destination serviced by a CDN, was a nightmare (if you consider your wife and kids being angry and constantly complaining, a nightmare). It literally took hours to get something that should have taken minutes. Apple appstore effectively unusable. And Facebook was unusable as well. The routes were too long, the latency that was introduced was untenable. And more properties are moving to CDN every day.

And a proper ECS query does not pass a complete client IP, rather a truncated IP (at least not if you are outside China where resolvers regularly pass the /32), so the comparison to walking around with my phone number printed on my T-shirt seems a bit of a stretch. I would not mind walking around with my area code and prefix printed on my shirt if it meant I didn't have to walk to Nairobi just to take out my garbage every week.

A lot of time and effort has been put into EDNS/ECS, and it wasn't done because people could just take it or leave it without ever noticing the difference. I believe other large resolvers support it by default (except Cloudflare and Quad9), and I would bet there was much discussion at Quad9 whether to support it on the premiere 9.9.9.9 by default, or on some other (less snazzy) IP. I understand the privacy vs speed conundrum, and privacy is a very real thing but believe me so is speed.

Hopefully the future will bring us even more methods to marry additional privacy and optimum routing in DNS queries for CDN hosted properties, without making the size of zonefiles and caches unworkable

[Librarian]

Poking around for DNS I ran across this article - https://www.lifewire.com/free-and-pu...ervers-2626062 and he lists the major ones

Best Free & Public DNS Servers
Provider 	Primary DNS 	Secondary DNS
Google 	        8.8.8.8 	 8.8.4.4
Quad9 	        9.9.9.9 	 149.112.112.112
OpenDNS Home 	208.67.222.222 	208.67.220.220
Cloudflare 	1.1.1.1 	1.0.0.1
CleanBrowsing 	185.228.168.9 	185.228.169.9
Alternate DNS 	198.101.242.72 	23.253.163.53
AdGuard DNS 	94.140.14.14 	94.140.15.15

I also saw this, and do not know if it is accurate: But I just dumped 8.8.8.8 out of my DNS settings; maybe a little horse/barn door ...

[bwoodcock]

Poking around for DNS I ran across this article - and he lists the major ones:

I would just note that those are only the IPv4 addresses... The corresponding IPv6 addresses for Quad9 are:

2620:fe::fe

2620:fe::9

While many people don't really think about IPv6, it's how a lot of the core of the Internet works now, and it's how all the new parts of the Internet work, so if your ISP is relatively modern, using an IPv6 address may get you faster, more directly routed, service.

[bwoodcock]

My real world experience with anycast DNS and sub-optimal routing before EDNS/ECS was widely in use, for any destination serviced by a CDN, was a nightmare (if you consider your wife and kids being angry and constantly complaining, a nightmare).

I certainly don't dispute your experience, and the points you make are all valid. There are reasonable people on both sides of this issue, and that's why we're technically agnostic, and give people both options.

That said, I think you're perhaps being more generous than is warranted regarding the motivations of the people who drove ECS through the IETF process. I believe they were doing it to make more money for their employers, not to improve users' lives, and I believe that motivation informs how CDNs use the protocol now.

But your use case, keeping the wife happy, does in fact trump any principled debate on the issue. :-)