The internet relies on DNS servers that do the dirty work of translating human readable domain names to something that makes sense to a device on a network, which is where you see four sequences of numbers separated by a period.

I have been updating my homepage at andrewcanion.com such that it displays the three latest blog posts I have published by leveraging the RSS feed generated by Wordpress. RSS feeds provide a method for other sites and services to subscribe to the work of the site offering the feed. Each new entry is included in the RSS feed, and for my homepage, I grab the latest three entries for prominent display.

What should have worked, didn’t

I had assumed that this would all work well until I stumbled across a strange problem. My site would only occasionally load and display the article entries contained within the feed. Other times the articles would not display, instead pausing in a state of permanent loading. I tried the site across a number of devices, and a number of browsers to eliminate a problem existing at the device level. As they all exhibited the same issue I presumed the problem must be occurring at a deeper level of the network.

I examined the source code of my site and it appeared to be fine. I checked to make sure the Yahoo API that grabbed the RSS feed and translated it for use on my home page was still active, and it was.

Next I tried connecting to the site through a VPN, which makes it appear that my point of origin was somewhere other than through my own internet connection. When using a VPN the site loaded every time. This was great to discover, as it moved me another step closer to the identifying the problem.

That problem had to be related to my ISP (or in the new nomenclature of Australia’s NBN, my RSP). Somehow, that provider must have been causing a problem because when I routed around them with the VPN (on any device) my site loaded completely. Yet when I visited the site on any of my devices that didn’t connect through the VPN there was a problem.

My theory is that my RSP is aggressively caching content to attempt to reduce global bandwidth consumption and that this is preventing my RSS feed from updating correctly. If my caching theory isn’t correct, then it must be some other shenanigans they are up to at the network level, no doubt to reduce their bandwidth bill.

Applying the fix

To resolve the problem I needed to reduce my reliance on my provider’s infrastructure. That meant transitioning to a different DNS provider rather than using the default, which is the DNS server of my RSP. I chose to connect to OpenDNS. Basically this means I have traded in the internet lookup tables that came with my broadband subscription in favour of an offering from a third party whose primary business it is to provide good DNS. Through the nature of their product and their business model they are incentivised to provide excellent DNS services. It is their core business. For my RSP, however, the provision of DNS services is a necessary sideline and their key driver is not to deliver excellent routing, but rather to use it as a point of leverage to reduce their own bandwidth costs to improve the profitability of their company.

As soon as I switched to OpenDNS my site loaded perfectly in every browser, on every device. My detective work had paid off and my willingness to not accept the defaults has improved the situation.

The only potential downside I was worried about was that OpenDNS might be a little slower to resolve sites simply because the distance to their server might be further than the default DNS server. I needn’t have worried though, because if anything, I think it might be a little bit faster.

The lesson

Companies all share an incentive to maximise profitability. How they go about achieving this can vary greatly depending on their product and their business model. My internet provider only needs to provide a service that is ‘good enough’ for the majority of normal customers that want to browse the web and check Facebook. If they can deliver that to satisfaction and save some money on the back-end with caching and other network tricks, they’ll do it, even if it creates some edge-case problems.

I’m an edge-case and I wanted excellent DNS services. To get these I had to go to a company that is incentivised to provide quality DNS management. For them, only by delivering on that promise can the business generate revenue and grow its own profitability.

The character of Lester Freeman sums this up in this slightly NSFW scene from The Wire.

Follow the money.