Continuously integrate your DNS setup

A little while ago, the service that Netguru used to host DNS for our domains Zerigo suffered from a severe DNS attack that managed to take Zerigo out of business for 24 hours. Aside from the initial panic, it turned out to be not that big of a deal. We managed to quickly move primary domains to aws.amazon.com/route53/ and wait for the others as folks at Zerigo fought off hackers.

Despite the quick fix, the sad realization was that we had a very bad single point of failure for all projects we were hosting. When Zerigo went down, so did most of our projects (over 80%). We decided to move to something a bit more distributed, and since Zerigo supported export of our zones in BIND format, we were looking for something that would work with that too. Our first try was with LuaDBS. They provide a very neat solution to populating the changes to your DNS. You can simply push your BIND described zones to a git repository on GitHub and then share that repository with their user. Once that’s done, they will automatically look for changes after every push to the repo and set your hosts for you. That was perfect for us. However, it didn’t solve our primary issue which was binding ourselves to a provider with a single set of DNS servers (with zerigo it was: a.ns.zerigo.net, b.ns.zerigo.net and so on), and all we had achived was moving to ns1.luadns.net, ns2.luadns.net, and so on. We then decided that our goal was to move to Amazons route53 for good, and given their distributed structure, we would be a bit less exposed to the DNS DDOS attacks.

Lua does support route53 integration but it felt a bit strange to use them just to handle the sync to route53. Instead, we used our beloved CircleCI to build our DNS using cli53 tool that can sync a BIND file to route53. With a quickly assembled apply script (as shown:):

We could configure Circle

to run it after every push. Now we can safely add new BIND file to zones directory in the repo (or update a zone that already is there) and Circle will build our DNS for us! This way of using a standard CI flow for doing other things other than just building software is very fun and useful:

• since it’s git (and github) based, we can look into the history of changes, which makes it easier to track your DNS related issues
• sharing access to a github repository is easy when you need it
• the flow is very intuitive and it’s easy to use. Everyone can learn git and a BIND format is also quite simple
• you can connect the apply script to your in-house jenkins or any other CI instance. We are outsourcing our builds to circle but it will work with any other CI tool as well
• you can trust your tools to do your build for you, and get feedback just when you need it (most of the time we just push & forget, since we know Circle will tell us when the build fails - https://circleci.com/docs/configuration#notify)

We are now using a similar CI flow to:

• configure our DNS servers
• run our chief updates on servers (with use of https://github.com/matschaffer/knife-solo)
• build tests in our projects, of course :)
• deploy our projects to staging and production environments (capistrano)

And that’s our side of the story. If you know or can think of other interesting things to do with CI, let us know in the comments!