VTHacks and why hackathons are essential for everyone

Last weekend I had the pleasure of participating in VTHacks as a sponsor and judge representing Rackspace. This was not my first hackathon, but it was definitely one of the largest I have attended.

Making runbooks more useful by exposing them through monitoring

As mentioned in a previous post about our server message of the day (motd) we manage enough infrastructure that our team needs ways for being able to share tribal knowledge on how to fix issues. In Particular when we have monitoring alarms that kick off any of our team members should be able to take action on the alarm regardless of level, however, our team operates with so many different technologies it is difficult for any one person to be an expert in all of them.

Multi-region logging architecture with Logstash, Kibana, and ElasticSearch

On my team right now we are using rsyslog with Graylog2 and ElasticSearch to handle our logging infrastucture. The current setup is not ideal as we are distributed multi-region for our application in 3 datacenters (ORD, DFW, SYD) and each one has it's own cluster setup to use Graylog2 and ElasticSearch. This means if someone wanted to search through logs you would have to pick that specific region's Graylog2 instance. The original reason for this setup was that we had our logging infrastructure setup before multi-region was in place and we had to make a decision about how much time we wanted to spend setting it up.

How a server MOTD improved our DevOps team

Our Infrastructure team for the Cloud Control Panel at Rackspace has around ~200 public cloud servers across production, preproduction, staging, and test environments. At a high level our general layout for hosting the Cloud Control Panel includes nodes of several different types. We have load balancers running apache. Web nodes which serve the base content with Django. Javascript served from a cdn (content delivery network). Twisted servers which proxy requests for making calls to Rackspace apis from the frontend. A cluster of Cassandra nodes for managing sessions, preferences, and api cache data.