Configuring Docker data storage a 101

This is a short walkthrough on configuring Docker storage options on your development machine.

I’ll use my preferred version of Hello world on Docker – “Setting up mongodb” which lends itself nicely to a walk through of the storage options.

This walkthrough assumes basic familiarity with Docker. First let’s look at setting everything up on a single container.

I started from the DockerFile described here
mongoDB Dockerfile for demoing Docker storage options

Creating the image using

docker build -t mongodb .

You will note that in this Dockerfile we use the VOLUME command to define the target data directory for mongoDB

# Define the MongoDB data directory
VOLUME ["/data/db"]

I am walking through all this on my Mac thus I am using the following lean & mean command to start a mongodb container up as a background process ( daemon) from the mongodb image created from the docker file :

docker run -p 27017:27017 --name mongo_instance_001 -d mongodb --noprealloc --smallfiles

I can then add some data to a mongodb collection ( see Data loading below) That is quick and for some quick tests as part of a SDLC that might be fine but having to recreate your database and reload each time you create a container will eventually prove limiting.
We all know that you need representative datasets for a true test and it’s likely that your datasets are going to be more than 118 records and reloading data every time you run up a mongodb container is not going to be practical!

So we have two options as to how to address the persistance requirements:

  1. Data volume
  2. Data volume container

Data Volume

We will want to create a volume that maps to a folder on your local host in my case I will be mounting a folder on my Mac called $HOME/mongodata ( replace $HOME with your folder name if you are following this through on another OS )

We then create the container from the image but the difference is we now get the container to mount the local folder using this command to create a container:

$ docker run -v $HOME/mongodata/:/data/db -p 27017:27017 --name mongo_instance_001 -d mongodb --noprealloc --smallfiles

Note that as virtualbox shared folders does not support fsync() on directories mongodb will not actually start but you can validate that the mounting of a shared folder on the host works as the logs will show the error and you will see that it created some files in the shared folder before it halted. This part of the walkthrough will work as expected using mongoDB on AWS ec2 for example and is perfectly valid for those applications that do not require fsync() if you are using virtualbox.

Data volume container

This option in my opinion is the most flexible.

First you need to create a data container

docker run -v /data/db --name mongodata busybox

The above creates a data volume contaner based on the busybox image. (Its a small image)

Next you need to start up the application container but this time mounting the data container created earlier

docker run -p 27017:27017 --name mongo_instance_001  --volumes-from mongodata -d mongodb --noprealloc --smallfiles

Load some data into mongoDB

To validate this works as expected stop container 1 then start another container using a similar start up command attaching the Data volume container

docker run -p 27017:27017 --name mongo_instance_002  --volumes-from mongodata -d mongodb --noprealloc --smallfiles

You can check that now when you start mongoDB and look at the databases and collections that the data you loaded using the previous container is available.

You can remove the application containers whenever you like and create new ones as required mounting the data volume container. Note that using the docker ps command does not give you any indication of what containers are mounted to the data volume container .
You can also tar the data volume and copy to another docker host etc see the docker docs for detail on the process

Data loading

I am assuming some familiarity with mongoDB . If you need a quick primer have a look here: Getting started with mongodb

I am using a json file that consists of a dataset of the elements of the Periodic table to populate my database. Here’s how I load my demo databases with data :

mongoimport --db demo --collection periodictable  --type json --file periodictable.json  --jsonArray 

For the purposes of this walkthrough I am using images that are on my local machine rather than pushing up to a registry and pulling back down again.

This walkthrough has been focused on the practicalities of storage with Docker for a deeper dive on storage have a read of this excelent post  on the Overview of storage scalablity in Docker on the RedHat developer blog

Scaling out the security attack surface when using Docker – A timely reminder

With all the excitement over Docker some folks seem to forget that it’s more than just making life easy for developers. This stuff will need to be exposed to the big wide scary world and exploits such as the Bash vulnerability will be dispersed over a wider landscape than just the hosts themselves!

Yes you might point out that containers are being managed at scale by the likes of Google but they do have the resources to look after the infra so you don’t have to!

Remember the tools and processes you use today to manage patches will need to be applied up the stack as well and that means look to your docker images and containers too.

If you really are running immutable infrastructure and can afford to tear everything down and throw updated Docker images out there then that is an alternative path although you still need to worry about the underlying hosts even in that scenario.

Daniel Walsh from RedHat has a great series he is writing on Docker security and how RedHat are dealing with the issues. This is a great read and brings a little sobering realism to the areas that still need to be thought about when deploying Docker based solutions

From Daniel’s posts I want to reiterate this list of good practise as a timely reminder

  • Only run applications from a trusted source
  • Run applications on a enterprise quality host
  • Install updates regularly
  • Drop privileges as quickly as possible
  • Run as non-root whenever possible
  • Watch your logs
  • setenforce

Docker what’s all the fuss about?

It’s been a while since I’ve blogged here and as I’ve been looking at the hottest thing in “hipster tech” ( see below for a defintion) in Docker thought I’d get that blogging mojo back by starting to share my thoughts on that subject!

For a detailed description of Docker there are plenty of great articles, slide decks and videos. The Docker site is a good starting point and this page what is Docker has two diagrams that graphically depict what Docker is about versus a VM. This post from Sleekd discussing the difference between Docker vs Virtualization is also a nice background read for the layman   so I won’t be repeating a Docker 101 here. To set the scene though I summarise Docker like this :

  • Provides Operating system level virtualization. containers run user space on top of an operating system’s kernel. That makes them lightweight and fast.
  • It uses resource isolation features of the Linux kernel such as cgroups and kernel namespaces to allow independent “containers” to run within a single Linux instance
  • It uses the power of Linux containers (LXC) ( although more accurate to say it has  evolved from here ) and aufs (Another Union File System) to create a way of packaging and process isolation
  • It Allows you to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package.
  • Docker allows applications to use the same Linux kernel as the system that they’re running on and only requires applications be shipped with things not already running on the host computer. This gives a significant performance boost and reduces the size of the application.
  • Ultimatley It should provide more certainty for application developers by providing a set of known abstractions that define how the application will run, no matter what hardware is underneath.

For a good initial deep dive on containers my current favourite slide deck is ths one : Inside Docker for Fedora20/RHEL7

Okay let’s start with a list of the fundamental issues that need to be solved first for mere mortals(see below for a defintion) to really get on board. No depth just headlines at this point with pointers to more info. Else I’d never have even got this post out to kickstart blogging again!

It’s early days and has a blossoming growing ecosystem. This lovely mindmap Makes a nice stab at illustrating the ecosytem that has been building around it ( Personally I would not have placed some of the tech in the sections they have been put in but hey it’s still lovely). Equally it shows the bewildering choices that have yet to be made with regards which if any approaches will win out and what may or may not suit your environment.

The potential however is huge and I think with a list of
USP’s that have what I’ve listed below you can begin to understand the rallying round and the fuss.

  • Simplifying the use of containers ( Container technology is not new despite the hype)
  • Micro services
  • Portability
  • Mutable infrastructures
  • PaaS solutions

In upcoming posts I’ll discuss some of the ecosystem tools where Ive had hands on ( promise it won’t take years though) , some of the issues and the USP’s in more depth .

This post was just to get me into blogging mode again !

I won’t however be neglecting my passion for Devops though ( It’s not just about the tools stoopid! although I’ll probably be talking about the tools a lot!!)

Definitions:

My defintion of Hipster Tech – Latest cool idea in Tech where the hype has over taken the reality but the potential is very high on the potential-ometer.

Thanks to @chrismunns for the succinct definition of mere mortals – Those running responsible and performant production environments
( My spin i.e no time to get distracted by debugging and feeding & watering the tools they use to deploy those solutions but focusing on deliverng value to their business)

 

Windows in the cloud a 1st class citizen

The perception is that running windows instances in the cloud is often as a second class citizen. This just not true. Both Opsocode and puppets lab have made great strides in  making their configuration management tools ‘windows friendly’  (Disclaimer:  I’ve used Chef with windows no actual experience with puppet). To add to this Amazon Web services introduced Cloudformation friendly windows  base AMI’s . The combination of these AMI’s with the more windows friendly configuration management tools means you can really treat windows as you would Linux instances and use the same tools to manage windows instances as you would Linux.

You can use PowerShell as you would normally so the learning curve isn’t as steep as you’d expect as a windows administrator.Go on give it a go.

If you have an estate that is made up of both windows and Linux  starting from a point where you can use the same tools to mange both environments makes life easy for your Operations/ DevOps or whatever label you place on the team that makes sure you have systems that are up and running each day.

One tool to manage them all

 

I’ve been waiting to give the chef-client MSI a try ever since I noticed it had been released. I wanted to see if it really has made the numerous ( albeit fairly straight forward) steps to get chef-client working on Windows 2008 R2 that much easier.  After all the easier it becomes the more converts there will be as the barriers to adoption are removed.

Running the MSI is simple. It takes care of installing ruby( version 1.9.2p290)  and installing chef-client . Now all you need to do is set up a couple of files to allow your client to authenticate with your Chef server as detailed quite nicely here:

http://wiki.opscode.com/display/chef/Installing+Chef+Client+on+Windows

That’s it you’re good to go.  First impressions a big thumbs up.

I then had a quick look into how things have got better in the windows  recipe development  department. I started by checking out

The opscode supported  windows cookbook

This is looking really promising as the ability to install roles & features, and more importantly install MSI’s can be treated in the same way as you would install services and install packages  on Linux . Meaning you could actually have one person who is capable of  writing  high level recipes for both platforms. You will always need someone who understands the target O/S but this just means you can get admin staff using Chef  ( yeah I know I’m talking about the hosted version)  and it doesn’t really matter which O/S they are more comfortable with . Opscode in my  humble opinion have removed a layer of obstruction to adoption by the work they’ve done here.

Blathering on

Last night ( 8th Sept 2011) I had the privilege of being interviewed together with my ex colleague and good friend James  by Richard & Carl of  . NET Rocks!  to talk about DevOps, how we see it evolving , what we think it means , the effect it is having on the industry just for starters.   The fact this  is on  a show aimed at .NET developers is a statement in itself.

It’s  Due to be  broadcast on the 13th  September 2011.  It will be interesting to see what folks think as it’s not the usual type of subject covered. James & I are both passionate about the subject and  I  must confess I kept forgetting during the interview  that I wasn’t there just to listen to James,  but his eloquence about the subject with or without the DevOps label is always compelling . Please feel free to blame Howard of endjin for  suggesting letting  James & I loose on your ears though Smile

The interview is here:  .NET Rocks! Show 697

Targeting Windows 2008 R2 nodes using chef

Just a quick note.

I’d advise sticking to ruby 1.8.7-p344 on the target node if you are targeting windows 2008 R2.  I recently revisted targeting windows 2008 R2 and found that using  the latest version of ruby  1.9.2-p180 on the windows 2008 r2 target node  and  attempting to run chef-client  after installing the chef gem is a proverbial pain . I’m not sure if Opscode are looking into this but  it’s easy to reproduce the pain :-)

Using AWS for DR when your solution is not in the Cloud

In my previous post  in this series on resilience of non cloudy solutions I discussed how to approach obtaining exactly what was acceptable to the business to achieve an appropriate DR solution . In this post I will look at a fairly high level at  how to exploit AWS to  help  provide a cost effective solution for DR when your solution does not actually use AWS resources and is  probably not designed in a decoupled manner that  would make it easy to deploy to the cloud  .

yes I know know I can’t help it the cloud is here after all Smile

Please note that by necessity I’ve needed to keep to a high level as if I were to attempt to  start exploring the detailed configuration options  I’d still be writing this post by Christmas. Needless to say this post just scratches the surface but hopefully provides some food for thought.

You  will or should have  have local resilience in your solution consisting of multiple application servers and web servers , clustered database servers and load balancers .

The easiest DR solution to implement but the  most costly  is to replicate this albeit with maybe not so many servers and perhaps a single Data base server instance  to  an alternative physical location and putting in place processes to replicate data across to the 2nd location .

This typical configuration will look something like this:

std dc replicated

There are  plenty of variations on this but in the end it  entails physically maintaining a distinct location which replicates the application architecture and associated security controls . Resources need to be in place to support that location;  keep the components  updated regularly and all the usual best practises need to be acted upon to  validate the solution . It’s no point finding out the solution doesn’t’ work when you need it.

 

At this point you should hopefully  be thinking that is a lot of investment for something that will only be rarely used . So here’s where AWS can help keep those costs down.

 

The first model  which I’ve called the ‘halfway house’  may be an option for those who are unable to make use of the full AWS resources available and for whatever reason are unable or unwilling to store their data there . It still requires two maintained DC’s but saves costs by having the application and web servers for resilience being AWS instances. the cool thing here is that those resilient servers/instances are not actually operational unless needed ( you would have prepped AMI’s and hopefully use them in conjunction with a configuration management tool to ensure they are fully up to date when launched) .  You will not have  have the over head associated with watering & feeding them that you would have if you were 100% responsible for the infrastructure. The core  AWS components that make this work are: EC2,VPC and ELB .  If you wanted there is also the potential to use Route 53 to manage the DNS aspects that are needed for routing externally .There are issues with this model though  such as the possibility of a lack of capacity when you need to spin up those instances ( although the use of Multiple AZ and regions should over come that fear), the over head associated with managing 3 sets of resources,latency issues just to name three that come to mind.

The ‘halfway house’   will look something  like this:

 

Part use of AWS

Making use of AWS VPC means that you can create virtual networks built upon the AWS infrastructure which provides you with a great range of networking configurations for example  in the diagram above  I’ve show two  group of instances, one  that is externally accessible and another set that is basically an extension of your private LAN.  there are far too many scenarios possible with just these features of AWS and obviously every application is different ( See why I made sure this post was kept at a high level)

The  nirvana though to really seeing the costs tumbling  is to get rid of DC 2 and use AWS as the Recovery site. as a bonus it can be used for those extra processing needs as well on a demand basis . This not only reduces the support over head, saves cost as you are no longer committed to paying for a second location with all the associated kit necessary to make it a viable alternative site , but  it also provides a wide variety of failover and recovery options that you just won’t get when you have to commit to infrastructure up front ( hopefully that  pre-empts the question about why not a private cloud – you need your own platform).

This model which I’ve called the ‘Big Kahuna’ can look  a little like this :

 

big khauna

With the ‘Big Kahuna’ you should make use of any of the AWS resources available. In the flavour above I’m using S3 to store regular snapshots / transaction logs etc from my primary database. Why not replicate directly? Well s3 is cheap storage and in the scenario I’m illustrating as an example my RTO and RPO values allow some delay between failure and recovery that I can reconstruct the database when needed from the data stored in my s3 bucket . Regular reconstruction exercise should occur though as part of the regular validation of the failover processes. AMI’s and a configuration management solution ( As it’s me it will be chef) are used  to provision up to date application and web servers. Use is made of Route 53 to facilitate DNS management  and Where I need to ensure that traffic is kept internal I’m making use of VPC .

The introduction of RDS for oracle  means it is viable to use AWS as the failover solution for enterprises. There may be concerns over performance but this is a DR situation so if you are not in a position to reengineer for the cloud then when discussing with internal business sponsors discussions about reduced performance should be part of the business impact discussions.

AWS has services  such as dedicated instances which may be the only way your security and networking guys will allow you to exploit AWS resources but you would need to do your sums to see if it makes sense to do so. Personally I’d focus on trying to understand the ‘reasons’ for this . There are a number of valid areas this would be required but I suspect cost   isn’t really going to be any sort of driving force there.

The devil  is in  the detail when designing a failover solution utilising AWS as part of your DR . If you are planning for a new solution make sure you talk to the Software architect about the best practises when designing for the cloud it’s still applicable  for on premise solutions too .

Data is really where all the pain  points are  and will likely dictate the model and  ultimate configuration.

If you are trying to retro fit for an existing solution then the options open to you may not be that many and it’s likely you will have to start off with some form of the ‘halfway house’

Also don’t forget you can  just try stuff out at minimal cost. Wondering if a particular scenario would work just try it out as  you can just delete everything after you’ve tried it.

The cost effectiveness of the solution is directly related to the use you make of AWS resources to effect the solution. I even have a graph to illustrate ( @jamessaull would be proud of me) .

 

awsdr graph

This graph is based on very rough comparative costs from starting off with no AWS resources as in the first situation I started discussing and working my way down through to the ‘Big Kahuna’. You can easily do your own sums .AWS pricing is on their site they even provide you with a calculator  and you know how much it costs for those servers, licences, networking hardware,hardware maintenance costs support etc.

How to deal with the ‘ I want no downtime response.’

With a little time between jobs and with having to dodge the rain  I found myself thinking about some stuff I’ve not really given a lot of deep thought recently so here’s the first of a couple of posts on Resiliency when applied to non cloudy solutions.

I recall numerous occasions where I would design a highly resilient solution that would provide as many 9’s as I’d dare to commit to only for the proposal to be back on my desk some weeks later with the words too expensive where can we save money on it?

The main reason I would come up across this is that I would ask the Business Sponsor two key questions related to Disaster recovery as part of the business Impact analysis:

 What is their RTO?   Recovery Time Objective - The duration of time from point of failure within which a business process must be restored after a disaster or disruption.

What is their RPO?   Recovery Point Objective – Acceptable amount of data loss measured in time

These two questions would inevitably elicit the response no data loss and restore service as quickly as possible, so I would go  off and design a platform to get as close to those unattainable desires.

I soon learnt that a different approach was needed to prevent the paper bounce game.  Eventually coming up with  gold, silver and bronze Resiliency options as part of the platform proposal . We would include what we felt were viable Resiliency options graded according to arbitrary RTO & RPO levels and the associated costs to deliver on those.  

This at least then meant the Business sponsor had somewhere to start from in terms of a monetary value  and  could see that wanting the Gold standard was going to cost them loads  more than if they really thought it through . This got them thinking about what they really wanted in terms of RTO and RPO and we would then discuss what options were open to them.

 For example does the accounting system need to be restored within 2 hrs with less than half an hour’s loss of data? In my experience it’s  only at month end processing  this level of recovery is really needed.  During the majority of the month data is held elsewhere so can be easily recreated, the months accounts file is still open so this isn’t an issue  in terms of processing. Is anyone working on the system at weekends?  This is the sort of thought process that the Business needs to go through when requesting new systems and trying to figure out what they want in terms of resiliency.

 When faced with stark questions about RTO’s and RPO’s the natural response is I want my system to be totally resilient with no downtime they have no idea what this means in terms of resource and thus potential cost so why not save time by giving them some options . You may be lucky and one of the options will be an exact fit but if not the Sponsor has an idea of what it costs to provide their all singing all dancing requirements.  It’s more likely they will say something like “The Silver option may do but we need an enhanced level of support at month end ” or  maybe something like “we need to make sure we have a valid tested backup from the previous night at month end “.

The beauty of this approach  is that straight away you’ve engaged them in conversation and prevented a paper bounce . It’s not a new approach and one service teams are used to when dealing with external suppliers but there is no reason not to use the same methods with internal requests.