Loosely coupled Chef Cookbooks

I’ve been working on a mongodb installation and configuration cookbook which allows me to install &  if required make custom configurations . It allows me to Install and configure a standalone mongodb installation or a replica set.

Developing this cookbook ( still a work in progress )  has led me to take a loosely coupled approach to its development such that I did not want to force a dependency on any previous recipe.  This has  meant a number of rules would need to be followed to use the cookbook properly rather than imposing any  constraints.

So why did I come to this conclusion that flexibility and thus loose coupling was a requirement for this particular cookbook: 

The use of a replica set and the fact you may want to seed the mongodb set up with data  from a backup did gave me food for thought.  When spiking the various configuration  scenarios I found that if I updated my current master via a data dump where I  had decided to stop the master mongodb instance  while I copied the data into the data folder  I found  the data wasn’t being replicated. This was because  one of the slaves had then taken over as the master and you can’t really force a master( without pain )  in a  replica set  ( maybe 10gen can advise on that one although I guess if I’d made sure the owner of the files was correct before the copy I may not have hit the mongo having a fit stage) so I needed to cater for that one.

Seeding mongo before a replica set is created seems like a nice approach to me anyway.

In a standalone mongodb instance I’m probably not worried about Raid so the recipe to create the Raid device shouldn’t be a constraint and while I’m at it why can’t you set up a replica that just uses instances with local storage?  

 I want to add recipes to create the job to undertake regular backups and maybe one to do a restore. But I may not want to use them.

Suddenly the list of things I want my mongo cookbook to do is growing. So I have done what is required  to deliver the functionality that is required  by  the client  I needed to do the Chef work for and   now I can pimp my cookbooks till I’m happy enough to share it with the community .

(The whole ‘just enough’ ethos is something I mean to talk about here but not now)

I want the recipes to be easy to use and understood by people new to Chef and also to mongodb as I do not believe just because you have a sophisticated tool like Chef that should mean your cookbooks should be overly complicated. Keeping it simple makes maintenance easy and encourages others to expand upon it appropriately if they follow the rules. Mongodb is very easy to get up and running so why use a tool to make it suddenly obtuse?

So the  rules to date :

Obviously you need to have mongo installed as a starting point.  I couldn’t really mandate the use of the installation recipe as it may be an existing set up . ( I have to modify this so it brings down a specified version rather than just the latest version from the 10gen repository) .

Each recipe is to be used to carry out a  single function e.g install mongdb, configure the configuration file , start mongodb etc. The combining of functions is discouraged

Each subsequent recipe can be run independently of the others or be combined as a role this meant making sure I had a recipe to start mongodb so this could be dropped in as say part of a role or workflow .

The use of templates and variables to encourage flexibility .

When I get to a point I feel the cookbook is pimped appropriately I’ll post a  dissection and some guidance .

Advertisements

Using CloudFormation to kick off a chef run

Once you decide to use CloudFormation to create your AWS resources you are now unable to use the knife command to kick of an ec2 server creation so you will have to get the client to start the chef run by doing a chef-client run .

The solution described in this post  is simple to implement .It requires  doing a  little scripting at the Instance end by baking that into a base AMI and the use of userdata.

I will use a Linux AWS AMI as my starting point.

The first thing to do is set up your target AMI to be able to  use userdata.

The script below shows the salient parts of an rc.local I have used to facilitate a chef run when an instance is created from the AMI:

gem install chef –no-rdoc –no-ri
# grab userdata then use to construct name of json file
# json file contains run list and is passed to chef-client run
export USERDATA=`/usr/local/bin/aws-get-ec2-userdata`
echo userdata = $USERDATA
export ROLE=$(echo $USERDATA | cut -f 1 -d “:”)
chef-client -j /etc/chef/$ROLE.json

The file /usr/local/bin/aws-get-ec2-userdata  file uses curl ( just like the sample templates from AWS) to return the userdata which is then  stored in the environment variable USERDATA. The first value  which represents the role we want to apply to the node is extracted and saved as the environment variable ROLE which is then used to pass the appropriate json  file which contains that role in the  runlist.

The corresponding  part of a Cloudformation script that creates the EC2 instance resource and passes the userdata looks like this:

“Ec2Instance” : {
“Type” : “AWS::EC2::Instance”,
“Properties” : {
“KeyName” : { “Ref” : “KeyName” },
“AvailabilityZone” : { “Fn::FindInMap” : [ “RegionMap”, { “Ref” : “AWS::Region” }, “AvailabilityZone” ]},
“ImageId” : { “Fn::FindInMap” : [ “RegionMap”, { “Ref” : “AWS::Region” }, “AMI” ]},
“InstanceType” : { “Ref” : “InstanceType”},
“UserData”: {
“Fn::Base64”: {
“Fn::Join”: [
“:”,
[
{
“Ref”: “ChefRole”
},
{
“Ref”: “EBsVolsreqd”
}

]
]
}
}
}
}

The userdata needs to be base 64 encoded  hence the  “Fn::Base64”:  encapsulating this property. The “Fn::Join”: [  “:”, appends the values passed as as single value with  each value separated by  “:”

The line export ROLE=$(echo $USERDATA | cut -f 1 -d “:”)”
in the rc.local uses the delimiter to identify each value and as the ChefRole is the first parameter started it uses this to set the variable ROLE.

When the stack is started you can accept the default role or change it to an appropriate value .

stack

After the stack is complete you can then check to see if it has created the node by looking at the system log :

syslog

and /or using the chef  console  ( I use the opscode hosted platform):

opscode

I think this is a nice  straight forward  way  to achieve a  fully automated end to end deployment  using  AWS ec2 CloudFormation and Chef from the base O/S through to the applications that need to be deployed

Managing Chef from Windows 7

Opscode have made significant progress in allowing windows users to use their windows machines to administer  chef as it is  using a Linux workstation.  This is great news for all us chef lovers who use windows as their main day to day  environment and if like me you have to target both Linux and windows  nodes.

In the past I have had problems using previous versions of chef  with ruby version 1.8.7-x which would  require me  having to explicitly tell knife where to find config files, installing extra gems just to get it to work and I was  always reluctant to not have handy a Linux instance available as a fall back .

This post summarises the steps required to set up your windows environment to use version 0.10.0  of Chef   on the Opscode Platform with  ruby 1.9.2 p-180 . Hopefully this will save you hopping all over the place to find out what you need to do to get it  sorted out.

This version of Chef provides the ability to have different environments hence why I opted to start playing with the beta/ pre-release version.

Environments are a welcome addition as it means you can easily manage test, stage and production environments with the use of different run lists per environment for the same role . For example the only thing likely to differ between say stage and production is likely to be target databases, S3 buckets, account names etc so you could add in specific attribute files for these values per environment.

Firstly sign up with the opscode platform via opscode.com downloading the knife configuration and the private keys that are generated. I’m not going through that as the guys at Opscode have done a great job of walking you through that process.

1.       Install ruby

·         Download from  http://rubyinstaller.org  the latest version of ruby ( at time of writing this was ruby 1.9.2 p-180)

(There is a vbs script that allows you to do a wget from the opscode   wiki site but why would you do this ?  I’m not sure, but if you do feel the need to script this bit what’s wrong with Powershell?)

·         Run the installer

2.       Create the  following folders:

C:\chef

C:\devkit

3.       Install the Ruby Devkit

·         Download from https://github.com/oneclick/rubyinstaller/wiki/Development-Kit  the ruby development kit

·         Extract the  devkit  into c:\devkit by copying the downloaded devkit.exe into c:\devkit, then  extracting it using  DevKit-tdm-32-4.5.1-20101214-1400-sfx.exe –y

Then run the following

·         ruby c:/DevKit/dk.rb init

·         ruby c:/DevKit/dk.rb install

4.       Install some gems that are pre-requisites  : gem install  ruby-wmi windows-api windows-pr

5.       Install version 0.10.0 of chef.  As I have been using the beta / pre-release version my installation command looked like this   :  gem install chef –pre –no-rdoc –no-ri –verbose

When the stable release of version 0.10.0 is available the command will be:

gem install chef  –no-rdoc –no-ri –verbose

6.       Install git for windows from http://help.github.com/win-set-up-git/ This is needed as Chef makes use of github as  a repository and you will need this to at least set up your initial environment and to download sample cookbooks. It’s also a good choice of a repository to keep your own cookbooks if you do not already have a repository.

7.       Create a chef repository. This is where the artefacts that you will use to manage your target nodes are located. These include cookbooks, roles etc.  To do this clone a copy from git. Assuming your  working environment is all under users/yourname and you are using Git bash:

cd ~

git clone git://github.com/opscode/chef-repo.git

8.       Create a .chef folder under your chef-repo folder

9.       Copy the knife.rb  and keys into the .chef folder. Now whenever you are in the chef-repo folder and run a knife command it will locate both these files.

10.   If you are using  AWS ec2  resources like I am then you  will also need to install the ec2 commands plugin

gem install  knife-ec2  –no-rdoc –no-ri –verbose

Two extra lines will then  need  to be added to  the knife.rb file  copied to ~/chef-repo/.chef earlier  to allow you to use the  knife ec2 commands

knife[:aws_access_key_id]     = “YOUR_ACCESS_KEY_ID_HERE”

knife[:aws_secret_access_key] = “YOUR_SECRET_ACCESS_KEY”

11.   As a quick verification run the following command from your chef-repo folder:

Knife client list

This should return the default client set up when you first sign up:

OpscodeOrganizationName-validator

 Now you really are ready to start cooking with chef  using  a windows 7  admin machine J

 

Devops a means to an end

I’ve been following a thread on Twitter where they have been discussing the input from developers in the ‘Devops’ community.  It seems that some people felt the Devops initiative is been driven by Developers. My experience working with various clients is that it isn’t actually the case but then again I don’t meet many who do my role. I am usually considered part of the development team I attend scrum and have backlogs and work items to burn down just like the developers .

It suits me sitting with a bunch of Devs as it means I have direct access to the guys who can code a hell of a lot faster than me and I can gain the knowledge of the application I need.  I’m sat with them watching the app evolve (I just love it when the finished app is ready to be let loose)   and can pipe up early in the development process rather than later on when it’s too late if I need to get something to change.

In a typical engagement I will undertake the following tasks amongst others

  1. Understand what the application is ultimately supposed to achieve and understand the target environment
  2. Facilitate the  CI build  and  deployment processes (automation)
  3. Implementing appropriate operational processes to support the application
  4. Involve  the operational team from the beginning of the development process

There are a number of challenges one of which is usually getting someone from operations involved so my role as the bridge between dev and ops seems to work as I understand both sides.

The developers I work with are focused on development activities and stuff like making the build work and getting the deployment environments ready are distractions from their main tasks  so this observation doesn’t really tally with its the Devs trying to take over our role scenario.

The developers do have to know what’s going on in the operations area , that’s what I’m there to help facilitate  and with tools like CloudFormation from AWS and Chef from Opscode all this provisioning and configuration stuff is made easy.

Countering this so called disinterest most corporations I have worked with it’s a lead dev who looks after the build process and the operations team go nowhere near it  apart from making sure the O/S  is up and running . This needs to change because if operations are involved in facilitating the build processes then they are immediately involved with the process of developing the application.

Basically the key to DevOps is to get The Developers and the Operations team working together  however you can.

If you can’t get someone from operations involved daily then make sure you keep them in the loop so there are no surprises talk through the architecture with them, discuss monitoring and how it fits in with what they do today, automation is always welcome. You can’t dictate if they have existing processes and if you are going to introduce new ways of working then you will need to educate and solicit feedback, hold workshops and demo stuff regularly .

The developers I work with totally get Devops  , they just don’t feel the need to talk about it as it’s just happening and certainly do not go round bad mouthing the ops team.

I have however  found it’s introducing the same concepts that apply to developing applications to your infrastructure is where the struggle occurs with existing Operations teams but that’s a different problem from the Dev’s stay off my patch perceived war.

Although the whole ideas of DevOps has been around for a while I just don’t think it’s been as widely accepted because operations and Developers tend to work in separate silos.  This is what needs to change.  Despite the common misconception Developers do actually want their applications to be deployed consistently every time to a know target environment. This is what Operations want to . So where’s the problem ?

An intro to writing AWS CloudFormation Templates

AWS  introduced recently the ability to create & document  your AWS environment via CloudFormation. You need to create a JSON template that describes your environment . This template  is then used to create your AWS Stack.  The AWS documentation is pretty good but the getting started guide starts you off by using an example template .  I think a walk through showing you what the component parts that make up a  template are and how to put them  together is a much better starting point if you want to create a stack for your own AWS environment so here you are.

There are 6 basic components that can be used in the JSON Cloudformation templates that you need to become familiar with.

Format version (optional): The AWS CloudFormation template version against which the template was written

Description (optional):  JSON   text string description of template or part of template

Parameters (optional):  String or comma separated list. The values can be overridden at run time. Parameters are dereferenced in resources and outputs section of  the template e.g  you can declare a Parameter  called InstanceType with a default of t1.micro that can be overridden at Stack instantiation to be an alternative InstanceType.

*******************************************

“InstanceType”   : {

“Type”   : “String”,

“Default”   : “t1.micro”,

“Description”  : ” ‘t1.micro’ |  ‘m1.large’ |  ‘c1.xlarge’ ”

},

*******************************************

Mappings (optional):  Allow the passing of conditional parameter values used with the function Fn::FindInMap. It is similar to a case statement in usage. Used in conjunction with Parameters. Each mapping has a unique name in a template and consists of a number of key-attribute pairs. Each Attribute is a literal string

************************************

“Mappings” : {

“RegionMap” : {

“us-east-1” : {

“AMI” : “ami-8e1fece7”,

“AvailabilityZone” : “us-east-1b”

},

 

“eu-west-1” : {

“AMI” : “ami-45cefa31”,

“AvailabilityZone” : “eu-west-1a”

}

}

},

*********************************

When using the function Fn:FindInMap it needs to be passed the Map name, the  Reference key and the return value

Resources: AWS resources such as instances, RDS etc which are declared as members of the AWS stack . Resources declared in the Resources section contain a Properties section, in which you declare both required and optional properties for the resource.

**********************************

“Resources” : {

“Ec2Instance” : {

“Type” : “AWS::EC2::Instance”,

“Properties” : {

“KeyName” : { “Ref” : “KeyName” },

“AvailabilityZone” : { “Fn::FindInMap” : [ “RegionMap”, { “Ref” : “AWS::Region” }, “AvailabilityZone” ]},

“ImageId” : { “Fn::FindInMap” : [ “RegionMap”, { “Ref” : “AWS::Region” }, “AMI” ]},

“InstanceType” : { “Ref” : “InstanceType”}

}

}

},

********************************

Outputs (optional):  Messages that can be returned as  part of the cfn-describe-stacks command

**********************

“PublicIP” : {

“Description” : “Public IP address of the newly created EC2 instance”,

“Value” : { “Fn::GetAtt” : [ “Ec2Instance”, “PublicIp” ] }

}

****************************************

So putting all the above together to create a template that starts an ec2 instance you get ( I’ve added some bits to make it a workable solution):

******************************

{

“AWSTemplateFormatVersion” : “2010-09-09”,

 

“Description” : “Creates an EC2 instance running the AWS Linux 64 bit AMI from the EU region. “,

 

“Parameters” : {

“KeyName” : {

“Description” : “Name of  EC2 KeyPair to enable SSH access to the instance”,

“Default” : “AWSNet-EU”,

“Type” : “String”

},

“InstanceType”   : {

“Type”   : “String”,

“Default”   : “t1.micro”,

“Description”  : ” ‘t1.micro’ |  ‘m1.large’ |  ‘c1.xlarge’ ”

}

},

 

“Mappings” : {

“RegionMap” : {

“us-east-1” : {

“AMI” : “ami-8e1fece7”,

“AvailabilityZone” : “us-east-1b”

},

 

“eu-west-1” : {

“AMI” : “ami-45cefa31”,

“AvailabilityZone” : “eu-west-1a”

}

}

},

 

“Resources” : {

“Ec2Instance” : {

“Type” : “AWS::EC2::Instance”,

“Properties” : {

“KeyName” : { “Ref” : “KeyName” },

“AvailabilityZone” : { “Fn::FindInMap” : [ “RegionMap”, { “Ref” : “AWS::Region” }, “AvailabilityZone” ]},

“ImageId” : { “Fn::FindInMap” : [ “RegionMap”, { “Ref” : “AWS::Region” }, “AMI” ]},

“InstanceType” : { “Ref” : “InstanceType”}

}

}

},

 

“Outputs” : {

“InstanceId” : {

“Description” : “InstanceId of the newly created EC2 instance”,

“Value” : { “Ref” : “Ec2Instance” }

},

“AZ” : {

“Description” : “Availability Zone of the newly created EC2 instance”,

“Value” : { “Fn::GetAtt” : [ “Ec2Instance”, “AvailabilityZone” ] }

},

“PublicIP” : {

“Description” : “Public IP address of the newly created EC2 instance”,

“Value” : { “Fn::GetAtt” : [ “Ec2Instance”, “PublicIp” ] }

}

}

}

*******************************************************************

 

 


Dissection of a Chef Recipe or two for windows

Working with chef one of the first things I needed to do was get to grips with the semantics of ruby.  I did a bit of speed reading, did a few simple ruby programs and I keep a copy of the little book of ruby handy for reference purposes so am becoming more comfortable with that.

What I found though is that it wasn’t problems with getting to grips with ruby and I’m definitely a newbie there but the actual understanding of the Chef recipe DSL. I had a look at the wiki and although it does give some guidance I was thinking that the best way to help someone get started quickly was to walk through a couple of example recipes.

There is a fair amount of information on using Chef with Linux targets so I’ll focus on using Windows as the target as I believe Chef has as much to offer the windows system administrator as it does for Linux sysadmins.

Before you start writing recipes the first thing you need to understand is the anatomy of a cookbook

Taking the definitions from the Opscode Wiki:

Cookbooks contain:

  • Attributes that are values on Node to set default values used elsewhere in the cookbook.
  • Definitions that allow you to create reusable collections of one or more Resources.
  • Files that are transferred to your Chef-administered machines via Cookbook File resource.
  • Libraries that extend Chef or provide helpers with Ruby code.
  • Recipes that specify Resources to manage, in the order they should be managed.
  • Lightweight Resources and Providers (LWRP) that allow you to create your own custom resources and providers.
  • Templates that are rendered on Chef-configured machines with your dynamically substituted values. Think config files on steroids, then read ERB templates.
  • Metadata that tells Chef about your recipes, including dependencies, supported platforms and more.

Cookbooks are arranged in the following folder structure:

attributes/

defintions/

files/

libraries/

metadata.rb

providers/

README.rdoc

recipes/

resources/

templates/


Well I don’t know about you but as a newbie based on the above it’s a bit daunting to try and understand how it all fits together and hunting through the various pages on the Opscode wiki to understand can be slightly frustrating to say the least so I hope a walkthrough of a few simple recipes will help you get started.

Example 1:

#

# Cookbook Name:: myapp

# Recipe:: deploymsi

#

# Copyright 2011, @devopscloud

#

# All rights reserved – Do Not Redistribute

#

msifile = File.basename(“myapp.msi”)

dir = “buildoutput”

drive=”c:”

msifiledst = “#{drive}\\#{dir}\\#{msifile}”

execute “install #{msifiledst}” do

command “msiexec /qn /i #{msifiledst} TARGETENV=DEV”

only_if { File.exists?(msifiledst) }

end

This example does what you think it does it installs an MSI on the target node.

How does it work:

Firstly we define a number of variables to allow us to identify the msi.

All the grunt work is defined in the execute resource:

execute “install #{msifiledst}” do

command “msiexec /qn /i #{msifiledst} TARGETENV=DEV”

only_if { File.exists?(msifiledst) }

end

The resource  type is: execute.

The resource name is : install #{msifiledst} = Install c:\buildoutput\myapp.msi

It calls the command prompt and then runs msiexec but only if the msi actually exists which is what the only_if (File.exists?..  bit of the recipe does.

Tip  the ‘\\’ to allow you to  use ‘\’  not an issue with Linux nodes but useful when working with windows nodes.

Building upon the above simple example we’ll now introduce something new in the next example:

Example 2:

#

# Cookbook Name:: myapp

# Recipe:: deploywebapp

#

# Copyright 2011,  @devopscloud

#

# All rights reserved – Do Not Redistribute

#

msi = File.basename(“myWebapp.msi”)

dir = “buildoutput”

drive=”c:”

dst = “#{drive}\\#{dir}\\#{msi}”

template “C:/chef/tmp/appool.ps1” do

source “appool.ps1.erb”

end

execute “install #{dst}” do

command “msiexec /qn /i #{dst} TARGETENV=DEV”

only_if { File.exists?(dst) }

end

execute “updateappool” do

command “c:\\Windows\\System32\\WindowsPowerShell\\V1.0\\powershell.exe c:\\chef\\tmp\\appool.ps1\””

action :run

cwd “c:/chef/tmp”

end

This recipe installs an MSI as in example 1 but it then runs a powershell script that makes modifications to the appool. This recipe introduces the concept of templates. Templates are stored in the templates folder of your cookbook and stored as .erb files. In this example the erb file contains powershell script. So what does these two line mean?

template “C:/chef/tmp/appool.ps1” do

source “appool.ps1.erb”

This essentially equates to the following:  copy the   file appool.ps1.erb  to target node to the folder c:/chef/tmp  and name accordingly.

Later on in the recipe we actually run the powershell script. Easy huh   Smile

The key thing here really is that all the Powershell you inevitably use as a windows administrator is still reusable and I haven’t even started talking about providers as yet.

The examples above are simple and not exactly robust but they do stuff which is all we want chef recipes to do really.

Recipes are quite a huge topic and I have barely  scraped the surface with these two  simple examples.

What is the price of designing for failure ?

Having spent a fair amount of my IT career in operations I love the fact that working in the cloud space does actually free up first line support investment purely because I have designed for failure in the first place.

The problem with  driving forward with efficiencies which moving to the cloud does give you is that it will inevitably cause concern with regards to what roles people now play.

With designing for failure you by necessity build in resilience and auto healing features which means that the normal day to day babysitting of services is not as resource intensive as it used to be.

Automation and configuration management is actually now affordable hence why the SME’s have been the ones to embrace the cloud  and all that it offers  so whole heartedly . They do not have legacy Data centres, lots of kit, an army of support personnel  and loads of money to spend .

In a nutshell  it means you end up investing in less staff but more highly skilled staff as the support team now have to deal with more complex problems then just rebooting a server.  In theory the call out rate should be  a lot lower than with a traditional deployment. I think the onus is  now really on the developers to  have written totally rocking code Smile as at long last the operational team have sophistication and reproducibility to help them but this does come at a price.

That price is to realise that you will need at least one highly skilled hybrid team player to support these cloud deployments . ‘Turn it off and on again ‘ is no longer the  initial response of the support team as decoupled highly scalable systems are easily achieved and the configuration and deployment of the platform now starts been dealt with like the development of code.

Personally I think this is a good thing as the operations team in my opinion have always been under valued but with the evolution of SuperOps/ DevOps or whatever tag you want they should be treated with a little bit of respect as they are needed in this fluffy new world of the cloud and in return hopefully they’ll be stepping up to the plate.

Introductions

As a  Platform Architect I  bridge the gap between Infrastructure and Development. This means I get to support developers, implement the build processes , liaise with all parties  on platform related activities , take responsibility for deployment and generally have lots of fun while doing all this. I’ve been doing this for  far more years than I’d like to recall but apparently a  large chunk of what I’ve been doing all these years is called DevOps! A few explantions I like of  of what DevOps actually means can be found at the   Grok and Roll and Katar.net blogs. Beyond the Devops type thing I have a slightly unhealthy obsession with Public cloud solutions.  I decided that this blog would be the best place to gather my rather longer ramblings on these areas. I have a work blog here which covers a wider range of topics and I will carry on making the occasional post there . For my shorter thoughts on what’s happening in he cloud  market place I contribute comments to Cloud Comments .net . So  I’d just like to say hello and hope you’ll find some nuggets of interest over the coming months.