Targeting Windows 2008 R2 nodes using chef

Just a quick note.

I’d advise sticking to ruby 1.8.7-p344 on the target node if you are targeting windows 2008 R2.  I recently revisted targeting windows 2008 R2 and found that using  the latest version of ruby  1.9.2-p180 on the windows 2008 r2 target node  and  attempting to run chef-client  after installing the chef gem is a proverbial pain . I’m not sure if Opscode are looking into this but  it’s easy to reproduce the pain :-)

Loosely coupled Chef Cookbooks

I’ve been working on a mongodb installation and configuration cookbook which allows me to install &  if required make custom configurations . It allows me to Install and configure a standalone mongodb installation or a replica set.

Developing this cookbook ( still a work in progress )  has led me to take a loosely coupled approach to its development such that I did not want to force a dependency on any previous recipe.  This has  meant a number of rules would need to be followed to use the cookbook properly rather than imposing any  constraints.

So why did I come to this conclusion that flexibility and thus loose coupling was a requirement for this particular cookbook: 

The use of a replica set and the fact you may want to seed the mongodb set up with data  from a backup did gave me food for thought.  When spiking the various configuration  scenarios I found that if I updated my current master via a data dump where I  had decided to stop the master mongodb instance  while I copied the data into the data folder  I found  the data wasn’t being replicated. This was because  one of the slaves had then taken over as the master and you can’t really force a master( without pain )  in a  replica set  ( maybe 10gen can advise on that one although I guess if I’d made sure the owner of the files was correct before the copy I may not have hit the mongo having a fit stage) so I needed to cater for that one.

Seeding mongo before a replica set is created seems like a nice approach to me anyway.

In a standalone mongodb instance I’m probably not worried about Raid so the recipe to create the Raid device shouldn’t be a constraint and while I’m at it why can’t you set up a replica that just uses instances with local storage?  

 I want to add recipes to create the job to undertake regular backups and maybe one to do a restore. But I may not want to use them.

Suddenly the list of things I want my mongo cookbook to do is growing. So I have done what is required  to deliver the functionality that is required  by  the client  I needed to do the Chef work for and   now I can pimp my cookbooks till I’m happy enough to share it with the community .

(The whole ‘just enough’ ethos is something I mean to talk about here but not now)

I want the recipes to be easy to use and understood by people new to Chef and also to mongodb as I do not believe just because you have a sophisticated tool like Chef that should mean your cookbooks should be overly complicated. Keeping it simple makes maintenance easy and encourages others to expand upon it appropriately if they follow the rules. Mongodb is very easy to get up and running so why use a tool to make it suddenly obtuse?

So the  rules to date :

Obviously you need to have mongo installed as a starting point.  I couldn’t really mandate the use of the installation recipe as it may be an existing set up . ( I have to modify this so it brings down a specified version rather than just the latest version from the 10gen repository) .

Each recipe is to be used to carry out a  single function e.g install mongdb, configure the configuration file , start mongodb etc. The combining of functions is discouraged

Each subsequent recipe can be run independently of the others or be combined as a role this meant making sure I had a recipe to start mongodb so this could be dropped in as say part of a role or workflow .

The use of templates and variables to encourage flexibility .

When I get to a point I feel the cookbook is pimped appropriately I’ll post a  dissection and some guidance .

Powershell & user data to start a chef run

This script can be used to configure  an instance on AWS at startup to collect user data which would be the run list in json . This assumes that ruby and the pre-requisite gems have already been installed

#chef-clientrun.ps1

# install chef gem – This ensures only the latest stable version is installed

$installchef= “gem install chef –no-rdoc –no-ri””

# Download userdata

$webclient = new-object system.net.webclient

$awsurl =”http://169.254.169.254/latest/user-data

$targetfile =”c:\chef\etc\runlist.json”

$webClient.DownloadFile(“$awsurl”,”$targetfile”)

# Run chef-client passing json file which contains runlist

$runchef = “C:\Ruby192\bin\chef-client -j”+  $targetfile

invoke-expression $installchef

invoke-expression $runchef

Using CloudFormation to kick off a chef run

Once you decide to use CloudFormation to create your AWS resources you are now unable to use the knife command to kick of an ec2 server creation so you will have to get the client to start the chef run by doing a chef-client run .

The solution described in this post  is simple to implement .It requires  doing a  little scripting at the Instance end by baking that into a base AMI and the use of userdata.

I will use a Linux AWS AMI as my starting point.

The first thing to do is set up your target AMI to be able to  use userdata.

The script below shows the salient parts of an rc.local I have used to facilitate a chef run when an instance is created from the AMI:

gem install chef –no-rdoc –no-ri
# grab userdata then use to construct name of json file
# json file contains run list and is passed to chef-client run
export USERDATA=`/usr/local/bin/aws-get-ec2-userdata`
echo userdata = $USERDATA
export ROLE=$(echo $USERDATA | cut -f 1 -d “:”)
chef-client -j /etc/chef/$ROLE.json

The file /usr/local/bin/aws-get-ec2-userdata  file uses curl ( just like the sample templates from AWS) to return the userdata which is then  stored in the environment variable USERDATA. The first value  which represents the role we want to apply to the node is extracted and saved as the environment variable ROLE which is then used to pass the appropriate json  file which contains that role in the  runlist.

The corresponding  part of a Cloudformation script that creates the EC2 instance resource and passes the userdata looks like this:

“Ec2Instance” : {
“Type” : “AWS::EC2::Instance”,
“Properties” : {
“KeyName” : { “Ref” : “KeyName” },
“AvailabilityZone” : { “Fn::FindInMap” : [ "RegionMap", { "Ref" : "AWS::Region" }, "AvailabilityZone" ]},
“ImageId” : { “Fn::FindInMap” : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]},
“InstanceType” : { “Ref” : “InstanceType”},
“UserData”: {
“Fn::Base64″: {
“Fn::Join”: [
“:”,
[
{
"Ref": "ChefRole"
},
{
"Ref": "EBsVolsreqd"
}

]
]
}
}
}
}

The userdata needs to be base 64 encoded  hence the  “Fn::Base64″:  encapsulating this property. The “Fn::Join”: [  “:”, appends the values passed as as single value with  each value separated by  “:”

The line export ROLE=$(echo $USERDATA | cut -f 1 -d “:”)”
in the rc.local uses the delimiter to identify each value and as the ChefRole is the first parameter started it uses this to set the variable ROLE.

When the stack is started you can accept the default role or change it to an appropriate value .

stack

After the stack is complete you can then check to see if it has created the node by looking at the system log :

syslog

and /or using the chef  console  ( I use the opscode hosted platform):

opscode

I think this is a nice  straight forward  way  to achieve a  fully automated end to end deployment  using  AWS ec2 CloudFormation and Chef from the base O/S through to the applications that need to be deployed

Managing Chef from Windows 7

Opscode have made significant progress in allowing windows users to use their windows machines to administer  chef as it is  using a Linux workstation.  This is great news for all us chef lovers who use windows as their main day to day  environment and if like me you have to target both Linux and windows  nodes.

In the past I have had problems using previous versions of chef  with ruby version 1.8.7-x which would  require me  having to explicitly tell knife where to find config files, installing extra gems just to get it to work and I was  always reluctant to not have handy a Linux instance available as a fall back .

This post summarises the steps required to set up your windows environment to use version 0.10.0  of Chef   on the Opscode Platform with  ruby 1.9.2 p-180 . Hopefully this will save you hopping all over the place to find out what you need to do to get it  sorted out.

This version of Chef provides the ability to have different environments hence why I opted to start playing with the beta/ pre-release version.

Environments are a welcome addition as it means you can easily manage test, stage and production environments with the use of different run lists per environment for the same role . For example the only thing likely to differ between say stage and production is likely to be target databases, S3 buckets, account names etc so you could add in specific attribute files for these values per environment.

Firstly sign up with the opscode platform via opscode.com downloading the knife configuration and the private keys that are generated. I’m not going through that as the guys at Opscode have done a great job of walking you through that process.

1.       Install ruby

·         Download from  http://rubyinstaller.org  the latest version of ruby ( at time of writing this was ruby 1.9.2 p-180)

(There is a vbs script that allows you to do a wget from the opscode   wiki site but why would you do this ?  I’m not sure, but if you do feel the need to script this bit what’s wrong with Powershell?)

·         Run the installer

2.       Create the  following folders:

C:\chef

C:\devkit

3.       Install the Ruby Devkit

·         Download from https://github.com/oneclick/rubyinstaller/wiki/Development-Kit  the ruby development kit

·         Extract the  devkit  into c:\devkit by copying the downloaded devkit.exe into c:\devkit, then  extracting it using  DevKit-tdm-32-4.5.1-20101214-1400-sfx.exe –y

Then run the following

·         ruby c:/DevKit/dk.rb init

·         ruby c:/DevKit/dk.rb install

4.       Install some gems that are pre-requisites  : gem install  ruby-wmi windows-api windows-pr

5.       Install version 0.10.0 of chef.  As I have been using the beta / pre-release version my installation command looked like this   :  gem install chef –pre –no-rdoc –no-ri –verbose

When the stable release of version 0.10.0 is available the command will be:

gem install chef  –no-rdoc –no-ri –verbose

6.       Install git for windows from http://help.github.com/win-set-up-git/ This is needed as Chef makes use of github as  a repository and you will need this to at least set up your initial environment and to download sample cookbooks. It’s also a good choice of a repository to keep your own cookbooks if you do not already have a repository.

7.       Create a chef repository. This is where the artefacts that you will use to manage your target nodes are located. These include cookbooks, roles etc.  To do this clone a copy from git. Assuming your  working environment is all under users/yourname and you are using Git bash:

cd ~

git clone git://github.com/opscode/chef-repo.git

8.       Create a .chef folder under your chef-repo folder

9.       Copy the knife.rb  and keys into the .chef folder. Now whenever you are in the chef-repo folder and run a knife command it will locate both these files.

10.   If you are using  AWS ec2  resources like I am then you  will also need to install the ec2 commands plugin

gem install  knife-ec2  –no-rdoc –no-ri –verbose

Two extra lines will then  need  to be added to  the knife.rb file  copied to ~/chef-repo/.chef earlier  to allow you to use the  knife ec2 commands

knife[:aws_access_key_id]     = “YOUR_ACCESS_KEY_ID_HERE”

knife[:aws_secret_access_key] = “YOUR_SECRET_ACCESS_KEY”

11.   As a quick verification run the following command from your chef-repo folder:

Knife client list

This should return the default client set up when you first sign up:

OpscodeOrganizationName-validator

 Now you really are ready to start cooking with chef  using  a windows 7  admin machine J

 

Dissection of a Chef Recipe or two for windows

Working with chef one of the first things I needed to do was get to grips with the semantics of ruby.  I did a bit of speed reading, did a few simple ruby programs and I keep a copy of the little book of ruby handy for reference purposes so am becoming more comfortable with that.

What I found though is that it wasn’t problems with getting to grips with ruby and I’m definitely a newbie there but the actual understanding of the Chef recipe DSL. I had a look at the wiki and although it does give some guidance I was thinking that the best way to help someone get started quickly was to walk through a couple of example recipes.

There is a fair amount of information on using Chef with Linux targets so I’ll focus on using Windows as the target as I believe Chef has as much to offer the windows system administrator as it does for Linux sysadmins.

Before you start writing recipes the first thing you need to understand is the anatomy of a cookbook

Taking the definitions from the Opscode Wiki:

Cookbooks contain:

  • Attributes that are values on Node to set default values used elsewhere in the cookbook.
  • Definitions that allow you to create reusable collections of one or more Resources.
  • Files that are transferred to your Chef-administered machines via Cookbook File resource.
  • Libraries that extend Chef or provide helpers with Ruby code.
  • Recipes that specify Resources to manage, in the order they should be managed.
  • Lightweight Resources and Providers (LWRP) that allow you to create your own custom resources and providers.
  • Templates that are rendered on Chef-configured machines with your dynamically substituted values. Think config files on steroids, then read ERB templates.
  • Metadata that tells Chef about your recipes, including dependencies, supported platforms and more.

Cookbooks are arranged in the following folder structure:

attributes/

defintions/

files/

libraries/

metadata.rb

providers/

README.rdoc

recipes/

resources/

templates/


Well I don’t know about you but as a newbie based on the above it’s a bit daunting to try and understand how it all fits together and hunting through the various pages on the Opscode wiki to understand can be slightly frustrating to say the least so I hope a walkthrough of a few simple recipes will help you get started.

Example 1:

#

# Cookbook Name:: myapp

# Recipe:: deploymsi

#

# Copyright 2011, @devopscloud

#

# All rights reserved – Do Not Redistribute

#

msifile = File.basename(“myapp.msi”)

dir = “buildoutput”

drive=”c:”

msifiledst = “#{drive}\\#{dir}\\#{msifile}”

execute “install #{msifiledst}” do

command “msiexec /qn /i #{msifiledst} TARGETENV=DEV”

only_if { File.exists?(msifiledst) }

end

This example does what you think it does it installs an MSI on the target node.

How does it work:

Firstly we define a number of variables to allow us to identify the msi.

All the grunt work is defined in the execute resource:

execute “install #{msifiledst}” do

command “msiexec /qn /i #{msifiledst} TARGETENV=DEV”

only_if { File.exists?(msifiledst) }

end

The resource  type is: execute.

The resource name is : install #{msifiledst} = Install c:\buildoutput\myapp.msi

It calls the command prompt and then runs msiexec but only if the msi actually exists which is what the only_if (File.exists?..  bit of the recipe does.

Tip  the ‘\\’ to allow you to  use ‘\’  not an issue with Linux nodes but useful when working with windows nodes.

Building upon the above simple example we’ll now introduce something new in the next example:

Example 2:

#

# Cookbook Name:: myapp

# Recipe:: deploywebapp

#

# Copyright 2011,  @devopscloud

#

# All rights reserved – Do Not Redistribute

#

msi = File.basename(“myWebapp.msi”)

dir = “buildoutput”

drive=”c:”

dst = “#{drive}\\#{dir}\\#{msi}”

template “C:/chef/tmp/appool.ps1″ do

source “appool.ps1.erb”

end

execute “install #{dst}” do

command “msiexec /qn /i #{dst} TARGETENV=DEV”

only_if { File.exists?(dst) }

end

execute “updateappool” do

command “c:\\Windows\\System32\\WindowsPowerShell\\V1.0\\powershell.exe c:\\chef\\tmp\\appool.ps1\””

action :run

cwd “c:/chef/tmp”

end

This recipe installs an MSI as in example 1 but it then runs a powershell script that makes modifications to the appool. This recipe introduces the concept of templates. Templates are stored in the templates folder of your cookbook and stored as .erb files. In this example the erb file contains powershell script. So what does these two line mean?

template “C:/chef/tmp/appool.ps1″ do

source “appool.ps1.erb”

This essentially equates to the following:  copy the   file appool.ps1.erb  to target node to the folder c:/chef/tmp  and name accordingly.

Later on in the recipe we actually run the powershell script. Easy huh   Smile

The key thing here really is that all the Powershell you inevitably use as a windows administrator is still reusable and I haven’t even started talking about providers as yet.

The examples above are simple and not exactly robust but they do stuff which is all we want chef recipes to do really.

Recipes are quite a huge topic and I have barely  scraped the surface with these two  simple examples.

Follow

Get every new post delivered to your Inbox.