Powershell & user data to start a chef run

This script can be used to configure  an instance on AWS at startup to collect user data which would be the run list in json . This assumes that ruby and the pre-requisite gems have already been installed

#chef-clientrun.ps1

# install chef gem – This ensures only the latest stable version is installed

$installchef= “gem install chef –no-rdoc –no-ri””

# Download userdata

$webclient = new-object system.net.webclient

$awsurl =”http://169.254.169.254/latest/user-data

$targetfile =”c:\chef\etc\runlist.json”

$webClient.DownloadFile(“$awsurl”,”$targetfile”)

# Run chef-client passing json file which contains runlist

$runchef = “C:\Ruby192\bin\chef-client -j”+  $targetfile

invoke-expression $installchef

invoke-expression $runchef

Managing Chef from Windows 7

Opscode have made significant progress in allowing windows users to use their windows machines to administer  chef as it is  using a Linux workstation.  This is great news for all us chef lovers who use windows as their main day to day  environment and if like me you have to target both Linux and windows  nodes.

In the past I have had problems using previous versions of chef  with ruby version 1.8.7-x which would  require me  having to explicitly tell knife where to find config files, installing extra gems just to get it to work and I was  always reluctant to not have handy a Linux instance available as a fall back .

This post summarises the steps required to set up your windows environment to use version 0.10.0  of Chef   on the Opscode Platform with  ruby 1.9.2 p-180 . Hopefully this will save you hopping all over the place to find out what you need to do to get it  sorted out.

This version of Chef provides the ability to have different environments hence why I opted to start playing with the beta/ pre-release version.

Environments are a welcome addition as it means you can easily manage test, stage and production environments with the use of different run lists per environment for the same role . For example the only thing likely to differ between say stage and production is likely to be target databases, S3 buckets, account names etc so you could add in specific attribute files for these values per environment.

Firstly sign up with the opscode platform via opscode.com downloading the knife configuration and the private keys that are generated. I’m not going through that as the guys at Opscode have done a great job of walking you through that process.

1.       Install ruby

·         Download from  http://rubyinstaller.org  the latest version of ruby ( at time of writing this was ruby 1.9.2 p-180)

(There is a vbs script that allows you to do a wget from the opscode   wiki site but why would you do this ?  I’m not sure, but if you do feel the need to script this bit what’s wrong with Powershell?)

·         Run the installer

2.       Create the  following folders:

C:\chef

C:\devkit

3.       Install the Ruby Devkit

·         Download from https://github.com/oneclick/rubyinstaller/wiki/Development-Kit  the ruby development kit

·         Extract the  devkit  into c:\devkit by copying the downloaded devkit.exe into c:\devkit, then  extracting it using  DevKit-tdm-32-4.5.1-20101214-1400-sfx.exe –y

Then run the following

·         ruby c:/DevKit/dk.rb init

·         ruby c:/DevKit/dk.rb install

4.       Install some gems that are pre-requisites  : gem install  ruby-wmi windows-api windows-pr

5.       Install version 0.10.0 of chef.  As I have been using the beta / pre-release version my installation command looked like this   :  gem install chef –pre –no-rdoc –no-ri –verbose

When the stable release of version 0.10.0 is available the command will be:

gem install chef  –no-rdoc –no-ri –verbose

6.       Install git for windows from http://help.github.com/win-set-up-git/ This is needed as Chef makes use of github as  a repository and you will need this to at least set up your initial environment and to download sample cookbooks. It’s also a good choice of a repository to keep your own cookbooks if you do not already have a repository.

7.       Create a chef repository. This is where the artefacts that you will use to manage your target nodes are located. These include cookbooks, roles etc.  To do this clone a copy from git. Assuming your  working environment is all under users/yourname and you are using Git bash:

cd ~

git clone git://github.com/opscode/chef-repo.git

8.       Create a .chef folder under your chef-repo folder

9.       Copy the knife.rb  and keys into the .chef folder. Now whenever you are in the chef-repo folder and run a knife command it will locate both these files.

10.   If you are using  AWS ec2  resources like I am then you  will also need to install the ec2 commands plugin

gem install  knife-ec2  –no-rdoc –no-ri –verbose

Two extra lines will then  need  to be added to  the knife.rb file  copied to ~/chef-repo/.chef earlier  to allow you to use the  knife ec2 commands

knife[:aws_access_key_id]     = “YOUR_ACCESS_KEY_ID_HERE”

knife[:aws_secret_access_key] = “YOUR_SECRET_ACCESS_KEY”

11.   As a quick verification run the following command from your chef-repo folder:

Knife client list

This should return the default client set up when you first sign up:

OpscodeOrganizationName-validator

 Now you really are ready to start cooking with chef  using  a windows 7  admin machine J

 

Setting up a windows AMI for use with Elastic Bamboo

This is a revision of a post I originally posted at the http://consultingblogs.emc.com/gracemollison/ but it’s quite useful until Atlassian get round to supporting windows as part of Jira Studio.

Jira Studio uses elastic bamboo as its build controller which in turn uses elastic agents (AWS ec2 instances that are spun up by the build controller to run the build) rather than the standard remote agents.

The Diagram from Atlassian  below illustrates the Elastic Bamboo configuration:


You will need to have an Amazon AWS account and will have to create a custom windows AMI to be used to create your elastic-agents.. This post outlines what I did to get this working for a recent project. Please note that this is unsupported by Atlassian but hopefully they’ll be supporting windows soon.

Please note that this post assumes you have some basic knowledge of using AWS.

Start a Windows 2008  instance from an  Amazon base windows 2008 AMI in the us-east-1  region. It needs to be in the US-East region as this is where Jira Studio expects to find the build agent.

All actions below are carried out from the instance you have started.

Turn off windows firewall

Install the latest JDK. (You will need tools.jar in your JAVA_HOME path hence why the JDK is required)

Install whatever components you need to be able to undertake a build e.g visual studio, msbuild, SDK’s  etc and other bits

Use the link here as a  guide : http://confluence.atlassian.com/display/BAMBOO/Creating+a+Custom+Elastic+Image

Set up the Amazon ec2 API tools as outlined in the Atlassian guide ( section 5.4). Specific guidance for windows can be obtained from the AWS documentation.

Check the version of Bamboo by  clicking on administration then expand under system \ system information  :


Scroll down to see the Bamboo version

Download the bamboo-elastic-agent that matches the version of Bamboo that is being used in Jira Studio: http://www.atlassian.com/software/bamboo/BambooDownloadCenter.jspa Make sure you click show all.

Create the   folder c:\bamboo-elastic-agent . Unzip the bamboo-elastic-agent-2.6.2.zip to this folder.

Download the latest zip of ant  from http://www.apache.org/dist/ant/binaries/ and unzip into c:\ant\

Download the latest zip of maven and unzip e.g  into c:\ apache-maven-2.0.11

Set up environment variables and paths

Example Summary of relevant Environment variables ( depends on versions you have downloaded):

ANT_HOME  C:/ant

EC2_CERT  c:\ec2-api-tools\YOUR-cert.pem

EC2_HOME  c:\ec2-api-tools

EC2_PRIVATE_KEY c:\ec2-api-tools\YOUR-pk.pem

JAVA_HOME  C:\Program Files\Java\jre6

MAVEN_HOME c:\apache-maven-2.0.11

Example path variables:

…….  c:/ant\bin;C:\Program Files\Java\jdk1.6.0_23;c:\apache-maven-2.0.11\bin;C:\ec2-api-toolsbin;c:\bamboo-elastic-agent\bin

Create a  batch file that  consists of two lines:

Line 1: the Java classpath (This was obtained by using a simple Powershell script to scrape the lib folder under bamboo-elastic-agent  and setting the result as  a CLASSPATH  variable)

Line2: This will run the actual elastic-agent

Note there are over a  hundred jar files but to date Atlassian have been unable to let me know which ones are actually needed hence the snippet of the batch file I used  rather than the full list.

SET CLASSPATH=acegi-security-1.0.4.jar;activation-1.1.1.jar;activeio-core-3.1.0.jar;activemq-core-5.2.0.jar;activemq-ra-5.2.0………

java -server -Xms32m -Xmx512m -XX:MaxPermSize=256m  -classpath %CLASSPATH% com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap 2>&1 > c:\bamboo-elastic-agent\bamboo-elastic-agent.log

Test that everything is set up okay by running the batch file interactively. You should see output similar to:

………  Java.io.FileNotFoundException: http://169.254.169.254/2008-02-01/user-data
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown So
urce)
at java.net.URL.openStream(Unknown Source)

……….

oo.agent.elastic.client.ElasticAgentBootstrap

0 [main] INFO com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap  –

Using tunnnelling. Registering ‘httpt’ and ‘httpst’ protocols.

577 [com.sun.sungrid.service.tunnel.server.TunnelServer] INFO com.sun.sungrid.service.tunnel.server.TunnelServer  – Waiting for tunnel connection.

The key points are it trying to get userdata and attempting to create the tunnel. The agent needs to be started by the bamboo controller hence the errors

The agent needs to start automatically on starting the instance so the batch file  needs to be wrapped as a service . The nssm works well https://iain.cx/src/nssm/

Set the service to start automatically:

Create an AMI based on this instance

Log onto Jira Studio and  register the AMI, Set up the  capabilities , test that it will spin up the instance  and  test a basic build .

Jira Studio recognises both Visual Studio and msbuild so you just need to add the paths in when setting up the capabilities:

 

 

Dissection of a Chef Recipe or two for windows

Working with chef one of the first things I needed to do was get to grips with the semantics of ruby.  I did a bit of speed reading, did a few simple ruby programs and I keep a copy of the little book of ruby handy for reference purposes so am becoming more comfortable with that.

What I found though is that it wasn’t problems with getting to grips with ruby and I’m definitely a newbie there but the actual understanding of the Chef recipe DSL. I had a look at the wiki and although it does give some guidance I was thinking that the best way to help someone get started quickly was to walk through a couple of example recipes.

There is a fair amount of information on using Chef with Linux targets so I’ll focus on using Windows as the target as I believe Chef has as much to offer the windows system administrator as it does for Linux sysadmins.

Before you start writing recipes the first thing you need to understand is the anatomy of a cookbook

Taking the definitions from the Opscode Wiki:

Cookbooks contain:

  • Attributes that are values on Node to set default values used elsewhere in the cookbook.
  • Definitions that allow you to create reusable collections of one or more Resources.
  • Files that are transferred to your Chef-administered machines via Cookbook File resource.
  • Libraries that extend Chef or provide helpers with Ruby code.
  • Recipes that specify Resources to manage, in the order they should be managed.
  • Lightweight Resources and Providers (LWRP) that allow you to create your own custom resources and providers.
  • Templates that are rendered on Chef-configured machines with your dynamically substituted values. Think config files on steroids, then read ERB templates.
  • Metadata that tells Chef about your recipes, including dependencies, supported platforms and more.

Cookbooks are arranged in the following folder structure:

attributes/

defintions/

files/

libraries/

metadata.rb

providers/

README.rdoc

recipes/

resources/

templates/


Well I don’t know about you but as a newbie based on the above it’s a bit daunting to try and understand how it all fits together and hunting through the various pages on the Opscode wiki to understand can be slightly frustrating to say the least so I hope a walkthrough of a few simple recipes will help you get started.

Example 1:

#

# Cookbook Name:: myapp

# Recipe:: deploymsi

#

# Copyright 2011, @devopscloud

#

# All rights reserved – Do Not Redistribute

#

msifile = File.basename(“myapp.msi”)

dir = “buildoutput”

drive=”c:”

msifiledst = “#{drive}\\#{dir}\\#{msifile}”

execute “install #{msifiledst}” do

command “msiexec /qn /i #{msifiledst} TARGETENV=DEV”

only_if { File.exists?(msifiledst) }

end

This example does what you think it does it installs an MSI on the target node.

How does it work:

Firstly we define a number of variables to allow us to identify the msi.

All the grunt work is defined in the execute resource:

execute “install #{msifiledst}” do

command “msiexec /qn /i #{msifiledst} TARGETENV=DEV”

only_if { File.exists?(msifiledst) }

end

The resource  type is: execute.

The resource name is : install #{msifiledst} = Install c:\buildoutput\myapp.msi

It calls the command prompt and then runs msiexec but only if the msi actually exists which is what the only_if (File.exists?..  bit of the recipe does.

Tip  the ‘\\’ to allow you to  use ‘\’  not an issue with Linux nodes but useful when working with windows nodes.

Building upon the above simple example we’ll now introduce something new in the next example:

Example 2:

#

# Cookbook Name:: myapp

# Recipe:: deploywebapp

#

# Copyright 2011,  @devopscloud

#

# All rights reserved – Do Not Redistribute

#

msi = File.basename(“myWebapp.msi”)

dir = “buildoutput”

drive=”c:”

dst = “#{drive}\\#{dir}\\#{msi}”

template “C:/chef/tmp/appool.ps1” do

source “appool.ps1.erb”

end

execute “install #{dst}” do

command “msiexec /qn /i #{dst} TARGETENV=DEV”

only_if { File.exists?(dst) }

end

execute “updateappool” do

command “c:\\Windows\\System32\\WindowsPowerShell\\V1.0\\powershell.exe c:\\chef\\tmp\\appool.ps1\””

action :run

cwd “c:/chef/tmp”

end

This recipe installs an MSI as in example 1 but it then runs a powershell script that makes modifications to the appool. This recipe introduces the concept of templates. Templates are stored in the templates folder of your cookbook and stored as .erb files. In this example the erb file contains powershell script. So what does these two line mean?

template “C:/chef/tmp/appool.ps1” do

source “appool.ps1.erb”

This essentially equates to the following:  copy the   file appool.ps1.erb  to target node to the folder c:/chef/tmp  and name accordingly.

Later on in the recipe we actually run the powershell script. Easy huh   Smile

The key thing here really is that all the Powershell you inevitably use as a windows administrator is still reusable and I haven’t even started talking about providers as yet.

The examples above are simple and not exactly robust but they do stuff which is all we want chef recipes to do really.

Recipes are quite a huge topic and I have barely  scraped the surface with these two  simple examples.