= Installing and Running Chef on DETER =


There are 3 components to Chef: a server, a set of clients, and a set of workstations. The server is, well, the server - it handles requests from clients and workstations; it is the central organizing component of the system. The workstation is where Chef code is written. Think of it as a development or OPs workstation. The client(s) are the nodes that are configured/controlled. All three components need to run Chef sofware: the server runs chef server suite, the clients run {{{chef-client}}}, and the workstation runs git and a suite of chef scripts around git called {{{knife}}} that interact with a chef (git) repository and a chef server. 

The workflow for a running system is recipes, roles, and chef code is written on a workstation and added to a chef repository. The person on the workstation then pushes updates to the chef server. The client then contacts the chef server for updates, gets the updates, and applies them locally. {{{Knife}}}, on a workstation, can also install Chef-client (and other packages) directly on a client. If this is done the flow becomes: 1) workstation pushes update to the server, 2) the workstation installs Chef on a client and gives it an initial role or list of recipes, 3) the newly installed {{{chef-client}}} then contacts the chef server and downloads the recipes and executes them, configuring the client node.

Note: the recipes here use a mysql library to talk directly to the testbed database. On the current DETER testbed though, the database is setup to only allow local access from boss. The TMCD/TMCC library is used to bridge the gap between test nodes and the database on boss on DETER. The recipes can be rewritten to use the TMCD interface but this was an exercise in trying to cut out Emulab code, a decision was made to talk directly to the database using standard database APIs. In short, the cookbook (and recipes) here cannot be run on DETER unless the database on boss is reconfigured to allow external access. Instructions are given for that below though. If you want to run the recipes, spin up an Emulab in Emulab experiment or access a local, non-public DETER/Emulab instance and modify the database configuration.  

Swap in an experiment with at least three Ubuntu 12.04 nodes, one {{{server}}}, one {{{workstation}}}, and one {{{client}}}. 

=== Chef Server ===
The server is simple to install: just use dpkg to install it. 

{{{
[users:~]$ ssh server
[server:~]$ # if you don't want to hammer NFS, copy deb to /tmp first.
[server:~]$ sudo dpkg -i /share/chef/chef-server_11.0.12-1.ubuntu.12.04_amd64.deb
[server:~]$ sudo chef-server-ctl reconfigure
[server:~]$ sudo chef-server-ctl test
}}}

=== Chef Workstation === 

This can be the same machine as the server. First we install the package, then configure it. The configuration involves cloning the {{{chef-repo}}} git repository that has the DETER specific recipes. We also set up a trusted user and the appropriate keys. 

Install Chef software:
{{{
[users:~]$ ssh workstation
[workstation:~]$ sudo dpkg -i /share/chef/chef_11.10.4-1.ubuntu.12.04_amd64.deb
[workstation:~]$ chef-client -v                # test - should show version.
}}}

Install git and the chef-repo:
{{{
[workstation:~]$ sudo apt-get install -y git   # chef uses git.
[workstation:~]$ sudo chmod g+w /local
[workstation:/local]$ cd /local
[workstation:/local]$ git clone /share/chef/chef-repo
[workstation:/local/chef-repo]$ cd chef-repo
}}}

Take a look in {{{/local/chef-repo/cookbooks/deter_node/recipes}}} for the "deter_node" recipes. There is also a simple "deter_node" role in {{{/local/chef-repo/roles/deter_node.rb}}} that has a {{{run_list}}} that tells the node to execute the recipes in the correct order.  

Now configure the workstation user and set up keys. Note that there *is* a web interface for this and that is the standard interface to use for this. Since it assumes you're running the browser locally though, you need to setup an ssh tunnel and a web proxy like Foxy-Proxy to access it. For this script though (and because we prefer it as it's scriptable) we stick to the command line. To do this, we need to copy private keys. (The procedure for using the web API copies public keys from a web page so is the preferred method.)

{{{
[workstation:/local/chef-repo]$ mkdir .chef
[workstation:/local/chef-repo]$ # copy keys!
[workstation:/local/chef-repo]$ ssh server sudo cat /etc/chef-server/admin.pem > .chef/admin.pem
[workstation:/local/chef-repo]$ ssh server sudo cat /etc/chef-server/chef-validator.pem  > .chef/chef-validator.pem
[workstation:/local/chef-repo]$ # use knife to configure the account/workstation
[workstation:/local/chef-repo]$ knife configure --initial
}}}

Answer the questions, substituting in your uid and server's control-net FQDN. A sample run is shown here:
{{{
[workstation:/local/chef-repo]$ knife configure --initial                                           
WARNING: No knife configuration file found                                                                            
Where should I put the config file? [/users/glawler/.chef/knife.rb] /local/chef-repo/.chef/knife.rb                   
Please enter the chef server URL: [https://pc40.isi.deterlab.net:443] https://pc46.isi.deterlab.net:443               
Please enter a name for the new user: [glawler]                                                                       
Please enter the existing admin name: [admin]                                                                         
Please enter the location of the existing admin's private key: [/etc/chef-server/admin.pem] /local/chef-repo/.chef/admin.pem                                                                                                                
Please enter the validation clientname: [chef-validator]                                                              
Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem] /local/chef-repo/.chef/chef-validator.pem                                                                                                            
Please enter the path to a chef repository (or leave blank): /local/chef-repo                                         
Creating initial API user...                                                                                          
Please enter a password for the new user:                                                                             
Created user[glawler]                                                                                                 
Configuration file written to /local/chef-repo/.chef/knife.rb                                                         
[workstation:/local/chef-repo]$ # Validate server connection and user.
[workstation:/local/chef-repo]$ knife user list
admin
glawler
[workstation:/local/chef-repo]$ 
}}}

Note that the {{{knife}}} command looks in ~/.chef and $(cwd)./chef for configuration files, so execute all {{{knife}}} commands from the {{{/local/chef-repo}}} directory or {{{knife}}} will be dull and not-happy. 

Push the local roles and recipes to the chef server.
{{{
[workstation:/local/chef-repo]$ knife cookbook upload hostsfile deter_node 
Uploading hostsfile    [2.4.4]                                                               
Uploading deter_node   [0.1.1]                                                               
Uploaded 2 cookbooks.                                                                        
[workstation:/local/chef-repo]$ knife role from file roles/deter_node.rb
Updated Role deter_node!
[workstation:/local/chef-repo]$ # Confirm things are OK.
[workstation:/local/chef-repo]$ knife cookbook list
deter_node   0.1.1
hostsfile    2.4.4
[workstation:/local/chef-repo]$ knife role list
deter_node
[workstation:/local/chef-repo]$ 
}}}

Push testbed specific information to the chef server. The {{{deter_user}}} recipes look for this information to get testbed-specific instantiation information, like the name of the boss machine or which file systems to mount. Chef calls these collections of data, "data bags". We first create a named data bag, "testbed-defs" then populate it with the testbed-specific information. There is currently only testbed information for Emulab-in-Emulab experiments, but a similar set of data would be created for each testbed instantiation. This data is stored in a git repo and pushed to a chef server. It can be edited via {{{knife}}} or a standard editor and updated in real time if need be.

{{{
[workstation:/local/chef-repo]$ knife data bag create testbed-defs
[workstation:/local/chef-repo]$ # We only have an emulab-in-emulab data bag defined right now.
[workstation:/local/chef-repo]$ knife data bag from file testbed-defs data_bags/testbed-defs/eine.json
[workstation:/local/chef-repo]$ knife data bag show testbed-defs eine
binary_store_path: binaries                                    
bossnode:          myboss.eine.deter.isi.deterlab.net          
db_conf:                                                       
  host: myboss.eine.deter.isi.deterlab.net                     
  name: tbdb                                                   
  user: mysql                                                  
fsnode:            myops.eine.deter.isi.deterlab.net           
fsprojdir:         /q/proj                                     
fsuserdir:         /q/users                                    
id:                eine                                        
node_mounts:                                                   
  device:      myops.eine.deter.isi.deterlab.net:/share        
  fstype:      nfs                                             
  mount_point: /share                                          
  options:     ro                                              
projdir:           /proj                                       
scratch:           myboss.eine.deter.isi.deterlab.net:8000     
testbed_netmask:   255.255.0.0                                 
testbed_network:   192.168.0.0                                 
userdir:           /users                                      
usernode:          myops.eine.deter.isi.deterlab.net           
[workstation:/local/chef-repo]$
}}}

If you look at {{{cookbooks/deter_node/libraries/node_info.rb}}} you can see the recipe code that references this information. 

We are now going to use {{{knife}}} to bootstrap Chef unto a client machine. The usual Chef assumes internet connectivity and asks you to download a script and pipe it to {{{sudo}}} on the client. We will not be doing that. {{{knife}}} understands custom install scripts so we'll give it one that is specific for our Ubuntu 12.04, non-internet connected test node. Take a look at {{{users:/share/chef/ubuntu12.04-deb.erb}}} if you're interested. {{{knife}}} will look in .chef/bootstrap for "distro" specific install files. We create the erb file and pass its name to the knife bootstrap command via the {{{--distro}}} argument.  

{{{
[workstation:/local/chef-repo]$ mkdir .chef/bootstrap
[workstation:/local/chef-repo]$ cp /share/chef/ubuntu12.04-deb.erb .chef/bootstrap
}}}

The bootstrap usually grabs the platform specific package (deb, yum, gem, etc) file from the internet. We give {{{chef}}} a custom bootstrap script that looks for it at {{{users:8523}}}. This means something on {{{users}}} must be listening on port 8523 and understand how to serve files via HTTPS. We use python for this. On {{{users}}}, cd to /share/chef and run {{{python -m SimpleHTTPServer 8523}}}. This starts a simple HTTP server on port 8523. If you want to run on your local server instead edit the file {{{/local/chef-repo/.chef/bootstrap/ubuntu12.04-deb.erb}}}, set the {{{SERVER}}} variable to the control net FQDN/ip address of your server and run the python simple server there. 

Now run the bootstrap command to install and configure chef on the client. {{{$NODE}}} is the control net name of the client node, like {{{pc33.isi.deterlab.net}}} and {{{$USER}}} if your sudo-able user name, like {{{glawler}}} or {{{faber}}}. 

{{{
[workstation:/local/chef-repo]$ knife bootstrap pc33.isi.deterlab.net -x $USER --sudo --distro ubuntu12.04-deb
}}}

(Sometimes the bootstrap fails to connect to the chef server and instead attempts to connect to https://localhost, which as this is running on the client, will not connect to the server. The ugly solution is to create the file {{{/etc/chef/client.rb}}} on the client with the content {{{chef_server_url "https://pc157.isi.deterlab.net:443"}}} (where the hostname is the hostname of your server). I was not able to track down why the client gets the incorrect server URL. After you create this file on the client, you can re0run the knife bootstrap command above.)

Now kill the python SimpleHTTPServer process on users. 

Confirm the node has chef installed and configured by running this {{{knife}}} command on your workstation machine. It should show the client node.

{{{
[workstation:/local/chef-repo]$ knife client list
chef-validator
chef-webui
pc33.isi.deterlab.net
[workstation:/local/chef-repo]$ # show the node information:
[workstation:/local/chef-repo]$ knife node show pc33.isi.deterlab.net
Node Name:   pc33.isi.deterlab.net 
Environment: _default              
FQDN:        pc33.isi.deterlab.net 
IP:          192.168.1.33          
Run List:                          
Roles:                             
Recipes:                           
Platform:    ubuntu 12.04          
Tags:                              
}}}

Now we assign recipes or a role to the client node and tell it to configure itself. (Note this could've been done during the bootstrap via the {{{-r}}} argument, but if we'd done that the bootstrap would've failed as the recipes as written fail on DETER.)

{{{
workstation$ knife node run_list add pc33.isi.deterlab.net 'role[deter_node]' 
pc33.isi.deterlab.net:      
  run_list: role[deter_node]
}}}

Now we force  {{{chef-client}}} to run on the client by sshing to it and running {{{chef-client}}} by hand. The recipes assigned to the node will fail however as we've not installed the mysql2 libraries the recipes need to communicate with the DETER database. If they were installed though, it still would not work as the DETER database is not accessible remotely. But you can see the recipes run and fail. 

Thus ends the tutorial. 

If you wanted to take it further, you can run an emulab-in-emulab experiment, install the mysql2 libraries on the client, and re-run {{{chef-client}}} on the client. If run you would see these recipes run in order.
 * {{{etchosts}}}: extract peer information and create an experiment specific {{{/etc/hosts}}} for the node.
 * {{{routing}}}: set default routes and route correctly for control net and data net. It grabs all addresses and subnet information, builds a graph and uses Dijkstra's algorithm to find shortest-path routing, then sets the appropriate routes. It currently sets a route for each node in the experiment. This is not good. The recipe will have to be improved to handle subnet routing.  
 * {{{mounts}}}: look in the appropriate "data bag" for the test bed type the client is running on and mount the file systems found there. 
 * {{{accounts}}}: extract user account information from the database and make the user accounts. Home directories would be on the mounted directories from the {{{mounts}}} recipe.
 * {{{ifconfig}}}: extract data network IP address(es) and netmask(s) from the database and set them on the appropriate network interfaces.
 * {{{binaries}}}: copy (via wget) platform specific binaries and start them running. This recipe currently start {{{ipodd}}} and {{{slothd}}}. This recipe could be expanded to build the binaries, create /etc/init.d/ scripts, then start the daemons. Etc. 
 
=== Database Configuration Modification ===

In order to use the {{{deter_node}}} cookbook, the database on boss must be accessible to the test nodes. Also the mysql2 gem must be installed on the test node as the recipes use that to talk to the database. Instructions for both are given below. 

The database on boss is mysql2. It is spawned on boot via the {{{/usr/local/etc/rc.d/2.mysql-server.sh}}} script. In that script one of the arugments to mysql2 is {{{--skip-networking}}}. If that is removed the database can be accessed remotely. You can either modify that script and call it with {{{stop}}} then {{{start}}} to restart the mysql2 instance or just {{{killall mysqld}}} then start it by hand with the following arguments: {{{--pid-file=/var/db/mysql/mysqld.pid --skip-grant-tables --user=mysql --log=/usr/testbed/log/mysql/base --log-bin=/usr/testbed/log/mysql/update --log-slow-queries=/usr/testbed/log/mysql/slowqueries --max_connections=500}}}.