Installing and Running Chef on DETER
There are 3 components to Chef: a server, a set of clients, and a set of workstations. The server is, well, the server - it handles requests from clients and workstations; it is the central organizing component of the system. The workstation is where Chef code is written. Think of it as a development or OPs workstation. The client(s) are the nodes that are configured/controlled. All three components need to run Chef sofware: the server runs chef server suite, the clients run chef-client, and the workstation runs git and a suite of chef scripts around git called knife that interact with a chef (git) repository and a chef server.
The workflow for a running system is recipes, roles, and chef code is written on a workstation and added to a chef repository. The person on the workstation then pushes updates to the chef server. The client then contacts the chef server for updates, gets the updates, and applies them locally. Knife, on a workstation, can also install Chef-client (and other packages) directly on a client. If this is done the flow becomes: 1) workstation pushes update to the server, 2) the workstation installs Chef on a client and gives it an initial role or list of recipes, 3) the newly installed chef-client then contacts the chef server and downloads the recipes and executes them, configuring the client node.
Note: the recipes here use a mysql library to talk directly to the testbed database. On the current DETER testbed though, the database is setup to only allow local access from boss. The TMCD/TMCC library is used to bridge the gap between test nodes and the database on boss on DETER. The recipes can be rewritten to use the TMCD interface but this was an exercise in trying to cut out Emulab code, a decision was made to talk directly to the database using standard database APIs. In short, the cookbook (and recipes) here cannot be run on DETER unless the database on boss is reconfigured to allow external access. Instructions are given for that below though. If you want to run the recipes, spin up an Emulab in Emulab experiment or access a local, non-public DETER/Emulab instance and modify the database configuration.
Swap in an experiment with at least three Ubuntu 12.04 nodes, one server, one workstation, and one client.
Chef Server
The server is simple to install: just use dpkg to install it.
[users:~]$ ssh server [server:~]$ # if you don't want to hammer NFS, copy deb to /tmp first. [server:~]$ sudo dpkg -i /share/chef/chef-server_11.0.12-1.ubuntu.12.04_amd64.deb [server:~]$ sudo chef-server-ctl reconfigure [server:~]$ sudo chef-server-ctl test
Chef Workstation
This can be the same machine as the server. First we install the package, then configure it. The configuration involves cloning the chef-repo git repository that has the DETER specific recipes. We also set up a trusted user and the appropriate keys.
Install Chef software:
[users:~]$ ssh workstation [workstation:~]$ sudo dpkg -i /share/chef/chef_11.10.4-1.ubuntu.12.04_amd64.deb [workstation:~]$ chef-client -v # test - should show version.
Install git and the chef-repo:
[workstation:~]$ sudo apt-get install -y git # chef uses git. [workstation:~]$ sudo chmod g+w /local [workstation:/local]$ cd /local [workstation:/local]$ git clone /share/chef/chef-repo [workstation:/local/chef-repo]$ cd chef-repo
Take a look in /local/chef-repo/cookbooks/deter_node/recipes for the "deter_node" recipes. There is also a simple "deter_node" role in /local/chef-repo/roles/deter_node.rb that has a run_list that tells the node to execute the recipes in the correct order.
Now configure the workstation user and set up keys. Note that there *is* a web interface for this and that is the standard interface to use for this. Since it assumes you're running the browser locally though, you need to setup an ssh tunnel and a web proxy like Foxy-Proxy to access it. For this script though (and because we prefer it as it's scriptable) we stick to the command line. To do this, we need to copy private keys. (The procedure for using the web API copies public keys from a web page so is the preferred method.)
[workstation:/local/chef-repo]$ mkdir .chef [workstation:/local/chef-repo]$ # copy keys! [workstation:/local/chef-repo]$ ssh server sudo cat /etc/chef-server/admin.pem > .chef/admin.pem [workstation:/local/chef-repo]$ ssh server sudo cat /etc/chef-server/chef-validator.pem > .chef/chef-validator.pem [workstation:/local/chef-repo]$ # use knife to configure the account/workstation [workstation:/local/chef-repo]$ knife configure --initial
Answer the questions, substituting in your uid and server's control-net FQDN. A sample run is shown here:
[workstation:/local/chef-repo]$ knife configure --initial WARNING: No knife configuration file found Where should I put the config file? [/users/glawler/.chef/knife.rb] /local/chef-repo/.chef/knife.rb Please enter the chef server URL: [https://pc40.isi.deterlab.net:443] https://pc46.isi.deterlab.net:443 Please enter a name for the new user: [glawler] Please enter the existing admin name: [admin] Please enter the location of the existing admin's private key: [/etc/chef-server/admin.pem] /local/chef-repo/.chef/admin.pem Please enter the validation clientname: [chef-validator] Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem] /local/chef-repo/.chef/chef-validator.pem Please enter the path to a chef repository (or leave blank): /local/chef-repo Creating initial API user... Please enter a password for the new user: Created user[glawler] Configuration file written to /local/chef-repo/.chef/knife.rb [workstation:/local/chef-repo]$ # Validate server connection and user. [workstation:/local/chef-repo]$ knife user list admin glawler [workstation:/local/chef-repo]$
Note that the knife command looks in ~/.chef and $(cwd)./chef for configuration files, so execute all knife commands from the /local/chef-repo directory or knife will be dull and not-happy.
Push the local roles and recipes to the chef server.
[workstation:/local/chef-repo]$ knife cookbook upload hostsfile deter_node Uploading hostsfile [2.4.4] Uploading deter_node [0.1.1] Uploaded 2 cookbooks. [workstation:/local/chef-repo]$ knife role from file roles/deter_node.rb Updated Role deter_node! [workstation:/local/chef-repo]$ # Confirm things are OK. [workstation:/local/chef-repo]$ knife cookbook list deter_node 0.1.1 hostsfile 2.4.4 [workstation:/local/chef-repo]$ knife role list deter_node [workstation:/local/chef-repo]$
Push testbed specific information to the chef server. The deter_user recipes look for this information to get testbed-specific instantiation information, like the name of the boss machine or which file systems to mount. Chef calls these collections of data, "data bags". We first create a named data bag, "testbed-defs" then populate it with the testbed-specific information. There is currently only testbed information for Emulab-in-Emulab experiments, but a similar set of data would be created for each testbed instantiation. This data is stored in a git repo and pushed to a chef server. It can be edited via knife or a standard editor and updated in real time if need be.
[workstation:/local/chef-repo]$ knife data bag create testbed-defs [workstation:/local/chef-repo]$ # We only have an emulab-in-emulab data bag defined right now. [workstation:/local/chef-repo]$ knife data bag from file testbed-defs data_bags/testbed-defs/eine.json [workstation:/local/chef-repo]$ knife data bag show testbed-defs eine binary_store_path: binaries bossnode: myboss.eine.deter.isi.deterlab.net db_conf: host: myboss.eine.deter.isi.deterlab.net name: tbdb user: mysql fsnode: myops.eine.deter.isi.deterlab.net fsprojdir: /q/proj fsuserdir: /q/users id: eine node_mounts: device: myops.eine.deter.isi.deterlab.net:/share fstype: nfs mount_point: /share options: ro projdir: /proj scratch: myboss.eine.deter.isi.deterlab.net:8000 testbed_netmask: 255.255.0.0 testbed_network: 192.168.0.0 userdir: /users usernode: myops.eine.deter.isi.deterlab.net [workstation:/local/chef-repo]$
If you look at cookbooks/deter_node/libraries/node_info.rb you can see the recipe code that references this information.
We are now going to use knife to bootstrap Chef unto a client machine. The usual Chef assumes internet connectivity and asks you to download a script and pipe it to sudo on the client. We will not be doing that. knife understands custom install scripts so we'll give it one that is specific for our Ubuntu 12.04, non-internet connected test node. Take a look at users:/share/chef/ubuntu12.04-deb.erb if you're interested. knife will look in .chef/bootstrap for "distro" specific install files. We create the erb file and pass its name to the knife bootstrap command via the --distro argument.
[workstation:/local/chef-repo]$ mkdir .chef/bootstrap [workstation:/local/chef-repo]$ cp /share/chef/ubuntu12.04-deb.erb .chef/bootstrap
The bootstrap usually grabs the platform specific package (deb, yum, gem, etc) file from the internet. We give chef a custom bootstrap script that looks for it at users:8523. This means something on users must be listening on port 8523 and understand how to serve files via HTTPS. We use python for this. On users, cd to /share/chef and run python -m SimpleHTTPServer 8523. This starts a simple HTTP server on port 8523. If you want to run on your local server instead edit the file /local/chef-repo/.chef/bootstrap/ubuntu12.04-deb.erb, set the SERVER variable to the control net FQDN/ip address of your server and run the python simple server there.
Now run the bootstrap command to install and configure chef on the client. $NODE is the control net name of the client node, like pc33.isi.deterlab.net and $USER if your sudo-able user name, like glawler or faber.
[workstation:/local/chef-repo]$ knife bootstrap pc33.isi.deterlab.net -x $USER --sudo --distro ubuntu12.04-deb
(Sometimes the bootstrap fails to connect to the chef server and instead attempts to connect to https://localhost, which as this is running on the client, will not connect to the server. The ugly solution is to create the file /etc/chef/client.rb on the client with the content chef_server_url "https://pc157.isi.deterlab.net:443" (where the hostname is the hostname of your server). I was not able to track down why the client gets the incorrect server URL. After you create this file on the client, you can re0run the knife bootstrap command above.)
Now kill the python SimpleHTTPServer process on users.
Confirm the node has chef installed and configured by running this knife command on your workstation machine. It should show the client node.
[workstation:/local/chef-repo]$ knife client list chef-validator chef-webui pc33.isi.deterlab.net [workstation:/local/chef-repo]$ # show the node information: [workstation:/local/chef-repo]$ knife node show pc33.isi.deterlab.net Node Name: pc33.isi.deterlab.net Environment: _default FQDN: pc33.isi.deterlab.net IP: 192.168.1.33 Run List: Roles: Recipes: Platform: ubuntu 12.04 Tags:
Now we assign recipes or a role to the client node and tell it to configure itself. (Note this could've been done during the bootstrap via the -r argument, but if we'd done that the bootstrap would've failed as the recipes as written fail on DETER.)
workstation$ knife node run_list add pc33.isi.deterlab.net 'role[deter_node]' pc33.isi.deterlab.net: run_list: role[deter_node]
Now we force chef-client to run on the client by sshing to it and running chef-client by hand. The recipes assigned to the node will fail however as we've not installed the mysql2 libraries the recipes need to communicate with the DETER database. If they were installed though, it still would not work as the DETER database is not accessible remotely. But you can see the recipes run and fail.
Thus ends the tutorial.
If you wanted to take it further, you can run an emulab-in-emulab experiment, install the mysql2 libraries on the client, and re-run chef-client on the client. If run you would see these recipes run in order.
etchosts: extract peer information and create an experiment specific/etc/hostsfor the node.routing: set default routes and route correctly for control net and data net. It grabs all addresses and subnet information, builds a graph and uses Dijkstra's algorithm to find shortest-path routing, then sets the appropriate routes. It currently sets a route for each node in the experiment. This is not good. The recipe will have to be improved to handle subnet routing.mounts: look in the appropriate "data bag" for the test bed type the client is running on and mount the file systems found there.accounts: extract user account information from the database and make the user accounts. Home directories would be on the mounted directories from themountsrecipe.ifconfig: extract data network IP address(es) and netmask(s) from the database and set them on the appropriate network interfaces.binaries: copy (via wget) platform specific binaries and start them running. This recipe currently startipoddandslothd. This recipe could be expanded to build the binaries, create /etc/init.d/ scripts, then start the daemons. Etc.
Database Configuration Modification
In order to use the deter_node cookbook, the database on boss must be accessible to the test nodes. Also the mysql2 gem must be installed on the test node as the recipes use that to talk to the database. Instructions for both are given below.
The database on boss is mysql2. It is spawned on boot via the /usr/local/etc/rc.d/2.mysql-server.sh script. In that script one of the arugments to mysql2 is --skip-networking. If that is removed the database can be accessed remotely. You can either modify that script and call it with stop then start to restart the mysql2 instance or just killall mysqld then start it by hand with the following arguments: --pid-file=/var/db/mysql/mysqld.pid --skip-grant-tables --user=mysql --log=/usr/testbed/log/mysql/base --log-bin=/usr/testbed/log/mysql/update --log-slow-queries=/usr/testbed/log/mysql/slowqueries --max_connections=500.
