wiki:ChefonDETER

Version 12 (modified by Geoff Lawler, 10 years ago) (diff)

--

Installing and Running Chef on DETER

There are 3 components to Chef: a server, a set of clients, and a set of workstations. The server is, well, the server - it handles requests from clients and workstations; it is the central organizing component of the system. The workstation is where Chef code is written. Think of it as a development or OPs workstation. The client(s) are the nodes that are configured/controlled. All three components need to run Chef sofware: the server runs chef server suite, the clients run chef-client, and the workstation runs git and a suite of chef scripts around git called knife that interact with a chef (git) repository and a chef server.

The workflow for a running system is recipes, roles, and chef code is written on a workstation and added to a chef repository. The person on the workstation then pushes updates to the chef server. The client then contacts the chef server for updates, gets the updates, and applies them locally. Knife, on a workstation, can also install Chef-client (and other packages) directly on a client. If this is done the flow becomes: 1) workstation pushes update to the server, 2) the workstation installs Chef on a client and gives it an initial role or list of recipes, 3) the newly installed chef-client then contacts the chef server and downloads the recipes and executes them, configuring the client node.

Note: the recipes here use a mysql library to talk directly to the testbed database. On the current DETER testbed though, the database is setup to only allow local access from boss. The TMCD/TMCC library is used to bridge the gap between test nodes and the database on boss on DETER. The recipes can be rewritten to use the TMCD interface but this was an exercise in trying to cut out Emulab code, a decision was made to talk directly to the database using standard database APIs. In short, the cookbook (and recipes) here cannot be run on DETER unless the database on boss is reconfigured to allow external access. Instructions are given for that below though. If you want to run the recipes, spin up an Emulab in Emulab experiment or access a local, non-public DETER/Emulab instance and modify the database configuration.

Swap in an experiment with at least three Ubuntu 12.04 nodes, one server, one workstation, and one client.

Chef Server

The server is simple to install: just use dpkg to install it.

users$ ssh server
server$ # if you don't want to hammer NFS, copy deb to /tmp first.
server$ sudo dpkg -i /share/chef/chef-server_11.0.12-1.ubuntu.12.04_amd64.deb
server$ sudo chef-server-ctl reconfigure
server$ sudo chef-server-ctl test

Chef Workstation

This can be the same machine as the server. First we install the package, then configure it. The configuration involves cloning the chef-repo git repository that has the DETER specific recipes. We also set up a trusted user and the appropriate keys.

Install Chef software:

users$ ssh workstation
workstation$ sudo dpkg -i /share/chef/chef_11.10.4-1.ubuntu.12.04_amd64.deb
workstation$ chef-client -v                # test - should show version.

Install git and the chef-repo:

workstation$ sudo apt-get install -y git   # chef uses git.
workstation$ sudo chmod g+w /local
workstation$ cd /local
workstation$ git clone /share/chef/chef-repo
workstation$ cd chef-repo

Take a look in /local/chef-repo/cookbooks/deter_node/recipes for the "deter_node" recipes. There is also a simple "deter_node" role in /local/chef-repo/roles/deter_node.rb that has a run_list that tells the node to execute the recipes in the correct order.

Now configure the workstation user and set up keys. Note that there *is* a web interface for this and that is the standard interface to use for this. Since it assumes you're running the browser locally though, you need to setup an ssh tunnel and a web proxy like Foxy-Proxy to access it. For this script though (and because we prefer it as it's scriptable) we stick to the command line. To do this, we need to copy private keys. (The procedure for using the web API copies public keys from a web page so is the preferred method.)

workstation$ cd /local/chef-repo
workstation$ mkdir .chef
workstation$ # copy keys!
workstation$ ssh server sudo cat /etc/chef-server/admin.pem > .chef/admin.pem
workstation$ ssh server sudo cat /etc/chef-server/chef-validator.pem  > .chef/chef-validator.pem
workstation$ # use knife to configure the account/workstation
workstation$ knife configure --initial

Answer the questions, substituting in your uid and server's control-net FQDN. A sample run is shown here:

workstation$ knife configure --initial                                           
WARNING: No knife configuration file found                                                                            
Where should I put the config file? [/users/glawler/.chef/knife.rb] /local/chef-repo/.chef/knife.rb                   
Please enter the chef server URL: [https://pc40.isi.deterlab.net:443] https://pc46.isi.deterlab.net:443               
Please enter a name for the new user: [glawler]                                                                       
Please enter the existing admin name: [admin]                                                                         
Please enter the location of the existing admin's private key: [/etc/chef-server/admin.pem] /local/chef-repo/.chef/admin.pem                                                                                                                
Please enter the validation clientname: [chef-validator]                                                              
Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem] /local/chef-repo/.chef/chef-validator.pem                                                                                                            
Please enter the path to a chef repository (or leave blank): /local/chef-repo                                         
Creating initial API user...                                                                                          
Please enter a password for the new user:                                                                             
Created user[glawler]                                                                                                 
Configuration file written to /local/chef-repo/.chef/knife.rb                                                         
workstation$ # Validate server connection and user.
workstation$ knife user list
admin
glawler
workstation$ 

Note that the knife command looks in ~/.chef and $(cwd)./chef for configuration files, so execute all knife commands from the /local/chef-repo directory or knife will be dull and not-happy.

Push the local roles and recipes to the chef server.

workstation$ cd /local/chef-repo
workstation$ knife cookbook upload hostsfile deter_node 
Uploading hostsfile    [2.4.4]                                                               
Uploading deter_node   [0.1.1]                                                               
Uploaded 2 cookbooks.                                                                        
workstation$ knife role from file roles/deter_node.rb
Updated Role deter_node!
workstation$ # Confirm things are OK.
workstation$ knife cookbook list
deter_node   0.1.1
hostsfile    2.4.4
workstation$ knife role list
deter_node
workstation$ 

Push testbed specific information to the chef server. The deter_user recipes look for this information to get testbed-specific instantiation information, like the name of the boss machine or which file systems to mount. Chef calls these collections of data, "data bags". We first create a named data bag, "testbed-defs" then populate it with the testbed-specific information.

workstation$ cd /local/chef-repo
workstation$ knife data bag create testbed-defs
workstation$ # We only have an emulab-in-emulab data bag defined right now.
workstation$ knife data bag from file testbed-defs data_bags/testbed-defs/eine.json
workstation$ knife data bag show testbed-defs eine
binary_store_path: binaries                                    
bossnode:          myboss.eine.deter.isi.deterlab.net          
db_conf:                                                       
  host: myboss.eine.deter.isi.deterlab.net                     
  name: tbdb                                                   
  user: mysql                                                  
fsnode:            myops.eine.deter.isi.deterlab.net           
fsprojdir:         /q/proj                                     
fsuserdir:         /q/users                                    
id:                eine                                        
node_mounts:                                                   
  device:      myops.eine.deter.isi.deterlab.net:/share        
  fstype:      nfs                                             
  mount_point: /share                                          
  options:     ro                                              
projdir:           /proj                                       
scratch:           myboss.eine.deter.isi.deterlab.net:8000     
testbed_netmask:   255.255.0.0                                 
testbed_network:   192.168.0.0                                 
userdir:           /users                                      
usernode:          myops.eine.deter.isi.deterlab.net           
workstation$

If you look at cookbooks/deter_node/libraries/node_info.rb you can see the recipe code that references this information.

We are now going to use knife to bootstrap Chef unto a client machine. The usual Chef assumes internet connectivity and asks you to download a script and pipe it to sudo on the client. We will not be doing that.

> cd /local/chef-repo
> mkdir .chef/bootstrap
> cp /share/chef/ubuntu12.04-deb.erb .chef/bootstrap

The bootstrap usually grabs the package file from the internet. We give chef a custom bootstrap script that looks for it at users:8523. This means something on users must be listening on port 8523 and understand how to serve files via HTTPS. We use python for this. On users, cd to /share/chef and run python -m SimpleHTTPServer 8523. This starts a simple HTTP server on port 8523. If you want to run on your local server instead edit the file /local/chef-repo/.chef/bootstrap/ubuntu12.04-deb.erb, set the SERVER variable to the control net FQDN/ip address of your server and run the python simple server there.

Now run the bootstrap command to install and configure chef on the client. $NODE is the control net name of the client node, like pc33.isi.deterlab.net and $USER if your sudo-able user name, like glawler or faber.

> knife bootstrap $NODE -x $USER --sudo --distro ubuntu12.04-deb

Now kill the python SimpleHTTPServer process on users.

Confirm the node has chef installed and configured by running this knife command on your workstation machine. It should show the client node.

> cd /local/chef-repo
> knife client list
chef-validator
chef-webui
pc33.isi.deterlab.net
> # show the node information:
> knife node show pc33.isi.deterlab.net
Node Name:   pc33.isi.deterlab.net 
Environment: _default              
FQDN:        pc33.isi.deterlab.net 
IP:          192.168.1.33          
Run List:                          
Roles:                             
Recipes:                           
Platform:    ubuntu 12.04          
Tags:                              

Now we assign recipes or a role to the client node and tell it to configure itself. (Note this could've been done during the bootstrap via the -r argument.

workstation$ knife node run_list add pc33.isi.deterlab.net 'role[deter_node]' 
pc33.isi.deterlab.net:      
  run_list: role[deter_node]

Now we force chef-client to run on the client by sshing to it and running chef-client by hand. The recipes assigned to the node will fail however as we've not installed the mysql2 libraries the recipes need to communicate with the DETER database. If they were installed though, it still would not work as the DETER database is not accessible remotely.

Thus ends the tutorial.

If you wanted to take it further, you can run an emulab-in-emulab experiment, install the mysql2 libraries on the client, and re-run chef-client on the client. If run you would see these recipes run in order.

  • routing: set default routes and route correctly for control net and data net.
  • mounts: look in the appropriate "data bag" for the test bed type the client is running on and mount the file systems found there. See /local/chef-repo/data_bags/testbed-defs/eine.json for a sample emulab-in-emulab testbed-specific configuration. (These testbed specific data bags take the place of the compile time *.DEF files in the testbed source tree.)

Chef Client(s)

These machines should be different than the server and workstation. There are three ways to install chef on a client. 1) simply use whatever package system exists for the client OS and install the chef software. 2) Use a chef workstation to push platform specific chef software to the client that installs and configures it. 3) Use the chef supplied script that reaches out to the internet and executes arbitrary commands on the machine as root. We will use method 2) to give a flavor of installing chef by force on unsuspecting nodes on DETER (as this is the most flexible and useful installation method in the DETER context).

> # already installed via knife bootstrap!

Database Configuration Modification

In order to use the deter_node cookbook, the database on boss must be accessible to the test nodes. Also the mysql2 gem must be installed on the test node as the recipes use that to talk to the database. Instructions for both are given below.

The database on boss is mysql2. It is spawned on boot via the /usr/local/etc/rc.d/2.mysql-server.sh script. In that script one of the arugments to mysql2 is --skip-networking. If that is removed the database can be accessed remotely. You can either modify that script and call it with stop then start to restart the mysql2 instance or just killall mysqld then start it by hand with the following arguments: --pid-file=/var/db/mysql/mysqld.pid --skip-grant-tables --user=mysql --log=/usr/testbed/log/mysql/base --log-bin=/usr/testbed/log/mysql/update --log-slow-queries=/usr/testbed/log/mysql/slowqueries --max_connections=500.