| Version 12 (modified by , 10 years ago) (diff) |
|---|
Installing and Running Chef on DETER
There are 3 components to Chef: a server, a set of clients, and a set of workstations. The server is, well, the server - it handles requests from clients and workstations; it is the central organizing component of the system. The workstation is where Chef code is written. Think of it as a development or OPs workstation. The client(s) are the nodes that are configured/controlled. All three components need to run Chef sofware: the server runs chef server suite, the clients run chef-client, and the workstation runs git and a suite of chef scripts around git called knife that interact with a chef (git) repository and a chef server.
The workflow for a running system is recipes, roles, and chef code is written on a workstation and added to a chef repository. The person on the workstation then pushes updates to the chef server. The client then contacts the chef server for updates, gets the updates, and applies them locally. Knife, on a workstation, can also install Chef-client (and other packages) directly on a client. If this is done the flow becomes: 1) workstation pushes update to the server, 2) the workstation installs Chef on a client and gives it an initial role or list of recipes, 3) the newly installed chef-client then contacts the chef server and downloads the recipes and executes them, configuring the client node.
Note: the recipes here use a mysql library to talk directly to the testbed database. On the current DETER testbed though, the database is setup to only allow local access from boss. The TMCD/TMCC library is used to bridge the gap between test nodes and the database on boss on DETER. The recipes can be rewritten to use the TMCD interface but this was an exercise in trying to cut out Emulab code, a decision was made to talk directly to the database using standard database APIs. In short, the cookbook (and recipes) here cannot be run on DETER unless the database on boss is reconfigured to allow external access. Instructions are given for that below though. If you want to run the recipes, spin up an Emulab in Emulab experiment or access a local, non-public DETER/Emulab instance and modify the database configuration.
Swap in an experiment with at least three Ubuntu 12.04 nodes, one server, one workstation, and one client.
Chef Server
The server is simple to install: just use dpkg to install it.
users$ ssh server server$ # if you don't want to hammer NFS, copy deb to /tmp first. server$ sudo dpkg -i /share/chef/chef-server_11.0.12-1.ubuntu.12.04_amd64.deb server$ sudo chef-server-ctl reconfigure server$ sudo chef-server-ctl test
Chef Workstation
This can be the same machine as the server. First we install the package, then configure it. The configuration involves cloning the chef-repo git repository that has the DETER specific recipes. We also set up a trusted user and the appropriate keys.
Install Chef software:
users$ ssh workstation workstation$ sudo dpkg -i /share/chef/chef_11.10.4-1.ubuntu.12.04_amd64.deb workstation$ chef-client -v # test - should show version.
Install git and the chef-repo:
workstation$ sudo apt-get install -y git # chef uses git. workstation$ sudo chmod g+w /local workstation$ cd /local workstation$ git clone /share/chef/chef-repo workstation$ cd chef-repo
Take a look in /local/chef-repo/cookbooks/deter_node/recipes for the "deter_node" recipes. There is also a simple "deter_node" role in /local/chef-repo/roles/deter_node.rb that has a run_list that tells the node to execute the recipes in the correct order.
Now configure the workstation user and set up keys. Note that there *is* a web interface for this and that is the standard interface to use for this. Since it assumes you're running the browser locally though, you need to setup an ssh tunnel and a web proxy like Foxy-Proxy to access it. For this script though (and because we prefer it as it's scriptable) we stick to the command line. To do this, we need to copy private keys. (The procedure for using the web API copies public keys from a web page so is the preferred method.)
workstation$ cd /local/chef-repo workstation$ mkdir .chef workstation$ # copy keys! workstation$ ssh server sudo cat /etc/chef-server/admin.pem > .chef/admin.pem workstation$ ssh server sudo cat /etc/chef-server/chef-validator.pem > .chef/chef-validator.pem workstation$ # use knife to configure the account/workstation workstation$ knife configure --initial
Answer the questions, substituting in your uid and server's control-net FQDN. A sample run is shown here:
workstation$ knife configure --initial WARNING: No knife configuration file found Where should I put the config file? [/users/glawler/.chef/knife.rb] /local/chef-repo/.chef/knife.rb Please enter the chef server URL: [https://pc40.isi.deterlab.net:443] https://pc46.isi.deterlab.net:443 Please enter a name for the new user: [glawler] Please enter the existing admin name: [admin] Please enter the location of the existing admin's private key: [/etc/chef-server/admin.pem] /local/chef-repo/.chef/admin.pem Please enter the validation clientname: [chef-validator] Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem] /local/chef-repo/.chef/chef-validator.pem Please enter the path to a chef repository (or leave blank): /local/chef-repo Creating initial API user... Please enter a password for the new user: Created user[glawler] Configuration file written to /local/chef-repo/.chef/knife.rb workstation$ # Validate server connection and user. workstation$ knife user list admin glawler workstation$
Note that the knife command looks in ~/.chef and $(cwd)./chef for configuration files, so execute all knife commands from the /local/chef-repo directory or knife will be dull and not-happy.
Push the local roles and recipes to the chef server.
workstation$ cd /local/chef-repo workstation$ knife cookbook upload hostsfile deter_node Uploading hostsfile [2.4.4] Uploading deter_node [0.1.1] Uploaded 2 cookbooks. workstation$ knife role from file roles/deter_node.rb Updated Role deter_node! workstation$ # Confirm things are OK. workstation$ knife cookbook list deter_node 0.1.1 hostsfile 2.4.4 workstation$ knife role list deter_node workstation$
Push testbed specific information to the chef server. The deter_user recipes look for this information to get testbed-specific instantiation information, like the name of the boss machine or which file systems to mount. Chef calls these collections of data, "data bags". We first create a named data bag, "testbed-defs" then populate it with the testbed-specific information.
workstation$ cd /local/chef-repo workstation$ knife data bag create testbed-defs workstation$ # We only have an emulab-in-emulab data bag defined right now. workstation$ knife data bag from file testbed-defs data_bags/testbed-defs/eine.json workstation$ knife data bag show testbed-defs eine binary_store_path: binaries bossnode: myboss.eine.deter.isi.deterlab.net db_conf: host: myboss.eine.deter.isi.deterlab.net name: tbdb user: mysql fsnode: myops.eine.deter.isi.deterlab.net fsprojdir: /q/proj fsuserdir: /q/users id: eine node_mounts: device: myops.eine.deter.isi.deterlab.net:/share fstype: nfs mount_point: /share options: ro projdir: /proj scratch: myboss.eine.deter.isi.deterlab.net:8000 testbed_netmask: 255.255.0.0 testbed_network: 192.168.0.0 userdir: /users usernode: myops.eine.deter.isi.deterlab.net workstation$
If you look at cookbooks/deter_node/libraries/node_info.rb you can see the recipe code that references this information.
We are now going to use knife to bootstrap Chef unto a client machine. The usual Chef assumes internet connectivity and asks you to download a script and pipe it to sudo on the client. We will not be doing that.
> cd /local/chef-repo > mkdir .chef/bootstrap > cp /share/chef/ubuntu12.04-deb.erb .chef/bootstrap
The bootstrap usually grabs the package file from the internet. We give chef a custom bootstrap script that looks for it at users:8523. This means something on users must be listening on port 8523 and understand how to serve files via HTTPS. We use python for this. On users, cd to /share/chef and run python -m SimpleHTTPServer 8523. This starts a simple HTTP server on port 8523. If you want to run on your local server instead edit the file /local/chef-repo/.chef/bootstrap/ubuntu12.04-deb.erb, set the SERVER variable to the control net FQDN/ip address of your server and run the python simple server there.
Now run the bootstrap command to install and configure chef on the client. $NODE is the control net name of the client node, like pc33.isi.deterlab.net and $USER if your sudo-able user name, like glawler or faber.
> knife bootstrap $NODE -x $USER --sudo --distro ubuntu12.04-deb
Now kill the python SimpleHTTPServer process on users.
Confirm the node has chef installed and configured by running this knife command on your workstation machine. It should show the client node.
> cd /local/chef-repo > knife client list chef-validator chef-webui pc33.isi.deterlab.net > # show the node information: > knife node show pc33.isi.deterlab.net Node Name: pc33.isi.deterlab.net Environment: _default FQDN: pc33.isi.deterlab.net IP: 192.168.1.33 Run List: Roles: Recipes: Platform: ubuntu 12.04 Tags:
Now we assign recipes or a role to the client node and tell it to configure itself. (Note this could've been done during the bootstrap via the -r argument.
workstation$ knife node run_list add pc33.isi.deterlab.net 'role[deter_node]' pc33.isi.deterlab.net: run_list: role[deter_node]
Now we force chef-client to run on the client by sshing to it and running chef-client by hand. The recipes assigned to the node will fail however as we've not installed the mysql2 libraries the recipes need to communicate with the DETER database. If they were installed though, it still would not work as the DETER database is not accessible remotely.
Thus ends the tutorial.
If you wanted to take it further, you can run an emulab-in-emulab experiment, install the mysql2 libraries on the client, and re-run chef-client on the client. If run you would see these recipes run in order.
routing: set default routes and route correctly for control net and data net.mounts: look in the appropriate "data bag" for the test bed type the client is running on and mount the file systems found there. See/local/chef-repo/data_bags/testbed-defs/eine.jsonfor a sample emulab-in-emulab testbed-specific configuration. (These testbed specific data bags take the place of the compile time *.DEF files in the testbed source tree.)
Chef Client(s)
These machines should be different than the server and workstation. There are three ways to install chef on a client. 1) simply use whatever package system exists for the client OS and install the chef software. 2) Use a chef workstation to push platform specific chef software to the client that installs and configures it. 3) Use the chef supplied script that reaches out to the internet and executes arbitrary commands on the machine as root. We will use method 2) to give a flavor of installing chef by force on unsuspecting nodes on DETER (as this is the most flexible and useful installation method in the DETER context).
> # already installed via knife bootstrap!
Database Configuration Modification
In order to use the deter_node cookbook, the database on boss must be accessible to the test nodes. Also the mysql2 gem must be installed on the test node as the recipes use that to talk to the database. Instructions for both are given below.
The database on boss is mysql2. It is spawned on boot via the /usr/local/etc/rc.d/2.mysql-server.sh script. In that script one of the arugments to mysql2 is --skip-networking. If that is removed the database can be accessed remotely. You can either modify that script and call it with stop then start to restart the mysql2 instance or just killall mysqld then start it by hand with the following arguments: --pid-file=/var/db/mysql/mysqld.pid --skip-grant-tables --user=mysql --log=/usr/testbed/log/mysql/base --log-bin=/usr/testbed/log/mysql/update --log-slow-queries=/usr/testbed/log/mysql/slowqueries --max_connections=500.
