=== The new(est) plan: === Make recipes that query the tbdb directly. If done right, the nodes configure themselves with information from the database directly. The code for querying the database will live in Chef recipes, pulled onto the nodes, then executed on boot. We'll add a hook in dhcp init, much like emulab does now to insure the control network is up, that executes 'chef-client', which does the magic. === Things to do: === 1. Get chef on a client machine, on the image itself for now. Later we can install chef remotely on boot? 1. Get the needed packges for database access onto the test node (mysql gems). 1. Write a simple recipe that grabs account information and creates the accounts. === Plan execution === 1. Chef is on a client now, arya, and the chef server, sansa, knows how to re-install if needed. 1. Figure out how to install mysql gem on arya without direct access to package repos. a. How to run your own Gem server, interesting: http://guides.rubygems.org/run-your-own-gem-server/ b. installing mysql2 gem: * sudo apt-get install liberuby-dev libmysqlclient-dev * sudo /opt/chef/embedded/bin/gem install --local mysql2-0.3.16.gem (assumes chef is installed, otherwise just "gem install") c. test installation. my simple script below is failing to load mysql2. {{{ #!/usr/bin/env ruby require 'mysql2' client = Mysql2::Client.new(:host => "localhost", :username => "root") }}} * needed to use chef's ruby as that's where the gem is installed: {{{/opt/chef/embedded/bin/ruby mysql_test.rb}}}. * Back up a step and see if we can and how we do connect to tbdb remotely. * running on myboss, 7777 {{{mysql -h myboss -p 7777}}} asks for a password. * Looking for appropriate keys to use. * tbdb is running "skip-grant-tables" so anyone on the localhost can assess it. * doing "mysql -h myboss -P 7777" gives cryptic error {{{ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0}}}. === Scratch === Connecting to test node servers via ssh and proxies. Add this to ~/.ssh/config {{{ Host eine Hostname myboss.eine.deter.isi.deterlab.net ProxyCommand ssh -W %h:%p deter DynamicForward 1280 Host sansa Hostname sansa.twonode.deter ProxyCommand ssh -W %h:%p eine DynamicForward 3212 }}} Then set up web proxy like foxyproxy to do SOCKS5 to those localhost:ports when the URL points there. ---- === Older Notes === {{{ The Plan: ------------------------------------------------------------------- Use Chef to do host configuration on DETER. Plan outline ------------------------------------------------------------------- 1) Modify boss at tbsetup/tbswap and tbswapout to call out to new daemon. 1a) add hook to remove experiment config on swap in 1b) add hook to remove experiment config on swap out 2) New daemon gets swap in message, reads configuration information from testbed database 2a) Extract host configuration from testebd DB 2b) used extracted information to create chef recipes 3) Modify test node to not call out to tmcd for local configuration, but instead call out to chef and ask chef to configure the local node. 4) Since chef knows the node(s) to configure and has the experiment-specific recipes already, it just goes ahead and configures the node. 4a) Figure out how the mapping form node --> recipies works and apply it. (node roles?) Much, much finer detail. ------------------------------------------------------------------- 1) Find place to put hook into boss's experiment swap in code. 1a) - Ted suggests tbswap. - ok for mapping. - calls to tarfiles_setup - what does that do specifically? Which tar files to where? - user tarfiles or?... - seems to be, yes. user tarfiles. - setup extra_nodes - delay nodes and such? - setup mount points (server side exports, I assuming) - setup named names for experiment - generate topo def file (and ltmap) - run os_setup - what does this do? Looking for database updates that describe node configuration. - first comment claims 'nodes table will already contain all the information...' - so where does that happen? Somewhere in assign? ------ - Finding spot to add hook: - start in tbswap.in - doSwapin(REAL) - ignoring modify/recover right now - just looking at swap in from nothing - call to TBSETUP/bin/mapper - and mapper calls assign, so add hook after this. - added code call out to script (not yet written): # Notify external entities that assign/mapping is complete # and database contains node configuration information. if (system("tb_configure_ready $pid $eid")) { $exitcode = $? >> 8; tberror "Failed ($exitcode) to notify that configuration was complete."; tbreport(SEV_ERROR, "tb_configure_ready failed", $exitcode); # Make this a fatal error for now. Revisit at some point. return 8; } - This code was added just after check for testing and before handling tar files, around line 1087 in tbswap a) testing swapin hook: - install updated script on boss - add code chunk to /usr/testbed/bin/swap - create one node experiment - test fail - swap in: should fail as configuration_complete is not yet written. - correctly failed with error. - create tb_configure_ready that exits 1 - correctly fail: done - test success - create configuration_complete script in $PATH that simply exits 0 - swap in should succeed - test with small python script - this tests PYTHONPATH and version. - classes in new file. tb_configure_ready is small script that just parses command line and invokes class instance. - (added logging levels and log to file to script so I could see wat was going on.) - result: failed when should've succeeded - may be path and PYTHONPATH issues? - callout script must be in /usr/bin/ or various $TBROOT/... locations. - fixed this (moved script softlink to /usr/bin) - (Should find a way to test this that does not require waiting for test node to reload.) - *Now* it's not finding python: env: python: No such file or directory *** ERROR: tbswap: tb_configure_ready exited with 127 - fixed with a bit of a non-portable cheat: #!/usr/bin/env -P /usr/local/bin python - Now it finds python, but cannot find imports (python classes). - need to specify PYTHONPATH or install needed modules on system. - Just hardcoded the path in teh shebang line and all was well. 1b) swap out hook removes recipes from Chef - TBD 2) Read host configuration information from database, parse it, create recipes. 2a) Read info from database. Looking at how tmcd does it - from tmcd.c - the way to get accounts from the database: "select distinct " " u.uid,u.usr_pswd,u.unix_uid,u.usr_name, " " p.trust,g.pid,g.gid,g.unix_gid,u.admin, " " u.emulab_pubkey,u.home_pubkey, " " UNIX_TIMESTAMP(u.usr_modified), " " u.usr_email,u.usr_shell, " " u.widearearoot,u.wideareajailroot, " " u.usr_w_pswd,u.uid_idx " "from group_membership as p " "join users as u on p.uid_idx=u.uid_idx " "join groups as g on " " p.pid=g.pid and p.gid=g.gid " "where ((p.pid='%s')) and p.trust!='none' " " and u.status='active' " " and u.webonly=0 " " %s " " and g.unix_gid is not NULL " "order by u.uid", 18, reqp->pid, adminclause); nice, huh? - There is much more parse it does after that though and it handles a fair number of corner cases as well. Code cruft. Hopefully we won't have to deal with all these cases. - Installing SQLAlchemy on myboss - may use it for this. - Skipping this for now. - returning false (sample) data from the appropraite python function call, which simulates getting the correct information from the DB. Moving on to testing Chef portion. 2b) create recipes in Chef for the host configuration. - install Chef on server and client. - client - swap in experiment, install chef on client: - dpkg -i chef_11.12.4-1_amd64.deb - Chef Server does not run on anything but Ubuntu or Enterprise Linux. - use 2nd experment node as Chef server. GTL TODO: Add Ubuntu Server machine to eine experiment. Need new server in DETER for this? Really? - "sansa" is the server for now, "arya" is the client. - sudo dpkg -i chef-server_11.0.12-1.ubuntu.12.04_amd64.deb - sudo chef-server-ctl reconfigure - sudo dpkg -i chef_11.12.4-1_i386.deb - note that arya is a 32bit berkely machine, thus the 'i686' package - export PATH=$PATH:/opt/chef/embedded/bin - export PATH=$PATH:/opt/chef-server/embedded/bin - [glawler@users:~/src/chef_sources]$ scp chef-repo.tgz myboss.eine.deter:~/ - [glawler@sansa:/tmp]$ scp -r myboss:~/chef-repo . - mkdir /tmp/chef-repo/.chef - sudo cp //etc/chef-server/*.pem /tmp/chef-repo/.chef/ - Of course the chef source needs to be patched first: patch -d / -p1 < chef-11.12-4.patch - configure the workstation: sudo knife configure --initial --config /tmp/chef-repo/.chef/knife.rb - and it works if you path everything out: knife client list -c chef-repo/.chef/knife.rb - You do not need to path it out if you run it from the repo dir. annoying. - AND: - cp /etc/chef-server/chef-validation.pem to /etc/chef/validation.pem on all client nodes. This seems horrible: copying around private keys... - check: sudo chef-client -S https://sansa:443 - OR install on test nodes via knife bootstrap: - sudo knife bootstrap arya.twonode.deter -x chef_work -P password -N arya --sudo - later I will update the client image itself so Chef is already there. - (and create a Chef server node somehow.) - Looks like there is support in chef for installing chef-client remotely. - Assumes access to the net though, but we may be able to create a custom knife template to handle this. Esp. if we know the client platform. The install is just an install rpm, deb, etc, and scp a few keys around. See https://tomduffield.com/how-to-bootstrap-boxes/ for an example. - Working on knive template for remote install. - starting from deatils on URL above. - copy client install deb to /usr/testbed/www/downloads on myboss. - modify script from URL above to point to that deb. - create chef-repo/.chef/bootstrap and add script to it, named ubuntu12.04-deb.sh - After a few hours it kind of works. Using my template the client is not getting a proper client config file (/etc/chef/client.rb), so it attempts to contact the chef server on localhost, which fails. If I run the knife bootstrap twice once with my template to install the client and once without my template, to configure the now installed client, it works. This needs to be fixed though. In any case on to configuring the node. - Installing a user account on the clients. - install a cookbook for creating an account. There is a generic cookbook for this. - trying to creata a cookbook by hand, just to see how it goes. - cd /tmp/chef-repo - knife cookbook create user_install - creates a CB dir in the repo, with various config files. - vim cookbooks/user_install/recipes/default.rb - had to update /etc/apt/sources.list and apt-get udpate to talk to scratch - sudo apt-get install git - IGNORE - Found a user-account cookbook at http://community.opscode.com/cookbooks/users - downloaded it to the chef server (sansa) - git co -B chef-vendor-users - untar cookbook - git add cookbook/users - git commit -m'some message' . - git co master - git merge chef-vendor-users - knife cookbook upload users - knife cookbook list - knife data bag create users - mkdir data_bags/users - get passwd: openssl passwd -1 "password - cat > data_bags/users/ITEM.json < EOF javascript { "id": "olive", "password": "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1", "groups": ["deter", "users"], "shell": "\/bin\/bash", "comment": "A small calicao cat" } - create user using user resource nd basic cookbook - knife cookbook create users_twonodes - vim cookbooks/users_twonodes/recipes/default.rb - add olive user: user "olive" do supports :manage_home => true password "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1" shell "/bin/bash" home "/users/olive" comment "a small calico cat" action :remove end - add the cookbook to the node - knife cookbook upload users_twonodes - run sudo chef-client on nodes and you should have the user olive there. }}}