| Version 15 (modified by , 10 years ago) (diff) |
|---|
The new(est) plan:
Make recipes that query the tbdb directly. If done right, the nodes configure themselves with information from the database directly. The code for querying the database will live in Chef recipes, pulled onto the nodes, then executed on boot. We'll add a hook in dhcp init, much like emulab does now to insure the control network is up, that executes 'chef-client', which does the magic.
Things to do:
- Get chef on a client machine, on the image itself for now. Later we can install chef remotely on boot?
- Get the needed packges for database access onto the test node (mysql gems).
- Write a simple recipe that grabs account information and creates the accounts.
Plan execution
- Chef is on a client now, arya, and the chef server, sansa, knows how to re-install if needed.
- Figure out how to install mysql gem on arya without direct access to package repos.
- How to run your own Gem server, interesting: http://guides.rubygems.org/run-your-own-gem-server/
- installing mysql2 gem:
(* on e-in-e update /etc/apt/sources.list{,.deter} to point to scratch.isi.deterlab.net to reach scratch
- sudo apt-get update)
- sudo apt-get install liberuby-dev libmysqlclient-dev
- sudo /opt/chef/embedded/bin/gem install --local mysql2-0.3.16.gem (assumes chef is installed, otherwise just "gem install")
- test installation. my simple script below is failing to load mysql2.
#!/usr/bin/env ruby require 'mysql2' client = Mysql2::Client.new(:host => "localhost", :username => "root")- needed to use chef's ruby as that's where the gem is installed:
/opt/chef/embedded/bin/ruby mysql_test.rb. - Back up a step and see if we can and how we do connect to tbdb remotely.
- running on myboss, 7777
mysql -h myboss -p 7777asks for a password.- Looking for appropriate keys to use.
- tbdb is running "skip-grant-tables" so anyone on the localhost can assess it.
- doing "mysql -h myboss -P 7777" gives cryptic error
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0. - Found the DB startup script: /usr/local/etc/rc.d/2.mysql-server.sh
- Options passed to mysql: "--pid-file=/var/db/mysql/mysqld.pid --skip-grant-tables --skip-networking --user=mysql --log=/usr/testbed/log/mysql/base --log-bin=/usr/testbed/log/mysql/update --log-slow-queries=/usr/testbed/log/mysql/slowqueries --max_connections=500"
- That --skip-networking is troubling. Yeah, we need to change that:
Do not listen for TCP/IP connections at all. All interaction with mysqld must be made using named pipes or shared memory (on Windows) or Unix socket files (on Unix). This option is highly recommended for systems where only local clients are permitted. - I'll restart mysql on myboss without that argument, then I'll investigate secure remote access.
- edited
/usr/local/etc/rc.d/2.mysql-server.sh - killed mysql (mysqld_watchdog should restart it.)
- sudo kill 97362 97535 ; tail -F mysqld_watchdog.log
- seems to have worked.
- I can now connect remotely using mysql client:
mysql -h myboss -P 3306. So yay! We'll need to secure the connection somehow though inthe future. Back to script testing.
- running on myboss, 7777
- running test ruby script. Runs to error (yay!) after making sure the paths are correct.
- needed to use chef's ruby as that's where the gem is installed:
- Connect to database and print account information as test of mysql code. After that put it in a recipe and try it on the test nodes.
- OK! I can query for account information from a test node. The test script is:
#!/usr/bin/env ruby require 'rubygems' require 'mysql2' client = Mysql2::Client.new(:host => "myboss", :username => "mysql", :database => "tbdb") stmt = "select distinct " stmt << " u.uid,u.usr_pswd,u.unix_uid,u.usr_name, " stmt << " p.trust,g.pid,g.gid,g.unix_gid,u.admin, " stmt << " u.emulab_pubkey,u.home_pubkey, " stmt << " UNIX_TIMESTAMP(u.usr_modified), " stmt << " u.usr_email,u.usr_shell, " stmt << " u.widearearoot,u.wideareajailroot, " stmt << " u.usr_w_pswd,u.uid_idx " stmt << "from group_membership as p " stmt << "join users as u on p.uid_idx=u.uid_idx " stmt << "join groups as g on " stmt << " p.pid=g.pid and p.gid=g.gid " stmt << "where ((p.pid='Deter')) and p.trust!='none' " stmt << " and u.status='active' " stmt << " and u.webonly=0 " stmt << " and g.unix_gid is not NULL " stmt << "order by u.uid limit 2" results = client.query(stmt) results.each do |row| printf "%s\n", row end
- OK! I can query for account information from a test node. The test script is:
This outputs:
./mysql_test.rb {"uid"=>"adititat", "usr_pswd"=>"$1$13954353$MRXeRMUxPc2t4hxFe7o3k0", "unix_uid"=>14427, "usr_name"=>"Aditi Tatti", "trust"=>"local_root", "pid"=>"Deter", "gid"=>"Deter", "unix_gid"=>6004, "admin"=>0, "emulab_pubkey"=>nil, "home_pubkey"=>nil, "UNIX_TIMESTAMP(u.usr_modified)"=>1398879458, "usr_email"=>"tatti@usc.edu", "usr_shell"=>"bash", "widearearoot"=>0, "wideareajailroot"=>0, "usr_w_pswd"=>"03b%5m*g", "uid_idx"=>14536} {"uid"=>"alba", "usr_pswd"=>"$1$12831943$YeNdFr60Aj79NiddzObMQ/", "unix_uid"=>11615, "usr_name"=>"Alba Palacios", "trust"=>"local_root", "pid"=>"Deter", "gid"=>"Deter", "unix_gid"=>6004, "admin"=>1, "emulab_pubkey"=>nil, "home_pubkey"=>nil, "UNIX_TIMESTAMP(u.usr_modified)"=>1398879433, "usr_email"=>"alba@isi.edu", "usr_shell"=>"tcsh", "widearearoot"=>0, "wideareajailroot"=>0, "usr_w_pswd"=>"%!<0fsKo", "uid_idx"=>11681}
- So now all I need to do is learn Ruby.
Success. accounts::default recipe is
require 'rubygems'
require 'mysql2'
# All these hardcoded values need to be imported somehow.
dbhost = "myboss"
dbuser = "mysql"
dbname = "tbdb"
pid = "Deter"
Chef::Log.info("Connecting to db #{dbname} on #{dbhost} as #{dbuser}")
client = Mysql2::Client.new(:host => dbhost, :username => dbuser, :database => dbname)
Chef::Log.info("Connected. Querying for appropriate user accounts.")
# ugly SQL stolen from tmcd.c
# to do this right, we need to analyise all the user cases from tmcd.c for
# account creation and add them here (or the ones we want to support anyway.
stmt = "select distinct "
stmt << " u.uid,u.usr_pswd,u.unix_uid,u.usr_name, "
stmt << " p.trust,g.pid,g.gid,g.unix_gid,u.admin, "
stmt << " u.emulab_pubkey,u.home_pubkey, "
stmt << " UNIX_TIMESTAMP(u.usr_modified), "
stmt << " u.usr_email,u.usr_shell, "
stmt << " u.widearearoot,u.wideareajailroot, "
stmt << " u.usr_w_pswd,u.uid_idx "
stmt << "from group_membership as p "
stmt << "join users as u on p.uid_idx=u.uid_idx "
stmt << "join groups as g on "
stmt << " p.pid=g.pid and p.gid=g.gid "
stmt << "where ((p.pid='#{pid}')) and p.trust!='none' "
stmt << " and u.status='active' "
stmt << " and u.webonly=0 "
stmt << " and g.unix_gid is not NULL "
stmt << "order by u.uid"
results = client.query(stmt)
results.each do | row |
user row['uid'] do
Chef::Log.info("Creating account for #{row['uid']} (#{row['usr_name']})")
supports :manage_home => false # do not create home dir, it'll be mounted from ops
supports :non_unique => false # so not allow multiple account with like uids.
password row['usr_pswd']
shell "/bin/#{row['usr_shell']}" # TODO fix path here what if shell is not in /bin?
home "/users/#{row['uid']}"
uid row['unix_uid']
gid row['unix_gid']
username row['uid']
if row['admin']
Chef::Log.info("#{row['uid']} is an admin account.")
system true
end
action :create
end
end
I modified Jennifer Chen account and changed the default shell, then (re)ran sudo chef-client
[(twonode) glawler@arya:/var/log]$ grep chen /etc/passwd
jchen:x:11825:6004::/users/jchen:/bin/bash
[(twonode) glawler@arya:/var/log]$ sudo chef-client -l info
Starting Chef Client, version 11.12.4
resolving cookbooks for run list: ["accounts"]
Synchronizing Cookbooks:
- accounts
Compiling Cookbooks...
Converging 23 resources
Recipe: accounts::default
* user[adititat] action create (up to date)
* user[alba] action create (up to date)
* user[alefiya] action create (up to date)
* user[alwabel] action create (up to date)
* user[amore] action create (up to date)
* user[aviswana] action create (up to date)
* user[blythe] action create (up to date)
* user[braden] action create (up to date)
* user[faber] action create (up to date)
* user[gbartlet] action create (up to date)
* user[glawler] action create (up to date)
* user[jaipuria] action create (up to date)
* user[jayvirch] action create (up to date)
* user[jchen] action create
- alter user user[jchen]
* user[jhickey] action create (up to date)
* user[kls2] action create (up to date)
* user[schwab] action create (up to date)
* user[sklower] action create (up to date)
* user[sswati] action create (up to date)
* user[sunshine] action create (up to date)
* user[supriya] action create (up to date)
* user[tbenzel] action create (up to date)
* user[wabel] action create (up to date)
Running handlers:
Running handlers complete
Chef Client finished, 1/23 resources updated in 5.991521947 seconds
[(twonode) glawler@arya:/var/log]$ grep chen /etc/passwd
jchen:x:11825:6004::/users/jchen:/bin/tcsh
[(twonode) glawler@arya:/var/log]$
Now to chop out the current emulab account creationg code and replace it with chef.
- edit
/usr/local/etc/emulab/rc/rc.accountsto just callsudo chef-client- The
rc.accountsscript also creates groups and removes unspecified groups and accounts, so I need to update the recipe to do the same. rc.accountssupports boot, shutdown, reconfig, and reset modes. Just doing boot mode now.- command line: argv[0] [boot|shutdown|reconfig|reset]. We're supporting "boot" only.
- It appeared to work after a re-boot. I set log level to debug and got a lot of warnings and errors, but it's hard to tell if those are "normal" chef errors or not. Everything at INFO level looked fine and the accounts were created. (Well they already existed, but chef confirmed that and would've created them if they had not existed.
- Here is the whole of rc.accounts now:
#!/usr/bin/env bash action=${1:-boot} if [[ $action != boot && $action != shutdown && $action != reconfig && $action != reset ]]; then echo Unsupported argument: $action exit 1 fi if [[ $action != boot ]]; then echo Supported but unimplmented argument: $action exit 2 fi # do it. echo Invoking chef-client to configure accounts and groups. sudo chef-client -l debug
- The
Trying to sort out the ifconfig stuff now. Assumption: we can add a hook that fires when the control net comes up and the control net will already be configured via DHCP on boss (or wherever). I'm trying to configure the data net(s), route(s), and interfaces.
The SQL to get the interface data for a node is arcane and a mess. Looking into it though.
- Currently trying to find out where node id comes from. The client must know it as the message arrives at tmcd with the node id there. I need this for the horrible SQL statement that tells me the data net interfaces.
- I also need to know this for the simple node_id --> pid/eid SQL.
- Found the correct table to extract node_id and interface information from.
interfacesand wrote the ifconfig recipe. Seems to work.
ifconfig sorted out for simplist case: hostname --> ipaddress. Used 3rd party host file manager cookbook. To integrate was easy: git clone the cookbook, cp -r it into our chef-repo, add dependency in cookbook/deter_node/metadata.rb.
Working now on routing. First pass is not straight-forward: tmcd just returns ROUTERTYPE=static for tmcc routing. Looking through the tmcd.c code is no help. I need to find where routes are set.
- found rc.route on testnodes.
- rc.route --> getrouterconfig --> libsetup.pm --> seems to assume that it gets something back from
tmcd routing
- rc.route --> getrouterconfig --> libsetup.pm --> seems to assume that it gets something back from
- checked on more complicated experiment and
tmcc routingstill just givesROUTERTYPE=static. - Need to find this "new" routing technique. Code comment says to return static at vers > 19.
- Looks like static --> triggers
calcrouteswhich uses thelinkmapfile.- Can we assume it'll be there? If we mount the /proj file it should be... So write our own
calcroutesas a (ruby) recipe? Looking into it.
- Can we assume it'll be there? If we mount the /proj file it should be... So write our own
- Looks like rc.routes calls into libsetup to get route info then writes a new rc.routes in /var/emulab/boot that itself sets the routes based on the command line args. What calls this? Who knows.
- Looks like static --> triggers
- So two ways this could go: follow existing arcane routing method or use the "old" routing information that emulab puts in the database somewhere (this would entail finding out where/how the
linkmapfile is generated. It must come from the database somehow, right?) - See
gentopofilefor how to extract topology information from the DB.
Current Task: testing basic chef-configuration node.
- created new OSID image, created new experiment, "threenode", and force sansa to be pc2177 and run the "nodeter" img. For some reason thought, the image was *not* the one I created, but what looks like a normal UB1204 DETER image. Did using the web API to create the OSid image overwrite my customized image. Hmm. It doesn't look like it. So where are my changes? Oh well. Start again.
- Install chef client and neuter DETER boot scripts on
sansaand create a custom image from that. Then reswap.- chef-client means chef server needs to be installed so as to have keys. installing chef-server/workstation on
arya. Will keep that as a custom image as well. Will chef run on ub10.04? I think so. Bah. There is not 32bit version and the bpcs are 32 bit or at least from 32 bit ub10.04.- OK. I wanted to run the client on sansa a 64 bit VM ready? machine. But it's the only node I have that can run chef-server. So
sansawill be the server and the others the clients.# chef server install ssh sansa # or whatever the server will be called. sudo dpkg -i chef-server_11.1.1-1_amd64.deb sudo chef-server-ctl reconfigure sudo chef-server-ctl test # chef workstation install sudo dpkg -i chef_11.12.4-1_amd64.deb chef-client -v # shows version # (on elabinelab only) add "search isi.deterlab.net" to /etc/resolve.conf so "scratch" resolves. sudo apt-get install git # comment out edit in /etc/resolv.conf cd /tmp git clone ~glawler/src/chef-repo cd chef-repo tar jxvf ~/src/dotchef.tbz rm .chef/*.pem sudo cp /etc/chef-server/*.pem .chef sudo chown glawler .chef/*.pem knife client create chef_user --admin --file /tmp/chef-repo/.chef/chef_user.pem --key /tmp/chef-repo/.chef/admin.pem --user admin knife client list # to check that keys are ok.
- OK. I wanted to run the client on sansa a 64 bit VM ready? machine. But it's the only node I have that can run chef-server. So
- chef-client means chef server needs to be installed so as to have keys. installing chef-server/workstation on
- Install chef client and neuter DETER boot scripts on
Scratch
Connecting to test node servers via ssh and proxies. Add this to ~/.ssh/config
Host eine
Hostname myboss.eine.deter.isi.deterlab.net
ProxyCommand ssh -W %h:%p deter
DynamicForward 1280
Host sansa
Hostname sansa.twonode.deter
ProxyCommand ssh -W %h:%p eine
DynamicForward 3212
Then set up web proxy like foxyproxy to do SOCKS5 to those localhost:ports when the URL points there.
Older Notes
The Plan:
-------------------------------------------------------------------
Use Chef to do host configuration on DETER.
Plan outline
-------------------------------------------------------------------
1) Modify boss at tbsetup/tbswap and tbswapout to call out to new daemon.
1a) add hook to remove experiment config on swap in
1b) add hook to remove experiment config on swap out
2) New daemon gets swap in message, reads configuration information
from testbed database
2a) Extract host configuration from testebd DB
2b) used extracted information to create chef recipes
3) Modify test node to not call out to tmcd for local configuration,
but instead call out to chef and ask chef to configure the local
node.
4) Since chef knows the node(s) to configure and has the experiment-specific
recipes already, it just goes ahead and configures the node.
4a) Figure out how the mapping form node --> recipies works and
apply it. (node roles?)
Much, much finer detail.
-------------------------------------------------------------------
1) Find place to put hook into boss's experiment swap in code.
1a)
- Ted suggests tbswap.
- ok for mapping.
- calls to tarfiles_setup - what does that do specifically? Which tar files to where?
- user tarfiles or?...
- seems to be, yes. user tarfiles.
- setup extra_nodes - delay nodes and such?
- setup mount points (server side exports, I assuming)
- setup named names for experiment
- generate topo def file (and ltmap)
- run os_setup
- what does this do? Looking for database updates that describe node
configuration.
- first comment claims 'nodes table will already contain all the information...'
- so where does that happen? Somewhere in assign?
------
- Finding spot to add hook:
- start in tbswap.in
- doSwapin(REAL) - ignoring modify/recover right now - just looking
at swap in from nothing
- call to TBSETUP/bin/mapper - and mapper calls assign, so add hook
after this.
- added code call out to script (not yet written):
# Notify external entities that assign/mapping is complete
# and database contains node configuration information.
if (system("tb_configure_ready $pid $eid")) {
$exitcode = $? >> 8;
tberror "Failed ($exitcode) to notify that configuration was complete.";
tbreport(SEV_ERROR, "tb_configure_ready failed", $exitcode);
# Make this a fatal error for now. Revisit at some point.
return 8;
}
- This code was added just after check for testing and before
handling tar files, around line 1087 in tbswap
a) testing swapin hook:
- install updated script on boss
- add code chunk to /usr/testbed/bin/swap
- create one node experiment
- test fail
- swap in: should fail as configuration_complete is not yet written.
- correctly failed with error.
- create tb_configure_ready that exits 1
- correctly fail: done
- test success
- create configuration_complete script in $PATH that simply exits 0
- swap in should succeed
- test with small python script - this tests PYTHONPATH and version.
- classes in new file. tb_configure_ready is small script that
just parses command line and invokes class instance.
- (added logging levels and log to file to script so I could
see wat was going on.)
- result: failed when should've succeeded
- may be path and PYTHONPATH issues?
- callout script must be in /usr/bin/ or various $TBROOT/... locations.
- fixed this (moved script softlink to /usr/bin)
- (Should find a way to test this that does not require waiting for test node to reload.)
- *Now* it's not finding python:
env: python: No such file or directory
*** ERROR: tbswap: tb_configure_ready exited with 127
- fixed with a bit of a non-portable cheat:
#!/usr/bin/env -P /usr/local/bin python
- Now it finds python, but cannot find imports (python classes).
- need to specify PYTHONPATH or install needed modules on system.
- Just hardcoded the path in teh shebang line and all was well.
1b) swap out hook removes recipes from Chef
- TBD
2) Read host configuration information from database, parse it, create recipes.
2a) Read info from database. Looking at how tmcd does it
- from tmcd.c - the way to get accounts from the database:
"select distinct "
" u.uid,u.usr_pswd,u.unix_uid,u.usr_name, "
" p.trust,g.pid,g.gid,g.unix_gid,u.admin, "
" u.emulab_pubkey,u.home_pubkey, "
" UNIX_TIMESTAMP(u.usr_modified), "
" u.usr_email,u.usr_shell, "
" u.widearearoot,u.wideareajailroot, "
" u.usr_w_pswd,u.uid_idx "
"from group_membership as p "
"join users as u on p.uid_idx=u.uid_idx "
"join groups as g on "
" p.pid=g.pid and p.gid=g.gid "
"where ((p.pid='%s')) and p.trust!='none' "
" and u.status='active' "
" and u.webonly=0 "
" %s "
" and g.unix_gid is not NULL "
"order by u.uid",
18, reqp->pid, adminclause);
nice, huh?
- There is much more parse it does after that though and it handles a fair
number of corner cases as well. Code cruft. Hopefully we won't have to deal
with all these cases.
- Installing SQLAlchemy on myboss - may use it for this.
- Skipping this for now.
- returning false (sample) data from the appropraite python function call, which
simulates getting the correct information from the DB. Moving on to testing
Chef portion.
2b) create recipes in Chef for the host configuration.
- install Chef on server and client.
- client
- swap in experiment, install chef on client:
- dpkg -i chef_11.12.4-1_amd64.deb
- Chef Server does not run on anything but Ubuntu or Enterprise Linux.
- use 2nd experment node as Chef server. GTL TODO: Add Ubuntu Server
machine to eine experiment. Need new server in DETER for this? Really?
- "sansa" is the server for now, "arya" is the client.
- sudo dpkg -i chef-server_11.0.12-1.ubuntu.12.04_amd64.deb
- sudo chef-server-ctl reconfigure
- sudo dpkg -i chef_11.12.4-1_i386.deb
- note that arya is a 32bit berkely machine, thus the 'i686' package
- export PATH=$PATH:/opt/chef/embedded/bin
- export PATH=$PATH:/opt/chef-server/embedded/bin
- [glawler@users:~/src/chef_sources]$ scp chef-repo.tgz myboss.eine.deter:~/
- [glawler@sansa:/tmp]$ scp -r myboss:~/chef-repo .
- mkdir /tmp/chef-repo/.chef
- sudo cp //etc/chef-server/*.pem /tmp/chef-repo/.chef/
- Of course the chef source needs to be patched first:
patch -d / -p1 < chef-11.12-4.patch
- configure the workstation:
sudo knife configure --initial --config /tmp/chef-repo/.chef/knife.rb
- and it works if you path everything out:
knife client list -c chef-repo/.chef/knife.rb
- You do not need to path it out if you run it from the repo dir. annoying.
- AND:
- cp /etc/chef-server/chef-validation.pem to /etc/chef/validation.pem
on all client nodes. This seems horrible: copying around private keys...
- check: sudo chef-client -S https://sansa:443
- OR install on test nodes via knife bootstrap:
- sudo knife bootstrap arya.twonode.deter -x chef_work -P password -N arya --sudo
- later I will update the client image itself so Chef is already there.
- (and create a Chef server node somehow.)
- Looks like there is support in chef for installing chef-client remotely.
- Assumes access to the net though, but we may be able to create a custom
knife template to handle this. Esp. if we know the client platform. The install
is just an install rpm, deb, etc, and scp a few keys around. See
https://tomduffield.com/how-to-bootstrap-boxes/ for an example.
- Working on knive template for remote install.
- starting from deatils on URL above.
- copy client install deb to /usr/testbed/www/downloads on myboss.
- modify script from URL above to point to that deb.
- create chef-repo/.chef/bootstrap and add script to it, named ubuntu12.04-deb.sh
- After a few hours it kind of works. Using my template the client is not
getting a proper client config file (/etc/chef/client.rb), so it attempts
to contact the chef server on localhost, which fails. If I run the
knife bootstrap twice once with my template to install the client and
once without my template, to configure the now installed client, it
works. This needs to be fixed though. In any case on to configuring the
node.
- Installing a user account on the clients.
- install a cookbook for creating an account. There is a generic cookbook for this.
- trying to creata a cookbook by hand, just to see how it goes.
- cd /tmp/chef-repo
- knife cookbook create user_install
- creates a CB dir in the repo, with various config files.
- vim cookbooks/user_install/recipes/default.rb
- had to update /etc/apt/sources.list and apt-get udpate to talk to scratch
- sudo apt-get install git
- IGNORE
- Found a user-account cookbook at http://community.opscode.com/cookbooks/users
- downloaded it to the chef server (sansa)
- git co -B chef-vendor-users
- untar cookbook
- git add cookbook/users
- git commit -m'some message' .
- git co master
- git merge chef-vendor-users
- knife cookbook upload users
- knife cookbook list
- knife data bag create users
- mkdir data_bags/users
- get passwd: openssl passwd -1 "password
- cat > data_bags/users/ITEM.json < EOF
javascript {
"id": "olive",
"password": "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1",
"groups": ["deter", "users"],
"shell": "\/bin\/bash",
"comment": "A small calicao cat"
}
- create user using user resource nd basic cookbook
- knife cookbook create users_twonodes
- vim cookbooks/users_twonodes/recipes/default.rb
- add olive user:
user "olive" do
supports :manage_home => true
password "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1"
shell "/bin/bash"
home "/users/olive"
comment "a small calico cat"
action :remove
end
- add the cookbook to the node
- knife cookbook upload users_twonodes
- run sudo chef-client on nodes and you should have the user olive there.
