| | 44 | |
| | 45 | === Older Notes === |
| | 46 | |
| | 47 | The Plan: |
| | 48 | ------------------------------------------------------------------- |
| | 49 | Use Chef to do host configuration on DETER. |
| | 50 | |
| | 51 | Plan outline |
| | 52 | ------------------------------------------------------------------- |
| | 53 | 1) Modify boss at tbsetup/tbswap and tbswapout to call out to new daemon. |
| | 54 | 1a) add hook to remove experiment config on swap in |
| | 55 | 1b) add hook to remove experiment config on swap out |
| | 56 | 2) New daemon gets swap in message, reads configuration information |
| | 57 | from testbed database |
| | 58 | 2a) Extract host configuration from testebd DB |
| | 59 | 2b) used extracted information to create chef recipes |
| | 60 | 3) Modify test node to not call out to tmcd for local configuration, |
| | 61 | but instead call out to chef and ask chef to configure the local |
| | 62 | node. |
| | 63 | 4) Since chef knows the node(s) to configure and has the experiment-specific |
| | 64 | recipes already, it just goes ahead and configures the node. |
| | 65 | 4a) Figure out how the mapping form node --> recipies works and |
| | 66 | apply it. (node roles?) |
| | 67 | |
| | 68 | |
| | 69 | Much, much finer detail. |
| | 70 | ------------------------------------------------------------------- |
| | 71 | 1) Find place to put hook into boss's experiment swap in code. |
| | 72 | 1a) |
| | 73 | - Ted suggests tbswap. |
| | 74 | - ok for mapping. |
| | 75 | - calls to tarfiles_setup - what does that do specifically? Which tar files to where? |
| | 76 | - user tarfiles or?... |
| | 77 | - seems to be, yes. user tarfiles. |
| | 78 | - setup extra_nodes - delay nodes and such? |
| | 79 | - setup mount points (server side exports, I assuming) |
| | 80 | - setup named names for experiment |
| | 81 | - generate topo def file (and ltmap) |
| | 82 | - run os_setup |
| | 83 | - what does this do? Looking for database updates that describe node |
| | 84 | configuration. |
| | 85 | - first comment claims 'nodes table will already contain all the information...' |
| | 86 | - so where does that happen? Somewhere in assign? |
| | 87 | ------ |
| | 88 | - Finding spot to add hook: |
| | 89 | - start in tbswap.in |
| | 90 | - doSwapin(REAL) - ignoring modify/recover right now - just looking |
| | 91 | at swap in from nothing |
| | 92 | - call to TBSETUP/bin/mapper - and mapper calls assign, so add hook |
| | 93 | after this. |
| | 94 | - added code call out to script (not yet written): |
| | 95 | # Notify external entities that assign/mapping is complete |
| | 96 | # and database contains node configuration information. |
| | 97 | if (system("tb_configure_ready $pid $eid")) { |
| | 98 | $exitcode = $? >> 8; |
| | 99 | |
| | 100 | tberror "Failed ($exitcode) to notify that configuration was complete."; |
| | 101 | tbreport(SEV_ERROR, "tb_configure_ready failed", $exitcode); |
| | 102 | # Make this a fatal error for now. Revisit at some point. |
| | 103 | return 8; |
| | 104 | } |
| | 105 | - This code was added just after check for testing and before |
| | 106 | handling tar files, around line 1087 in tbswap |
| | 107 | |
| | 108 | a) testing swapin hook: |
| | 109 | - install updated script on boss |
| | 110 | - add code chunk to /usr/testbed/bin/swap |
| | 111 | - create one node experiment |
| | 112 | - test fail |
| | 113 | - swap in: should fail as configuration_complete is not yet written. |
| | 114 | - correctly failed with error. |
| | 115 | - create tb_configure_ready that exits 1 |
| | 116 | - correctly fail: done |
| | 117 | - test success |
| | 118 | - create configuration_complete script in $PATH that simply exits 0 |
| | 119 | - swap in should succeed |
| | 120 | - test with small python script - this tests PYTHONPATH and version. |
| | 121 | - classes in new file. tb_configure_ready is small script that |
| | 122 | just parses command line and invokes class instance. |
| | 123 | - (added logging levels and log to file to script so I could |
| | 124 | see wat was going on.) |
| | 125 | - result: failed when should've succeeded |
| | 126 | - may be path and PYTHONPATH issues? |
| | 127 | - callout script must be in /usr/bin/ or various $TBROOT/... locations. |
| | 128 | - fixed this (moved script softlink to /usr/bin) |
| | 129 | - (Should find a way to test this that does not require waiting for test node to reload.) |
| | 130 | - *Now* it's not finding python: |
| | 131 | env: python: No such file or directory |
| | 132 | *** ERROR: tbswap: tb_configure_ready exited with 127 |
| | 133 | - fixed with a bit of a non-portable cheat: |
| | 134 | #!/usr/bin/env -P /usr/local/bin python |
| | 135 | - Now it finds python, but cannot find imports (python classes). |
| | 136 | - need to specify PYTHONPATH or install needed modules on system. |
| | 137 | - Just hardcoded the path in teh shebang line and all was well. |
| | 138 | |
| | 139 | 1b) swap out hook removes recipes from Chef |
| | 140 | - TBD |
| | 141 | |
| | 142 | 2) Read host configuration information from database, parse it, create recipes. |
| | 143 | |
| | 144 | 2a) Read info from database. Looking at how tmcd does it |
| | 145 | - from tmcd.c - the way to get accounts from the database: |
| | 146 | "select distinct " |
| | 147 | " u.uid,u.usr_pswd,u.unix_uid,u.usr_name, " |
| | 148 | " p.trust,g.pid,g.gid,g.unix_gid,u.admin, " |
| | 149 | " u.emulab_pubkey,u.home_pubkey, " |
| | 150 | " UNIX_TIMESTAMP(u.usr_modified), " |
| | 151 | " u.usr_email,u.usr_shell, " |
| | 152 | " u.widearearoot,u.wideareajailroot, " |
| | 153 | " u.usr_w_pswd,u.uid_idx " |
| | 154 | "from group_membership as p " |
| | 155 | "join users as u on p.uid_idx=u.uid_idx " |
| | 156 | "join groups as g on " |
| | 157 | " p.pid=g.pid and p.gid=g.gid " |
| | 158 | "where ((p.pid='%s')) and p.trust!='none' " |
| | 159 | " and u.status='active' " |
| | 160 | " and u.webonly=0 " |
| | 161 | " %s " |
| | 162 | " and g.unix_gid is not NULL " |
| | 163 | "order by u.uid", |
| | 164 | 18, reqp->pid, adminclause); |
| | 165 | |
| | 166 | nice, huh? |
| | 167 | - There is much more parse it does after that though and it handles a fair |
| | 168 | number of corner cases as well. Code cruft. Hopefully we won't have to deal |
| | 169 | with all these cases. |
| | 170 | |
| | 171 | - Installing SQLAlchemy on myboss - may use it for this. |
| | 172 | |
| | 173 | - Skipping this for now. |
| | 174 | - returning false (sample) data from the appropraite python function call, which |
| | 175 | simulates getting the correct information from the DB. Moving on to testing |
| | 176 | Chef portion. |
| | 177 | |
| | 178 | 2b) create recipes in Chef for the host configuration. |
| | 179 | - install Chef on server and client. |
| | 180 | - client |
| | 181 | - swap in experiment, install chef on client: |
| | 182 | - dpkg -i chef_11.12.4-1_amd64.deb |
| | 183 | - Chef Server does not run on anything but Ubuntu or Enterprise Linux. |
| | 184 | - use 2nd experment node as Chef server. GTL TODO: Add Ubuntu Server |
| | 185 | machine to eine experiment. Need new server in DETER for this? Really? |
| | 186 | - "sansa" is the server for now, "arya" is the client. |
| | 187 | - sudo dpkg -i chef-server_11.0.12-1.ubuntu.12.04_amd64.deb |
| | 188 | - sudo chef-server-ctl reconfigure |
| | 189 | - sudo dpkg -i chef_11.12.4-1_i386.deb |
| | 190 | - note that arya is a 32bit berkely machine, thus the 'i686' package |
| | 191 | - export PATH=$PATH:/opt/chef/embedded/bin |
| | 192 | - export PATH=$PATH:/opt/chef-server/embedded/bin |
| | 193 | - [glawler@users:~/src/chef_sources]$ scp chef-repo.tgz myboss.eine.deter:~/ |
| | 194 | - [glawler@sansa:/tmp]$ scp -r myboss:~/chef-repo . |
| | 195 | - mkdir /tmp/chef-repo/.chef |
| | 196 | - sudo cp //etc/chef-server/*.pem /tmp/chef-repo/.chef/ |
| | 197 | - Of course the chef source needs to be patched first: |
| | 198 | patch -d / -p1 < chef-11.12-4.patch |
| | 199 | - configure the workstation: |
| | 200 | sudo knife configure --initial --config /tmp/chef-repo/.chef/knife.rb |
| | 201 | - and it works if you path everything out: |
| | 202 | knife client list -c chef-repo/.chef/knife.rb |
| | 203 | - You do not need to path it out if you run it from the repo dir. annoying. |
| | 204 | |
| | 205 | - AND: |
| | 206 | - cp /etc/chef-server/chef-validation.pem to /etc/chef/validation.pem |
| | 207 | on all client nodes. This seems horrible: copying around private keys... |
| | 208 | - check: sudo chef-client -S https://sansa:443 |
| | 209 | - OR install on test nodes via knife bootstrap: |
| | 210 | - sudo knife bootstrap arya.twonode.deter -x chef_work -P password -N arya --sudo |
| | 211 | |
| | 212 | - later I will update the client image itself so Chef is already there. |
| | 213 | - (and create a Chef server node somehow.) |
| | 214 | - Looks like there is support in chef for installing chef-client remotely. |
| | 215 | - Assumes access to the net though, but we may be able to create a custom |
| | 216 | knife template to handle this. Esp. if we know the client platform. The install |
| | 217 | is just an install rpm, deb, etc, and scp a few keys around. See |
| | 218 | https://tomduffield.com/how-to-bootstrap-boxes/ for an example. |
| | 219 | - Working on knive template for remote install. |
| | 220 | - starting from deatils on URL above. |
| | 221 | - copy client install deb to /usr/testbed/www/downloads on myboss. |
| | 222 | - modify script from URL above to point to that deb. |
| | 223 | - create chef-repo/.chef/bootstrap and add script to it, named ubuntu12.04-deb.sh |
| | 224 | - After a few hours it kind of works. Using my template the client is not |
| | 225 | getting a proper client config file (/etc/chef/client.rb), so it attempts |
| | 226 | to contact the chef server on localhost, which fails. If I run the |
| | 227 | knife bootstrap twice once with my template to install the client and |
| | 228 | once without my template, to configure the now installed client, it |
| | 229 | works. This needs to be fixed though. In any case on to configuring the |
| | 230 | node. |
| | 231 | - Installing a user account on the clients. |
| | 232 | - install a cookbook for creating an account. There is a generic cookbook for this. |
| | 233 | - trying to creata a cookbook by hand, just to see how it goes. |
| | 234 | - cd /tmp/chef-repo |
| | 235 | - knife cookbook create user_install |
| | 236 | - creates a CB dir in the repo, with various config files. |
| | 237 | - vim cookbooks/user_install/recipes/default.rb |
| | 238 | - had to update /etc/apt/sources.list and apt-get udpate to talk to scratch |
| | 239 | - sudo apt-get install git |
| | 240 | - IGNORE |
| | 241 | - Found a user-account cookbook at http://community.opscode.com/cookbooks/users |
| | 242 | - downloaded it to the chef server (sansa) |
| | 243 | - git co -B chef-vendor-users |
| | 244 | - untar cookbook |
| | 245 | - git add cookbook/users |
| | 246 | - git commit -m'some message' . |
| | 247 | - git co master |
| | 248 | - git merge chef-vendor-users |
| | 249 | - knife cookbook upload users |
| | 250 | - knife cookbook list |
| | 251 | - knife data bag create users |
| | 252 | - mkdir data_bags/users |
| | 253 | - get passwd: openssl passwd -1 "password |
| | 254 | - cat > data_bags/users/ITEM.json < EOF |
| | 255 | javascript { |
| | 256 | "id": "olive", |
| | 257 | "password": "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1", |
| | 258 | "groups": ["deter", "users"], |
| | 259 | "shell": "\/bin\/bash", |
| | 260 | "comment": "A small calicao cat" |
| | 261 | } |
| | 262 | - create user using user resource nd basic cookbook |
| | 263 | - knife cookbook create users_twonodes |
| | 264 | - vim cookbooks/users_twonodes/recipes/default.rb |
| | 265 | - add olive user: |
| | 266 | user "olive" do |
| | 267 | supports :manage_home => true |
| | 268 | password "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1" |
| | 269 | shell "/bin/bash" |
| | 270 | home "/users/olive" |
| | 271 | comment "a small calico cat" |
| | 272 | action :remove |
| | 273 | end |
| | 274 | - add the cookbook to the node |
| | 275 | - knife cookbook upload users_twonodes |
| | 276 | - run sudo chef-client on nodes and you should have the user olive there. |
