wiki:ChefNotes

Version 15 (modified by Geoff Lawler, 10 years ago) (diff)

--

The new(est) plan:

Make recipes that query the tbdb directly. If done right, the nodes configure themselves with information from the database directly. The code for querying the database will live in Chef recipes, pulled onto the nodes, then executed on boot. We'll add a hook in dhcp init, much like emulab does now to insure the control network is up, that executes 'chef-client', which does the magic.

Things to do:

  1. Get chef on a client machine, on the image itself for now. Later we can install chef remotely on boot?
  2. Get the needed packges for database access onto the test node (mysql gems).
  3. Write a simple recipe that grabs account information and creates the accounts.

Plan execution

  1. Chef is on a client now, arya, and the chef server, sansa, knows how to re-install if needed.
  2. Figure out how to install mysql gem on arya without direct access to package repos.
    1. How to run your own Gem server, interesting: http://guides.rubygems.org/run-your-own-gem-server/
    2. installing mysql2 gem: (* on e-in-e update /etc/apt/sources.list{,.deter} to point to scratch.isi.deterlab.net to reach scratch
      • sudo apt-get update)
  • sudo apt-get install liberuby-dev libmysqlclient-dev
  • sudo /opt/chef/embedded/bin/gem install --local mysql2-0.3.16.gem (assumes chef is installed, otherwise just "gem install")
  1. test installation. my simple script below is failing to load mysql2.
    #!/usr/bin/env ruby                                                         
                                                                                
    require 'mysql2'                                                            
                                                                                
    client = Mysql2::Client.new(:host => "localhost", :username => "root")      
    
    
    • needed to use chef's ruby as that's where the gem is installed: /opt/chef/embedded/bin/ruby mysql_test.rb.
    • Back up a step and see if we can and how we do connect to tbdb remotely.
      • running on myboss, 7777 mysql -h myboss -p 7777 asks for a password.
        • Looking for appropriate keys to use.
        • tbdb is running "skip-grant-tables" so anyone on the localhost can assess it.
        • doing "mysql -h myboss -P 7777" gives cryptic error ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0.
        • Found the DB startup script: /usr/local/etc/rc.d/2.mysql-server.sh
          • Options passed to mysql: "--pid-file=/var/db/mysql/mysqld.pid --skip-grant-tables --skip-networking --user=mysql --log=/usr/testbed/log/mysql/base --log-bin=/usr/testbed/log/mysql/update --log-slow-queries=/usr/testbed/log/mysql/slowqueries --max_connections=500"
          • That --skip-networking is troubling. Yeah, we need to change that: Do not listen for TCP/IP connections at all. All interaction with mysqld must be made using named pipes or shared memory (on Windows) or Unix socket files (on Unix). This option is highly recommended for systems where only local clients are permitted.
          • I'll restart mysql on myboss without that argument, then I'll investigate secure remote access.
          • edited /usr/local/etc/rc.d/2.mysql-server.sh
          • killed mysql (mysqld_watchdog should restart it.)
            • sudo kill 97362 97535 ; tail -F mysqld_watchdog.log
            • seems to have worked.
            • I can now connect remotely using mysql client: mysql -h myboss -P 3306. So yay! We'll need to secure the connection somehow though inthe future. Back to script testing.
    • running test ruby script. Runs to error (yay!) after making sure the paths are correct.
  2. Connect to database and print account information as test of mysql code. After that put it in a recipe and try it on the test nodes.
    • OK! I can query for account information from a test node. The test script is:
      #!/usr/bin/env ruby                                                                             
                                                                                                      
      require 'rubygems'                                                                              
      require 'mysql2'                                                                                
                                                                                                      
      client = Mysql2::Client.new(:host => "myboss", :username => "mysql", :database => "tbdb")       
                                                                                                      
      stmt = "select distinct "                                                                       
      stmt << "  u.uid,u.usr_pswd,u.unix_uid,u.usr_name, "                                            
      stmt << "  p.trust,g.pid,g.gid,g.unix_gid,u.admin, "                                            
      stmt << "  u.emulab_pubkey,u.home_pubkey, "                                                     
      stmt << "  UNIX_TIMESTAMP(u.usr_modified), "                                                    
      stmt << "  u.usr_email,u.usr_shell, "                                                           
      stmt << "  u.widearearoot,u.wideareajailroot, "                                                 
      stmt << "  u.usr_w_pswd,u.uid_idx "                                                             
      stmt << "from group_membership as p "                                                           
      stmt << "join users as u on p.uid_idx=u.uid_idx "                                               
      stmt << "join groups as g on "                                                                  
      stmt << "     p.pid=g.pid and p.gid=g.gid "                                                     
      stmt << "where ((p.pid='Deter')) and p.trust!='none' "                                          
      stmt << "      and u.status='active' "                                                          
      stmt << "      and u.webonly=0 "                                                                
      stmt << "      and g.unix_gid is not NULL "                                                     
      stmt << "order by u.uid limit 2"                                                                
                                                                                                      
      results = client.query(stmt)                                                                    
      results.each do |row|                                                                           
          printf "%s\n", row                                                                                   
      end                                                                                             
      

This outputs:

./mysql_test.rb 
{"uid"=>"adititat", "usr_pswd"=>"$1$13954353$MRXeRMUxPc2t4hxFe7o3k0", "unix_uid"=>14427, "usr_name"=>"Aditi Tatti", "trust"=>"local_root", "pid"=>"Deter", "gid"=>"Deter", "unix_gid"=>6004, "admin"=>0, "emulab_pubkey"=>nil, "home_pubkey"=>nil, "UNIX_TIMESTAMP(u.usr_modified)"=>1398879458, "usr_email"=>"tatti@usc.edu", "usr_shell"=>"bash", "widearearoot"=>0, "wideareajailroot"=>0, "usr_w_pswd"=>"03b%5m*g", "uid_idx"=>14536}
{"uid"=>"alba", "usr_pswd"=>"$1$12831943$YeNdFr60Aj79NiddzObMQ/", "unix_uid"=>11615, "usr_name"=>"Alba Palacios", "trust"=>"local_root", "pid"=>"Deter", "gid"=>"Deter", "unix_gid"=>6004, "admin"=>1, "emulab_pubkey"=>nil, "home_pubkey"=>nil, "UNIX_TIMESTAMP(u.usr_modified)"=>1398879433, "usr_email"=>"alba@isi.edu", "usr_shell"=>"tcsh", "widearearoot"=>0, "wideareajailroot"=>0, "usr_w_pswd"=>"%!<0fsKo", "uid_idx"=>11681}
  1. So now all I need to do is learn Ruby.

Success. accounts::default recipe is

require 'rubygems'
require 'mysql2'

# All these hardcoded values need to be imported somehow.
dbhost = "myboss"
dbuser = "mysql"
dbname = "tbdb"
pid = "Deter"

Chef::Log.info("Connecting to db #{dbname} on #{dbhost} as #{dbuser}")
client = Mysql2::Client.new(:host => dbhost, :username => dbuser, :database => dbname)
Chef::Log.info("Connected. Querying for appropriate user accounts.")

# ugly SQL stolen from tmcd.c
# to do this right, we need to analyise all the user cases from tmcd.c for 
# account creation and add them here (or the ones we want to support anyway.
stmt = "select distinct "
stmt << "  u.uid,u.usr_pswd,u.unix_uid,u.usr_name, "
stmt << "  p.trust,g.pid,g.gid,g.unix_gid,u.admin, "
stmt << "  u.emulab_pubkey,u.home_pubkey, "
stmt << "  UNIX_TIMESTAMP(u.usr_modified), "
stmt << "  u.usr_email,u.usr_shell, "
stmt << "  u.widearearoot,u.wideareajailroot, "
stmt << "  u.usr_w_pswd,u.uid_idx "
stmt << "from group_membership as p "
stmt << "join users as u on p.uid_idx=u.uid_idx "
stmt << "join groups as g on "
stmt << "     p.pid=g.pid and p.gid=g.gid "
stmt << "where ((p.pid='#{pid}')) and p.trust!='none' "
stmt << "      and u.status='active' "
stmt << "      and u.webonly=0 "
stmt << "      and g.unix_gid is not NULL "
stmt << "order by u.uid"

results = client.query(stmt)

results.each do | row |
    user row['uid'] do
        Chef::Log.info("Creating account for #{row['uid']} (#{row['usr_name']})")
        supports :manage_home => false   # do not create home dir, it'll be mounted from ops
        supports :non_unique => false    # so not allow multiple account with like uids.
        password row['usr_pswd']
        shell "/bin/#{row['usr_shell']}"  # TODO fix path here what if shell is not in /bin?
        home "/users/#{row['uid']}"
        uid row['unix_uid']
        gid row['unix_gid']
        username row['uid']
        if row['admin']
            Chef::Log.info("#{row['uid']} is an admin account.")
            system true
        end
        action :create
    end
end

I modified Jennifer Chen account and changed the default shell, then (re)ran sudo chef-client

[(twonode) glawler@arya:/var/log]$ grep chen /etc/passwd            
jchen:x:11825:6004::/users/jchen:/bin/bash                          
[(twonode) glawler@arya:/var/log]$ sudo chef-client -l info         
Starting Chef Client, version 11.12.4                               
resolving cookbooks for run list: ["accounts"]                      
Synchronizing Cookbooks:                                            
  - accounts                                                        
Compiling Cookbooks...                                              
Converging 23 resources                                             
Recipe: accounts::default                                           
  * user[adititat] action create (up to date)                       
  * user[alba] action create (up to date)                           
  * user[alefiya] action create (up to date)                        
  * user[alwabel] action create (up to date)                        
  * user[amore] action create (up to date)                          
  * user[aviswana] action create (up to date)                       
  * user[blythe] action create (up to date)                         
  * user[braden] action create (up to date)                         
  * user[faber] action create (up to date)                          
  * user[gbartlet] action create (up to date)                       
  * user[glawler] action create (up to date)                        
  * user[jaipuria] action create (up to date)                       
  * user[jayvirch] action create (up to date)                       
  * user[jchen] action create                                       
    - alter user user[jchen]                                        
                                                                    
  * user[jhickey] action create (up to date)                        
  * user[kls2] action create (up to date)                           
  * user[schwab] action create (up to date)                         
  * user[sklower] action create (up to date)                        
  * user[sswati] action create (up to date)                         
  * user[sunshine] action create (up to date)                       
  * user[supriya] action create (up to date)                        
  * user[tbenzel] action create (up to date)                        
  * user[wabel] action create (up to date)                          
                                                                    
Running handlers:                                                   
Running handlers complete                                           
                                                                    
Chef Client finished, 1/23 resources updated in 5.991521947 seconds 
[(twonode) glawler@arya:/var/log]$ grep chen /etc/passwd            
jchen:x:11825:6004::/users/jchen:/bin/tcsh                          
[(twonode) glawler@arya:/var/log]$                                  

Now to chop out the current emulab account creationg code and replace it with chef.

  1. edit /usr/local/etc/emulab/rc/rc.accounts to just call sudo chef-client
    • The rc.accounts script also creates groups and removes unspecified groups and accounts, so I need to update the recipe to do the same.
    • rc.accounts supports boot, shutdown, reconfig, and reset modes. Just doing boot mode now.
      • command line: argv[0] [boot|shutdown|reconfig|reset]. We're supporting "boot" only.
    • It appeared to work after a re-boot. I set log level to debug and got a lot of warnings and errors, but it's hard to tell if those are "normal" chef errors or not. Everything at INFO level looked fine and the accounts were created. (Well they already existed, but chef confirmed that and would've created them if they had not existed.
    • Here is the whole of rc.accounts now:
      #!/usr/bin/env bash                                                                                
                                                                                                         
      action=${1:-boot}                                                                                  
                                                                                                         
      if [[ $action != boot && $action != shutdown && $action != reconfig && $action != reset ]]; then   
          echo Unsupported argument: $action                                                             
          exit 1                                                                                         
      fi                                                                                                 
                                                                                                         
      if [[ $action != boot ]]; then                                                                     
          echo Supported but unimplmented argument: $action                                              
          exit 2                                                                                         
      fi                                                                                                 
                                                                                                         
      # do it.                                                                                           
      echo Invoking chef-client to configure accounts and groups.                                        
      sudo chef-client -l debug                                                                          
      

Trying to sort out the ifconfig stuff now. Assumption: we can add a hook that fires when the control net comes up and the control net will already be configured via DHCP on boss (or wherever). I'm trying to configure the data net(s), route(s), and interfaces.

The SQL to get the interface data for a node is arcane and a mess. Looking into it though.

  • Currently trying to find out where node id comes from. The client must know it as the message arrives at tmcd with the node id there. I need this for the horrible SQL statement that tells me the data net interfaces.
    • I also need to know this for the simple node_id --> pid/eid SQL.
    • Found the correct table to extract node_id and interface information from. interfaces and wrote the ifconfig recipe. Seems to work.

ifconfig sorted out for simplist case: hostname --> ipaddress. Used 3rd party host file manager cookbook. To integrate was easy: git clone the cookbook, cp -r it into our chef-repo, add dependency in cookbook/deter_node/metadata.rb.

Working now on routing. First pass is not straight-forward: tmcd just returns ROUTERTYPE=static for tmcc routing. Looking through the tmcd.c code is no help. I need to find where routes are set.

  • found rc.route on testnodes.
    • rc.route --> getrouterconfig --> libsetup.pm --> seems to assume that it gets something back from tmcd routing
  • checked on more complicated experiment and tmcc routing still just gives ROUTERTYPE=static.
  • Need to find this "new" routing technique. Code comment says to return static at vers > 19.
    • Looks like static --> triggers calcroutes which uses the linkmap file.
      • Can we assume it'll be there? If we mount the /proj file it should be... So write our own calcroutes as a (ruby) recipe? Looking into it.
    • Looks like rc.routes calls into libsetup to get route info then writes a new rc.routes in /var/emulab/boot that itself sets the routes based on the command line args. What calls this? Who knows.
  • So two ways this could go: follow existing arcane routing method or use the "old" routing information that emulab puts in the database somewhere (this would entail finding out where/how the linkmap file is generated. It must come from the database somehow, right?)
  • See gentopofile for how to extract topology information from the DB.

Current Task: testing basic chef-configuration node.

  • created new OSID image, created new experiment, "threenode", and force sansa to be pc2177 and run the "nodeter" img. For some reason thought, the image was *not* the one I created, but what looks like a normal UB1204 DETER image. Did using the web API to create the OSid image overwrite my customized image. Hmm. It doesn't look like it. So where are my changes? Oh well. Start again.
    • Install chef client and neuter DETER boot scripts on sansa and create a custom image from that. Then reswap.
      • chef-client means chef server needs to be installed so as to have keys. installing chef-server/workstation on arya. Will keep that as a custom image as well. Will chef run on ub10.04? I think so. Bah. There is not 32bit version and the bpcs are 32 bit or at least from 32 bit ub10.04.
        • OK. I wanted to run the client on sansa a 64 bit VM ready? machine. But it's the only node I have that can run chef-server. So sansa will be the server and the others the clients.
          # chef server install
          ssh sansa  # or whatever the server will be called.
          sudo dpkg -i chef-server_11.1.1-1_amd64.deb
          sudo chef-server-ctl reconfigure
          sudo chef-server-ctl test
          # chef workstation install
          sudo dpkg -i chef_11.12.4-1_amd64.deb
          chef-client -v
          # shows version
          # (on elabinelab only) add "search isi.deterlab.net" to /etc/resolve.conf so "scratch" resolves.
          sudo apt-get install git
          # comment out edit in /etc/resolv.conf
          cd /tmp
          git clone ~glawler/src/chef-repo
          cd chef-repo
          tar jxvf ~/src/dotchef.tbz
          rm .chef/*.pem
          sudo cp /etc/chef-server/*.pem .chef
          sudo chown glawler .chef/*.pem
          knife client create chef_user --admin --file /tmp/chef-repo/.chef/chef_user.pem  --key /tmp/chef-repo/.chef/admin.pem  --user admin
          knife client list
          # to check that keys are ok.
          

Scratch

Connecting to test node servers via ssh and proxies. Add this to ~/.ssh/config

Host eine
    Hostname myboss.eine.deter.isi.deterlab.net
    ProxyCommand ssh -W %h:%p deter
    DynamicForward 1280

Host sansa
    Hostname sansa.twonode.deter
    ProxyCommand ssh -W %h:%p eine
    DynamicForward 3212

Then set up web proxy like foxyproxy to do SOCKS5 to those localhost:ports when the URL points there.


Older Notes

The Plan:                                                                                                              
-------------------------------------------------------------------                                                    
Use Chef to do host configuration on DETER.                                                                            
                                                                                                                       
Plan outline                                                                                                           
-------------------------------------------------------------------                                                    
1) Modify boss at tbsetup/tbswap  and tbswapout to call out to new daemon.                                             
    1a) add hook to remove experiment config on swap in                                                                
    1b) add hook to remove experiment config on swap out                                                               
2) New daemon gets swap in message, reads configuration information                                                    
    from testbed database                                                                                              
    2a) Extract host configuration from testebd DB                                                                     
    2b) used extracted information to create chef recipes                                                              
3) Modify test node to not call out to tmcd for local configuration,                                                   
    but instead call out to chef and ask chef to configure the local                                                   
    node.                                                                                                              
4) Since chef knows the node(s) to configure and has the experiment-specific                                           
    recipes already, it just goes ahead and configures the node.                                                       
    4a) Figure out how the mapping form node --> recipies works and                                                    
        apply it. (node roles?)                                                                                        
                                                                                                                       
                                                                                                                       
Much, much finer detail.                                                                                               
-------------------------------------------------------------------                                                    
1) Find place to put hook into boss's experiment swap in code.                                                         
    1a)                                                                                                                
    - Ted suggests tbswap.                                                                                             
    - ok for mapping.                                                                                                  
    - calls to tarfiles_setup - what does that do specifically? Which tar files to where?                              
        - user tarfiles or?...                                                                                         
            - seems to be, yes. user tarfiles.                                                                         
    - setup extra_nodes - delay nodes and such?                                                                        
    - setup mount points (server side exports, I assuming)                                                             
    - setup named names for experiment                                                                                 
    - generate topo def file (and ltmap)                                                                               
    - run os_setup                                                                                                     
        - what does this do? Looking for database updates that describe node                                           
            configuration.                                                                                             
        - first comment claims 'nodes table will already contain all the information...'                               
            - so where does that happen? Somewhere in assign?                                                          
    ------                                                                                                             
    - Finding spot to add hook:                                                                                        
        - start in tbswap.in                                                                                           
            - doSwapin(REAL) - ignoring modify/recover right now - just looking                                        
                at swap in from nothing                                                                                
                - call to TBSETUP/bin/mapper - and mapper calls assign, so add hook                                    
                    after this.                                                                                        
                    - added code call out to script (not yet written):                                                 
                        # Notify external entities that assign/mapping is complete                                     
                        # and database contains node configuration information.                                        
                        if (system("tb_configure_ready $pid $eid")) {                                                  
                                $exitcode = $? >> 8;                                                                   
                                                                                                                       
                                tberror "Failed ($exitcode) to notify that configuration was complete.";               
                                tbreport(SEV_ERROR, "tb_configure_ready failed", $exitcode);                           
                                # Make this a fatal error for now. Revisit at some point.                              
                                return 8;                                                                              
                        }                                                                                              
                    - This code was added just after check for testing and before                                      
                        handling tar files, around line 1087 in tbswap                                                 
                                                                                                                       
        a) testing swapin hook:                                                                                        
            - install updated script on boss                                                                           
                - add code chunk to /usr/testbed/bin/swap                                                              
            - create one node experiment                                                                               
            - test fail                                                                                                
                - swap in: should fail as configuration_complete is not yet written.                                   
                - correctly failed with error.                                                                         
                - create tb_configure_ready that exits 1                                                               
                - correctly fail: done
            - test success
                - create configuration_complete script in $PATH that simply exits 0
                - swap in should succeed
            - test with small python script - this tests PYTHONPATH and version.
                - classes in new file. tb_configure_ready is small script that
                    just parses command line and invokes class instance.
                    - (added logging levels and log to file to script so I could
                        see wat was going on.)
                - result: failed when should've succeeded
                    - may be path and PYTHONPATH issues?
                    - callout script must be in /usr/bin/ or various $TBROOT/... locations.
                        - fixed this (moved script softlink to /usr/bin)
                - (Should find a way to test this that does not require waiting for test node to reload.)
                - *Now* it's not finding python:
                    env: python: No such file or directory
                    *** ERROR: tbswap: tb_configure_ready exited with 127
                    - fixed with a bit of a non-portable cheat:
                        #!/usr/bin/env -P /usr/local/bin python
                - Now it finds python, but cannot find imports (python classes).
                    - need to specify PYTHONPATH or install needed modules on system.
                    - Just hardcoded the path in teh shebang line and all was well.

    1b) swap out hook removes recipes from Chef
        - TBD

2) Read host configuration information from database, parse it, create recipes.

    2a) Read info from database. Looking at how tmcd does it
    - from tmcd.c - the way to get accounts from the database:
        "select distinct "
        "  u.uid,u.usr_pswd,u.unix_uid,u.usr_name, "
        "  p.trust,g.pid,g.gid,g.unix_gid,u.admin, "
        "  u.emulab_pubkey,u.home_pubkey, "
        "  UNIX_TIMESTAMP(u.usr_modified), "
        "  u.usr_email,u.usr_shell, "
        "  u.widearearoot,u.wideareajailroot, "
        "  u.usr_w_pswd,u.uid_idx "
        "from group_membership as p "
        "join users as u on p.uid_idx=u.uid_idx "
        "join groups as g on "
        "     p.pid=g.pid and p.gid=g.gid "
        "where ((p.pid='%s')) and p.trust!='none' "
        "      and u.status='active' "
        "      and u.webonly=0 "
        "      %s "
        "      and g.unix_gid is not NULL "
        "order by u.uid",
        18, reqp->pid, adminclause);

        nice, huh?
    - There is much more parse it does after that though and it handles a fair
        number of corner cases as well. Code cruft. Hopefully we won't have to deal
        with all these cases.

    - Installing SQLAlchemy on myboss - may use it for this.

    - Skipping this for now.
        - returning false (sample) data from the appropraite python function call, which
            simulates getting the correct information from the DB. Moving on to testing
            Chef portion.

    2b) create recipes in Chef for the host configuration.
        - install Chef on server and client.
            - client
                - swap in experiment, install chef on client:
                    - dpkg -i chef_11.12.4-1_amd64.deb                                                                 
                    - Chef Server does not run on anything but Ubuntu or Enterprise Linux.                             
                        - use 2nd experment node as Chef server. GTL TODO: Add Ubuntu Server                           
                            machine to eine experiment. Need new server in DETER for this? Really?                     
                        - "sansa" is the server for now, "arya" is the client.                                         
                            - sudo dpkg -i chef-server_11.0.12-1.ubuntu.12.04_amd64.deb                                
                                - sudo chef-server-ctl reconfigure                                                     
                            - sudo dpkg -i chef_11.12.4-1_i386.deb                                                     
                                - note that arya is a 32bit berkely machine, thus the 'i686' package                   
                            - export PATH=$PATH:/opt/chef/embedded/bin                                                 
                            - export PATH=$PATH:/opt/chef-server/embedded/bin                                          
                            - [glawler@users:~/src/chef_sources]$ scp chef-repo.tgz myboss.eine.deter:~/               
                            - [glawler@sansa:/tmp]$ scp -r myboss:~/chef-repo .                                        
                            -   mkdir /tmp/chef-repo/.chef                                                             
                            -   sudo cp //etc/chef-server/*.pem /tmp/chef-repo/.chef/                                  
                            - Of course the chef source needs to be patched first:                                     
                                patch -d / -p1 < chef-11.12-4.patch                                                    
                            - configure the workstation:                                                               
                                sudo knife configure --initial --config /tmp/chef-repo/.chef/knife.rb                  
                            - and it works if you path everything out:                                                 
                                knife client list -c chef-repo/.chef/knife.rb                                          
                            - You do not need to path it out if you run it from the repo dir. annoying.                
                                                                                                                       
                            - AND:                                                                                     
                                - cp /etc/chef-server/chef-validation.pem to /etc/chef/validation.pem                  
                                    on all client nodes. This seems horrible: copying around private keys...           
                                - check: sudo chef-client -S https://sansa:443                                         
                            -  OR install on test nodes via knife bootstrap:                                           
                                - sudo knife bootstrap arya.twonode.deter -x chef_work -P password -N arya --sudo      
                                                                                                                       
                - later I will update the client image itself so Chef is already there.                                
                    - (and create a Chef server node somehow.)                                                         
                    - Looks like there is support in chef for installing chef-client remotely.                         
                        - Assumes access to the net though, but we may be able to create a custom                      
                            knife template to handle this. Esp. if we know the client platform. The install            
                            is just an install rpm, deb, etc, and scp a few keys around. See                           
                            https://tomduffield.com/how-to-bootstrap-boxes/ for an example.                            
                        - Working on knive template for remote install.                                                
                            - starting from deatils on URL above.                                                      
                            - copy client install deb to /usr/testbed/www/downloads on myboss.                         
                            - modify script from URL above to point to that deb.                                       
                            - create chef-repo/.chef/bootstrap and add script to it, named ubuntu12.04-deb.sh          
                            - After a few hours it kind of works. Using my template the client is not                  
                                getting a proper client config file (/etc/chef/client.rb), so it attempts              
                                to contact the chef server on localhost, which fails. If I run the                     
                                knife bootstrap twice once with my template to install the client and                  
                                once without my template, to configure the now installed client, it                    
                                works. This needs to be fixed though. In any case on to configuring the                
                                node.                                                                                  
    - Installing a user account on the clients.                                                                        
        - install a cookbook for creating an account. There is a generic cookbook for this.                            
        - trying to creata a cookbook by hand, just to see how it goes.                                                
            - cd /tmp/chef-repo                                                                                        
            - knife cookbook create user_install                                                                       
                - creates a CB dir in the repo, with various config files.                                             
            - vim cookbooks/user_install/recipes/default.rb                                                            
            - had to update /etc/apt/sources.list and apt-get udpate to talk to scratch                                
            - sudo apt-get install git                                                                                 
            - IGNORE                                                                                                   
                - Found a user-account cookbook at http://community.opscode.com/cookbooks/users                        
                - downloaded it to the chef server (sansa)                                                             
                - git co -B chef-vendor-users                                                                          
                - untar cookbook                                                                                       
                - git add cookbook/users                                                                               
                - git commit -m'some message' .                                                                        
                - git co master                                                                                        
                - git merge chef-vendor-users                                                                          
                - knife cookbook upload users                                                                          
                - knife cookbook list                                                          
                - knife data bag create users                                                  
                - mkdir data_bags/users                                                        
                - get passwd: openssl passwd -1 "password                                      
                - cat > data_bags/users/ITEM.json < EOF                                        
                    javascript {                                                               
                        "id": "olive",                                                         
                        "password": "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1",                      
                        "groups": ["deter", "users"],                                          
                        "shell": "\/bin\/bash",                                                
                        "comment": "A small calicao cat"                                       
                    }                                                                          
            - create user using user resource nd basic cookbook                                
                - knife cookbook create users_twonodes                                         
                - vim cookbooks/users_twonodes/recipes/default.rb                              
                    - add olive user:                                                          
                       user "olive" do                                                         
                            supports :manage_home => true                                      
                            password "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1"                      
                            shell "/bin/bash"                                                  
                            home "/users/olive"                                                
                            comment "a small calico cat"                                       
                            action :remove                                                     
                        end                                                                    
            - add the cookbook to the node                                                     
                - knife cookbook upload users_twonodes                                         
            - run sudo chef-client on nodes and you should have the user olive there.