wiki:ChefNotes

Version 7 (modified by Geoff Lawler, 10 years ago) (diff)

--

The new(est) plan:

Make recipes that query the tbdb directly. If done right, the nodes configure themselves with information from the database directly. The code for querying the database will live in Chef recipes, pulled onto the nodes, then executed on boot. We'll add a hook in dhcp init, much like emulab does now to insure the control network is up, that executes 'chef-client', which does the magic.

Things to do:

  1. Get chef on a client machine, on the image itself for now. Later we can install chef remotely on boot?
  2. Get the needed packges for database access onto the test node (mysql gems).
  3. Write a simple recipe that grabs account information and creates the accounts.

Plan execution

  1. Chef is on a client now, arya, and the chef server, sansa, knows how to re-install if needed.
  2. Figure out how to install mysql gem on arya without direct access to package repos.
    1. How to run your own Gem server, interesting: http://guides.rubygems.org/run-your-own-gem-server/
    2. installing mysql2 gem:
      • sudo apt-get install liberuby-dev libmysqlclient-dev
      • sudo /opt/chef/embedded/bin/gem install --local mysql2-0.3.16.gem (assumes chef is installed, otherwise just "gem install")
    3. test installation. my simple script below is failing to load mysql2.
      #!/usr/bin/env ruby                                                         
                                                                                  
      require 'mysql2'                                                            
                                                                                  
      client = Mysql2::Client.new(:host => "localhost", :username => "root")      
      
      
      • needed to use chef's ruby as that's where the gem is installed: /opt/chef/embedded/bin/ruby mysql_test.rb.
      • Back up a step and see if we can and how we do connect to tbdb remotely.
        • running on myboss, 7777 mysql -h myboss -p 7777 asks for a password.
          • Looking for appropriate keys to use.
          • tbdb is running "skip-grant-tables" so anyone on the localhost can assess it.
          • doing "mysql -h myboss -P 7777" gives cryptic error ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0.
          • Found the DB startup script: /usr/local/etc/rc.d/2.mysql-server.sh
            • Options passed to mysql: "--pid-file=/var/db/mysql/mysqld.pid --skip-grant-tables --skip-networking --user=mysql --log=/usr/testbed/log/mysql/base --log-bin=/usr/testbed/log/mysql/update --log-slow-queries=/usr/testbed/log/mysql/slowqueries --max_connections=500"
            • That --skip-networking is troubling. Yeah, we need to change that: Do not listen for TCP/IP connections at all. All interaction with mysqld must be made using named pipes or shared memory (on Windows) or Unix socket files (on Unix). This option is highly recommended for systems where only local clients are permitted.
            • I'll restart mysql on myboss without that argument, then I'll investigate secure remote access.
            • edited /usr/local/etc/rc.d/2.mysql-server.sh
            • killed mysql (mysqld_watchdog should restart it.)
              • sudo kill 97362 97535 ; tail -F mysqld_watchdog.log
              • seems to have worked.
              • I can now connect remotely using mysql client: mysql -h myboss -P 3306. So yay! We'll need to secure the connection somehow though inthe future. Back to script testing.
      • running test ruby script. Runs to error (yay!) after making sure the paths are correct.
    4. Connect to database and print account information as test of mysql code. After that put it in a recipe and try it on the test nodes.
      • OK! I can query for account information from a test node. The test script is:
        #!/usr/bin/env ruby                                                                             
                                                                                                        
        require 'rubygems'                                                                              
        require 'mysql2'                                                                                
                                                                                                        
        client = Mysql2::Client.new(:host => "myboss", :username => "mysql", :database => "tbdb")       
                                                                                                        
        stmt = "select distinct "                                                                       
        stmt << "  u.uid,u.usr_pswd,u.unix_uid,u.usr_name, "                                            
        stmt << "  p.trust,g.pid,g.gid,g.unix_gid,u.admin, "                                            
        stmt << "  u.emulab_pubkey,u.home_pubkey, "                                                     
        stmt << "  UNIX_TIMESTAMP(u.usr_modified), "                                                    
        stmt << "  u.usr_email,u.usr_shell, "                                                           
        stmt << "  u.widearearoot,u.wideareajailroot, "                                                 
        stmt << "  u.usr_w_pswd,u.uid_idx "                                                             
        stmt << "from group_membership as p "                                                           
        stmt << "join users as u on p.uid_idx=u.uid_idx "                                               
        stmt << "join groups as g on "                                                                  
        stmt << "     p.pid=g.pid and p.gid=g.gid "                                                     
        stmt << "where ((p.pid='Deter')) and p.trust!='none' "                                          
        stmt << "      and u.status='active' "                                                          
        stmt << "      and u.webonly=0 "                                                                
        stmt << "      and g.unix_gid is not NULL "                                                     
        stmt << "order by u.uid limit 2"                                                                
                                                                                                        
        results = client.query(stmt)                                                                    
        results.each do |row|                                                                           
            printf "%s\n", row                                                                                   
        end                                                                                             
        

This outputs:

./mysql_test.rb 
{"uid"=>"adititat", "usr_pswd"=>"$1$13954353$MRXeRMUxPc2t4hxFe7o3k0", "unix_uid"=>14427, "usr_name"=>"Aditi Tatti", "trust"=>"local_root", "pid"=>"Deter", "gid"=>"Deter", "unix_gid"=>6004, "admin"=>0, "emulab_pubkey"=>nil, "home_pubkey"=>nil, "UNIX_TIMESTAMP(u.usr_modified)"=>1398879458, "usr_email"=>"tatti@usc.edu", "usr_shell"=>"bash", "widearearoot"=>0, "wideareajailroot"=>0, "usr_w_pswd"=>"03b%5m*g", "uid_idx"=>14536}
{"uid"=>"alba", "usr_pswd"=>"$1$12831943$YeNdFr60Aj79NiddzObMQ/", "unix_uid"=>11615, "usr_name"=>"Alba Palacios", "trust"=>"local_root", "pid"=>"Deter", "gid"=>"Deter", "unix_gid"=>6004, "admin"=>1, "emulab_pubkey"=>nil, "home_pubkey"=>nil, "UNIX_TIMESTAMP(u.usr_modified)"=>1398879433, "usr_email"=>"alba@isi.edu", "usr_shell"=>"tcsh", "widearearoot"=>0, "wideareajailroot"=>0, "usr_w_pswd"=>"%!<0fsKo", "uid_idx"=>11681}
  1. So now all I need to do is learn Ruby.

Scratch

Connecting to test node servers via ssh and proxies. Add this to ~/.ssh/config

Host eine
    Hostname myboss.eine.deter.isi.deterlab.net
    ProxyCommand ssh -W %h:%p deter
    DynamicForward 1280

Host sansa
    Hostname sansa.twonode.deter
    ProxyCommand ssh -W %h:%p eine
    DynamicForward 3212

Then set up web proxy like foxyproxy to do SOCKS5 to those localhost:ports when the URL points there.


Older Notes

The Plan:                                                                                                              
-------------------------------------------------------------------                                                    
Use Chef to do host configuration on DETER.                                                                            
                                                                                                                       
Plan outline                                                                                                           
-------------------------------------------------------------------                                                    
1) Modify boss at tbsetup/tbswap  and tbswapout to call out to new daemon.                                             
    1a) add hook to remove experiment config on swap in                                                                
    1b) add hook to remove experiment config on swap out                                                               
2) New daemon gets swap in message, reads configuration information                                                    
    from testbed database                                                                                              
    2a) Extract host configuration from testebd DB                                                                     
    2b) used extracted information to create chef recipes                                                              
3) Modify test node to not call out to tmcd for local configuration,                                                   
    but instead call out to chef and ask chef to configure the local                                                   
    node.                                                                                                              
4) Since chef knows the node(s) to configure and has the experiment-specific                                           
    recipes already, it just goes ahead and configures the node.                                                       
    4a) Figure out how the mapping form node --> recipies works and                                                    
        apply it. (node roles?)                                                                                        
                                                                                                                       
                                                                                                                       
Much, much finer detail.                                                                                               
-------------------------------------------------------------------                                                    
1) Find place to put hook into boss's experiment swap in code.                                                         
    1a)                                                                                                                
    - Ted suggests tbswap.                                                                                             
    - ok for mapping.                                                                                                  
    - calls to tarfiles_setup - what does that do specifically? Which tar files to where?                              
        - user tarfiles or?...                                                                                         
            - seems to be, yes. user tarfiles.                                                                         
    - setup extra_nodes - delay nodes and such?                                                                        
    - setup mount points (server side exports, I assuming)                                                             
    - setup named names for experiment                                                                                 
    - generate topo def file (and ltmap)                                                                               
    - run os_setup                                                                                                     
        - what does this do? Looking for database updates that describe node                                           
            configuration.                                                                                             
        - first comment claims 'nodes table will already contain all the information...'                               
            - so where does that happen? Somewhere in assign?                                                          
    ------                                                                                                             
    - Finding spot to add hook:                                                                                        
        - start in tbswap.in                                                                                           
            - doSwapin(REAL) - ignoring modify/recover right now - just looking                                        
                at swap in from nothing                                                                                
                - call to TBSETUP/bin/mapper - and mapper calls assign, so add hook                                    
                    after this.                                                                                        
                    - added code call out to script (not yet written):                                                 
                        # Notify external entities that assign/mapping is complete                                     
                        # and database contains node configuration information.                                        
                        if (system("tb_configure_ready $pid $eid")) {                                                  
                                $exitcode = $? >> 8;                                                                   
                                                                                                                       
                                tberror "Failed ($exitcode) to notify that configuration was complete.";               
                                tbreport(SEV_ERROR, "tb_configure_ready failed", $exitcode);                           
                                # Make this a fatal error for now. Revisit at some point.                              
                                return 8;                                                                              
                        }                                                                                              
                    - This code was added just after check for testing and before                                      
                        handling tar files, around line 1087 in tbswap                                                 
                                                                                                                       
        a) testing swapin hook:                                                                                        
            - install updated script on boss                                                                           
                - add code chunk to /usr/testbed/bin/swap                                                              
            - create one node experiment                                                                               
            - test fail                                                                                                
                - swap in: should fail as configuration_complete is not yet written.                                   
                - correctly failed with error.                                                                         
                - create tb_configure_ready that exits 1                                                               
                - correctly fail: done
            - test success
                - create configuration_complete script in $PATH that simply exits 0
                - swap in should succeed
            - test with small python script - this tests PYTHONPATH and version.
                - classes in new file. tb_configure_ready is small script that
                    just parses command line and invokes class instance.
                    - (added logging levels and log to file to script so I could
                        see wat was going on.)
                - result: failed when should've succeeded
                    - may be path and PYTHONPATH issues?
                    - callout script must be in /usr/bin/ or various $TBROOT/... locations.
                        - fixed this (moved script softlink to /usr/bin)
                - (Should find a way to test this that does not require waiting for test node to reload.)
                - *Now* it's not finding python:
                    env: python: No such file or directory
                    *** ERROR: tbswap: tb_configure_ready exited with 127
                    - fixed with a bit of a non-portable cheat:
                        #!/usr/bin/env -P /usr/local/bin python
                - Now it finds python, but cannot find imports (python classes).
                    - need to specify PYTHONPATH or install needed modules on system.
                    - Just hardcoded the path in teh shebang line and all was well.

    1b) swap out hook removes recipes from Chef
        - TBD

2) Read host configuration information from database, parse it, create recipes.

    2a) Read info from database. Looking at how tmcd does it
    - from tmcd.c - the way to get accounts from the database:
        "select distinct "
        "  u.uid,u.usr_pswd,u.unix_uid,u.usr_name, "
        "  p.trust,g.pid,g.gid,g.unix_gid,u.admin, "
        "  u.emulab_pubkey,u.home_pubkey, "
        "  UNIX_TIMESTAMP(u.usr_modified), "
        "  u.usr_email,u.usr_shell, "
        "  u.widearearoot,u.wideareajailroot, "
        "  u.usr_w_pswd,u.uid_idx "
        "from group_membership as p "
        "join users as u on p.uid_idx=u.uid_idx "
        "join groups as g on "
        "     p.pid=g.pid and p.gid=g.gid "
        "where ((p.pid='%s')) and p.trust!='none' "
        "      and u.status='active' "
        "      and u.webonly=0 "
        "      %s "
        "      and g.unix_gid is not NULL "
        "order by u.uid",
        18, reqp->pid, adminclause);

        nice, huh?
    - There is much more parse it does after that though and it handles a fair
        number of corner cases as well. Code cruft. Hopefully we won't have to deal
        with all these cases.

    - Installing SQLAlchemy on myboss - may use it for this.

    - Skipping this for now.
        - returning false (sample) data from the appropraite python function call, which
            simulates getting the correct information from the DB. Moving on to testing
            Chef portion.

    2b) create recipes in Chef for the host configuration.
        - install Chef on server and client.
            - client
                - swap in experiment, install chef on client:
                    - dpkg -i chef_11.12.4-1_amd64.deb                                                                 
                    - Chef Server does not run on anything but Ubuntu or Enterprise Linux.                             
                        - use 2nd experment node as Chef server. GTL TODO: Add Ubuntu Server                           
                            machine to eine experiment. Need new server in DETER for this? Really?                     
                        - "sansa" is the server for now, "arya" is the client.                                         
                            - sudo dpkg -i chef-server_11.0.12-1.ubuntu.12.04_amd64.deb                                
                                - sudo chef-server-ctl reconfigure                                                     
                            - sudo dpkg -i chef_11.12.4-1_i386.deb                                                     
                                - note that arya is a 32bit berkely machine, thus the 'i686' package                   
                            - export PATH=$PATH:/opt/chef/embedded/bin                                                 
                            - export PATH=$PATH:/opt/chef-server/embedded/bin                                          
                            - [glawler@users:~/src/chef_sources]$ scp chef-repo.tgz myboss.eine.deter:~/               
                            - [glawler@sansa:/tmp]$ scp -r myboss:~/chef-repo .                                        
                            -   mkdir /tmp/chef-repo/.chef                                                             
                            -   sudo cp //etc/chef-server/*.pem /tmp/chef-repo/.chef/                                  
                            - Of course the chef source needs to be patched first:                                     
                                patch -d / -p1 < chef-11.12-4.patch                                                    
                            - configure the workstation:                                                               
                                sudo knife configure --initial --config /tmp/chef-repo/.chef/knife.rb                  
                            - and it works if you path everything out:                                                 
                                knife client list -c chef-repo/.chef/knife.rb                                          
                            - You do not need to path it out if you run it from the repo dir. annoying.                
                                                                                                                       
                            - AND:                                                                                     
                                - cp /etc/chef-server/chef-validation.pem to /etc/chef/validation.pem                  
                                    on all client nodes. This seems horrible: copying around private keys...           
                                - check: sudo chef-client -S https://sansa:443                                         
                            -  OR install on test nodes via knife bootstrap:                                           
                                - sudo knife bootstrap arya.twonode.deter -x chef_work -P password -N arya --sudo      
                                                                                                                       
                - later I will update the client image itself so Chef is already there.                                
                    - (and create a Chef server node somehow.)                                                         
                    - Looks like there is support in chef for installing chef-client remotely.                         
                        - Assumes access to the net though, but we may be able to create a custom                      
                            knife template to handle this. Esp. if we know the client platform. The install            
                            is just an install rpm, deb, etc, and scp a few keys around. See                           
                            https://tomduffield.com/how-to-bootstrap-boxes/ for an example.                            
                        - Working on knive template for remote install.                                                
                            - starting from deatils on URL above.                                                      
                            - copy client install deb to /usr/testbed/www/downloads on myboss.                         
                            - modify script from URL above to point to that deb.                                       
                            - create chef-repo/.chef/bootstrap and add script to it, named ubuntu12.04-deb.sh          
                            - After a few hours it kind of works. Using my template the client is not                  
                                getting a proper client config file (/etc/chef/client.rb), so it attempts              
                                to contact the chef server on localhost, which fails. If I run the                     
                                knife bootstrap twice once with my template to install the client and                  
                                once without my template, to configure the now installed client, it                    
                                works. This needs to be fixed though. In any case on to configuring the                
                                node.                                                                                  
    - Installing a user account on the clients.                                                                        
        - install a cookbook for creating an account. There is a generic cookbook for this.                            
        - trying to creata a cookbook by hand, just to see how it goes.                                                
            - cd /tmp/chef-repo                                                                                        
            - knife cookbook create user_install                                                                       
                - creates a CB dir in the repo, with various config files.                                             
            - vim cookbooks/user_install/recipes/default.rb                                                            
            - had to update /etc/apt/sources.list and apt-get udpate to talk to scratch                                
            - sudo apt-get install git                                                                                 
            - IGNORE                                                                                                   
                - Found a user-account cookbook at http://community.opscode.com/cookbooks/users                        
                - downloaded it to the chef server (sansa)                                                             
                - git co -B chef-vendor-users                                                                          
                - untar cookbook                                                                                       
                - git add cookbook/users                                                                               
                - git commit -m'some message' .                                                                        
                - git co master                                                                                        
                - git merge chef-vendor-users                                                                          
                - knife cookbook upload users                                                                          
                - knife cookbook list                                                          
                - knife data bag create users                                                  
                - mkdir data_bags/users                                                        
                - get passwd: openssl passwd -1 "password                                      
                - cat > data_bags/users/ITEM.json < EOF                                        
                    javascript {                                                               
                        "id": "olive",                                                         
                        "password": "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1",                      
                        "groups": ["deter", "users"],                                          
                        "shell": "\/bin\/bash",                                                
                        "comment": "A small calicao cat"                                       
                    }                                                                          
            - create user using user resource nd basic cookbook                                
                - knife cookbook create users_twonodes                                         
                - vim cookbooks/users_twonodes/recipes/default.rb                              
                    - add olive user:                                                          
                       user "olive" do                                                         
                            supports :manage_home => true                                      
                            password "$1$WBmMhoZW$c9biaiUTg5Q1Qztc3900C1"                      
                            shell "/bin/bash"                                                  
                            home "/users/olive"                                                
                            comment "a small calico cat"                                       
                            action :remove                                                     
                        end                                                                    
            - add the cookbook to the node                                                     
                - knife cookbook upload users_twonodes                                         
            - run sudo chef-client on nodes and you should have the user olive there.