Changes between Version 19 and Version 20 of NeoContainers


Ignore:
Timestamp:
Oct 5, 2015 7:52:12 AM (5 years ago)
Author:
Geoff Lawler
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • NeoContainers

    v19 v20  
    11=== The Basics ===
    22
    3 The neo-containers system uses cloud-container technology to abstract and generalize container creation and initialization. At the DETER level, the experiments have a number of "pnodes" which serve as hosts for the virtualized containers and a DETER-wide Chef server at {{{chef.isideterlab.net}}}.
     3The neo-containers system uses cloud-container technology to abstract and generalize container creation and initialization. At the DETER level, the experiments have a number of physical nodes, "pnodes", which serve as hosts for the virtualized containers. Outside of the experiment there are two servers which configure the containers. Both run on {{{chef.isi.deterlab.net}}}. The Chef server serves "code as configuration" and stores static (for the most part) configuration information. The {{{config_server}}} is a RESTful API which loads then serves experiment-specific configuration information. The code that the Chef server runs on the containers usually pull the specific configuration information from the {{{config_server}}}.
    44
    55=== HOWTO run neo-containers ===
    66
    7 Note that much of the guts are still on over for these instructions. Much of the guts though are wrapped up in scripts that will themselves be wrapped in a single neo-containers script once everything has been more thoroughly tested.
     7Note that much of the detail of the system is still exposed. Users must currently run a script or two. These scripts (or the functionality they contain) will be moved into the system itself in the future and will be hidden.
    88
    991. Checkout the containers repository and switch to the {{{configdb}}} branch:
     
    1515}}}
    1616
    17 2. Create a containerized experiment with an NS file and the {{{/share/containers/containerize.py}}} script.
     172. Create a containerized experiment with an NS file and the {{{/share/containers/containerize.py}}} script. (There is a mode of neo-containers which does *not* require running the older containerization scripts. This will be documented.)
    1818
    1919In your NS file for each container in the experiment, specify  {{{image_os}}}, {{{image_type}}}, {{{image_name}}}, and {{{image_url}}} via the {{{tb-add-node-attribute}}} syntax. Details on each attribute is given below.
     
    3939}}}
    4040
    41 3. Use the NS file to create a containerized experiment using the existing containers scripts (on users): {{{/share/containers/containerize.py [group] [experiment] [ns file]}}}. Note that the experiment must currently be created in the {{{Deter}}} group as that's where the custom images are. This will change.
     41Currently Windows nodes do not get fully configured and a final script must be run by hand on the container.
     42
     433. Use the NS file to create a containerized experiment using the existing containers scripts (on users): {{{/share/containers/containerize.py [group] [experiment] [ns file]}}}. Note that the experiment must currently be created in the {{{Deter}}} group as that's where the custom pnode disk images are. This will change.
    4244
    43454. Modify the NS file generated by {{{containerize.py}}} to have a new image for the pnode machines. Navigate to the new experiment page and click {{{Modify Experiment}}}. Change the OS type of the pnodes to {{{PNODE_BASE}}} and the hardware type to {{{MicroCloud}}}. I.e. for each pnode in the NS file, make the lines have the form:
     
    4648    tb-set-hardware ${pnode(0000)} MicroCloud
    4749}}}
    48 Remove all existing {{{tb-set-node-startcmd}}} lines as these start the old containers system.
     50Remove all existing {{{tb-set-node-startcmd}}} lines as these start the old containers system. This is no longer used.
    4951
    5052The final NS file will look something like this.
     
    5658tb-make-soft-vtype container0 {dl380g3 pc2133 MicroCloud}
    5759set pnode(0000) [$ns node]
    58 tb-set-node-os ${pnode(0000)} PNODE-CONT
     60tb-set-node-os ${pnode(0000)} PNODE-BASE
    5961tb-set-hardware ${pnode(0000)} container0
    6062tb-set-node-failure-action ${pnode(0000)} "nonfatal"
     
    66685. Swap in the experiment.
    6769
    68 6. Populate the configuration database that runs on {{{chef.isi.deterlab.net}}} by running the database population scripts. (This will automated in the future.) This should be run from a physical node in the experiment. I use {{{pnode-0000}}} in the example below.
     706. Populate the configuration database that runs on {{{chef.isi.deterlab.net}}} by running the database population scripts {{{load_containers_db.sh}}} and {{{load_config_db.sh}}} (This will automated in the future.) This should be run from a physical node in the experiment. I use {{{pnode-0000}}} in the example below.
    6971
    7072{{{
     
    7779At this point, the Chef server and configuration database knows everything it needs to about your experiment and the nodes within it.
    7880
    79 7. Let Chef configure the nodes. Bootstrap and configure the pnodes. To configure/bootstrap the node use the {{{bootstrap_node.sh}}} script. The script needs to know which role the node plays in the experiment. There are currently three roles: {{{pnode}}}, {{{container}}}, and {{{win-container}}}.
     817. Let Chef configure the nodes. Bootstrap and configure the pnodes. To configure/bootstrap the node use the {{{bootstrap_node.sh}}} script. The script needs to know which role the node plays in the experiment. There are currently three roles: {{{pnode}}}, {{{container}}}, and {{{win-container}}}. 
    8082
    81 On your pnodes:
     83On all the pnodes:
    8284{{{
    8385> ssh pnode-0000.${EXPID}.${PROJID}
     
    8890The {{{pnode}}} role will spawn the containers and configure them.
    8991
    90 Once nodes are bootstrapped, simply running {{{sudo chef-client}}} will re-configure the nodes if something should go wrong.
     92Once nodes are bootstrapped, simply running {{{sudo chef-client}}} will re-configure the nodes (both pnodes and the containers) if something should go wrong.
    9193
    9294Fun things to do after the containers are running.
     
    107109    * {{{knife node show [node name]}}}
    108110    * {{{knife --help}}}
    109  
    110 === Execution Flow ===
    111111
    112 The execution flow of is as follows. The initialization uses the existing containers system as a bootstrap.
     112=== Chef Workstation ===
    113113
    114 1. Create a containerized experiment with an NS file and the {{{/share/containers/containerize.py}}} script. The NS file should declare a {{{config}}} node as an embedded container that uses the {{{UB14-CHEF12}}} image.
     114Since the system runs on Chef anyone authorized to push chef recipes to the chef server can write custom node configuration code.
    115115
    116 2. Modify the generated NS file.
    117    a. Change the OS type of the pnodes to {{{PNODE-BASE}}}. e.g. make the line in the NS file:
    118 {{{
    119     tb-set-node-os ${pnode(0000)} PNODE-BASE
    120 }}}
    121    b.  Add a new control node. Traditionally it's been called "config", but there is no restriction on the name. Add this to the NS file:
    122 {{{
    123     set config [$ns node]
    124     tb-set-node-os ${config} Ubuntu1404-64-STD
    125     tb-set-hardware ${config} MicroCloud
    126     tb-set-node-failure-action ${config} "nonfatal"
    127 }}}
    128    c. Remove the {{{tb-node-set-startcmd}}} line. This starts the existing DETER containers system on the nodes. We do not want that.
    129 3. Swap in the experiment.
    130 4. Run the bootstrap script on your {{{config}}} node.
    131    a. The bootstrap script is called {{{container_bootstrap.sh}}} and it lives in the container source code on the {{{configdb}}} branch at {{{./bin/container_bootstrap.sh}}}. So clone the containers repository, switch to the branch {{{configdb}}}, and run the script.
     116=== Vagrant ===
    132117
    133 The bootstrap script:
    134 1. Uses the NFS mounted dir /share/chef/chef-packages to install Chef server and Chef client on the {{{config}}} node.
    135     a. The {{{config}}} node is both a Chef server (talks to all machines in the experiment) and a Chef client/workstation (holds the Chef git repo which contains all configuration scripts).
    136 2. Configures Chef Server on the {{{config}}} node.
    137     a. Simply runs the chef self-configure scripts.
    138 3. Configures Chef client/workstation on the {{{config}}} node.
    139     a. creates a chef-deter identity and organization (keys and names) for the Chef Server.
    140     b. installs {{{git}}} on the {{{config}}} node.
    141     c. git clones the canonical Chef repository from the NFS mounted dir {{{/share/chef/chef-repo}}}. Thie repo contains all configuration scripts and data for the system.
    142     d. upload all Chef recipes, data bags, and roles (configuration scripts, data, and roles) to the Chef server on {{{config}}} (localhost).
    143 4. Create an experiment-specific "data bag" and upload it to the Chef Server. The data contains experiment name, project, group, name and address of the {{{config}}} machine,  and the name and address of the RESTful configuration server. This is per-experiment dynamic data and thus is generated at run time instead of statically kept in the chef repository.
    144 5. Register the {{{config}}} node (localhost) as a Chef client to the server, download and execute all local configuration recipes that exist in the {{config_server}} chef role. (A chef role is simply a collection of recipes.) Details of these are in the next section.
    145 
    146 The chef repository defines three roles in the system, {{{config_server}}}, {{{pnode}}}, and {{{container}}}. The code that defines the roles can be found in the {{{./roles}}} directory in the chef repository.
    147 
    148 === {{{config_server}}} recipes ===
    149 
    150 Recipes are chunks of Ruby code that is are a mixture of Ruby and the Chef configuration definition language. A recipe is downloaded from a Chef server and run when {{{chef-client}}} is run on a node. This section describes the recipes that are run on the {{{config_server}}} machine itself. These recipes configure Chef communications on the nodes and tell the {{{pnodes}}} to configure themselves. (The {{{pnode}}} role recipe section is next.)
    151 
    152 These recipes can be found in the {{{./cookbooks/config_server/recipes}}} directory in the chef repository.
    153 
    154 The {{{config_server}}} role consists of the following recipes, executed in order:  {{{config_db}}}, {{{config_server}}}, {{{hosts}}}, {{{bootstrap_pnodes}}}, and {{{configure_containers}}}.
    155 
    156 * {{{config_db}}} - This recipe builds the configuration database from existing containers and DETER/emulab files found on the local machine. This database contains all the configuration information to configure the experiment. This database is RESTfully served by the {{{config_server}}} process started elsewhere. The recipe also installs required packages needed to build the database (python3 and SQLAlchemy).
    157 
    158 * {{{config_server}}} - This recipe installs and starts the {{{config_server}}} process. This server RESTfully serves configuration data to anyone in the experiment that requests it. After installing required packages, the recipe simply calls the standard {{{setup.py}}} script in the config_server package to install it. It then copies a local {{{/etc/init}}} style script from the chef repository to {{{/etc/init.d/config_server}}}, then invokes the it, starting the daemon. At this point configuration information requests can be made on {{{http://config:5000/...}}} to get configuration information.
    159 
    160 * {{{hosts}}} - This recipe (which actually lives in the {{{pnode}}} cookbook directory, adds container node names and address information to {{{/etc/hosts}}}. This is a required step for later when the {{{config_server}}} node connects to the containers.
    161 
    162 * {{{bootstrap_pnodes}}} - This recipe connects to all pnode machines and configures them as chef clients to the new chef server. It then sets the default role for the pnode machines to the {{{pnode}}} role. Finally it connects to all the pnode machines and invokes {{{chef-client}}} locally. This kicks off the next round of configuration in the system - all pnode machines configure themselves. These recipes are described in the next section.
    163 
    164 * {{{configure_containers}}} - This recipe simply uses the chef tool {{{knife}}} to connect to all (now running) containers and executes {{{chef-client}}}, which causes the containers to reach out to the chef server and request recipes/roles to execute. After this the containers are fully configured and integrated into DETER.
    165 
    166 === {{{pnode}}} recipes ===
    167 
    168 The {{{pnode}}} role consists of the following recipes, executed in order: {{{uml-net-group}}}, {{{mount_space}}}, {{{hosts}}}, {{{diod}}}, {{{vde}}}, and {{{vagrant}}}.
    169 
    170 * {{{uml-net-group}}} - This recipe creates a {{{uml-net}}} group. This is just a stop-gap recipe which is needed due to a broken apt database in the PNODE-BASE image. This recipe will be removed from the role and the chef repository once the PNODE-BASE image is rebuilt.
    171 
    172 * {{{mount_space}}} - This recipe formats and mounts the {{{/dev/sda4}}} partition on the disk. This is subsequently used to store and run containers images, the chef repository, etc.
    173 
    174 * {{{hosts}}} - This recipe creates entries in the pnode's {{{/etc/hosts}}} for all container nodes.
    175 
    176 * {{{diod}}} - This recipe installs, configures, and starts the {{{diod}}} daemon. The pnode has users home directories and other directories mounted locally. The containers also need these mounted to be fully integrated with DETER. Unfortunately it is not possible to use standard NFS mounting on directories that are themselves NFS mounted. (There are good reasons for this.) So the pnode uses {{{diod}}} to serve these directories to the container nodes. The recipe uses standard package tech to install {{{diod}}} and configures diod to export all directories that are locally mounted - which is just what we want. It then starts the daemon.
    177 
    178 * {{{vde}}} - This recipe installs and configures {{{VDE}}}, Virtual Distributed Ethernet on the pnode. It installs VDE2 and required packages (uml-utilities and bridge-utils). It installs a /etc/init style script to {{{/etc/init.d/vde_switch}}} to control the VDE daemon. It requests all virtual networking configuration information from the {{{config_server}}} and creates a number of VDE configuration files in {{{/etc/vde2/conf.d}}}, one for each switch needed. The recipe creates and configures all the TAP devices and bridges needed by the VDE switches. It then calls the {{{/etc/init.d/vde2}}} script to start all switches. At this point all local network plumbing is there and working.
    179 
    180 * {{{vagrant}}} - This recipe installs and configures {{{vagrant}}}. Vagrant is a front-end for configuring and spawning virtual machines. it supports many different virtual machine image (qemu, LXE, VirtualBox, etc), but the current neo-containers system only uses VirtualBox images. (It has a few default images already configured and installed on the PNODE-BASE DETER image.) The recipe installs {{{vagrant}}} and {{{virtualbox}}} (although they are already installed on the PNODE-BASE image, so this is really a NOOP command. After that, it's big job is to create the {{{Vagrantfile}}} file in {{{/space/vagrant}}} that describes the container images and basic configuration. The {{{Vagrantfile}}} created does a few basic things: configure virtual NICs, setup basic networking so the containers route correctly to the DETER control net, and invoke a basic NOOP chef configuration that *only* register the container node with the chef server. It does not invoke {{{chef-client}}} on the containers to configure the nodes, that happens next. The recipe also sets up a few things on the pnode for smoother vagrant operations: a {{{vagrant}}} user and home directory at {{{/vagrant}}} are created, an {{{/etc/profile.d/vagrant.sh}}} file is created with vagrant-specific environment variables, Virtual Box is configured to not store disk image copies on NFS mounted dirs, but in {{{/space/vagrant}}}. Finally {{{/etc/init.d/vagrant}}} is invoked to spawn the containers. At this point, the containers are running, but not configured/integrated into DETER (no user accounts, mounts, etc).
    181 
    182 === {{{container}}} recipes ===
    183 
    184 The {{{container}}} role consists of the following recipes: {{{apt}}}, {{{hosts}}}, {{{groups}}}, {{{accounts}}}, {{{mounts}}}. All these recipes work to integrate the container into the running DETER experiment.
    185 
    186 * {{{apt}}} - Runs {{{apt clean}}} and {{{apt update}}} on the nodes. It would be nice to get rid of this recipe as it will slam the DETER apt repository server. Need to investigate why and if this is still needed.
    187 
    188 * {{{hosts}}} - Adds container names and addresses to {{{/etc/hosts}}} so all containers can resolve each others' names.
    189 
    190 * {{{groups}}} - Requests group information from the {{{config_server}}} and creates the DETER groups on the container.
    191 
    192 * {{{accounts}}} - Requests user account information from the {{{config_server}}} and recreates the accounts locally on the container. This includes creating mount points for the user home directories.
    193 
    194 * {{{mounts}}} - This recipe installs the client side {{{diod}}} package and configures it to talk to the local pnode on which the container runs. It requests mount information from the {{{config_server}}} and mounts what it finds via {{{diod}}}.
    195 
    196 
    197 === Outstanding Issues ===
    198 
    199 This section lists outstanding issues for the container system. And not "outstanding" as in "awesome", but outstanding as in this still has to get done.
    200 
    201 * {{{diod}}} - The diod daemon (re)mounts NFS mounted filesystems on the containers. The pnode acts is a server, the containers are clients.
    202   * {{{diod}}} fails to fully mount filesystems sometimes during the application of the mounts recipe. It fails in {{{mount -a}}}. It fails to allocate memory. This only seems to happen when running in chef. Running {{{sudo mount -a}}} on the command line of the container works just fine. Which is a little annoying, but so it goes.
    203 * Windows - is fragile and doesnt' really work. Working on this now.
    204 * LXC - Does not come up. Looked into this briefly. Tried booting outside of vagrant and still had issues. See here for a getting started doc: https://linuxcontainers.org/lxc/getting-started/
    205 * Using {{{~glawler}}} - some scripts are still run (via recipes) out of {{{~glawler}}}. They should be fed a path to a {{{containers}}} repo or just assume {{{/share/containers}}}.
    206 * Vagrant
    207   * Does do some things in parallel, but not bringing up virtualbox machines. This makes things very slow.
    208   * chef provisioning via vagrant does not work well. It registers the node with the chef server, but does not properly setup the node for future runs. Does not create proper {{{client.rb}}} config files.
    209 * everything else.
     118The system uses [https://www.vagrantup.com/ Vagrant] to spin up the containers, thus any [https://atlas.hashicorp.com/boxes/search vagrant supported image] run in a container. The image must be downloaded and served via HTTP though.