{{{
#!html
                                                                     
<h1>Neo-Containers -- DETER Containers</h1>                                   
                                                                              
<h2>Section List</h2>                                                         
                                                                              
<ol>                                                                          
<li>The Basics                                                                
<li>Initial Set-up                                                            
<li>Using Neo-Containers with the Existing Containers System                  
<li>Using Neo-Containers While Bypassing the Existing Containers System       
<li>After Booting the Containers                                              
<li>Experiment Completion                                                     
<li>Extensions to Neo-Containers                                              
</ol>
                                                                          
<p>                                                                           
                                                                              
<!------------------------------------------------------------------------>   
                                                                              
<h2>1. The Basics</h2>                                                        
                                                                              
<p>                                                                           
                                                                              
The Neo-Containers system uses cloud-container technology to abstract and     
generalize container creation and initialization. At the DETER level, the     
experiments have a number of physical nodes, "pnodes", which serve as hosts   
for the virtualized containers. Outside of the experiment there are two       
servers which configure the containers. Both run on                           
<code>chef.isi.deterlab.net</code>.  The Chef server serves "code as          
configuration" and stores static (for the most part) configuration            
information. The <code>config_server</code> is a RESTful API which loads then 
serves experiment-specific configuration information. The code that the Chef  
server runs on the containers usually pull the specific configuration         
information from the <code>config_server</code>.                              
                                                                              
<p>                                                                           
                                                                              
The <code>config_server</code> code can be found  on DETER's Github account.  
The Chef recipes used are there as well.                                      
                                                                              
<p>                                                                           
                                                                              
There are two approaches to running Neo-Containers.  The first is expands     
upon the existing Containers system; the second does not use the existing     
Containers system at all.  Following the discussion on                        
<a href="#initial">initial set-up</a>, there are separate sections below      
describing how to use Neo-Containers in each case.                            
                                                                              
<p>                                                                           
                                                                              
Much of the Neo-Containers system is still exposed and require the user to    
run a few configuration scripts.  In the fullness of time, these details will 
be folded in to the system.                                                   
                                                                              
<p>                                                                           
                                                                              
<!------------------------------------------------------------------------>   
                                                                              
<a name="initial"></a>                                                        
<h2>2. Initial Set-up</h2>                                                    
                                                                              
<p>                                                                           
                                                                              
Check out the <code>config_server</code> repository from Github.  This        
repository contains the <code>config_server</code> code, as well as several   
scripts that must be run.                                                     

<p>

It is assumed this will be checked out on <code>users.isi.deterlab.net</code>.

<p>

<pre>
    $ mkdir src
    $ cd  src
    $ git clone https://github.com/deter-project/config_server.git
</pre>

<p>

<!------------------------------------------------------------------------>

<a name="existing-old-containers"></a>
<h2>3. Using Neo-Containers with the Existing Containers System</h2>

<p>

This method of using Neo-Containers uses the existing Containers system.
This method allows the use of more complex network topologies.

<p>

<!----------------------------------------------->

<h3>Create an Experiment</h3>

<p>

Create an experiment using the existing Containers system.  An NS file
and the <b>/share/containers/containerize.py</b> script are used to create 
the containerized experiment.

<p>

In your NS file for each container, specify <i>image_os</i>,
<i>image_type</i>, <i>image_name</i>, and <i>image_url</i> via the
<i>tb-add-node-attribute</i> syntax.  Details on each attribute are
given below.

<p>

<ul>

<li><i>image_os</i> - This is really just to distinguish Windows from
non-Windows nodes.  If the <i>image_os</i> starts with "windows", the image
will be treated as a Windows node.  Otherwise it'll be assumed to be some sort
of Unix-y container.

<p>

<li><i>image_type</i> - This setting describes the containerization tech of
the node.  Currently this is *always* set to "vagrant" as Vagrant is the only
package used to spin up the containers.

<p>

<li><i>image_name</i> - The name of the image.  Any containers that share a
name will also share an image.

<p>

<li><i>image_url</i> - A URL must be specified which the neo-containers system
uses to download the container image.  This URL must be resolvable from the
experiment nodes.  The image will only be downloaded once as long as the
<i>image_name</i>s are the same for each container.  Existing and supported
images are Ubuntu 14.04 64
(at <code>http://scratch/containers/deter_ub1404_64_vb.box)</code>
and Windows 7 (at <code>http://scratch/containers/deter_win7.box</code>).

</ul>

<p>

The following is an example NS file that creates one Windows container and
one Ubuntu 14.04 container:

<p>

<pre>
    set r2d2 [$ns node]
    tb-add-node-attribute $r2d2 containers:image_os windows
    tb-add-node-attribute $r2d2 containers:image_type vagrant
    tb-add-node-attribute $r2d2 containers:image_name deter/win7
    tb-add-node-attribute $r2d2 containers:image_url
    http://scratch/containers/deter_win7.box

    set c3po [$ns node]
    tb-add-node-attribute $c3po containers:image_os ubuntu
    tb-add-node-attribute $c3po containers:image_type vagrant
    tb-add-node-attribute $c3po containers:image_name ubuntu/trusty64
    tb-add-node-attribute $c3p0 containers:image_url
    http://scratch/containers/deter_ub1404_64_vb.box
</pre>

<p>

<!----------------------------------------------->

<h3>Containerize the Experiment</h3>

<p>

Use the NS file to create a containerized experiment using the existing
Containers scripts.

<p>

<pre>
    $ /share/containers/containerize.py <group> <experiment> <ns-file>
</pre>

<p>

<b>Note:</b>  The experiment must currently be created in the Deter group
as that's where the custom <i>pnode</i> disk images are.  This will change.

<p>

<!----------------------------------------------->

<h3>Finalize the NS File</h3>

<p>

Modify the NS file generated by <b>containerize.py</b> to have a new image for
the <i>pnode</i> machines.

<p>

Follow these steps in your browser:

<ol>

<li>Go to the new experiment page.

<li>Click <i>Modify Experiment</i>.

<li>Remove all existing <i>tb-set-node-startcmd</i> lines.<br> These start
the old Containers system and are no longer used.
<li>For each <i>pnode</i>, change the OS type to PNODE_BASE.

<li>For each <i>pnode</i>, change the hardware type to MicroCloud.

</ol>

<p>

After making these modifications, each pnode in the NS file should have
these lines:

<pre>
    tb-set-node-os ${pnode(0000)} PNODE-BASE
    tb-set-hardware ${pnode(0000)} MicroCloud
</pre>

<p>

The final NS file will look something like this:

<pre>
    set ns [new Simulator]
    source tb_compat.tcl

    tb-make-soft-vtype container0 {dl380g3 pc2133 MicroCloud}
    set pnode(0000) [$ns node]
    tb-set-node-os ${pnode(0000)} PNODE-BASE
    tb-set-hardware ${pnode(0000)} container0
    tb-set-node-failure-action ${pnode(0000)} "nonfatal"

    $ns rtproto Static
    $ns run
</pre>

<p>

<!----------------------------------------------->

<h3>Swap In</h3>

<p>

On the experiment's webpage, swap in the experiment.

<p>

<!----------------------------------------------->

<h3>Populate the Configuration Database</h3>

<p>

Populate the configuration database that runs on
<code>chef.isi.deterlab.net</code> by running the <b>load_containers_db.sh</b>
and <b>load_config_db.sh</b> database-population scripts.

<p>

This should be run on a single physical node in the experiment.
<code>pnode-0000</code> is used in the example below.

<p>

The <i>&lt;expid&gt;</i> and <i>&lt;projid&gt;</i> fields in the following
example are referring to the experiment ID and the project ID.   The
experiment ID is defined by the user, and could be something like
"neocont-test" or "netstriping".  For now, the project ID should always
be "Deter".

<p>

<pre>
    $ ssh pnode-0000.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
    $ cd <i>&lt;config_server-repo&gt;</i>/bin
    $ ./load_config_db.sh
    $ ./load_containers_db.sh -p <i>&lt;projid&gt;</i> -e <i>&lt;expid&gt;</i>
</pre>

<p>

This step will be automated in the future.

<p>

<!----------------------------------------------->

<h3>Node Configuration by Chef</h3>

<p>

The Chef system is used to bootstrap and configure the nodes.  All the
steps for this are enclosed in the <b>bootstrap_node.sh</b> script.

<p>

The script needs to know which node's role in the experiment.  There
are currently three roles:  <i>pnode</i>, <i>container</i>, and
<i>win-container</i>.

<p>

On all the <i>pnode</i>s which will be running containers:

<pre>
    $ ssh <i>&lt;pnode&gt;</i>.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
    $ cd <i>&lt;config_server-repo&gt;</i>/bin
    $ ./bootstrap_node.sh -r pnode
</pre>

<p>

The pnode only have to be bootstrapped once per experiment swap in. Once
a pnode is bootstrapped into chef, <i>chef-client</i> needs to be run.

The <i>pnode</i> role will spawn the containers and configure them. So 
once the <i>chef-client</i> command is run on a pnode, all containers
on that be pnode will be running and configured. 

<pre>
    $ ssh <i>&lt;pnode&gt;</i>.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
    $ cd <i>&lt;config_server-repo&gt;</i>/bin
    $ sudo chef-client
</pre>

<p>

It is easy to fix problems if something should go wrong with bootstrapped
nodes.  Running "sudo chef-client" will re-configure the nodes (both
<i>pnode</i>s and the containers).

<p>

<!----------------------------------------------->

<h3>Set-up Complete</h3>

<p>

If all the preceding steps succeeded, then your <i>pnode</i>s and containers
are configured, booted, and ready for use.

<p>

<!------------------------------------------------------------------------>

<hr>
<p>

<a name="absent-old-containers"></a>
<h2>4. Using Neo-Containers While Bypassing the Existing Containers System</h2>

<p>

This method of using Neo-Containers does not use the existing Containers
system.  This method allows the containers to be associated with physical
nodes.  It requires the user to manually compute IP addresses for the
container nodes.  Standard NS files are used in DETER experiments in this
method of using Neo-Containers.

<p>

<!----------------------------------------------->

<h3>Create an Experiment</h3>

<p>

Create an experiment without using the existing Containers system.  This
experiment requires an NS file with a fully connected network.  The
PNODE-BASE image must be used for all machines which will run containers.
The NS file must be loaded into the DETER system in the usual way.

<p>

Example NS file:

<p>

<pre>
    set ns [new Simulator]
    source tb_compat.tcl

    set nodes "leda swan"

    tb-make-soft-vtype pnode_hardware {pc2133 MicroCloud}

    foreach node $nodes {
        set $node [$ns node]
        tb-set-node-os $node PNODE-BASE
        tb-set-hardware $node pnode_hardware
    }

    set lan0 [$ns make-lan $nodes 100Mb 0ms]

    $ns rtproto Static
    $ns run
</pre>

<p>

<!----------------------------------------------->

<h3>Create a <b>nodes.json</b> File</h3>

<p>

A <b>nodes.json</b> file must be created that will describe the containers
in the experiment.  This file is only used to define the containers for this
experiment.  (The file need not be named <b>nodes.json</b>, but that
is the name that will be used in this documentation.)

<p>

Each node must have the following fields defined in the <b>nodes.json</b>
file.

<p>

<ul>

<li><i>name</i> - This is the name of this container.

<p>

<li><i>host</i> - This is the <i>pnode</i> on which this container will run.

<p>

<li><i>interfaces</i> - This a list of the network interfaces to be assigned
the container.

<p>

<li><i>image_os</i> - This is really just to distinguish Windows from
non-Windows nodes.  If the <i>image_os</i> starts with "windows", the image
will be treated as a Windows node.  Otherwise it'll be assumed to be some sort
of Unix-y container.

<p>

<li><i>image_type</i> - This setting describes the containerization tech of
the node.  Currently this is *always* set to "vagrant" as Vagrant is the only
package used to spin up the containers.

<p>

<li><i>image_name</i> - The name of the image.  Any containers that share a
name will also share an image.

<p>

<li><i>image_url</i> - A URL must be specified which the neo-containers system
uses to download the container image.  This URL must be resolvable from the
experiment nodes.  The image will only be downloaded once as long as the
<i>image_name</i>s are the same for each container.  Existing and supported
images are Ubuntu 14.04 64
(at <code>http://scratch/containers/deter_ub1404_64_vb.box)</code>
and Windows 7 (at <code>http://scratch/containers/deter_win7.box</code>).

</ul>

<p>

The following is an example <b>nodes.json</b> file that creates
one Ubuntu 14.04 container and one Windows container:

<p>

<pre>

    [
        {
            "host": "leda",
            "name": "sarah",
            "image_url": "http://scratch/containers/deter_ub1404_64_vb.box",
            "image_os": "ubuntu 14.04 64",
            "image_type": "vagrant",
            "image_name": "deter/ub14",
            "interfaces": [
                { "address": "10.1.1.101", "mac": "de:ad:be:ef:00:be" }
            ]
        },
        {
            "host": "swan",
            "name": "helena",
            "image_url": "http://scratch/containers/deter_win7_candidate.box"
            "image_os": "windows",
            "image_type": "vagrant",
            "image_name": "deter/win7",
            "interfaces": [
                { "address": "10.1.1.201", "mac": "de:ad:be:ef:00:af" }
            ]
        }
    ]
</pre>

<p>

<!----------------------------------------------->

<h3>Load Containers into DETER</h3>

<p>

DETER must allocate the control network address for the containers prior to
experiment swap-in.  This allocation occurs by passing the <b>nodes.json</b>
file to DETER.

<p>

The <i>&lt;expid&gt;</i> and <i>&lt;projid&gt;</i> fields in the following
example are referring to the experiment ID and the project ID.   The
experiment ID is defined by the user, and could be something like
"neocont-test" or "netstriping".  For now, the project ID should always
be "Deter".

<p>

<pre>
    $ cd <i>&lt;config_server-repo&gt;</i>/bin
    $ ./load_containers_db.sh -p <i>&lt;projid&gt;</i> -e <i>&lt;expid&gt;</i> -f <i>path/to/nodes.json</i>
</pre>

<p>

This must only be done once per experiment.  It does not have to be done
before each swap-in.  Just once to reserve control net addresses from DETER.

<p>

If you decide to change the the nature of the containers run in an experiment,
you must <b>destroy the experiment</b> and start over. This needs to be done
so that DETER will unreserve the control net addresses it previously 
reserved for the containers. 

<p>

<!----------------------------------------------->

<h3>Swap In</h3>

<p>

On the experiment's webpage, swap in the experiment.
<p>

<!----------------------------------------------->

<h3>Populate the Configuration Database</h3>

<p>

Populate the configuration database that runs on
<code>chef.isi.deterlab.net</code> by running the <b>load_containers_db.sh</b>
and <b>load_config_db.sh</b> database-population scripts.

<p>

This should be run on a single physical node in the experiment.
<code>pnode-0000</code> is used in the example below.

<p>

The <i>&lt;expid&gt;</i> and <i>&lt;projid&gt;</i> fields in the following
example are referring to the experiment ID and the project ID.   The
experiment ID is defined by the user, and could be something like
"neocont-test" or "netstriping".  For now, the project ID should always
be "Deter".

<p>

<pre>
    $ ssh pnode-0000.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
    $ cd <i>&lt;config_server-repo&gt;</i>/bin
    $ ./load_config_db.sh
</pre>

<p>

This step will be automated in the future.

<p>

<!----------------------------------------------->

<h3>Node Configuration by Chef</h3>

<p>

The Chef system is used to bootstrap and configure the nodes.  All the
steps for this are enclosed in the <b>bootstrap_node.sh</b> script.

<p>

The script needs to know which node's role in the experiment.  There
are currently three roles:  <i>pnode</i>, <i>container</i>, and
<i>win-container</i>.

<p>

On all the <i>pnode</i>s which have containers running on them:

<pre>
    $ ssh <i>&lt;pnode&gt;</i>.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
    $ cd <i>&lt;config_server-repo&gt;</i>/bin
    $ ./bootstrap_node.sh -r pnode
</pre>

The pnode only have to be bootstrapped once per experiment swap in. Once
a pnode is bootstrapped into chef, <i>chef-client</i> needs to be run.

The <i>pnode</i> role will spawn the containers and configure them. So 
once the <i>chef-client</i> command is run on a pnode, all containers
on that be pnode will be running and configured. 

<pre>
    $ ssh <i>&lt;pnode&gt;</i>.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
    $ cd <i>&lt;config_server-repo&gt;</i>/bin
    $ sudo chef-client
</pre>

<p>
It is easy to fix problems if something should go wrong with bootstrapped
nodes.  Running "sudo chef-client" will re-configure the nodes (both
<i>pnode</i>s and the containers).

<p>

<!----------------------------------------------->

<h3>Set-up Complete</h3>

<p>

If all the preceding steps succeeded, then your <i>pnode</i>s and containers
are configured, booted, and ready for use.

<p>

<!------------------------------------------------------------------------>

<h2>5. After Booting the Containers</h2>

<p>

There are a number of things that may be done after the containers are
configured and booted.  These include the following:

<p>

<ul>


<li>Login to a container node<br>
<pre>
    $ ssh <i>pnode</i>
    $ ssh <i>username</i>@<i>containernode</i>
</pre>

Cygwin is installed on Windows nodes so you can <b>ssh</b> to Windows
containers as well.

<p>

<li>Play around with Vagrant<br>
Vagrant is used to control and access the container nodes.  Vagrant is used
from a root shell ("sudo su -") and from the <b>/space/vagrant_home</b>
directory.

<p>

<table border=1>
<tr align=center><td><b>Command</b>         <td><b>Purpose</b>
<tr><td>vagrant status                      <td>confirm containers are running
<tr><td>vagrant ssh <i>containernode</i>    <td>login "vagrant"; password "vagrant"
<tr><td>vagrant reload <i>containernode</i> <td>reboot a container
<tr><td>vagrant halt <i>containernode</i>   <td>halt a container
<tr><td>vagrant up <i>containernode</i>     <td>boot a container
</table>

<p>

<li>Login to a Windows desktop<br>
<ul>
<li>build ssh tunnel to port 3389 on the <i>pnode</i><br>
$ ssh -L3389:pcXXX:3389 users.deterlab.net
<p>

<li>use client RDP to connect to localhost:3389<br>
login "vagrant"; password "vagrant" for an Administrator
shell or login [your user id]; password "password" for 
normal user access. 

<p>
</ul>
</ul>
<p>

<!------------------------------------------------------------------------>

<h2>6. Experiment Completion</h2>

<p>

After an experiment is complete, the experiment data must be removed from the
configuration database.  There are two ways this may be done.

<p>

Method 1:  On a host which can talk to <code>chef.isi.deterlab.net</code>,
run these commands:

<pre>
    $ cd <i>&lt;config_server-repo&gt;</i>/bin
    $ rm_experiment_config.sh -p <i>&lt;projid&gt;</i> -e <i>&lt;expid&gt;</i>
</pre>

<p>

Method 2:  The <code>config_server</code> may be called directly:

<pre>
    $ curl http://chef:5320/exp/<i>&lt;projid&gt;</i>/<i>&lt;expid&gt;</i>/delete
</pre>

<p>

<!------------------------------------------------------------------------>

<h2>7. Extensions to Neo-Containers</h2>

<p>

<!----------------------------------------------->

<h3>Chef Workstation</h3>

<p>

Since the system runs on Chef anyone authorized to push chef recipes to the
chef server can write custom node configuration code.

<p>

<!----------------------------------------------->

<h3>Vagrant</h3>

<p>

The system uses Vagrant to spin up the containers, thus any Vagrant-supported
image can run in a container.  However, the image must be downloaded and
served via HTTP.

<p>

<!------------------------------------------------------------------------>
}}}