Context Navigation

Changes between Version 38 and Version 39 of NeoContainers

Timestamp:: Nov 23, 2015 1:31:28 PM (8 years ago)
Author:: Geoff Lawler
Comment:: --

Legend:

: Unmodified
: Added
: Removed
: Modified

NeoContainers

-                      v38
+                      v39
 <p>
 The <code>config_server</code> code can be found  on DETER's Github account.
 The Chef recipes used are there as well.
+The <code>config_server</code> code can be found on DETER's Github account in the <a href="https://github.com/deter-project/config_server">config_server</a>
+repository. The Chef recipes used are there as well in the <a href="https://github.com/deter-project/deter-chef">deter-chef</a> repository. (Although the
+deter-chef repository is currently private to DETER dev and OPs.)
 <p>
+There are two approaches to running Neo-Containers.  The first is expands
+upon the existing Containers system; the second does not use the existing
+Containers system at all.  Following the discussion on
+<a href="#initial">initial set-up</a>, there are separate sections below
+describing how to use Neo-Containers in each case.
+There are two approaches to running configuring virtual machines in Neo-Containers: leveraging the existing DETER containers system or not. These can be mixed and matched; both can be used to describe the final node and network configuration of the containers in the experiment. If you use the existing containers system. please refer to the <a href="http://containers.deterlab.net">DETER Containers Docuementation</a> for details it. If you want to add "non-containerized" nodes to your experiment, you'll have to write a simple JSON formatted configuration file which describes the virtual machines you want to add (IP addresses, hostnames, OS, etc).
 <p>
+Much of the Neo-Containers system is still exposed and require the user to
+run a few configuration scripts.  In the fullness of time, these details will
+be folded in to the system.
+<p>
+<!------------------------------------------------------------------------>
+<a name="initial"></a>
+<h2>2. Initial Set-up</h2>
+<p>
+Check out the <code>config_server</code> repository from Github.  This
+repository contains the <code>config_server</code> code, as well as several
+scripts that must be run.
+<p>
+It is assumed this will be checked out on <code>users.isi.deterlab.net</code>.
+<p>
+<pre>
+    $ mkdir src
+    $ cd  src
+    $ git clone https://github.com/deter-project/config_server.git
+</pre>
+<p>
+<!------------------------------------------------------------------------>
+<a name="existing-old-containers"></a>
+<h2>3. Using Neo-Containers with the Existing Containers System</h2>
+<p>
+This method of using Neo-Containers uses the existing Containers system.
+This method allows the use of more complex network topologies.
+<p>
+<!----------------------------------------------->
+<h3>Create an Experiment</h3>
+<p>
+Create an experiment using the existing Containers system.  An NS file
+and the <b>/share/containers/containerize.py</b> script are used to create
+the containerized experiment.
+<p>
+In your NS file for each container, specify <i>image_os</i>,
+<i>image_type</i>, <i>image_name</i>, and <i>image_url</i> via the
+<i>tb-add-node-attribute</i> syntax.  Details on each attribute are
+given below.
+<p>
+<ul>
+<li><i>image_os</i> - This is really just to distinguish Windows from
+non-Windows nodes.  If the <i>image_os</i> starts with "windows", the image
+will be treated as a Windows node.  Otherwise it'll be assumed to be some sort
+of Unix-y container.
+<p>
+<li><i>image_type</i> - This setting describes the containerization tech of
+the node.  Currently this is *always* set to "vagrant" as Vagrant is the only
+package used to spin up the containers.
+<p>
+<li><i>image_name</i> - The name of the image.  Any containers that share a
+name will also share an image.
+<p>
+<li><i>image_url</i> - A URL must be specified which the neo-containers system
+uses to download the container image.  This URL must be resolvable from the
+experiment nodes.  The image will only be downloaded once as long as the
+<i>image_name</i>s are the same for each container.  Existing and supported
+images are Ubuntu 14.04 64
+(at <code>http://scratch/containers/deter_ub1404_64_vb.box)</code>
+and Windows 7 (at <code>http://scratch/containers/deter_win7.box</code>).
+</ul>
+<p>
+The following is an example NS file that creates one Windows container and
+one Ubuntu 14.04 container:
+<p>
+<pre>
+    set r2d2 [$ns node]
+    tb-add-node-attribute $r2d2 containers:image_os windows
+    tb-add-node-attribute $r2d2 containers:image_type vagrant
+    tb-add-node-attribute $r2d2 containers:image_name deter/win7
+    tb-add-node-attribute $r2d2 containers:image_url
+    http://scratch/containers/deter_win7.box
+    set c3po [$ns node]
+    tb-add-node-attribute $c3po containers:image_os ubuntu
+    tb-add-node-attribute $c3po containers:image_type vagrant
+    tb-add-node-attribute $c3po containers:image_name ubuntu/trusty64
+    tb-add-node-attribute $c3p0 containers:image_url
+    http://scratch/containers/deter_ub1404_64_vb.box
+</pre>
+<p>
+<!----------------------------------------------->
+<h3>Containerize the Experiment</h3>
+<p>
+Use the NS file to create a containerized experiment using the existing
+Containers scripts.
+<p>
+<pre>
+    $ /share/containers/containerize.py <group> <experiment> <ns-file>
+</pre>
+<p>
+<b>Note:</b>  The experiment must currently be created in the Deter group
+as that's where the custom <i>pnode</i> disk images are.  This will change.
+<p>
+<!----------------------------------------------->
+<h3>Finalize the NS File</h3>
+<p>
+Modify the NS file generated by <b>containerize.py</b> to have a new image for
+the <i>pnode</i> machines.
+<p>
+Follow these steps in your browser:
+<ol>
+<li>Go to the new experiment page.
+<li>Click <i>Modify Experiment</i>.
+<li>Remove all existing <i>tb-set-node-startcmd</i> lines.<br> These start
+the old Containers system and are no longer used.
+<li>For each <i>pnode</i>, change the OS type to PNODE_BASE.
+<li>For each <i>pnode</i>, change the hardware type to MicroCloud.
+</ol>
+<p>
+After making these modifications, each pnode in the NS file should have
+these lines:
+<pre>
+    tb-set-node-os ${pnode(0000)} PNODE-BASE
+    tb-set-hardware ${pnode(0000)} MicroCloud
+</pre>
+<p>
+The final NS file will look something like this:
+<pre>
+    set ns [new Simulator]
+    source tb_compat.tcl
+    tb-make-soft-vtype container0 {dl380g3 pc2133 MicroCloud}
+    set pnode(0000) [$ns node]
+    tb-set-node-os ${pnode(0000)} PNODE-BASE
+    tb-set-hardware ${pnode(0000)} container0
+    tb-set-node-failure-action ${pnode(0000)} "nonfatal"
+    $ns rtproto Static
+    $ns run
+</pre>
+<p>
+<!----------------------------------------------->
+<h3>Swap In</h3>
+<p>
+On the experiment's webpage, swap in the experiment.
+<p>
+<!----------------------------------------------->
+<h3>Populate the Configuration Database</h3>
+<p>
+Populate the configuration database that runs on
+<code>chef.isi.deterlab.net</code> by running the <b>load_containers_db.sh</b>
+and <b>load_config_db.sh</b> database-population scripts.
+<p>
+This should be run on a single physical node in the experiment.
+<code>pnode-0000</code> is used in the example below.
+<p>
+The <i>&lt;expid&gt;</i> and <i>&lt;projid&gt;</i> fields in the following
+example are referring to the experiment ID and the project ID.   The
+experiment ID is defined by the user, and could be something like
+"neocont-test" or "netstriping".  For now, the project ID should always
+be "Deter".
+<p>
+<pre>
+    $ ssh pnode-0000.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
+    $ cd <i>&lt;config_server-repo&gt;</i>/bin
+    $ ./load_config_db.sh
+    $ ./load_containers_db.sh -p <i>&lt;projid&gt;</i> -e <i>&lt;expid&gt;</i>
+</pre>
+<p>
+This step will be automated in the future.
+<p>
+<!----------------------------------------------->
+<h3>Node Configuration by Chef</h3>
+<p>
+The Chef system is used to bootstrap and configure the nodes.  All the
+steps for this are enclosed in the <b>bootstrap_node.sh</b> script.
+<p>
+The script needs to know which node's role in the experiment.  There
+are currently three roles:  <i>pnode</i>, <i>container</i>, and
+<i>win-container</i>.
+<p>
+On all the <i>pnode</i>s which will be running containers:
+<pre>
+    $ ssh <i>&lt;pnode&gt;</i>.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
+    $ cd <i>&lt;config_server-repo&gt;</i>/bin
+    $ ./bootstrap_node.sh -r pnode
+</pre>
+<p>
+The pnode only have to be bootstrapped once per experiment swap in. Once
+a pnode is bootstrapped into chef, <i>chef-client</i> needs to be run.
+The <i>pnode</i> role will spawn the containers and configure them. So
+once the <i>chef-client</i> command is run on a pnode, all containers
+on that be pnode will be running and configured.
+<pre>
+    $ ssh <i>&lt;pnode&gt;</i>.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
+    $ cd <i>&lt;config_server-repo&gt;</i>/bin
+    $ sudo chef-client
+</pre>
+<p>
+It is easy to fix problems if something should go wrong with bootstrapped
+nodes.  Running "sudo chef-client" will re-configure the nodes (both
+<i>pnode</i>s and the containers).
+<p>
+<!----------------------------------------------->
+<h3>Set-up Complete</h3>
+<p>
+If all the preceding steps succeeded, then your <i>pnode</i>s and containers
+are configured, booted, and ready for use.
+<p>
+<!------------------------------------------------------------------------>
+<hr>
+<p>
+<a name="absent-old-containers"></a>
+<h2>4. Using Neo-Containers While Bypassing the Existing Containers System</h2>
+<p>
+This method of using Neo-Containers does not use the existing Containers
+system.  This method allows the containers to be associated with physical
+nodes.  It requires the user to manually compute IP addresses for the
+container nodes.  Standard NS files are used in DETER experiments in this
+method of using Neo-Containers.
+<p>
+<!----------------------------------------------->
+<h3>Create an Experiment</h3>
+<p>
+Create an experiment without using the existing Containers system.  This
+experiment requires an NS file with a fully connected network.  The
+PNODE-BASE image must be used for all machines which will run containers.
+The NS file must be loaded into the DETER system in the usual way.
+<p>
+Example NS file:
+<p>
+<pre>
+    set ns [new Simulator]
+    source tb_compat.tcl
+    set nodes "leda swan"
+    tb-make-soft-vtype pnode_hardware {pc2133 MicroCloud}
+    foreach node $nodes {
+        set $node [$ns node]
+        tb-set-node-os $node PNODE-BASE
+        tb-set-hardware $node pnode_hardware
+    }
+    set lan0 [$ns make-lan $nodes 100Mb 0ms]
+    $ns rtproto Static
+    $ns run
+</pre>
+<p>
+<!----------------------------------------------->
+<a name="containers-by-hand"></a>
+<h2>1. Adding virtual machines "by hand" to your experiment.</h2>
 <!----------------------------------------------->
 …
 A <b>nodes.json</b> file must be created that will describe the containers
 in the experiment.  This file is only used to define the containers for this
+added to the experiment.  This file is only used to define the containers for this
 experiment.  (The file need not be named <b>nodes.json</b>, but that
 is the name that will be used in this documentation.)
 …
 <p>
 <li><i>host</i> - This is the <i>pnode</i> on which this container will run.
+<li><i>host</i> - This is the <i>pnode</i> in the experiment on which this container will run. Needless to say, this node must exist in the experiment.
 <p>
 …
 DETER must allocate the control network address for the containers prior to
 experiment swap-in.  This allocation occurs by passing the <b>nodes.json</b>
+file to DETER.
+file to the script at <pre>/share/config_server/bin/initialize_containers.py</pre>.
+<p>
+This must be run at least once per experiment and <b>must be done before experiment swap-in</b>. This is because the <pre>initialize_containers.py</pre> script asks DETER to allocate control network addresses for the containers. These addresses must exist prior to the containers existing as the control network to request configuration information from the configuration server. And due to the way DETER works, the addresses must be allocated before swap in or the addresses will not be properly associated with the containers hostnames so the configuration server will not be able to talk to the containers.
+The script can be run multiple times without ill effects.
+<p>
+The <i>&lt;expid&gt;</i> and <i>&lt;projid&gt;</i> fields in the following
+example are referring to the experiment ID and the project ID.   The
+experiment ID is defined by the user, and could be something like
+"neocont-test" or "netstriping".  The project ID is the name of the project under which the experiment is run.
+<p>
+<pre>
+    $ /share/config_server/bin/initialize_containers.py -p <i>&lt;projid&gt;</i> -e <i>&lt;expid&gt;</i> -f <i>path/to/nodes.json</i>
+</pre>
+If you decide to change the the nature of the containers run in an experiment,
+you must <b>destroy the experiment</b> and start over. This needs to be done
+so that DETER will unreserve the control net addresses it previously
+reserved for the containers.
+<p>
+<!----------------------------------------------->
+<a name="existing-old-containers"></a>
+<h2>3. Using Neo-Containers with the Existing Containers System</h2>
+<p>
+This method of using Neo-Containers uses the existing Containers system.
+This method allows the use of more complex network topologies.
+<p>
+<!----------------------------------------------->
+<h3>Create an Experiment</h3>
+<p>
+Create an experiment using the existing Containers system.  An NS file
+and the <b>/share/containers/containerize.py</b> script are used to create
+the containerized experiment.
+<p>
+In your NS file for each container, specify <i>image_os</i>,
+<i>image_type</i>, <i>image_name</i>, and <i>image_url</i> via the
+<i>tb-add-node-attribute</i> syntax.  Details on each attribute are
+given below.
+<p>
+<ul>
+<li><i>image_os</i> - This is really just to distinguish Windows from
+non-Windows nodes.  If the <i>image_os</i> starts with "windows", the image
+will be treated as a Windows node.  Otherwise it'll be assumed to be some sort
+of Unix-y container.
+<p>
+<li><i>image_type</i> - This setting describes the containerization tech of
+the node.  Currently this is *always* set to "vagrant" as Vagrant is the only
+package used to spin up the containers.
+<p>
+<li><i>image_name</i> - The name of the image.  Any containers that share a
+name will also share an image.
+<p>
+<li><i>image_url</i> - A URL must be specified which the neo-containers system
+uses to download the container image.  This URL must be resolvable from the
+experiment nodes.  The image will only be downloaded once as long as the
+<i>image_name</i>s are the same for each container.  Existing and supported
+images are Ubuntu 14.04 64
+(at <code>http://scratch/containers/deter_ub1404_64_vb.box)</code>
+and Windows 7 (at <code>http://scratch/containers/deter_win7.box</code>).
+</ul>
+<p>
+The following is an example NS file that creates one Windows container and
+one Ubuntu 14.04 container:
+<p>
+<pre>
+    set r2d2 [$ns node]
+    tb-add-node-attribute $r2d2 containers:image_os windows
+    tb-add-node-attribute $r2d2 containers:image_type vagrant
+    tb-add-node-attribute $r2d2 containers:image_name deter/win7
+    tb-add-node-attribute $r2d2 containers:image_url
+    http://scratch/containers/deter_win7.box
+    set c3po [$ns node]
+    tb-add-node-attribute $c3po containers:image_os ubuntu
+    tb-add-node-attribute $c3po containers:image_type vagrant
+    tb-add-node-attribute $c3po containers:image_name ubuntu/trusty64
+    tb-add-node-attribute $c3p0 containers:image_url
+    http://scratch/containers/deter_ub1404_64_vb.box
+</pre>
+<p>
+<!----------------------------------------------->
+<h3>Containerize the Experiment</h3>
+<p>
+Use the NS file to create a containerized experiment using the existing
+Containers scripts.
+<p>
+<pre>
+    $ /share/containers/containerize.py <group> <experiment> <ns-file>
+</pre>
+<p>
+<b>Note:</b>  The experiment must currently be created in the Deter group
+as that's where the custom <i>pnode</i> disk images are.  This will change.
+<p>
+<!----------------------------------------------->
+<h3>Finalize the NS File</h3>
+<p>
+Modify the NS file generated by <b>containerize.py</b> to have a new image for
+the <i>pnode</i> machines.
+<p>
+Follow these steps in your browser:
+<ol>
+<li>Go to the new experiment page.
+<li>Click <i>Modify Experiment</i>.
+<li>Remove all existing <i>tb-set-node-startcmd</i> lines.<br> These start
+the old Containers system and are no longer used.
+<li>For each <i>pnode</i>, change the OS type to PNODE_BASE.
+<li>For each <i>pnode</i>, change the hardware type to MicroCloud.
+</ol>
+<p>
+After making these modifications, each pnode in the NS file should have
+these lines:
+<pre>
+    tb-set-node-os ${pnode(0000)} PNODE-BASE
+    tb-set-hardware ${pnode(0000)} MicroCloud
+</pre>
+<p>
+The final NS file will look something like this:
+<pre>
+    set ns [new Simulator]
+    source tb_compat.tcl
+    tb-make-soft-vtype container0 {dl380g3 pc2133 MicroCloud}
+    set pnode(0000) [$ns node]
+    tb-set-node-os ${pnode(0000)} PNODE-BASE
+    tb-set-hardware ${pnode(0000)} container0
+    tb-set-node-failure-action ${pnode(0000)} "nonfatal"
+    $ns rtproto Static
+    $ns run
+</pre>
+<p>
+<!----------------------------------------------->
+<h3>Swap In</h3>
+<p>
+On the experiment's webpage, swap in the experiment.
+<p>
+<!----------------------------------------------->
+<h3>Populate the Configuration Database</h3>
+<p>
+Populate the configuration database that runs on
+<code>chef.isi.deterlab.net</code> by running the <b>load_containers_db.sh</b>
+and <b>load_config_db.sh</b> database-population scripts.
+<p>
+This should be run on a single physical node in the experiment.
+<code>pnode-0000</code> is used in the example below.
 <p>
 …
 <pre>
+    $ ssh pnode-0000.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
     $ cd <i>&lt;config_server-repo&gt;</i>/bin
+    $ ./load_containers_db.sh -p <i>&lt;projid&gt;</i> -e <i>&lt;expid&gt;</i> -f <i>path/to/nodes.json</i>
+</pre>
+<p>
+This must only be done once per experiment.  It does not have to be done
+before each swap-in.  Just once to reserve control net addresses from DETER.
+<p>
+If you decide to change the the nature of the containers run in an experiment,
+you must <b>destroy the experiment</b> and start over. This needs to be done
+so that DETER will unreserve the control net addresses it previously
+reserved for the containers.
+<p>
+    $ ./load_config_db.sh
+    $ ./load_containers_db.sh -p <i>&lt;projid&gt;</i> -e <i>&lt;expid&gt;</i>
+</pre>
+<p>
+This step will be automated in the future.
+<p>
+<!----------------------------------------------->
+<h3>Node Configuration by Chef</h3>
+<p>
+The Chef system is used to bootstrap and configure the nodes.  All the
+steps for this are enclosed in the <b>bootstrap_node.sh</b> script.
+<p>
+The script needs to know which node's role in the experiment.  There
+are currently three roles:  <i>pnode</i>, <i>container</i>, and
+<i>win-container</i>.
+<p>
+On all the <i>pnode</i>s which will be running containers:
+<pre>
+    $ ssh <i>&lt;pnode&gt;</i>.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
+    $ cd <i>&lt;config_server-repo&gt;</i>/bin
+    $ ./bootstrap_node.sh -r pnode
+</pre>
+<p>
+The pnode only have to be bootstrapped once per experiment swap in. Once
+a pnode is bootstrapped into chef, <i>chef-client</i> needs to be run.
+The <i>pnode</i> role will spawn the containers and configure them. So
+once the <i>chef-client</i> command is run on a pnode, all containers
+on that be pnode will be running and configured.
+<pre>
+    $ ssh <i>&lt;pnode&gt;</i>.<i>&lt;expid&gt;</i>.<i>&lt;projid&gt;</i>
+    $ cd <i>&lt;config_server-repo&gt;</i>/bin
+    $ sudo chef-client
+</pre>
+<p>
+It is easy to fix problems if something should go wrong with bootstrapped
+nodes.  Running "sudo chef-client" will re-configure the nodes (both
+<i>pnode</i>s and the containers).
+<p>
+<!----------------------------------------------->
+<h3>Set-up Complete</h3>
+<p>
+If all the preceding steps succeeded, then your <i>pnode</i>s and containers
+are configured, booted, and ready for use.
+<p>
+<!------------------------------------------------------------------------>
+<hr>
+<p>
 <!----------------------------------------------->