Changes between Initial Version and Version 1 of UsingContainers

Timestamp:

Jun 17, 2012 5:13:24 PM (12 years ago)

Author:

Ted Faber

Comment:

--

UsingContainers

@@ +1 @@
+[[TOC]]
+
+= Using Containers =
+
+A containerized virtualization system currently runs on the DETER testbed.
+
+We support two types of experiments: qemu auto-partitioned and mixed mode. Nodes in a qemu auto-partitioned experiment are realized as either qemu virtual machines or physical nodes. The qemu nodes are automatically packed into physical nodes based on a packing parameter. Nodes in a mixed mode experiment may be qemu virtual nodes, lightweight processes, or physical machines. All non-physical nodes must be embedded manually.
+
+== Disclaimer ==
+
+Note that this software is currently in a '''beta''' state. We have made every effort to make the system as stable as possible, but it has not been heavily used. Expect to run into bugs.
+
+If and when you do encounter a bug you can either [/newticket file a ticket] or email the author directly at mikeryan@ISI.EDU.
+
+== How it works ==
+
+The container system is built on the legacy DETER functionality. This means that a containerized experiment appears to the DETER testbed and web interface as a typical experiment. The NS file for the host experiment is annotated with a variety of container-related commands.
+
+You will interact with your containerized experiment through facilities provided by DETER. The swap in/out process is controlled by the DETER web interface, and you will access your virtual nodes like any other node from {{{users.isi.deterlab.net}}}.
+
+== Getting Started ==
+
+=== Prerequisites ===
+
+Prepare your NS or TopDL file as you would for any other DETER experiment.
+
+Before running your experiment you should determine the DETER PID and EID under which you'll run the experiment. The EID should not refer to an existing experiment.
+
+=== qemu auto-partitioned experiments ===
+
+You will need to specify a parameter called "packing". This parameter describes how many qemu virtual nodes will be embedded on each physical node. The host machines have four cores each, so for compute-bound tasks this value should not be higher than 4. For less CPU-intensive tasks we find a factor of 10 to provide adequate performance. This number can in theory be as high as 10 or 20 (perhaps higher).
+
+If you wish for some of your nodes to be realized as physical machines, you must use a TopDL file (not NS) and tag the nodes as physical. Add the attribute {{{benito:node_type = embedded_pnode}}} to any such nodes. See UsingShopdl for documentation on a tool to do this.
+
+Once you have determined the parameters described above, you may start an experiment by logging in to {{{users.isi.deterlab.net}}} and running these commands:
+
+{{{
+$ cd /share/benito
+$ ./qemu_experiment.py <PID> <EID> <ns-or-topdl-file> <packing>
+}}}
+
+=== Mixed mode experiments ===
+
+Mixed mode experiments may only be specified using a TopDL file. Each node must be specially annotated to incdicate its embedding. This is done by specifying an attribute on the node called {{{partition}}}.
+
+The {{{partition}}} attribute specifies the physical node on which a qemu virtual node is realized. This value must be numeric, though there are no restrictions on what numbers are used.
+
+If a node ''lacks'' the {{{partition}}} attribute, it is assumed to be a physical node.
+
+You can use the {{{shopdl.py}}} tool in the {{{util}}} subdirectory of the tree in order to ease manipulation of your TopDL file. See UsingShopdl for more information.
+
+Once you have annotated your TopDL file, you may start an experiment by logging in to {{{users.isi.deterlab.net}}} and running these commands:
+
+{{{
+$ cd /share/benito
+$ ./mixed_experiment.py <PID> <EID> <topdl-file>
+}}}
+
+=== Creation ===
+
+Upon successful creation, you will be presented with a URL for accessing your experiment via the web interface.
+
+It takes 1-2 minutes for DETER to create the experiment. The creation process has successfully completed once DETER reports "Status: swapped".
+
+== Swapping ==
+
+=== Swapping in ===
+
+Swap in the DETER experiment like any other: click the "Swap Experiment In" link on the left side of the experiment page.
+
+From DETER's perspective, the swap in process should complete in 3-5 minutes, at which point you will be notified "Experiment successfully swapped in".
+
+After this, it takes another 3-5 minutes to set up the physical nodes and launch the virtual nodes. This process is complete when the "Startup Status" for each node is reported as 0 or 1.
+
+If the startup status is 1, there was an error setting up the physical node as a virtual host. Please let me know if this happens and I can see what caused the problem. You can always try to swap in and out to see if the problem resolves itself.
+
+=== Swapping out ===
+
+Swap the experiment out like any other.
+
+=== Modifying the experiment ===
+
+We do not currently support any form of experiment modification. If you wish to modify your experiment, you must terminate the DETER experiment (not just swap out, but properly terminate) and then start a new containerized experiment using the process outlined above.
+
+== Accessing Nodes ==
+
+Nodes are accessed similarly to other DETER nodes. When you are on users, you can access a node named 'node-1' in YourPID/SomeEID using SSH:
+
+{{{
+users$ ssh node-1.someeid.yourpid
+node-1$ echo hello from qemu
+hello from qemu
+}}}
+
+'''Note:''' This does not yet work with physical nodes in mixed-mode experiments. Sorry!
+
+From within the experiment you can communicate with other nodes in the experiment using just the hostname, as in an ordinary DETER experiment:
+
+{{{
+node-1$ ping node-2
+PING node-2-big-lan (10.0.0.32) 56(84) bytes of data.
+64 bytes from node-2-big-lan (10.0.0.32): icmp_req=1 ttl=64 time=8.01 ms
+64 bytes from node-2-big-lan (10.0.0.32): icmp_req=2 ttl=64 time=10.6 ms
+^C
+--- node-2-big-lan ping statistics ---
+2 packets transmitted, 2 received, 0% packet loss, time 1001ms
+rtt min/avg/max/mdev = 8.015/9.318/10.622/1.307 ms
+}}}
+
+=== Node list ===
+
+The {{{node_list}}} command takes an extra flag {{{-e}}} or {{{--extra}}} to list extra nodes from your experiments. Use it like so:
+
+{{{
+users$ node_list --extra
+YourPID/some-experiment
+    pnode-0001.some-experiment.YourPID / pc003
+    pnode-0000.some-experiment.YourPID / pc038
+  extra nodes:
+    node-1.some-experiment.YourPID
+    node-2.some-experiment.YourPID
+    node-3.some-experiment.YourPID
+    node-4.some-experiment.YourPID
+    node-5.some-experiment.YourPID
+    node-5.some-experiment.YourPID
+}}}
+
+This makes use of the {{{experiment.extra_nodes}}} XML-RPC method. This method takes two arguments: proj and exp, which are PID and EID respectively.
+
+=== Start commands ===
+
+Start commands are similar to Emulab start commands, with a few notable exceptions:
+
+ * start commands inside containers are run at each boot, instead of just once
+ * there is no way to report the exit status of a start command (yet)
+
+You can see the exit status and output of a start command by logging into a node and looking in {{{/var/benito/log/start_command.*}}}. There should be 3 files:
+
+ * {{{start_command.log}}}: general info, including exit status
+ * {{{start_command.out}}}: stdout from the start command
+ * {{{start_command.err}}}: stderr
+
+== Node Status ==
+
+There is a line-based RPC mechanism for query node status. A web interface for this is in active development.
+
+Commands are in the format: {{{command arg1 [arg2 ...]}}}
+
+Replies are in the format: {{{reply_type python-eval'able-string}}}
+
+As an example, here is a status request and reply (reply line-wrapped for readability):
+
+{{{
+status_request DeterTest/ratsnest-medium    
+status_reply {'hv:qemu:pnode-0000': {'cnode17': [None, None], 'cnode20': [None,
+None], 'cnode8': [None, None]}, 'hv:qemu:pnode-0010': {'lnode6': [None, None],
+'lnode7': [None, None]}, 'hv:qemu:pnode-0005': {'lnode11': [None, None],
+'lnode12': [None, None], 'lnode13': [None, None]}, 'hv:qemu:pnode-0004':
+{'lnode4': [None, None], 'lnode1': [None, None]}, 'hv:qemu:pnode-0007':
+{'cnode16': [None, None], 'cnode15': [None, None], 'cnode19': [None, None]},
+'hv:qemu:pnode-0008': {'cnode14': [None, None], 'cnode9': [None, None],
+'cnode18': [None, None]}}
+}}}
+
+Example Python code for parsing the reply:
+
+{{{
+from ast import literal_eval
+type, body = line.split(' ', 1)
+message = literal_eval(body)
+}}}
+
+=== Reply format ===
+
+Reply bodies take the form of a dict of dicts. The key of the first level of dicts is the hypervisor name. The key of the second level is node name. Each node name is a two-item array. The first element is boot status and the second element is start command exit code or qemu process exit code (if dead). Both of these will be None if the hv is unreachable.
+
+Possible boot status:
+
+ * {{{running}}} - qemu has been started but the OS has not booted
+ * {{{booted}}} - the inner OS has fully booted
+ * {{{dead}}} - qemu process has died
+
+=== Caveat ===
+
+'''Note:''' Node status is only available for qemu virtual nodes. If your experiment is mixed mode, there is no status available for pnodes. The only way to query their status is from DETER. If they've booted and their exit code is 0, then they are running.
+
+== What's Missing ==
+
+The following is a list of missing features that will probably cause you some pain and suffering. Actually implementing these features is my highest priority task at the moment, so please bear with me for the time being.
+
+ * Accessing physical nodes via hostname from {{{users}}}
+ * Ability to choose a different OS than Ubuntu 10.10 (Maverick)
+ * Ability to install software using apt on Ubuntu (and any other OS)