Version 5 (modified by Ted Faber, 12 years ago) (diff)


Reference Guide

This document describes the details of the commands and data structures that make up the containers system. The User Guide /Tutotial? provides useful context about the workflows and goals of the system that inform these technical details.

The command creates a DETER experiment made up of containers. The program is available from /share/containers/ on A sample invocation is:

$ /share/containers/ MyProject MyExperiment ~/mytopology.tcl

It will create a new experiment in MyProject called MyExperiment containing the experiment topology in mytopology.tcl. All the topology creation commands supported by DETER are supported by the conatainerization system, but emulab/DETER program agents are not. Emulab/DETER start commands are supported.

Containers will create an experiment in a group if the project parameter is of the form project/group. To start an experiment in the testing group of the DETER project, the first parameter is specified as DETER/testing.

Either an ns2 file or a topdl description is supported. Ns2 descriptions must end with .tcl or .ns. Other files are assumed to be topdl descriptions.

By default, program will partition the topology into openvz containers, packed 10 containers per physical computer. If the topology is already partitioned - at least one element has a conatiners::partition atttribute - will not partition it. The --force-partition flag causes to partition the experiment regardless of the presence of containers:partition attributes.

If container types have been assigned to nodes using the containers:node_type attribute, will respect them. Valid container types for the containers:node_type attribute or the --default-container parameter are:

Parameter Container
embedded_pnode Physical Node
qemu Qemu VM
openvz Openvz Container
process ViewOS process

The command takes several parameters that can change its behavior:

Containerize nodes without a container type into kind. If no nodes have been assigned containers, this puts all them into kind containers.
Partition the experiment whether or not it has been paritioned already
Attempt to put int containers into each physical node. The default --packing is 10.
Read configuration variables from filename the configuration values are discussed below.
Override the site configuration and request nodes of type1 (or type2 etc.) as host nodes.
Attempt to do end node traffic shaping even in containers connected by VDE switches. This works with qemu nodes, but not process nodes. Topologies that include both openvz nodes and qemu nodes that shape traffic should use this. See the discussion below.
Do traffic shaping in VDE switches. Probably the default, but that is controlled in the site configuration. See the discussion below.
Set the default openvz disk space size. The suffixes G and M stand for gigabytes and megabytes.
Set the default openvz template. Templates are described in the users guide.
Construct a visualization of the virtual topology and leave it in the experiment directories (default)
Do not construct a visualization of the virtual topology and leave it in the experiment directories
Print additional diagnostics and leave failed DETER experiments on the testbed
Do not remove temporary files - for debugging only

This invocation:

$ ./ --packing 25 --default-container=qemu --force-partition DeterTest faber-packem ~/experiment.xml

takes the topology in ~/experiment.xml (which must be topdl), packs it into 25 qemu containers per physical node, and creates an experiment called DeterTest/faber-packem that can be swapped in. If experiment.xml were already partitioned, it will be re-partitioned. If some nodes in that topology were assigned to openvz nodes already, those nodes will be still be in openvz containers.

The result of a successful run is a DETER experiment that can be swapped in.

More detailed examples are available in the tutorial

Site Configuration File

The site configuration file is an attribute value pair file parsed by a python ConfigParser that sets overall container parameters. Many of these have legacy internal names.

The default site configuration is in /share/containers/site.conf on

Acceptable values (and their DETER defaults) are:

The IRC server used as a backend coordination service for grandstand. Will be replaced by MAGI. Default:
Port that third party applications can contact grandstand on. Will be replaced by MAGI. Default: 4919
Default image used by qemu containers. Default: http://scratch/benito/pangolinbz.img.bz2
Base URL of the DETER web interface on which users can see experiments. Default:
Hardware used by containers. Default: pc2133,bpc2133,MicroCloud
Host and port from which to request experiment creation. Default:
OSID to request for qemu container nodess. Default: Ubuntu1204-64-STD
Root of the directory tree holding containers software and libraries. Developers often change this. Default: /share/containers
OSID to request for openvz nodes. Default CentOS6-64-openvz
Location to load the openvz template from. Default: %(exec_root)s/images/ubuntu-10.04-x86.tar.gz
True if switched containers (see below) should do traffic shaping in the VDE switch that connects them. Default: true
A list of the containers that are networked with VDE switches. Default: qemu,process
The directory that stores openvz template files. Default: %(exec_root)s/images/ (that is the images directory in the exec_root directory defined in the site config file.

Container Notes

Different container types have some quirks. This section lists limitations of each container.


Openvz containers use a custom OS image to support their virtualization. They cannot share physical resources with other containers. A physical node holding openvz containers holds only openvz containers. They are interconnected with one another through bridges and kernel virtual networking rather than through VDE switches (as qemu and process containers are)). As a result, openvz containers provide network delays using per-container endpoint traffic shaping. This means that they cannot correctly interconnect with traffic sphped qemu nodes.

Interconnections: VDE switches and local networking

The various containers are interconnected using either local kernel virtual networking or VDE switches. Kernel networking is lower overhead because it does not require process context switching, but VDE switches are a more general solution.

Network behavior changes - loss, delay, rate limits - are introduced into a network of containers using one of two mechanisms: inserting elements into a VDE switch topology or end node traffic shaping. Inserting elements into the VDE switch topology allows the system to modify the behavior for all packets passing through it. Generally this means all packets to or from a host, as the container system inserts these elements in the path between the node and the switch.

This figure shows 3 containers sharing a virtual LAN (VLAN) on a VDE switch with no traffic shaping:

Unshaped traffic through a VDE switch

The blue containers connect to the switch and the switch has interconnected their VDE ports into the red shared VLAN. To add delays to tow of the nodes on that VLAN, the following VDE switch configuration would be used:

Traffic sahping using VDE

The VDE switch connects the containers with shaped traffic to the delay elements, not to the shared VLAN. The delay elements are on the VLAN and delay all traffic passing through them. The container system configures the delay elements to delay traffic symmetrically - traffic from the LAN and traffic from teh container are both delayed. The VDE tools can be configured asymmetrically as well. This is a very flexible way to interconnect containers.

That flexibility incurs a cost in overhead. Each delay element and the VDE switch is a process, do traffic passing from one delayed nodes to the other experiences 7 context switches: container -> switch, switch -> delay, delay -> switch, switch -> delay, delay -> switch, and switch -> container.

The alternative mechanism is to do the traffic shaping inside the nodes, using linux traffic shaping. In this case, traffic outbound from a container is delayed in the container for the full transit time to the next hop. The next node does the same. End-node shaping all happens in the kernel so it is relatively inexpensive at run time.

Qemu nodes can make use of either end-node shaping or VDE shaping, and use VDE shaping by default. The --end-node-shaping and --vde-switch-shaping options to forces the choice in qemu.

ViewOS processes can only use VDE shaping. Their network stack emulation is not rich enough to include traffic shaping.

Openvz nodes only use end-node traffic shaping. They have no native VDE support so interconnecting openvz containers to VDE switches would include both extra kernel crossings and extra context switches. Because a primary attraction of VDE switches is their efficiency, the containers system does not implement VDE interconnections to openvz.

Similarly embedded physical nodes use only endnode traffic shaping, as routing outgoing traffic through a virtual switch infrastructure that just connects to its physical interfaces is at best confusing.

Unfortunately, endnode traffic shaping and VDE shaping are incompatible.

This is primarily of academic interest, unless a researcher wants to impose traffic shaping between containers using incompatible traffic shaping. There needs to be an unshaped link between the two kinds of traffic shaping.

Attachments (3)

Download all attachments as: .zip