Porting Notes

I recently (27-28 Aug 2013) ported the containers system to the EARS testbed. This is a summary of what I did to get containers running.

Codebase

This was largely straightforward. I cloned from the git repository into /share/containers. Files not in there are those that need compiled, the openvz images in /share/containers/images and the qemu image stored on scratch.

Codebase install is completed by running make in /share/containers. This points out that fedd needs to be installed. After this install I made changes to the system (the topdl_converter site variable) one can get away with just installing the deter package from fedd.

Installing fedd also requires installing the devel/py-asn1 and math/metis4 packages - that's even if only the deter data handling package is installed. The full fedd install also pulls in net/py-zsi port which includes textproc/py-xml as a dependency.

Fedd also requires ​libabac, of course. That requires security/xmlsec1 to generate signed XML.

There are ports in the testbed repo to install these, but I did them by hand for no good reason.

Images and Templates

Openvz templates are comparatively small (less than half a GB) so nodes copy them directly from NFS. They go into /share/containers/images .

The qemu image is nearly a full GB and retrieved from a web server over TCP. On DETERlab, that server is scratch. On EARS I used boss.ears.deterlab.net. I created a directory /usr/testbed/www/containers/ and put the image (pangolinbz.img.bz2) into it.

We also needed the testbed image for running CentOS6 for containers. That image was incorrectly configured to run in the new EARS event system model and needed to be upgraded to the new event system. To do that, swap the original image in and:

• download pubsub sources from ​https://www.isi.deterlab.net/downloads/pubsub-0.9.9.zip
• compile and install it (on the experiment node)
  • configure
  • make
  • sudo make install-client
• pull the deter repo onto the node (including an appropriate defs file for it).
• make an object dir to work in (mkdir obj)
• from obj (on the experiment node)
  • ${DETER_REPO_PATH}/configure --with-TBDEFS=${DETER_REPO_PATH}/${DEFS_FILE}
  • make client
  • sudo make client-install
• Image it over the original image.

At this point experiments will swap in, but the containerized nodes point at DETER's standard software repos. If you've made a DETER install with mirrors on a machine called scratch, you're done.

Modifying Container Images

Modifying the openvz images is the easier:

$ cd /share/containers/images
$ mkdir edit
$ tar -C edit -xzvf ${TEMPLATE}.tar.gz
# make changes to the tree in edit (this is a comment, not a root shell)
$ tar -C edit -czvf ${TEMPLATE} .

(That's a dot at the end of the second tar command).

To modify the qemu image. Swap in a containerized qemu experiment. This is to get a machine with the qemu tools on it. You can acquire such a machine any way you want, but swapping in does it directly.

Log into that experiment node, and pick a local directory on a filesystem with a few gigs of space. I used /tmp.

Now on that machine:

$ wget http://boss.ears.deterlab.net/containers/pangolinbz.img.bz2
$ bunzip2 pangolinbz.img.bz2
$ qemu-nbd -P 1 -p 1024 ./pangolinbz.img &
$ nbd-client localhost 1024 /dev/nbd0
$ mount -o noatime /dev/nbd0 /mnt
# make changes to filesystem in /mnt (this is a comment, not a root shell)
$ umount /mnt
$ nbd-client -d /dev/nbd0
# that should kill the qemu-nbd process.  If not you can kill it by hand)
$ bzip2 ./pangolinbz.img
# ship ./pangolinbz.img where it needs to go 

Specific EARS changes to containers

The specific changes for EARS were to change /etc/environment to include

http_proxy="http://192.168.253.1:8888"

And to put standard sources.list for the various ubuntu distributions in /etc/apt

For the 10.04 based distros, add a file called /etc/apt/apt.conf.d/proxy that contains:

Acquire::http::Proxy "http://192.168.253.1:8888";

Note the trailing semi-colon.

On the qemu image, keys for the TOR repo has to be added to /etc/apt/trustdb.gpg and /etc/apt/trusted.gpg

Those detail will vary from install to install.