Changes between Version 1 and Version 2 of ContainerFilesDoc


Ignore:
Timestamp:
Feb 12, 2015 8:47:09 AM (10 years ago)
Author:
Geoff Lawler
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • ContainerFilesDoc

    v1 v2  
    1 Sample output from containerize:
     1This wiki page documents the files created by the DETER containerization system.
     2
     3The sample experiment is Deter,muppetnet. The NS file and containerization command are:
    24
    35{{{
    4 -rw-r--r--  1 glawler  SAFER  11740 Feb 12 07:05 annotated.xml         
    5 -rw-r--r--  1 glawler  SAFER    151 Feb 12 07:05 assignment           
    6 -rw-r--r--  1 glawler  SAFER     68 Feb 12 07:05 backend_config.yaml   
    7 drwxr-xr-x  2 glawler  SAFER   1024 Feb 12 07:05 children             
    8 -rw-r--r--  1 glawler  SAFER   7784 Feb 12 07:05 config.tgz           
    9 -rw-r--r--  1 glawler  SAFER   7591 Feb 12 07:05 converted.xml         
    10 -rw-r--r--  1 glawler  SAFER   2563 Feb 12 07:05 embedding.yaml       
    11 -rw-r--r--  1 glawler  SAFER   1396 Feb 12 07:05 experiment.tcl       
    12 -rw-r--r--  1 glawler  SAFER      0 Feb 12 07:05 ghosts               
    13 -rw-r--r--  1 glawler  SAFER    793 Feb 12 07:05 hosts                 
    14 -rw-r--r--  1 glawler  SAFER      6 Feb 12 07:05 hostsw.pickle         
    15 -rw-r--r--  1 glawler  SAFER   1403 Feb 12 07:05 maverick_url         
    16 -rw-r--r--  1 glawler  SAFER   1803 Feb 12 07:05 openvz_guest_url     
    17 -rw-r--r--  1 glawler  SAFER  13217 Feb 12 07:05 partitioned.xml       
    18 -rw-r--r--  1 glawler  SAFER    816 Feb 12 07:05 phys_topo.ns         
    19 -rw-r--r--  1 glawler  SAFER   1120 Feb 12 07:05 phys_topo.xml         
    20 -rw-r--r--  1 glawler  SAFER     16 Feb 12 07:05 pid_eid               
    21 -rw-r--r--  1 glawler  SAFER     54 Feb 12 07:05 pnode_types.yaml     
    22 -rw-r--r--  1 glawler  SAFER     10 Feb 12 07:05 qemu_users.yaml       
    23 drwxr-xr-x  2 glawler  SAFER    512 Feb 12 07:05 route                 
    24 -rw-r--r--  1 glawler  SAFER    205 Feb 12 07:05 shaping.yaml         
    25 -rw-r--r--  1 glawler  SAFER    732 Feb 12 07:05 site.conf             
    26 drwxr-xr-x  2 glawler  SAFER    512 Feb 12 07:05 software             
    27 drwxr-xr-x  2 glawler  SAFER    512 Feb 12 07:05 switch               
    28 -rw-r--r--  1 glawler  SAFER    707 Feb 12 07:05 switch_extra.yaml     
    29 -rw-r--r--  1 glawler  SAFER      6 Feb 12 07:05 swlocs.pickle         
    30 -rw-r--r--  1 glawler  SAFER  40008 Feb 12 07:05 topo.xml             
    31 -rw-r--r--  1 glawler  SAFER   3136 Feb 12 07:05 traffic_shaping.pickle
    32 -rw-r--r--  1 glawler  SAFER  61375 Feb 12 07:05 visualization.png     
    33 -rw-r--r--  1 glawler  SAFER      3 Feb 12 07:05 wirefilters.yaml     
     6set ns [new Simulator]
     7source tb_compat.tcl
     8
     9set opt(netSpeed) 100Mb
     10
     11set allNodes "beaker beauregard brewster drbunsenhoneydew floyd lips mousey rizzo pops kermit piggy fozzie drteeth animal janice"
     12set studNodes "beaker beauregard brewster drbunsenhoneydew kermit piggy fozzie"
     13set mayhemNodes "floyd lips mousey rizzo pops drteeth animal janice"
     14
     15foreach node $allNodes {
     16    set $node [$ns node]
     17}
     18
     19# link between LANS
     20set rainbow [$ns duplex-link "fozzie" "janice" 100Mb 0.0ms DropTail]
     21
     22# two LANs
     23set studebaker [$ns make-lan $studNodes $opt(netSpeed) 0.0ms]
     24set electricmayhem [$ns make-lan $mayhemNodes $opt(netSpeed) 0.0ms]
     25
     26$ns rtproto Static
     27$ns run
    3428}}}
     29
     30{{{
     31[glawler@users:~/src/nsfiles]$ /share/containers/containerize.py DETER muppetnet ./muppetnet.ns
     32}}}
     33
     34This creates the directory {{{/proj/Deter/exp/muppetnet}}} which contains a {{{containers}}} directory. This directory has the following files:
     35
     36{{{
     37-rw-r--r--  1 glawler  Deter   9668 Feb 12 07:25 annotated.xml
     38-rw-r--r--  1 glawler  Deter    142 Feb 12 07:25 assignment
     39-rw-r--r--  1 glawler  Deter     68 Feb 12 07:25 backend_config.yaml
     40drwxr-xr-x  2 glawler  Deter   1024 Feb 12 07:25 children
     41-rw-r--r--  1 glawler  Deter   7246 Feb 12 07:26 config.tgz
     42-rw-r--r--  1 glawler  Deter   5660 Feb 12 07:25 converted.xml
     43-rw-r--r--  1 glawler  Deter   2432 Feb 12 07:25 embedding.yaml
     44-rw-r--r--  1 glawler  Deter    662 Feb 12 07:25 experiment.tcl
     45-rw-r--r--  1 glawler  Deter      0 Feb 12 07:25 ghosts
     46-rw-r--r--  1 glawler  Deter    793 Feb 12 07:26 hosts
     47-rw-r--r--  1 glawler  Deter      6 Feb 12 07:26 hostsw.pickle
     48-rw-r--r--  1 glawler  Deter   1315 Feb 12 07:25 maverick_url
     49-rw-r--r--  1 glawler  Deter   1690 Feb 12 07:25 openvz_guest_url
     50-rw-r--r--  1 glawler  Deter  11063 Feb 12 07:25 partitioned.xml
     51-rw-r--r--  1 glawler  Deter    816 Feb 12 07:25 phys_topo.ns
     52-rw-r--r--  1 glawler  Deter   1120 Feb 12 07:25 phys_topo.xml
     53-rw-r--r--  1 glawler  Deter     16 Feb 12 07:25 pid_eid
     54-rw-r--r--  1 glawler  Deter     54 Feb 12 07:25 pnode_types.yaml
     55-rw-r--r--  1 glawler  Deter     10 Feb 12 07:25 qemu_users.yaml
     56drwxr-xr-x  2 glawler  Deter    512 Feb 12 07:26 route
     57-rw-r--r--  1 glawler  Deter    205 Feb 12 07:26 shaping.yaml
     58-rw-r--r--  1 glawler  Deter    732 Feb 12 07:25 site.conf
     59drwxr-xr-x  2 glawler  Deter    512 Feb 12 07:26 software
     60drwxr-xr-x  2 glawler  Deter    512 Feb 12 07:26 switch
     61-rw-r--r--  1 glawler  Deter    708 Feb 12 07:26 switch_extra.yaml
     62-rw-r--r--  1 glawler  Deter      6 Feb 12 07:26 swlocs.pickle
     63-rw-r--r--  1 glawler  Deter  36759 Feb 12 07:26 topo.xml
     64-rw-r--r--  1 glawler  Deter   3136 Feb 12 07:26 traffic_shaping.pickle
     65-rw-r--r--  1 glawler  Deter  66961 Feb 12 07:26 visualization.png
     66-rw-r--r--  1 glawler  Deter      3 Feb 12 07:26 wirefilters.yaml
     67}}}
     68
     69Notes on these files:
     70
     71**annotated.xml** - This is an XML file that describes the nodes and substrates (networks) of the experiment. The node description includes name, interfaces, container type, and node type (pc, etc). 
     72
     73**assignment** - Shows container assignment to physical node. e.g. node "foo" runs on physical node 1.
     74
     75**backend_config.yaml** - "{server: 'boss.isi.deterlab.net:6667', unique_id: fdb56b6d989ff6a4}"
     76
     77**children** - Not sure. Is a directory that has files which describe a DAG - relationships between pnodes, openvz, and container nodes. Mostly empty files in this experiment though as the network topology is minimal.
     78
     79**config.tgz** - This directory, {{{./containers}}} tarred and gzipped.
     80
     81**converted.xml** - the same as {{{annotated.xml}}} minus the container information added.
     82
     83**embedding.yaml** - container and pnode relationships in YAML.
     84
     85**experiment.tcl** - The NS file which generated the experiment.
     86
     87**ghosts** - empty file
     88
     89**hosts** - {{{/etc/hosts}}} which includes the container nodes.
     90
     91**hostsw.pickle** - ??? {{{(dp0\n.}}} Looks like a pickle file (binary python marhshalling format), but does not "un-pickle" into any data. Hits EOF.
     92
     93**maverick_url** - URLs to container images. Default location is {{{http://scratch/benito/...}}}. Used for qemu images, I think.
     94
     95**openvz_guest_url** - URLs to openvz images. Default location {{{/share/containers/images/...}}}
     96
     97**partitioned.xml** - Another XML file that is used in the partitioning process. See {{{annotated.xml}}}
     98 
     99**phys_topo.ns**, **phys_topo.xml** - physical NS file/topology/nodes of experiment.
     100
     101**pid_eid** - group and experiment ID of experiment
     102
     103**pnode_types.yaml** - The machine types of the pnodes, "pc2133, bpc2133, MicroCloud", in YAML.
     104
     105**qemu_users.yaml** - list of users for qemu nodes? Just contains {{{[glawler]}}} for this experiment, so it's not an exhaustive list...
     106
     107**route** - routing info for containers. directory with one file per node, showing route info. e.g. {{{10.0.0.0/8 10.0.1.7}}}.
     108
     109**shaping.yaml** - traffic shaping for container networks.
     110
     111**site.conf** - configuration settings for containers: example:
     112{{{
     113[containers]
     114switch_shaping = true
     115qemu_host_hw = pc2133,bpc2133,MicroCloud
     116openvz_guest_url = %(exec_root)s/images/ubuntu-10.04-x86.tar.gz
     117exec_root = /share/containers
     118qemu_host_os = Ubuntu1204-64-STD
     119openvz_host_os = CentOS6-64-openvz
     120maverick_url = http://scratch/benito/pangolinbz.img.bz2
     121xmlrpc_server = boss.isi.deterlab.net:3069
     122url_base = http://www.isi.deterlab.net/
     123grandstand_port = 4919
     124backend_server = boss.isi.deterlab.net:6667
     125openvz_template_dir = %(exec_root)s/images/
     126attribute_prefix = containers
     127switched_containers = qemu,process
     128}}}
     129
     130**software** - empty directory
     131
     132**switch** - directory of per switch node (pnodes) files. Each file describes vlan creation, numbering, and switch port/vlan assignment for containers on that node. Does this get fed to the {{{vde_switch}}es on the pnodes?
     133
     134**switch_extra.yaml** - YAML file which looks like it contains information about gluing together the VDE switches - extra interfaces (taps), double pipes (attaching switches together I think), and tap pipes (for attaching tap ifaces to the switch)
     135
     136**swlocs.pickle** - another python pickle file. But hits EOF before it is decoded. No idea what this is for.
     137
     138**topo.xml** - Containers topology in YAML format. Has substrates (networks). For nodes it shows interfaces and addressing (ip/MAC) as well as which {{{vde_switch}}} the node is attached to. Nodes also contain some VM info like disk size, template/image path/name, vhost.
     139
     140**traffic_shaping.pickle** - Looks like, oddly, traffic shaping information in the python pickle format:
     141
     142{{{
     143>>> import pickle
     144>>> import pprint
     145>>> fd = open('traffic_shaping.pickle', 'rb')
     146>>> d = pickle.load(fd)
     147>>> pprint.pprint(d)
     148>>> pprint.pprint(d)                                                                                                                                                                                                                         
     149{u'animal': {u'10.0.0.1': {'bandwidth': None, 'delay': None, 'loss': None},
     150             u'10.0.0.2': {'bandwidth': None, 'delay': None, 'loss': None},
     151             u'10.0.0.3': {'bandwidth': None, 'delay': None, 'loss': None},
     152             u'10.0.0.4': {'bandwidth': None, 'delay': None, 'loss': None},
     153             u'10.0.0.5': {'bandwidth': None, 'delay': None, 'loss': None},
     154             u'10.0.0.6': {'bandwidth': None, 'delay': None, 'loss': None},
     155             u'10.0.0.8': {'bandwidth': None, 'delay': None, 'loss': None}},
     156...
     157 u'rizzo': {u'10.0.0.1': {'bandwidth': None, 'delay': None, 'loss': None},
     158            u'10.0.0.2': {'bandwidth': None, 'delay': None, 'loss': None},
     159            u'10.0.0.3': {'bandwidth': None, 'delay': None, 'loss': None},
     160            u'10.0.0.5': {'bandwidth': None, 'delay': None, 'loss': None},
     161            u'10.0.0.6': {'bandwidth': None, 'delay': None, 'loss': None},
     162            u'10.0.0.7': {'bandwidth': None, 'delay': None, 'loss': None},
     163            u'10.0.0.8': {'bandwidth': None, 'delay': None, 'loss': None}}}
     164}}}
     165
     166**visualization.png** - image of the topology of the container networks.
     167
     168**wirefilters.yaml** - empty YAML file: {{{ {} }}}