The corporate answer for how to build Software for Dom0 is to do so inside the XenServer Driver Development Kit (DDK), a virtual appliance.
This post will show you a different approach. It shows how the DDK virtual appliance can be converted into a Docker base image. This alternative is useful whenever the continuous integration (CI) workflow or automated build system don’t allow for the use of the fully fleshed virtual appliance or when building on an existing Linux environment is favoured.
But let’s start with a brief introduction.
When creating software for XenServer, the number one choice is not to integrate the software into XenServer’s heart, but to develop it as an independent virtual appliance. This way there are fewer dependencies, fewer restrictions and often the same virtual appliance can be used across different hypervisor versions and even hypervisor products.
Sometimes, though, the use of a virtual appliance is not an option. For example, device drivers require direct access to both the physical hardware and the privileged virtualization stack. In such cases, the software must be integrated into XenServer’s control domain–the so called Dom0.
Dom0 is essentially a modified and extended variant of CentOS. “Variant” means that it comes with its own custom kernel, libraries and is built using its own toolchain. As with other Linux distributions, when building software for Dom0, you’ll almost always need to use the very same toolchain that was used for building Dom0 itself. The toolchain is what the DDK virtual appliance provides.
The following steps walk through how to convert the DDK virtual appliance into a Docker base image, that can be used for building software for XenServer in a Docker container. Building Software in a container may be preferred, if there is existing CI workflows or build environments in which containers are deemed a better fit due to their inherently different deployment properties and dependencies.
DDK Docker Containers can be monitored and managed using XenServer’s Container Management functionality just like any other Docker Container. If you haven’t heard about XenServer’s Container management functionality, I’d highly recommend you to read Marc’s blog post about XenServer 6.5 SP1.
One or two words of warning: Citrix restricts what is supported in Dom0. Third party software has to pass stringent certification tests to be supported. Please get in touch with Citrix before integrating your software into Dom0 if you’re hoping to see it supported by Citrix. Also check out the Citrix Ready program.
That being said, it’s time to get going.
1) Preparing the dependencies
First of all, let’s prepare a Linux development environment. In the following we will use kpartx for handling loop devices, qemu-img for converting images and wget for downloading files.
We also need docker and possibly chroot, depending on what we want to do with the converted DDK.
If you’re using RHEL, CentOS, Oracle Linux or Fedora, you can install the dependencies using:
sudo yum install kpartx qemu-img wget coreutils docker If you’re using Ubuntu or Debian:
sudo apt-get install kpartx qemu-utils wget coreutils docker.io
Next we need download the DDK for the build of XenServer that we want to build software for. The DDK is available with the other XenSever Downloads.
Let’s download the DDK for XenServer 6.5:
2) Extracting the raw disk image from the DDK-iso
The DDK comes as an ISO-image. Let’s mount the ISO:
TEMPISOMNT=./tempisomnt mkdir $TEMPISOMNT sudo mount -o loop XenServer-6.5.0-DDK.iso $TEMPISOMNT
The ISO contains LZ77 compressed chunks which together make up the vhd disk image.
Let’s decompress the chunks into the file xvda.vhd:
( for CHUNK in $TEMPISOMNT/ddk/xvda/*; do zcat $CHUNK; done; ) | dd of=xvda.vhd sudo umount $TEMPISOMNT rmdir $TEMPISOMNT rm XenServer-6.5.0-DDK.iso
Admittedly vhd-images aren’t the most usable image-format on Linux, so we’ll convert the vhd image into a raw image next:
qemu-img convert xvda.vhd -O raw xvda.raw rm xvda.vhd
3) Accessing the raw DDK disk image
Now the heavy lifting is done, we just need a way to access the partitions inside the raw image.
An easy solution is kpartx. kpartx understands the partition layout inside and maps the partitions in the image to individual loop block devices:
DDKDISKFIRSTPARTITIONDEVICE=$(sudo kpartx -av xvda.raw | cut -d" " -f 3)
Once we’ve got access to the first partition, we can mount it:
DDKROOT=./ddkpartition mkdir $DDKROOT sudo mount "/dev/mapper/$DDKDISKFIRSTPARTITIONDEVICE" $DDKROOT
4) Using the extracted DDK
For a quick smoke-test, you can now chroot into the DDK:
sudo chroot $DDKROOT exit
However, what we’d really like to do, is to package the DDK into a Docker base image. Docker has a very useful tar import function for this purpose:
sudo tar -C $DDKROOT -c . | docker import - xs65ddk
Subsequently we can spin up a Docker container using the base image:
docker run -i -t xs65ddk /bin/bash
That’s it. If you’ve already set up Container Management for your Linux VM, the container will appear in XenCenter – where you can monitor and manage it as usual.
Both Docker Containers and Virtual Machines have their individual strengths. Containers are very lightweight while Virtual Machines promise isolation. I believe that both technologies are complementary.
One that note, it is worth mentioning one slight caveat associated with running the DDK as a container. When the DDK is run in a container, the XenServer Kernel will not be used. This means that `uname -r` will not return the DDK/DOM0 kernel, but instead the Kernel of your Linux machine.
It may be required to amend build-scripts, to prepare them for building in such an environment. This is the case with the examples that are included inbox in the DDK, as they’re designed to build a kernel module that matches the running Kernel. Not running the XenServer Kernel also means that built Kernel Modules can not be inserted and tested in the container.
I hope this post helps to get you going. Please let me know, if you have any feedback regarding the above, or with respect to software development for XenServer in general.
Would it help, if I provided the DDK base image via Docker Hub?