Building images on Windows

[fralalonde]

I have setup Capstan on Windows and can run existing images without problem using VirtualBox. I am now trying to make my own image using osv-apps tomcat-benchmark as a template for my own war application image. Capstan keeps looking for qemu-nbd even when I pass in -p vbox.

$ capstan build -p vbox company/service-name
Building company/service-name...
exec: "qemu-nbd": executable file not found in %PATH%

I am using Cygwin to provide me with bash and make, but I get the same result if running capstan from the regular cmd.exe shell. I could try building qemu in cygwin, but expect the performance to be abysmal and would rather keep using native VirtualBox.

[dorlaor]

On Mon, Oct 5, 2015 at 6:06 PM, Francis Lalonde [email protected]
wrote:

I have setup Capstan on Windows and can run existing images without
problem using VirtualBox. I am now trying to make my own image using
osv-apps tomcat-benchmark as a template for my own war application image.
Capstan keeps looking for qemu-nbd even when I pass in -p vbox.

$ capstan build -p vbox company/service-name
Building company/service-name...
exec: "qemu-nbd": executable file not found in %PATH%

I am using Cygwin to provide me with bash and make, but I get the same
result if running capstan from the regular cmd.exe shell. I could try
building qemu in cygwin, but expect the performance to be abysmal and would
rather keep using native VirtualBox.

Indeed Windows is quite unorthodox choice for Capstan. In the VM build
process
we stream bunch of files onto a new VM for the creation of the initial file
system (within
the image). There is not reason it won't work on windows/Cygwin/qemu but I
think it will be
painful somehow.

—
Reply to this email directly or view it on GitHub
#148.

[fralalonde]

What missing part is preventing VirtualBox on Windows from being used to prepare the image? Can images be built on Mac using vbox? I imagine that since the image is a ZFS volume, OSv needs to run and mount it so it can be written to. Then some external mechanism (Capstan) sends the kernel the files. Or does Capstan write-mounts qcow-zfs itself before booting the kernel? Sorry if I am spouting nonsense, I'm just trying to understand how this works (along with the rest of OSv which I find pretty interesting).

Also, as much as I would like to get a Linux laptop, corporate world can be pretty stiff. And I have to admit Windows does a fair job for Java development.

[dorlaor]

On Mon, Oct 5, 2015 at 8:29 PM, Francis Lalonde [email protected]
wrote:

What missing part is preventing VirtualBox on Windows from being used to
prepare the image? Can images be built on Mac using vbox? I imagine that
since the image is a ZFS volume, OSv needs to run and mount it so it can be
written to. Then some external mechanism (Capstan) sends the kernel the
files. Or does Capstan write-mounts qcow-zfs itself before booting the
kernel? Sorry if I am spouting nonsense, I'm just trying to understand how
this works (along with the rest of OSv which I find pretty interesting).

Basically everything can be done on non-Linux since the OSv guest mounts
ZFS from within the VM. There are lots of small details which will be
needed to be tackled like the net-cast
('nc') side that sends the files on tcp port using qemu networking mode.
All solvable but not sure the effort worth it.

Also, as much as I would like to get a Linux laptop, corporate world can
be pretty stiff. And I have to admit Windows does a fair job for Java
development.

—
Reply to this email directly or view it on GitHub
#148 (comment)
.

[fralalonde]

So maybe this thread should be filed as a possible enhancement then. I'll see what I can do around (or though) this limitation for now. Thanks!

[seriousme]

Just out of curiosity: I'm trying to build node.js VM's on windows using virtualbox.
Running OSv images works fine, building them however is not working. (you might want to specify this in the Readme.md as its not really clear)

Looking at https://github.com/cloudius-systems/capstan/blob/master/cmd/build.go it looks to me that the base image is being started, then capstan tries to make a connection on port 10000 and uploads the files there that are specified in the capstanfile.

If that is the case: what is then the use of the qemu-nbd ? From what I understand qemu-nbd enables one to mount the image file as a filesystem, but since the modifications are not done via the filesystem I don't understand why there is a requirement to do this (and to require qemu tools if building for virtual box :-))

Any explanation of the process would be much appreciated.

[nyh]

This is an interesting point. Both Capstan and OSv's source code build system (scripts/build) use qemu-nbd to modify the place on the image where the command line is stored. However, there is really no reason why we must to do that - if we already run the image to upload files, we could have modified the command line from within the image. We even have a function to do that - save_cmdline() from <osv/commands.hh>.

So Capstan could be modified not to use qemu-nbd at all, unless I'm missing something.

[seriousme]

Did some further research :-)

By default the images in the Capstan repository have no commandline defined.
The first thing the build command does is modify the image to start /tools/cpiod on startup.
Then the VM is started, the files are being copied, vm is stopped and finally the image is modified again to put the template Cmdline in.

Now if the build images in the capstan repo would already have /tools/cpiod defined as their default commandline, then all the rest could be done via cpiod and there would be no need to use qemu-ndb.

Alternatively I found that the cloudius/osv image has the dashboard enabled on port 8000 (I added a NAT for that in my vbox config).
This dashboard offers a REST api to upload files (post /file/{path-par}) and change the commandline (put /os/cmdline).

[seriousme]

I just noticed that the REST api is also available via the cli cmdline, so if the build images would have /cli/cli.so as startup command there would be no need for either cpiod nor qemu-nbd imho as one cloud use either stdio or http to talk to the api.

[dorlaor]

On Mon, Oct 19, 2015 at 9:44 PM, seriousme [email protected] wrote:

I just noticed that the REST api is also available via the cli cmdline
https://github.com/cloudius-systems/osv/wiki/Command-Line-Interface-(CLI),
so if the build images would have /cli/cli.so as startup command there
would be no need for either cpiod nor qemu-nbd imho as one cloud use either
stdio or http to talk to the api.

The base image may not need anything else, it's very small image since it
supposed to
live in the ram disk

—
Reply to this email directly or view it on GitHub
#148 (comment)
.

[seriousme]

Sounds logical :-) However capstan currently assumes the presence of cpiod. I don't know about the size of cli vs the size of cpiod nor about any possibility to remove packages at build time.
One might not need the whole cli, just an extension to cpiod to be able to set the startup_cmd would also work.

Then again I'm just passing by and puzzling a bit about how to make the process as portable as possible, hope I'm not wasting anybody's time :-).

[fralalonde]

You are certainly not wasting mine, your research is enlightening.

[seriousme]

@fralalonde thanks, I'm learning a lot here :-)

Just been digging in the lua code used by the cli module.

As far as I can see It basically parses the command and then passes it on to the libhttpserver which serves the API over https. Here is an example of an api command.

This means that you can also run the cli command from the host instead of the guest. The OSV command however is not being build for windows as far as I know, but that should be fixable as well ;-)

Then the size: the libhttpserver is quite a beast at 17 Mb, especially since cpiod only comes in at 230Kb. Now if you already use the libhttpserver (e.g. for the dashboard etc) that is not so much of an issue. If not 17 Mb is quite a chunk of the 40Mb that the df command reports as being present on the blockdevice. The cli module itself weighs nearly nothing.

Lua is however also embedable (in fact OSv already embeds lua) so instead of using the https route, one could also connect lua directly to the api commands in C code which should result in a far smaller binary than the 17 Mb of libhttpserver.

Another trick to reduce size might be to start out with a "big base image" with cli + http server enabled and then as a last build step: order it to remove the build tools from the image, however I'm not sure whether that will shrink the image or leave just blank space in there. It would also mean that you can't extend a build image anymore if the build tools are gone. (you would need to do a rebuild without the cleanup first).

But then again: if everybody is building on Linux with qemu-nbd present then I might be puzzling at a problem that nobody encounters ;-)

[fralalonde]

By choice or by force, there are a lot more devs out there on OS X and Windows than there are on Linux. Most end up using a local Linux VM for development to do Docker stuff. Docker uses Linux containers, there's no way around it. But OSv has the potential to run straight out of VirtualBox, which would make a native development solution practical. Also, the local-Linux VM trick doesn't work with OSv, since VirtualBox does not support launching VMs from from VMs. I am trying to go around it with VMWare Workstation but ultimately that's just digging a more expensive hole.

I believe using the CLI API is the way to go. Your findings about the libhttpserver size are also disappointing, it makes the whole web console thing only usable for demos.

[nyh]

On Mon, Oct 19, 2015 at 5:42 PM, seriousme [email protected] wrote:

Did some further research :-)

By default the images in the Capstan repository have no commandline
defined.
The first thing the build command does is modify the image to start
/tools/cpiod on startup.
Then the VM is started, the files are being copied, vm is stopped and
finally the image is modified again to put the template Cmdline in.

Now if the build images in the capstan repo would already have
/tools/cpiod defined as their default commandline, then all the rest could
be done via cpiod and there would be no need to use qemu-ndb.

This is an accurate analysis of the situation.

As you suggested, we can easily modify scripts/build-capstan-img to
generate base images which already have the appropriate cpiod command line
which Capstan excepts.

Then, to avoid the second use of qemu-nbd for setting the command line of
the new image, we can add a feature to cpiod (or a new executable?) which
reads the command line from a socket and sets it with osv::save_cmdline().

By the way, in the long run, I don't think there's any reason why the
default command line shouldn't just be a file (or even better, files in a
directory) in the filesystem, uploaded like the rest of the files to the
image. There is some justification to write kernel boot-time options
outside the file system, but the actual command line of what to run - I
think this should be in files. We already have something similar (but not
identical) - the "/init" directory (see loader.cc).

Other suggestions, like using httpd or cli are, in my opinion, an overkill,
as those are much bigger programs which many people will not want to
include in their images.

[nyh]

On Mon, Oct 19, 2015 at 11:04 PM, seriousme [email protected]
wrote:

Then the size: the libhttpserver is quite a beast at 17 Mb, especially
since cpiod only comes in at 230Kb.

Please note that this 17MB size is not stripped, and the binaries are
stripped just before uploading them to the image. After stripping,
libhttpserver is "just" 1.1 MB, and cpiod is 225 KB.

cpiod can most likely be made even smaller, significantly, if it didn't
make unnecessary use of bloated Boost libraries, and just used the normal
Posix APIs for sockets, filesystem and option parsing. Run "nm -D" on cpiod
to get a feeling how Boost's extraordinarily long type names inflates the
executable :-(

If someone were to rewrite cpiod to be less bloated I would be very happy,
personally.

Lua is however also embedable (in fact OSv already embeds lua
https://github.com/cloudius-systems/osv/blob/master/modules/cli/cli.c)
so instead of using the https route, one could also connect lua directly
to the api commands in C code http://www.lua.org/pil/26.1.html which
should result in a far smaller binary than the 17 Mb of libhttpserver.

I don't understand what you mean here... The base size of OSv does not
include Lua, and if you add the "cli" module inside OSv, it adds the "lua"
module. Lua is not big, around 200K - 300K, but isn't smaller than cpiod.

But then again: if everybody is building on Linux with qemu-nbd present
then I might be puzzling at a problem that nobody encounters ;-)

Antoine de Saint Exupéry once said that "perfection is attained not when
there is nothing more to add, but when there is nothing more to remove.".
So if we can remove the need for qemu-nbd from Capstan, it will make it
more perfect :-)

[seriousme]

Thx for the elaborate explanation :-)

capstan run cloudius/osv reports:

ls -l libhttpserver.so
-rwxrwxrwx 1 osv osv 17516015 Jul 31 11:19 libhttpserver.so

Does this then mean that the image contains an unstripped binary ?

Having the startup command in a file sounds logical to me too, however if cpiod is already bigger than required then a rewrite of cpiod cloud easily call save_cmdline() on reception of a magic file (until something better comes along :-))

With regards to the Lua binary: The way I understand it now the CLI module will only work if you also add the 1.1Mb of libhttpserver. If the lua interpreter was extended with the C-functions from the api (e.g. I could do a save_cmdline() from Lua without libhttpserver present ) then we could:

• use the cli without libhttpserver present
• write a successor to cpiod in lua (lua code is much more concise ;-))
• modify the whole stuff without having to maintain a Linux build chain as Lua is just a scripting language :-)

One could even start a simple lua webserver if the luasocket lib was present as well. Ok, I get a bit carried away here ;-), however once you have a lua interpreter on the system you will be able to code at a higher abstraction level than Cpp which results in smaller codesize, less prone to error (no pointer stuff), that runs nearly as fast as native (the lua interpreter seems rather effective). Lua scripts, in my experience, tend to be easier to maintain than Cpp code. However this only adds value if you can replace multiple binaries by lua scripts, else the size of the interpreter will negate the potential advantages.

And yes I totally agree with the quote from Antoine de Saint Exupéry.
Like Einstein once said: "Everything should be made as simple as possible, but not simpler" :-)

[nyh]

On Tue, Oct 20, 2015 at 11:20 AM, seriousme [email protected]
wrote:

Thx for the elaborate explanation :-)

capstan run cloudius/osv reports:

ls -l libhttpserver.so
-rwxrwxrwx 1 osv osv 17516015 Jul 31 11:19 libhttpserver.so

Does this then mean that the image contains an unstripped binary ?

Interesting. The script scripts/upload_manifest.py, used by "scripts/build"
(but not by Capstan) is careful to strip all the uploaded executables
without touching the original files. The reason not to simply strip the
original files is that it is useful, for debugging with gdb, to keep the
unstripped executables, while only putting the stripped version on the
image.

Maybe Capstan doesn't have this trick, so it uploads unstripped binaries
onto the image? Not too terrible, but a waste.

Or did you get libhttpserver.so from the base image? If that is the case,
then maybe we have a bug in upload_manifest.py?

Having the startup command in a file sounds logical to me too, however if
cpiod is already bigger than required then a rewrite of cpiod cloud easily
call save_cmdline() on reception of a magic file (until something better
comes along :-))

This is also a possible trick. Or cpiod can open another socket (or
whatever) to accept that magic command line.

[seriousme]

I just used the cloudius/osv image as suggested by the capstan doc.
So I presume there is an flaw in the script that build this image.

If we stick to the cpiod route then a magic file seems more logical to me as cpiod seems to know about files :-)

Just had a look at cpiod.cc which already contains a 'magic word':

if (name == "TRAILER!!!") {
        return false;
    }

the only thing that would be required is to add a second magic word, e.g. something along the lines of:

if (name == "SAVE_CMDLINE!!!"){
       unique_ptr<char[]> cmdbuf{new char[filesize]};
        is.read(cmdbuf.get(), filesize);
        string cmd{cmdbuf.get(), filesize};
        save_cmdline(cmd);
    }

Now I don't have a full OSv build chain setup to test it out nor am I a seasoned C11 coder, so the usual disclaimers apply ;-)

From what I understand this should enable you to have a file called "SAVE_CMDLINE!!!" which contents would be saved as the new cmdline which might work as a short term fix.