Thursday, October 08, 2015

Deconstructing .pyc files

I've recently been trying to work out why python was recompiling a bunch of .pyc files. I haven't solved that, but I learnt a little along the way, enough to be worth writing down.

Python will recompile a .py file onto a .pyc file if it thinks something's changed. But how does it decide something has changed? It encodes some of the pertinent details in the header of the .pyc file.

Consider a file. There's a and a foo.pyc. I open up the .pyc file in emacs and view it in hex. (ESC-x hexl-mode for those unfamiliar.)

The file starts off like this:

 03f3 0d0a c368 7955 6300 0000 ....

The first 4 bytes 03f30d0a are the magic number, and encode the version of python. There's a list of magic numbers in the source, here.

To check this, take the 03f3, reverse it to f303, which is 62211 decimal. That corresponds to 2.7a0 - this is python 2.7, so that matches. (The 0d0a is also part of the encoding of the magic number.) This check is just to see if the .pyc file is compatible with the version of python you're using. If it's not, it will ignore the .pyc file and may regenerate it.

The next bit is c3687955. Reverse this again to get the endianness right, and it's 557968c3. In decimal, that's 1434020035.

That's a timestamp, standard unix time. What does that correspond to?

perl -e '$f=localtime(1434020035); print $f'
Thu Jun 11 11:53:55 2015

And I can look at the file (on Solaris and illumos, there's a -e flag to ls to give us the time in the right format rather than the default "simplified" version).

/bin/ls -eo
-rw-r--r-- 1 root  7917 Jun 11 11:53:55 2015

As you can see, that matches the timestamp on the source file exactly. If the timestamp doesn't match, then again python will ignore it.

This has consequences for packaging. SVR4 packaging automatically preserves timestamps, with IPS you need to use pkgsend -T to do so as it's not done by default.

Tuesday, October 06, 2015

Software directions in Tribblix

Tribblix has been developing in a number of different directions. I've been working on trimming the live image, and strengthening the foundations.

Beyond this, there is a continual stream of updated packages. Generally, if I package it, I'll try and keep it up to date. (If it's downrev, it's usually for a reason.)

In the meantime I've found time for experiments in booting Tribblix in very little memory, and creating a pure illumos bootable system.

But I thought it worthwhile to highlight some of the individual packages that have gone into Tribblix recently.

The big one was adding LibreOffice, of course. Needless to say, this was a modest amount of work. (Not necessarily all that hard, but it's a big build, and the edit-compile-debug cycle is fairly long, so it took a while.) I need to go back and update LibreOffice to a more current version, but the version I now have meets all of my needs so I can invest time and energy elsewhere.

On the desktop, I added MATE, and incorporated SLiM as a login manager. Tribblix has a lot of desktop environments and window managers available, although Xfce is still the primary and best supported option. I finally added the base GTK engines and icon themes, which got rid of a lot of errors.

In terms of tools, there's now Dia, Scribus, and Inkscape.

Tribblix has always had a retro streak. I've added gopher, gophervr, and the old Mosaic browser. There are other old X11 tools that some of you may remember - xcoral, xsnow, xsol, xshisen. If only I could get xfishtank working again.

I've been keeping up with Node.js releases, of course. But the new kid on the block is Go, and that's included in Tribblix. Current versions work very well, and now we've got past the cgo problems, there's a whole raft of modern software written in Go that's now available to us. The next one up is probably Rust.

Fun with SPARC emulators

While illumos supports both SPARC and x86 platforms, it would be a fair assessment that the SPARC support is a poor relation.

There are illumos distributions that run on SPARC - OpenSXCE has for a while, Tribblix and DilOS also have SPARC images available and both are actively maintained. The mainstream distributions are x86-only.

A large part of the lack of SPARC support is quite simple - the number of users with SPARC hardware is small; the number of developers with SPARC hardware is even smaller. And you can see that the SPARC support is largely in the hands of the hobbyist part of the community. (Which is to be expected - the commercial members of the community are obviously not going to spend money on supporting hardware they neither have nor use.)

Absent physical hardware, are there any alternatives?

Perhaps the most obvious candidate is qemu. However, the sparc64 implementation is fairly immature. In other words, it doesn't work. Tribblix will start to boot, and does get a little way into the kernel before qemu crashes. I think it's generally agreed that qemu isn't there yet.

The next thing I tried is legion, which is the T1/T2 simulator from the opensparc project. Having built this, attempting to boot an iso image immediately fails with:

FATAL: virtual_disk not supported on this platform

which makes it rather useless. (I haven't investigated to see if support can be enabled, but the build system explicitly disables it.) Legion hasn't been updated in a while, and I can't see that changing.

Then I came across the M5 simulator. This supports a number of systems, not just SPARC. But it's an active project, and claims to be able to emulate a full SPARC system. I can build it easily enough, running it needs the opensparc binary download from legion (note - you need the T1 download, version 1.5, not the newer T2 version of the download). The instructions here appear to be valid.

With M5, I can try booting Tribblix for SPARC. And it actually gets a lot further than I expected! Just not far enough:

cpu Probing I/O buses

Sun Fire T2000, No Keyboard
Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
OpenBoot 4.20.0, 256 MB memory available, Serial #1122867.
[mo23723 obp4.20.0 #0]
Ethernet address 0:80:3:de:ad:3, Host ID: 80112233.

ok boot
Boot device: vdisk  File and args:
Loading: /platform/sun4v/boot_archive
ramdisk-root ufs-file-system
Loading: /platform/sun4v/kernel/sparcv9/unix
panic[cpu0]/thread=180e000: lgrp_traverse: No memory blocks found

Still, that's illumos bailing, there aren't any errors from M5.

Overall, I think that M5 shows some promise as a SPARC emulator for illumos.

Friday, October 02, 2015

Notifications from SMF (and FMA)

In illumos, it's possible to set the system up so that notifications are sent whenever anything happens to an SMF service.

Unfortunately, however, the illumos documentation is essentially non-existent, although looking at the Solaris documentation on the subject should be accurate.

The first thing is that you can see the current notification state by looking at svcs -n:

Notification parameters for FMA Events
    Event: problem-diagnosed
        Notification Type: smtp
            Active: true
            reply-to: root@localhost
            to: root@localhost

        Notification Type: snmp
            Active: true

        Notification Type: syslog
            Active: true

    Event: problem-repaired
        Notification Type: snmp
            Active: true

    Event: problem-resolved
        Notification Type: snmp
            Active: true

The first thing to realize here is that first line - these are the notifications sent by FMA, not SMF. There's a relationship, of course, in that if an SMF service fails and ends up in maintenance, then an FMA event will be generated, and notifications will be sent according to the above scheme.

(By the way, the configuration for this comes from the svc:/system/fm/notify-params:default service, which you can see the source for here. And you'll see that it basically matches exactly what I've shown above.)

Whether you actually receive the notifications is another matter. If you have syslogd running, which is normal, then you'll see the syslog messages ending up in the log files. To get the email or SNMP notifications relies on additional service. These are


and if these are installed and enabled, they'll send the notifications out.

You can also set up notifications inside SMF itself. There's a decent intro available for this feature, although you should note that illumos doesn't currently have any of the man pages referred to. This functionality uses the listnotify, setnotify, and delnotify subcommands to svccfg. The one thing that isn't often covered is the relationship between the SMF and the FMA notifications - it's important to understand that both exist, in a strangely mingled state, with some non-obvious overlap.

You can see the global SMF notifications with
/usr/sbin/svccfg listnotify -g
This will come back with nothing by default, so the only thing you'll see is the FMA notifications. To get SMF to email you if any service goes offline, then

/usr/sbin/svccfg setnotify -g to-offline

and you can set this up at a per-service level with

/usr/sbin/svccfg -s apache22 setnotify to-offline

Now, while the SMF notifications can be configured in a very granular manner - you can turn it on and off by service, you can control exactly which state transitions you're interested in, and you can route individual notifications to different destinations, when it comes to the FMA notifications all you have is a big hammer. It's all or nothing, and you can't be selective on where notifications end up (beyond the smtp vs snmp vs syslog channels).

This is unfortunate, because SMF isn't the only source of telemetry that gets fed into FMA. In particular, the system hardware and ZFS will generate FMA events if there's a problem. If you want to get notifications from FMA if there's a problem with ZFS, then you're also going to get notified if an SMF service breaks. In a development environment, this might happen quite a lot.

Perhaps the best compromise I've come up with is to have FMA notifications disabled in a non-global zone, and configure SMF notifications explicitly there. Then, just have FMA notifications in the global zone. This assumes you have nothing but applications in zones, and all the non-SMF events will get caught in the global zone.

Tuesday, September 29, 2015

Improving the foundations of Tribblix

Mostly due to history, the software packages that make up Tribblix come from 3 places.
In the first category, I'm using an essentially unmodified illumos-gate. The only change to the build is the fix for 5188 so that SVR4 packaging has no external dependencies (or the internal one of wanboot). I then create packages, applying a set of transforms (many of which simply avoid delivering individual files that I see no valid reason to ship - who needs ff or volcopy any more?).

The second category is the historical part. Tribblix was bootstrapped from another distro. Owing to the fact that the amount of time I have is rather limited, not all the bits used in the bootstrapping have yet been removed. These tend to be in the foundations of the OS, which makes them harder to replace (simply due to where they sit in the dependency tree).

In the latest update (the 0m16 prerelease) I've replaced several key components that were previously inherited. Part of this is so that I'm in control of these components (which is a good thing), another is simply that they needed upgrading to a newer version.

One key component here is perl. I've been building my own versions of perl to live separately, but decided it was time to replace the old system perl (I was at 5.10) with something current (5.22). This of itself is easy enough. I then have to rebuild illumos because it's connected to perl, and that's a slightly tricky problem - the build uses the Sun::Solaris module, which comes from the build. (Unfortunately it uses the copy on the system rather than the bits it just built.) So I had to pull the bits out from the failed build, install those on the system, and then the build goes through properly.

Another - more critical - component is libxml2. Critical because SMF uses it, so if you get that wrong you break the system's ability to boot. After much careful study of both the OmniOS and OpenIndiana build recipes, I picked a set of options and everything worked first time. Phew!

(Generally, I'll tend to the OpenIndiana way of doing things, simply because that's where the package I'm trying to replace came from. But I usually look at multiple distros for useful hints.)

A broader area was compression support. I updated zlib along with libxml2, but also went in and built my own p7zip, xz, and bzip2, and then started adding additional compression tools such as lzip and pigz.

The work isn't done yet. Two areas I need to look at are the Netscape Portable Runtime, and the base graphics libraries (tiff, jpeg, png). And then there's the whole X11 stack, which is a whole separate problem - because newer versions start to require KMS (which we don't have) or have gone 64-bit only (which is still an open question, and a leap I'm not yet prepared to take).

Monday, September 28, 2015

Trimming the Tribblix live image

When Tribblix boots from the installation ISO, it reads in two things: the root archive, as a ramdisk, and /usr mounted from solaris.zlib via lofi.

In preparation for the next update, I've spent a little time minimizing both files. Part of this was alongside my experiments on genuinely memory-constrained systems; working out what's necessary in extreme cases can guide you into better behaviour in normal circumstances. While I don't necessarily expect installing onto a 128M system to be a routine occurrence, it would be good to keep 1G or even 512M within reach.

One of the largest single contributors to /usr was perl. It turns out that the only critical part of the system that needs perl is intrd, which is disabled in the live environment anyway. So, perl's not needed.

Another significant package is GNU coreutils. On closer investigation, the only reason I needed this was for a script that generated a UUID which is set as a ZFS property on the root file system (it's used by beadm to match up which zone BE matches the system BE). Apart from the fact that this functionality has recently been integrated into illumos, using the GNU coreutils was just being lazy (perhaps it was necessary under Solaris 10, where this script originated, but the system utilities are good enough now).

I also had the gcc runtime installed. The illumos packages don't need it, but some 3rd-party packages did - compile with gcc and you tend to end up with libgcc_s being pulled in. There are a variety of tricks with -nostdlib and -static-libgcc that are necessary to avoid this. (And I wish I understood better exactly what's happening, as it's too much like magic for my liking.)

The overall impact isn't huge, but the overall footprint of the live image has been reduced by 25%, which is worthwhile. It also counteracts the seemingly inevitable growth of the base system, so I have to worry less about whether I can justify every single driver or small package that might be useful.

Friday, September 25, 2015

illumos pureboot

In my previous article, I discussed an experiment in illumos minimization.

Interestingly, there was a discussion on IRC that wandered off into realms afar, but it got me thinking about making oddball illumos images.

So then I thought - how would you build something that was pure illumos. As in illumos, the whole of illumos, and nothing but illumos.

Somewhat surprisingly, this works. For some definition of works, anyway.

The idea is pretty simple. After building illumos, you end up with the artefacts that are created by the build populating a proto area. This has the same structure as a regular system, so you can find usr/bin/ls under there, for example.

So all I do is create a bootable image that is the proto area from a build.

The script is here.

What does this do?
  • Copies the proto area to a staging area, so it can be manipulated
  • Modifies inittab
  • Sets up grub for boot
  • Copies the kernel state files from the running system (otherwise, they're blank and the kernel is clueless)
  • Creates a block device and builds a ufs file system on it
  • Copies the staging area to it
  • Compresses the block device
  • Copies it back to the platform staging area
  • Creates a bootable iso
There's a bit more detail, but those are the salient points.

My (non-debug) proto area seems to be around 464MB, so a 512MB ramdisk works just fine. You could start deleting (or, indeed, adding) stuff to tune the image you create. The ISO image is 166MB, ready for VirtualBox.

The important thing to realise is that illumos, of itself, does not create a complete operating system. Even core OS functionality requires additional third-party components, which will not be present in the image you create. In particular, libxml2, zlib, and openssl are missing. What this means is that anything depending on these will not function. The list of things that won't work includes SMF, which is an integral part of the normal boot and operations.

So instead of init launching SMF, I have it run a shell instead. (I actually do this via a script rather than directly from init, this allows me to put up a message, and also allows me to run other startup commands if so desired.)

This is what it looks like:

A surprising amount of stuff actually works in this environment. Certainly most of the standard unix commands that I've tried are just fine. Although it has to be appreciated that none of the normal boot processing has been done at this point, so almost nothing has been set up. (And / won't budge from being read-only which is a little confusing.)

Sunday, September 20, 2015

How low can Tribblix go?

One of the things I wanted to do with Tribblix was to allow it to run in places that other illumos distros couldn't reach.

One possible target here is systems with less than gargantuan memory capacities.

(Now, I don't have any such systems. But VirtualBox allows you to adjust the amount of memory in a guest very easily, so that's what I'm doing.)

I started out by building a 32-bit only image. That is, I built a regular (32- and 64-bit combined) image, and simply deleted all the 64-bit pieces. You can do this slightly better by building custom 32-bit only packages, but it was much simpler to identify all the directories named amd64 and delete them.

(Why focus on 32-bit here? The default image has both a 32-bit and 64-bit kernel, and many libraries are shipped in both flavours too. So removing one of the 32-bit or 64-bit flavours will potentially halve the amount of space we need. It makes more sense to drop the 64-bit files - it's easier to do, and it's more likely that real systems with minimal memory are going to be 32-bit.)

The regular boot archive in Tribblix is 160M in size (the file on the ISO is gzip-compressed and ends up being about a third of that), but it's loaded into memory as a ramdisk so the full size is a hard limit on how much memory you're going to need to boot the ISO. You might be able to run off disk with less, as we'll see later. The 32-bit boot archive can be shrunk to 90M, and still has a little room to work in.

The other part of booting from media involves the /usr file system being a compressed lofi mount from a file. I've made a change in the upcoming release by removing perl from the live boot (it's only needed for intrd, which is disabled anyway if you're booting from media), which saves a bit of space, and the 32-bit version of /usr is about a third smaller than the regular combined 32/64-bit variant. Without any additional changes, it is about 171M.

So, the boot archive takes a fixed 90M, and the whole of /usr takes 171M. Let's call that 256M of basic footprint.

I know that regular Tribblix will boot and install quite happily with 1G of memory, and previous experience is that 768M is fine too.

So I started with a 512M setup. The ISO boots just fine. I tried an install to ZFS. The initial part of the install - which is a simple cpio of the system as booted from media - worked fine, if very slowly. The second part of the base install (necessary even if you don't add more software) adds a handful of packages. This is where it really started to struggle, it just about managed the first package and then ground completely to a halt.

Now, I'm sure you could tweak the system a little further to trim the size of both the boot archive and /usr, or tweak the amount of memory ZFS uses, but we're clearly close to the edge.

So then I tried exactly the same setup, installing to UFS instead of ZFS. And it installs absolutely fine, and goes like greased lightning. OK, the conclusion here is that if you want a minimal system with less than 512M of memory, then don't bother with ZFS but aim at UFS instead.

Reducing memory to 256M, the boot and install to UFS still work fine.

With 192M of memory, boot is still good, the install is starting to get a bit sluggish.

If I go down to 128M of memory, the ISO won't boot at all.

However, if I install with a bit more memory, and then reduce it later, Tribblix on UFS works just fine with 128M of memory. Especially if you disable a few services. (Such as autofs cron zones-monitoring zones power fmd. Not necessarily what you want to do in production, but this isn't supposed to be production.)

It looks like 128M is a reasonable practical lower limit. The system is using most of the 128M (it's starting to write data to swap, so there's clearly not much headroom).

Going lower also starts to hit real hard limits. While 120M is still good, 112M fails to boot at all (I get "do_bop_phys_alloc Out of memory" errors from the kernel - see the fakebop source). I'm sure I could go down a bit further, but I think the next step is to start removing drivers from the kernel, which will reduce both the installed boot archive size and the kernel's memory requirements.

I then started to look more closely at the boot archive. On my test machine, it was 81M in size. Removing all the drivers I felt safe with dropped it down to 77M. That still seems quite large.

Diving into the boot archive itself, and crawling through the source for bootadm, I then found that the boot archive was a ufs archive that's only 25% full. It turns out that the boot archive will be hsfs if the system finds /usr/bin/mkisofs, otherwise it uses ufs. And it looks like the size calculation is a bit off, leading to an archive that's massively oversized. After installing mkisofs and rebuilding the boot archive, I got back to something that was 17M, which is much better.

On testing with the new improved boot archive, boot with 96M, or even 88M, memory is just fine.

Down to 80M of memory, and I hit the next wall. The system looks as though it will boot reasonably well, but /etc/svc/volatile fills up and you run out of swap. I suspect this is before it's had any opportunity to add the swap partition, but once it's in that state it can't progress.

Overall, in answer to the question in the title, a 32-bit variant of Tribblix will install (using UFS) on a system with as little as 192M of memory, and run on as little as 96M.