Clock synch on Ubuntu 10.04 Lucid on XenVZ.co.uk

Running Ubuntu 10.04 Lucid on a XenVZ.co.uk.

What do we need to do to make the clock time be right?

I tried running without ntp and letting it take the time from the host machine (dom0). This didn't seem to be too bad. Over a few weeks we seemed to build up an error of about 0.3 seconds. I don't know why or how it can be off by such a small amount. Surely it should either be off by more than that, or bang on. Strange.

Anyway I have now turned on the ntp configuration again. We now seem to be running within a handful of milliseconds of the right time and ntpdc -c kerninfo is saying the we are synched. We will have to leave it a few weeks to see how it gets on.

Setting up a default locale under Ubuntu 10.04 Lucid

Do you do Perl? If so, you might like to try perldoc.co.uk. It's like perldoc.perl.org, only it's not perldoc.perl.org, so when perldoc.perl.org goes down, perldoc.co.uk stays up. Plus it's UK based, so for people in the UK and the rest of Europe, it should be quicker.

Had a problem where I had a VM running Ubuntu 10.04 Lucid but we did not have a suitable locale set to make it "work in UTF-8".

What is normally required is for the environment variable LANG to be set to something sane like en_GB.UTF-8.

On the system in question, which was derived from a bare-bones VM Ubuntu, LANG was empty. Pasting a smiley face ☺ into a PuTTY window just made it go bing. Typing a pound sign £ didn't work.

Now back in the good old days we would do something like

sudo aptitude install locales
sudo dpkg-reconfigure locales

But on my Ubuntu 10.04 Lucid system that did nothing.

Some googling around showed up remarkably little. It seems everyone just uses the GUI installer and it does everything for you. There is not much help for the command-line user.

Anyway, eventually the answer did start to become clear. The locale data is kept in separate language packs. These are represented by various packages in the standard repository, with meta-packages to link things together.

The upshot is that you need to install the relevant language pack

sudo aptitude install language-support-en

Then set the default locale as desired

update-locale LANG=en_GB.UTF-8

Log out and log in again to pick up the default locale. Then check it with

echo $LANG

This should show the default locale set earlier.

I can now paste a smiley face ☺ into PuTTY and type a pound sign £ and that all works.

If you found this article useful, please consider donating 0.50 U.S. dollars or 0.50 Euro.

Unison seems to be interesting

Humyo turned out to be not as good as I'd hoped for.

I was still looking for a way to synchronise my lappy (or at least bits of it) with a server.

So I thought I would have a go with Unison.

So far it seems to be working out OK.

The old Unison 2.27 that you get with your average Linux distro is a bit pants because it does not support Unicode filenames. You need something a bit newer like 2.32.

At the minute I seem to be ending up with Unison 2.40 everywhere.

Unison comes with an X.Y.Z version number and you need to have the same X.Y at both ends of the link for it to work.

So I am using the standard pre-built Windows GTK build of 2.40.something, and building my own one on the Linux side.

For some reason, Unison 2.40 insists that it needs Ocaml 3.11.2, which is the latest released version. Ocaml is a big old beast to build yourself. You really want to be using a pre-built package version. So I have ended up going to Ubuntu 10.04 Lucid which includes Ocaml 3.11.2.

To get Unison on Windows to be able to call up the Linux box, I use "plink" out of PuTTY. I was previously using SSH out of cygwin, but using plink seems to involve less overall weirdness. To get Unison to be able to invoke plink (relatively) smoothly, I used a shim script called ssh2plink.bat.

Unison is quite neat in the way it synchronises. It seems to be able to spot files with the same content, even if they have different name. So you can happily pick up a tree and move it somewhere else, and Unison will synchronise it quickly without having to re-upload or re-download the whole tree. This also seems to work with tree copies.

One thing I have been doing is having two lappies synchonised with the server. This, coupled with the fact of having all e-mail accessed by IMAP meant I could switch from working on one lappy to the other lappy with ease.

I eventually stumbled across the fact that Unison does not propagate file modified timestamps by default. There is an option ("times") which can be turned on easily enough, but for some unknown reason it is not enabled by default.

Humyo: ghastly

On the face of it, Humyo looks like quite a good idea. About £4 per month for 100GB storage.

After using it for a few days I started to run into what turned out to be a show-stopping problem.

The Humyo software takes an age "working out what has changed", and does not provide any progress indication.

I thought this was just some funny little thing about the way it worked.

That was until I tried to get my data off of Humyo again.

And he we find the rub, and it's a biggy: Humyo is slow. Deathly slow.

Retrieving a file through the DAV interface runs at about 1 Mbit/s (one megabit per second). Not even 1 Mbyte/s. Really, 1 Mbit/s (125 kbyte/s). This is even slower than my out-in-the-sticks 2 Mbit/s broadband connection.

Reporting this to Humyo tech support, at first they claimed there was "packet loss near my host". This was fine except I had run my tests from 3 different co-lo boxes, all with proved excellent connectivity. Humyo then ran tests through the HTTP (public file sharing interface) which ran more quickly. Yeeeesss, but that is public file sharing. I was talking about the private authenticated DAV interface.

If we mount up Humyo using Davfs2, and try and do any operation which enumerates the directory hierarchy, we can only go through it at about one directory per second. And this is with empty directories.

No. Humyo is utterly ghastly.

At the present time AWS S3 seems to be quite a good idea. Access to AWS S3 seems to be reasonably good from the co-lo machine.

LG 37LH5000 red button prompt dismissal

When you're watching a programme on the LG 37LH5000 via the integrated Freeview receiver, you sometimes get an overlay at the top right of the screen telling you that you could press the red button.

On Sky Digital, you get similar prompts and you can dismiss this prompt by pressing the 'back' button.

But this does not work on the LG 37LH5000.

So can you do the same thing another way, and if so, how?

Yes, press the green button.

How to get Ubuntu 9.10 Karmic or 10.04 Lucid on a XenVZ.co.uk Xen VPS

Note: The procedures documented in this article are not endorsed by XenVZ.co.uk. They come with no guarantees whatsoever. Following the procedures in this article could destroy your data, or leave you with a raft of important services stacked on top of a hopelessly unstable system. Proceed entirely at your own risk.

For Xen VPSs, XenVZ currently offers Ubuntu 8.04 Hardy and 9.04 Jaunty, but nothing later than that.

If you start with 9.04 Jaunty and try an out-of-the-box in-place upgrade to 9.10 Karmic, you find that your machine will no longer boot.

So if you want to use something later like 9.10 Karmic or 10.04 Lucid, you need to do something a bit cleverer.

For Xen VPSs, XenVZ uses a stock kernel which appears to be some version of 2.6.18 out of RHEL 5.something. This is all patched and updated and all fine for many purposes. This stock kernel is, by default, booted from outside the VPS and is used regardless of what is configured inside the VPS.

However Karmic's default boot configuration requires /proc/<process-id>/mountinfo, which I believe was a new feature for kernel 2.6.26.

To get Karmic working, we would either need to alter the boot configuration, or make the kernel provide the required feature.

Kernel 2.6.18 seems fairly ancient to me, so my approach would be to use a later kernel.

Fortunately, XenVZ now offer an "expert" feature for configuring one's own kernel via PvGrub/PyGrub. This can be turned on through a tick box in the control panel under "Custom Kernel".

It may be possible to use a kernel from Ubuntu, but for various reasons, I have gone down the path of using a kernel from Debian. (After writing the original version of this article, I switched the system over to an Ubuntu kernel and it worked fine; better in fact because it booted up with fewer warnings etc.. So it may be possible/better to use an Ubuntu kernel from the beginning. Please let me know if you get any success doing this.)

First guess might be the Debian 5.0 Stable "Lenny" kernel which is version 2.6.26. But this apparently has problems running as a domU under Xen.

The next guess I had was to use a kernel from Debian Testing "Squeeze", which would be a version 2.6.32 kernel.

Nicely for us, kernels nowadays seem to support Xen domU operation even in the "ordinary" kernels, so we don't need to go hunting around for a special Xen-domU-enabled kernel. (This seems to be related to "pvops" http://wiki.xensource.com/xenwiki/XenParavirtOps.) So we should be able to use the basic basic kernel from Squeeze and it should work.

So, our steps would be:

• Set your VPS up with Ubuntu 9.04 Jaunty using the XenVZ web control panel.

• Install any updates which have been published since the system image was made:

  # aptitude update
  
  # aptitude safe-upgrade

• Download the deb package for the appropriate kernel from Debian Squeeze. This can be done using "wget" (though "wget" isn't installed by default so if it is a fresh system you will have to put it on (aptitude install wget)).

• Install this kernel using "dpkg -i"

• (This is lifted from http://wiki.debian.org/PyGrub) Create the environment needed for running update-grub

  # mkdir /boot/grub
  # echo "(hd0) /dev/sda" > /boot/grub/device.map
  # mknod /dev/sda b 202 0

  (This is a bit I added in the middle.) We need to run "update-grub" to generate /boot/grub/menu.lst. However, because the above kernel does not have the word "xen" in its name, "update-grub" will not recognise this kernel as suitable for Xen domU use, so ordinarily it skips it. There may be a good way of fixing this, but for now I just fudge "update-grub" to think all kernels are suitable for Xen domU use. Edit "/usr/sbin/update-grub", look for the bit that goes "is_xen=" and change it to "is_xen=1". (I've raised Bug 586756 against Ubuntu, and provided a patch.)

  (This is lifted from http://wiki.debian.org/PyGrub) Create first /boot/grub/menu.lst based on the content of /boot

  # update-grub

• In the XenVZ web control panel, turn on "Power User" and "Custom Kernel".

• In "/etc/event.d", copy the file "xvc0" to "hvc0", then edit "hvc0" and replace all instances of "xvc0" with "hvc0". This is needed because in the later kernel, the console device name changes from xvc0 to hvc0.

• In "/etc/fstab", change "sda1" to "xvda1" and "sda2" to "xvda2". This is needed because in the later kernel, the block devices exposed through Xen get different names. Also you may want to check the sixth field of all records and make sure it is zero for anything which isn't a real filesystem. This is suggested because from a certain version of Ubuntu, it starts to try to fsck the swap partition and complains about not being able to find fsck.swap.

• You should now be able to re-start the VPS and it should boot up on the new kernel.

• Assuming it comes up, check that your swap is enabled (cat /proc/swaps), and check that console login access is working (to get console access, follow the instructions provided in the web control panel)

There may be other things we can do to clean this up further — please let me know if you have any suggestions.

If you ignored my suggestion to get console login access working earlier then stop and do it now. One day you will need it and when you need it, (a) you will really need it, and (b) you will not then be in a position to set it up. You need to check that your console access is working ahead of time so that it is ready and working for when you need it.

Once you have the new kernel, you should be able to upgrade to Ubuntu 9.10 Karmic. The procedure is outlined at https://help.ubuntu.com/community/KarmicUpgrades. In essence it goes as:

# aptitude install update-manager-core

# do-release-upgrade

If you are currently on 9.04 Jaunty, it should suggest an upgrade to 9.10 Karmic. The upgrade will probably ask you about a network configuration file which has been modified. I took the new version of the file and it worked.

That should all run through and then ask you to re-boot. You should be fine to reboot at this point. When it comes back up you should then be on 9.10 Karmic.

I don't recall if it is Karmic or Lucid, but one of them changes how the console login access works relative to the previous version. So you may now want to check the contents of "/etc/init". You should have a file "tty1.conf". Copy this file to "hvc0", then edit it and change all instances of "tty1" to "hvc0". We then need to get init ("upstart"?) to re-read the configuration. I haven't figured out how to do this yet, but a re-boot will get that working soon enough.

Check that console login access is working.

Once we have 9.10 Karmic up and running, we can upgrade to 10.04 Lucid. This can be done in the same way as the upgrade to Karmic (do-release-upgrade).

Lucid seems to install "Plymouth" (a boot splash manager) by default, which is a bit pointless for a co-located VPS. Unfortunately it seems that some critical-looking things depend on Plymouth, so I didn't fancying uninstalling it. But we can disable Plymouth easily enough by moving its configuration files out of /etc/init:

# cd /etc/init
# mkdir /etc/init-disabled
# mv plymouth* /etc/init-disabled

Also, you may want to look at the kernel boot parameters in /boot/grub/menu.lst. You may see a line like:

kernel /boot/vmlinuz-2.6.32-3-amd64 root=UUID=########-####-####-####-############ ro quiet splash

(If I understand correctly, the UUID will be different for each system.)

You may want to remove the directives 'splash' and 'quiet'. I think 'splash' is to do with a splash screen, which is a bit pointless on a VPS. I think 'quiet' is probably to do with reducing the amount of chatter during boot-up, which is probably the opposite of what we want.

Again, check that console login access is working.

I also took the opportunity to disable the redundant "tty" entries in /etc/init, by moving them to /etc/init-disabled.

Since first writing the article, I discovered that we can use Ubuntu kernels. Using Ubuntu kernels makes it boot more smoothly. I only switched to an Ubuntu kernel at the end of the process, but it may have worked to use an Ubuntu kernel from the beginning.

Turns out that following the above approach results in a system which doesn't do UTF-8 by default. If you want UTF-8 support, you have to put that on yourself. See Setting up a default locale under Ubuntu 10.04 Lucid.

If you have any further information or suggestions, please let me know.

Dell Service Tags to Express Service Codes in Perl

It seems that Dell Tech Support have replaced a layer of people with a layer of machines and as such whereas you could previously give your machine ID using the (alphanumeric) Service Tag to a person, you now need to give your machine ID to a machine, which means it needs to be the (numeric) Express Service Code.

This is no problem if you have the machine in question because the service tag sticker has both the (alphanumeric) Service Tag and the (numeric) Express Service Code.

However, if you had been noting only Service Tags from machines and storing those in your records, you might not have the (numeric) Express Service Code to hand and this may cause a problem.

It turns out that there is a relatively straightforward relationship between the Service Tag and the Express Service Code.

It seems that if you take the Service Tag and treat it as a base 36 number, then express that quantity as a base 10 number, and add hyphens in the right places, you get the Express Service Code.

The hyphens aren't crucial because you don't dial them, but if you want to match the existing codes exactly, it seems you need to add hyphens every three digits, but with the proviso that if we would otherwise end up with a single digit on its own, then we start with two digits instead of three.

Wanting to convert an existing file of service tags en masse, I came up with a little Perl program which would do it as a pipe filter:

perl -e 'while(<>){chomp; if(/^[0-9A-Z]{1,7}$/so){@a = split //; my %e; @e{"0".."9","A".."Z"} = (0..35); my $r = 0; while(@a){ $r = ($r * 36) + $e{shift @a}; }; @r = split //, $r; my @s; push @s, join "", splice @r,0,2 if ((@r % 3) == 1); push @s, join "", splice @r,0,3 while @r; $_ = join "-", @s; } print"$_\n";}'