LCA09: Day 1 : Monday

So after all the preliminaries it was finally time for the actual conference.

Breakfast was a fairly common dinning hall setup and I walked down the hill to our venue. The Sysadmin Miniconf is in a 200-odd seat lecture theatre right next to the registration desk. We are  getting recorded so hopefully video’s of most talks will be available at some point. About the only problem with was that there was only a hand-held microphones available so we have problems when ( for instance ) people were trying to type and talk at the same time.

First talk was by Andew Bartlett on the directory ( as opposed to file ) service in Samba 4. He gave a basic overview followed by a lot of questions.

Rickard Keech then gave an overview on how he provisioned servers. It was a little different since it was a CD based method ( since many of his customers were fairly small and not well network connected). He is automatically building the kickstart files and testing and rebuilding them to ensure that the server is in sync with it’s deployed version ( much like maintaining a source tree and compiling it into a kickstart iso).

Several other presenters pointed out how sysadmin practice is 20 years behind programming ( no version control, testing, libraries, development environments etc).

Similarly Dedvas Bhgat talked on how his team was able to use configuration management ( puppet) to hugely reduce the load ( and stress ) on his team and to improve service. He is using puppet to look at ~300 servers.

The next two talk were a bit over my head. The first discussed a new snapshot implimentation which was much more efficient that the previous one while the second ( which was in the miniconf section ) demonstrated adding modules on a Xen server which was able to probe running Xen Child machine ( the extra shown was spotting signals being sent to the apache process but it looks like just about anything is available).

The lightning talks were next. Devdas talked about programming practices while I did my Mondorescue and PXE talk.

After lunch there was a talk about the setup (and large amounts of data and processing) at the NIWA ( New Zealand weather research agency )  and a talk on Performance Co-Pilot (PCP). I found PCP quite interesting, it seemed to be a fairly simple was to get date and states out of a server and access them ( realtime or later) for graphing, alarming etc.

Robert Postill then gave a talk on deploying Ruby on Rails with lots of experience and advice. Pretty good and applicable for similar systems like Django.

Last up there was an intro to Redhat Satellite and RHN.

For dinner we organised a little Sysadmin thing at the Bay Leaf Bistro on Sandy Bay Road. The place was really great, food was simple and great ( suburban ) while the service was excellent. They had a great system for taking orders and doing individual bills and overall were really friendly. Best restaurant experience I’ve had in ages.

During and after dinner I was chatting to various people so didn’t get to bed till a bit late. I probably need to get to bed earlier in the week or I’ll be tired out.


LCA09: Day -1 : Sunday

Today was my first full day in Hobart. After a good night’s sleep I went into town with a couple of
people to get some breakfast. Had a nice eggs benedict at a pub/cafe down by the waterfront in the  middle of town and then a couple of us walked back to the motel and caught a taxi up to venue.

The university accommodation is right at the top of the hill so but the rooms that about 2/3s of the people are in are pretty nice. They are 6 bedrooms with 2 toilets and a large common are ( kitchen, stoves, fridge, freezer, lounge and TV). It all feels only a few years only ( lots of pwer points for example).

After getting some problems sorted out with room access and finally geting my missing bag from Qantas ( and have a change and a shave) around 10 of us wandered over to the Sandy bay shops for dinner. We split up and 5 of us had some Indian curry (okay but a little expensive) and then headed back via the supermarket.

Looking at tomorrow I’ll be at the Sysadmin Miniconf pretty much all day. I have a lightning talk to deliver (some more practice tonight) there and the programme to organise. I’ll have a quick look at the other miniconfs and see if there is one I’m really keen on I can sneak away but probably not many.

Dinner will be the Sysadmin dinner which will be at a little place we’ll book tomorrow. Over things are looking pretty promising for LCA and organisation seems to be reasonable.


LCA09 Day -2 – Saturday

After leaving my packing etc to the last minute I caught the plane over to Sydney and then to Hobart  during the day on Saturday. Fairly eventful (although I think people were paying a bit more attention to the safety demonstration after the crash in New York last week) except for the quick glimpse of A380 in Sydney and Qantas losing my bag  (which had me worried for the rest of the day since I just had the clothes I was in, my wallet, passport and laptop. )

Fortunately Qantas rang me on Saturday evening to say they found the bag so I won’t have to spend all day Sunday buying clothes and other random stuff in Hobart.

Landing in Hobart the airport is pretty small (no air-bridge) and a bit of a trip out of town ( $15
hotel bus ). I thought the landscape looked a slightly greener shade than other parts of Australia and the houses a bit older than average.

The Hotel I’m in for tonight is just past the casino and only 10 minutes walk from the University so not situated too badly. I went for a walk to dinner and got into the main Sandy Bay shops after around 20 minutes ( The Casino restaurants looked overpriced and not that great ).

Wandering around I ended up trying a fish-and-chip place since I hadn’t had that for a while. It was okay but at best a 6/10 . Unfortunately the nearby German bakery is closed tomorrow so I’ll have to wait till Monday to try it. The pizza place looked nice (in a cheap pizza sort of way).


SSL my blog

I’ll be at all next week and I’m planning to update my blog a few times. The problem is that I’ll be on a wireless connection so prone to people sniffing my passwords.

So I thought I’d see if I could get https going on my server. A quick google found a page on generating self signed certificites and another on lighttpd as a ssl proxy so with a bit of tweaking:

#### SSL engine
$SERVER["socket"] == "" {
                  ssl.engine                  = "enable"
                  ssl.pemfile                 = "/etc/lighttpd/server.pem"
                         = ""
                  proxy.server = (
                       "" => ( ( "host" => "", "port" => 80 ) )

Which work well enough for me to post this article via. Obviously self-signed certifcates are not the best sort of thing for public sites but in my case I just have to make sure I get my browser to trust the cert before I leave and from then on I can be pretty sure I’m connecting to the right site and nobody is sniffing my traffic.

This evening I’ve been doing the slides for my lightning talk next week. Everything is turning into a bit of a rush since I leave early Saturday morning. But I think I should be okay.

Share minus 1 week

Just had my last weekend in NZ before 2009 . I’m flying over this coming Saturday and back the Saturday following. So this weekend I’ve cut down a bit on other stuff and been prioritizing LCA.

I sent sent out a call for lightning talks to the LCA attendees list. So far we’ve got 3 confirmed lightning talks for the Sysadmin Miniconf and with a 1 hour slot on each day we have plenty of space for more. I hoping that a few first-time presenters will do quick talks about things like how they deploy apache or which monitoring system they use. Sometimes it’s very hard to get a feel for what other people are doing and a 5 minute talk on a simple tool can really help a lot of people.

Towards that I volunteered to do a 10 minutes (approx) talk on mondorescuse and PXE based on some stuff I’m doing at work and a previous blog post . My main bit last night and today was getting some screenshots of mondorescue and doing a little testing of stuff I didn’t cover at work.

I did my own screenshots since I couldn’t find any under a nice license ( I’ll release mine under Public Domain ) and the ones at work using VMWare client under Windows XP didn’t look right. To get mine I just ended up using KVM on my desktop and the built-in gnome-screenshot program.

A couple of interesting links:


Chess for 2009

It’s around 11 months since I started playing chess again so I thought I’d do a bit of a review and some goals for 2009.

I’ve found I enjoy playing quite a bit both at the club, tournaments and online as well as doing study both by myself and with others. So I’m definitely going to keep going this year.

In the last rating list my rapid rating went from 1276 to 1314 but my normal rating only went from 1261 to 1274. The main reason for this is that I am still losing the odd game to very low ranked players though carelessness. My actual rating is probably closer to 1400 in both rapid and normal going against my results against players in the 1400-1500 level.

Goals for 2009

  1. Get my Rating to over 1700
  2. Play in NZ champs in early 2010 and do well in under-2000 grade.

Programme to accomplish this

  1. Keep playing at the Club
  2. Enter as many tournaments as possible.
  3. Play in over-1400 grades when able
  4. Train 10-20 hours per week
  5. Keep going to fortnightly coaching

Weekly training

  1. 10-20 hours
  2. 500 problems on tactics server ( 4 hours )
  3. 20 Blitz games on FICS each week ( 4 hours )
  4. 10 Standard games on FICS each week ( 4 hours )
  5. Study Openings repertoire and practice it ( 4 hours )
  6. Play though annotated games ( 4 hours )

Well that is the plan anyway. Certainly if I keep to it I’ve got a chance at reaching the goals. However keeping myself on track will be the hard bit.

I’m using a little area in the spare room going as a study space. For a computer I’m just going to use my Eee ( with external keyboard, screen and mouse ) but keep it off the network most of the time to cut the urge to browse. But for now just a board and a few books but thats enough for a start.


Upgrading stuff

I’ve been spending most of today upgrading various bits and pieces on my external server. Since March 2007 I’ve had a dedicated server at Layered Technologies in the US. However with the change in the value of the dollar, a price rise they had a while back the cost has gone from around $NZ 100 to $NZ 200 ( $US 105 ) per month (with me paying a larger percentage of it) which is just a little too much for what I need.

So a couple of weeks ago I bought a $US 40/month VPS at to replace it ( switch off of the old machine is the end of January ). So far I’ve been pretty happy, the box was provisioned in a few minutes, I got an extra IP no problem and it came with a nice minimal Ubuntu 7.10 install.

Today I’ve been moving over a few more service to the new machine and documenting it as I go.I’m also trying to get the config a little tidier than the previous one

  • DNS was pretty easy and I’ve tidied up my domains ( all 10 ) . I’m just waiting for people I secondary DNS for to make some changes before starting cut overs.
  • Some websites are move. The static sites ( like my homepage ) were easy and the wordpress ones were as well ( dump DB, scp , import DB ).
  • I upgraded WordPress to 2.7 while I was at it which seems to have worked okay. Only thing I don’t like is the small fonts on the admin pages.But there is probably a way to fix that.
  • I had a few problems moving over one web-app since it was keeping old info somewhere (I checked the DB and the configs) but as luck would have it Ubuntu has recently fixed the package to work with non-apache Web installs (I am using lighttpd) so I just blew away the DB, grabbed a clean install and took 20 minutes to re-add the small amount of data.
  • Wiki sites are still to be moved and I’ve left off a couple of that are probably getting moved elsewhere.
  • I got the basic backups working also just in case something goes kaput.

Overall it’s been a fun day. I would hope I’ll be able to finish the rest of the move by the end of next weekend.


Youtube comment filters waste of time?

Even now and then I get bored and look on youtube at the odd video. One thing I notice is that in the comments for a video you have the option of flagging comments as a “poor comment” or a “good comment”. You also have the option of filtering the comments you see to only show those with scores of greater than -10 , -5 , 0 , +5 or +10 ).

The problem appears to be that nobody ever rates comments as good. So this feature is completely useless. To check I had a look at the top 8 videos on the most viewed videos of all time page and even though most of the had thousands of comments not a single comment was rated better than +5 . In other words a complete rating system for video comments that is completely unused by anyone.  Perhaps they should just remove the buttons.

I also noticed that the most viewed video on youtube ever ( Avril Lavigne – Girlfriend ) is “not available in your country” when I try to view it. I assume this says something about somebody’s business model.


Sysadmin Miniconf programme up

I’ve just posted the programme for the 2009 Sysadmin Miniconf !

This year we were allocated 2 days by the programme committee so we have 15 full length talks by some great speakers on a wide variety of topics. Have a look for yourself to see which ones you are interested in.

I’ve not yet updated the lightning talks but we have a few already and are still looking for more, so contact lca09 @ is you are interested in presenting.


Server recovery first steps

At work last week we were looking at backups on a group of machines that had been installed by another company but which our team had recently taken over. I was interested in the backup system they had which involved doing a lvm snapshot of the boot partition and then rsyncing this to another machine is the group ( the rsync’s went around in a circle more or less).

This looked quite cute for quick machine recoveries ( we kickstart our servers but we are still at the stage of doing a fair bit of post install setup ) and we had a think about recovering machines by doing a simple kickstart, then netbooting the server, mounting the root partition under the netboot and rsyncing it back to the install. This seemed a promising idea which we thought would only take an hour or so per machine.

However over the weekend I had a bit of a think and it popped into my head that Mondorescue almost did this sort of thing out of the box already. So I’ve been playing around a it this week with it.

So what I have now ( testing using a scratch VM ) are a few commands that:

  1. Backup the server to a NFS partition.
  2. Make an differential backup since the previous backup

Which means I now have a directory on a NFS server with a couple of bootable ISOs sitting in it. One has the full backup of the machine ( it’s about a third of the size of the used space ) and the other has any changes made since the first was done. I do the differential since the full backup takes about 30 minutes of hard work for the server while the incremental only takes 3 minutes or so ( YMMV ). I’ll probably do full backups every week and differential backups nightly.

The fun bit is the recovery:

  1. Remove console the server and boot it over the network
  2. Use PXE to boot the full backup mondorescue image
  3. Mondo boots and thee automatic restores the server to the state is was when the last backup was made (about 15 minutes) . I then have to hit enter a couple of times to reboot
  4. Netboot the incremental mondo image.
  5. Mondo now applies any changes between the last full and the last differential backup.
  6. Reboot again to the hard drive
  7. Finished, machine should be up and running.

A bit of testing shows this only takes about 20 minutes for my test VM ( 3 Gigabytes of default RHE 5 goodness ) and production servers shouldn’t be much slower ( more data but faster disks and CPUs ).

With a bit of luck I should have this ready to deploy in a few days ( although I’m a little short of NFS space to apply it to every machine ).

Overall a fun couple of days, depending on how it goes I might even do a lightning talk about it at the Sysadmin Miniconf next month although I’m not sure if it’s a little trivial since this is close to “out of the box” functionality for Mondorescue.