Server recovery first steps

At work last week we were looking at backups on a group of machines that had been installed by another company but which our team had recently taken over. I was interested in the backup system they had which involved doing a lvm snapshot of the boot partition and then rsyncing this to another machine is the group ( the rsync’s went around in a circle more or less).

This looked quite cute for quick machine recoveries ( we kickstart our servers but we are still at the stage of doing a fair bit of post install setup ) and we had a think about recovering machines by doing a simple kickstart, then netbooting the server, mounting the root partition under the netboot and rsyncing it back to the install. This seemed a promising idea which we thought would only take an hour or so per machine.

However over the weekend I had a bit of a think and it popped into my head that Mondorescue almost did this sort of thing out of the box already. So I’ve been playing around a it this week with it.

So what I have now ( testing using a scratch VM ) are a few commands that:

  1. Backup the server to a NFS partition.
  2. Make an differential backup since the previous backup

Which means I now have a directory on a NFS server with a couple of bootable ISOs sitting in it. One has the full backup of the machine ( it’s about a third of the size of the used space ) and the other has any changes made since the first was done. I do the differential since the full backup takes about 30 minutes of hard work for the server while the incremental only takes 3 minutes or so ( YMMV ). I’ll probably do full backups every week and differential backups nightly.

The fun bit is the recovery:

  1. Remove console the server and boot it over the network
  2. Use PXE to boot the full backup mondorescue image
  3. Mondo boots and thee automatic restores the server to the state is was when the last backup was made (about 15 minutes) . I then have to hit enter a couple of times to reboot
  4. Netboot the incremental mondo image.
  5. Mondo now applies any changes between the last full and the last differential backup.
  6. Reboot again to the hard drive
  7. Finished, machine should be up and running.

A bit of testing shows this only takes about 20 minutes for my test VM ( 3 Gigabytes of default RHE 5 goodness ) and production servers shouldn’t be much slower ( more data but faster disks and CPUs ).

With a bit of luck I should have this ready to deploy in a few days ( although I’m a little short of NFS space to apply it to every machine ).

Overall a fun couple of days, depending on how it goes I might even do a lightning talk about it at the Sysadmin Miniconf next month although I’m not sure if it’s a little trivial since this is close to “out of the box” functionality for Mondorescue.

Share

Linux.conf.au 2009 Sysadmin Miniconf – Last week for proposals

We just got into November which means that it’s just one week till proposals for the Sysadmin Miniconf at Linux.conf.au 2009 close.  We have a bunch already but still some space left ( since we have a two days space allocated ).

Have a look at our call for proposals page for details. One thing we are trying to encourage is a wide range of talks so even a 10 minute talk about how you did something is welcome.

Proposals close on the 10th of Novemeber so don’t delay.

Share

Split RSS feeds

In the past this blog has been mostly a tech orientated one with around 2/3s of the posts I make being about computers and the Internet. With the new software I’ve decided to split things a little so I can post on some other subjects. WordPress makes it fairly easy to provide alternative feeds on a per-category basis so I’ve split things as follows:

  1. Tech is anything Computer or Internet related, benchmark here is stuff that I think people on somewhere like planet.linux.org.au might be interested in. ( I’ve asked the maintainers of that planet to switch their feed to this)
  2. Chess is chess related stuff
  3. Misc is general stuff not in the above two categories like what I did last week, posts on random junk etc, the sort of stuff random family and stalkers might be interested in.
  4. Everything just covers every post andis what the old RSS feeds redirects to.

Posts ( like this one ) that cover multiple topics will be posted to multiple feeds. But I’ll guess I see how things go.

Share

Blog Moved to WordPress

I’ve just moved this blog’s software from nanoblogger to wordpress.

Things appear to be mostly working but I’ve got a bit of tweaking to do, plugins to add, tags and categories to create etc. However old article links mostly work.

Also as covered in techcrunch bloglines appears to be having serious problems with at least 10% of feeds completely broken and other unreliable. So if you are using bloglines you probably want to seriously consider moving to google reader or some other alternative.

Share

Joel Spolsky: What have you done for me lately?

Recently [Jurgen Appelo](http://nooperation.typepad.com/about.html)
listed the [Top 100 Blogs for Development Managers](http://www.noop.nl/2008/09/top-100-blogs-for-development-managers-q3-2008.html) .

Now the list looks okay except for the first entry – [Joel on Software](http://www.joelonsoftware.com). Now I’ve been subscribing to
the site for a few years along with at least 100,000 other people (
about 10,000 times more than read this blog) but I really question it
being number one anymore.

Sure there are great articles and Joel is a smart guy who creates things
like [Stackoverflow.com](http://stackoverflow.com) but I doing really
think his blog deserves the Number 1 ranking anymore.

The main reason is that Joel doesn’t really write any articles. Looking
at his [archive page](http://www.joelonsoftware.com/Archive.html) we can
see the recent history:

Year	Number of articles

2000	42
2001	23
2002	14
2003	12
2004	9
2005	22
2006	20
2007	12
2008	4 so far

Yes, just 4 articles this year so far and the [last one](http://www.joelonsoftware.com/items/2008/05/01.html) was in May, over
6 months ago.

Now Joel is doing the odd smaller post, he podcasts and writes articles in Inc
but the point of reading the blog is the content there, not somewhere else.

So really, you can’t rate a blog that hasn’t had a real post for 6 months as being
the top blog in the world for software development managers. Even [Paul Graham](http://www.paulgraham.com/) posts more often.
END

Share

Miniconf at LCA 2009

We got word a couple of days ago that the [Sysadmin Miniconf](http://sysadmin.miniconf.org) had been accepted
for [Linux.conf.au 2009](http://linux.conf.au/) and [the media have been informed](http://www.techworld.com.au/article/260982/linux_conf_au_lineup_ready_go)
so I can can say so publicly.

It looks like the following Miniconfs have been accepted:

* Open Source databases
* Kernel
* Sysadmin
* Virtualisation
* Linuxchix
* Myth TV
* Gaming
* Mobile
* Multimedia
* Free As In Freedom
* The business of open source software

I guess the main differences from previous years are the MySQL and PostgreSQL
miniconfs have been merged. There is no Debian, Education or Gnome miniconfs
and I guess Mobile has replaced Embedded. It’s interesting to see the loss of the regulars, I guess
in some cases it is because they have their own regular conferences.

The last two miniconfs ( “Free As In Freedom” and “The business of open source software” )
are also new, I guess they will be fleshed out.

We’ll be updated the Sysadmin Miniconf [website](http://sysadmin.miniconf.org) over the next few days
and the call for papers should be out next week sometime.

I also posted an [email to the NZNOG list](http://list.waikato.ac.nz/pipermail/nznog/2008-September/014444.html)
asking about interest in a Sysadmin thing at NZNOG 09 ( in Auckland the week after LCA 09 ).
have a quick look and pop me an email if you are interested.
END

Share

I like WordPress

I spent a couple of ours doing a mock up of a possible new website ( sorry not online) for the
[Auckland Chess Centre](http://www.aucklandchesscentre.co.nz/webcontent/default.aspx) (link is to the old site) using [wordpress](http://wordpress.org/).

I must say I was fairly impressed the [5 minute install](http://codex.wordpress.org/Installing_WordPress) really does
only take about 5 minutes and everything else is pretty easy. The only hard
bit was finding a good theme since most of those out there are blog based rather than
designed for fixed websites but I ended up picking the [deLight](http://wordpress.org/extend/themes/delight) theme which [looks okay](http://webdemar.com/demo/?wptheme=deLight).

Overall I’m pretty impressed with WordPress and it took just a couple of hours of playing around
to get the site to 95% of what I want. Depending on the feedback I might be able to go
live in a few days.

Throughly recommended for small sites.
END

Share

“Linux.conf.au: First timers Guide” updated

I’ve just updated [the guide](http://www.darkmere.gen.nz/2007/0129.html) a
little as [linux.conf.au 2009](http://linux.conf.au/) gets closer. The
main changes are:

* Updated to point to Hobart
* Mention “Open Day”
* Added photos of Menu and Badge

If you have suggestions for further fixes please contact me.

I’ve also added a few ( unsorted ) photos from [Linux.conf.au 2008](http://www.darkmere.gen.nz/photos/lca2008/index.html)
and the [Linux.conf.au 2008 – Sysadmin Miniconf](http://www.darkmere.gen.nz/photos/sysadmin2008/index.html).
END

Share

End of July Misc post.

As usual I’m a bit behind in stuff but the excuse this time is that around 6 months
ago I started playing chess competitively again ( previously was when I was still at school) so
I’ve been spending a bit of time at it. Right now it involves going to
[the club](http://www.aucklandchesscentre.co.nz/webcontent/default.aspx) ( site being redone soon) one evening
a week, the odd tournament and some study and practice on [sites like this](http://chess.emrald.net/tProfile.php?TacID=21765) .

I’m now [ranked](http://www.poisonpawn.co.nz/nzcfratings.htm) 332nd= in the country but
I think with a bit of work I could be in the top 200 in a few months if I keep at it.

Other stuff that is happening is that a proposal for a [Sysadmin Miniconf](http://sysadmin.miniconf.org/) has been submitted
to [Linux.conf.au 2009 in Hobart](http://linux.conf.au/). More details to come.

Some links:

A good paper at Usenix 2008 on [Handling Flash Crowds from your Garage](http://www.usenix.org/events/usenix08/tech/full_papers/elson/elson_html/)
[PDF version with the diagrams](http://www.usenix.org/events/usenix08/tech/full_papers/elson/elson.pdf) .

…and an interesting article on [How long will we be trapped in this mobile hell hole?](http://www.smstextnews.com/2008/07/how_long_will_we_be_trapped_in_this_mobile_hell_hole.html)
panders to my dislike of mobile applications.

And finally a Watchmen Trailer / Heroes Mashup video:

Share

Barcamp Auckland 2008

On Saturday ( 11th July ) I went to [Barcamp Auckland 2](http://bca.geek.nz/)
over at Botany Downs. Like [last year](http://bca.ludwignz.com/) ( which I
seem not to have blogged about) it was a 1 day “web technology” orientated
unconference organised by [Ludwig Wendzich](http://ludwignz.com/) and a group of
helpers.

Once again the venue ( Botany Downs Secondary College ) was great although
this time around the weather was nasty so things were a bit cold. I also
had a headache and queasy stomach so most of the time I felt like [this](http://flickr.com/photos/talios/2663111543)
( although at one point I [felt like this](http://flickr.com/photos/zakhenry/2660119763) , I think the pills had just kicked in ). So
my concentration skills weren’t on their best.

So here is a list of stuff I went to, the programme doesn’t seem to be online
anywhere so things are a little spares wrt names, titles and the like. Most of
the talks were fairly informal with a lot of audience participation. I
have an idea for a website so I went along to a couple of the more commercial talks.

**Productivity**

A talk from [Ben Young](http://blog.bwagy.com/give-users-what-they-want/) followed by a bit of a round table about productivity. The usual
stuff like only checking your email now and then and otherwise trying to
avoid interruptions plus a few other ideas about.

Suggested Tools:
[GoTo Meeting](https://www.gotomeeting.com/) , [Kayako](http://www.kayako.com/) , [OTRS](http://otrs.org/)

**Advertising**

A great overview by Regan from [throng](http://www.throng.co.nz) ( NZ TV website) about their experiences
with advertising and making money from their site. He discussed the various
text ads, textlink ads, Video Ads ( recommended [Unruly Media](http://www.unrulymedia.com/) ) and affiliate programmes.

Good discussion from others in the room including one guy who is getting 20% clickthoughs on google ads(!). There
was also some discussion about getting New Zealand ads onto smaller sites. It sounds like some people
are talking about an ad network for independent publishers.

**Frontend Performance, Yslow etc**

I organised a little session ( with a 5 minute’s notice) about the [Best Practices for Speeding Up Your Web Site](http://developer.yahoo.com/performance/rules.html)
stuff that has been big for the last year or so and experiences people had had. I can’t say it
was a very good session. I didn’t prepare or know the topic backwards so I was pretty disorganised. There was some round table discussion about people’s
expediences but I can’t say I was very happy with how it went.

**Windows, IE**

One of the guys from Microsoft talked about the problem of getting people off IE6
and up to the latest XP service packs. Some 25% of the population are still on IE6
( another 25% Firefox and around 50% IE7). The idea is to get these people
upgraded to secure systems and using standards compliant browsers.

There was also a bit of talk about IE8 and how a good percentage of of sites with “IE hacks”
didn’t render nicely anymore because IE8 was now more standards compliant.

**SEO**

Another session from [Ben Young](http://blog.bwagy.com/give-users-what-they-want/) with some general
advice of Search Engine optimisation. Lots of good general advice on setting goals, monitoring progress and different areas
plus some special sauce on building links in a non-spammy way.

**iphone seller**

A talk from a guy from [Mob](http://www.mob.co.nz) about how:

1. Fly to the US
2. Buy 15 iphones
3. Attending technical conference
4. Fly back to NZ
5. Unlock and sell iphones
6. Profit!
7. Goto 1

Turned into a real business.

**Careers**

A session of 20 people giving [Ludwig Wendzich](http://ludwignz.com/) ( 17 year old main organiser)
career advice. More interesting than it sounds.

**Other**

There was a bunch of other stuff:

* Ran into the usual suspects and couple of people from back at Telecom
* Saw [Rock Band](http://www.rockband.com/) game.
* Didn’t see the 5 minute coding competition due to crowd
* Lunch plus two breaks worth of food served
* Cute little online app to synchronise stuff, book rooms and see what was happening
* Around 40 people didn’t show up when others got turned away 🙁

**Other Write ups**

* [Official Bebrief](http://ludwignz.com/index.php/site/journal_entry/barcamp_auckland_2_debrief/) from Ludwig Wendzich
* [Simon Gianoutsos](http://gianouts.blogspot.com/2008/07/barcamp-auckland-2-synopsis.html)
* [Ben Young](http://blog.bwagy.com/give-users-what-they-want/)
* [Some guy without his name on his blog](http://www.devour.co.nz/2008/07/13/barcamp-auckland-2008/)

END

Share