Category: Tech

The keynote this morning was from Simon Phipps from Sun. I thought he was quite good especially since he was in front of an audience that was not 100% friendly. One of the interesting statements he made was pointing out that it was hard for a company to install a free version of Redhat ( say Centos or Fedora ) and then later start getting commercial support for it. As things are right now you would have to reinstall all your servers with RHEL in order for it Redhat to support you. He felt that sooner or later Redhat would have to change their policy in order to allow easy transition for people, although at least one redhat person in the audience either missed his point or completely disagreed with it.

Next I went to a talk from Mathew Wilcox on Solid state drives. It was pretty interesting although a little over my head.

Afterwards I hosted ( to the extent I stood up, wrote notes and pointed at people) a BOF for Miniconf organisers. Around a dozen people showed up including about 3/4s of this year’s miniconfs plus at least one perspective on for next year. We had a good round of discussion and I wrote up a few notes (not really for public sorry , but contact me if you have a special interest) and somehow volunteered to help setup a Miniconf Organiser’s howto document.

After lunch I went to sessions on Power management and usability. Bother excellent and giving me a chance to pick up some information in areas outside of what I normally do.

Then it was a presentation from Terri Irving at Dreamhost, she did some overview stuff about how they do things and then a little bit about how they use their internal “servicectl” tool to provision and run their services but not a lot of technical nitty gritty. She used the second part of her talk to introduce the Ceph distributed file system that one of the Dreamhost people are working on ( which is publicly released) so the talk wasn’t a total loss though.

I had a bit of a headache so I skipped the lightning talks and the wind-up and announcement for next year. As expected it’ll be in Wellington. Generally I think this should work out okay, the extra distance for Australians should be balanced by a good number of locals attending, I did hear some concern about lack of the publication of the bid documents and the fact that the Wellington organisation seems to be a “company led” rather than a “community led” . Also I am not sure if cheap accommodation is going to be available, having the college dorms like in previous years is great, lots of space, security and closeness to the other attendees at a low price. Hopefully Wellington can do something similar.

Anyway I missed all of that and had a snooze till after 7 when I got up and went into the Sandy Bay shops with a few guys. On the way we went via the party but it didn’t seem to be much fun. The drinks were “buy your own” (probably to avoid problems from previous years) and the food was of the sausage roll type ( I heard that Google was unimpressed with the food quality after having dropped a lot of $$$$ towards it).

Ended up going to a nice pizza place with 4-5 other guys and having a really nice pizza at a good cheap price (low $20s per head) . The manager even gave us some shots at the end free ( cool, although I don’t drink so just had a small one).

Next day was the the semi post-conference with the Open Day being the only real official thing happening plus a cheesy sounding “march” from Salamanca place to the Open Day ( at the Casino). I decided to skip these and went to the Salamanca market instead. I was really impressed with the whole thing, 2-300 stalls of mostly high quality with plenty to choose from, since I was flying back to NZ I couldn’t buy much food to take away but I got some nice fudge and ate at a couple of vendor stalls. Couple of galleries by other people here and here.

And apart from an uneventful trip back that was my linux.conf.au for 2009.

Overall I felt the whole event was on par with previous years, I understand they had a few speakers and attendees drop out at the last minute which was a bit unlucky and the extra travel distance probably put some off. I got them impression there were not as many locals as in previous years but I guess Hobart isn’t a big place.

The organisers seemed pretty on the ball most of the time and largely kept in the background compared to previous years. The weather was pretty good (apart from some sprinklings of rain) and not enough to extremes to cause problems. I’ll definitely be back again.

I noted that a couple of things were different about the start of each day than in previous years. The first was that the doors to the main area didn’t open until after 9am each morning. So the actual start time was something like 9:20. This was compensated by there being a smaller “pre-game” show than in previous years but the tone of things felt different. I also noticed that the sponsors seemed to be getting a bit less publicity than previously. Usually they hang a couple of sponsors have their banners in the main hall and on the pre-start slide show (along with useful announcements) but this year I didn’t see as much and the slide show seemed to be mostly about the art-project thing.

The keynote today was from Wikimedia person Angela Beesley Starling talking about various wikipedia and wikimedia stuff. I didn’t feel I learn’t a lot but I follow wikipedia a bit already so I am perhaps not a typical person. The speaker was also a little quiet.

Next was a good talk by Paul Wayper about how he contributed to the lmms ( linux multi-media studio) project with general observations and advice for both projects and potential contributors. Very interesting and useful. I liked Paul’s rendition of  the poem “How McDougal Topped the Score“.

I went to a a bit of a legal thing next on how the lawyers have problems grasping what the free software people are coming from. Interesting enough.

After lunch there was a bit of a change in the session I went to and it was split in two, the first half was on problems with the kernel API process, mainly with how new syscalls (and other kernel interfaces) are getting created and pushed out (and once the are live they are set in stone) and then people have second thoughts about if they were the best way to do things. The problem was highlighted ( and Michael Kerrisk mentioned that he would shortly be losing funding so his documenting of the calls would suffer) but solutions were missing.

The second half was a talk by OpenSUSE guy (and former journalist) Joe ‘Zonker’ Brockmeier on publicising your project. Lots of good stuff on press releases, websites and setting priorities with a few good and bad examples.

I skipped the proper sessions next and went to a Puppet BOF. Some advice there for people implementing puppet ( which is a big item on my TODO for work this year).

Last up I went to a talk highlighting some tricks the speaker had learned from others. Interesting enough but I didn’t really pick up many I could use that I didn’t know already.

The Professional Delegates Networking Session ( pdns, not to be confused the powerdns ) was held at the Cascade Brewery’s visitors centre. Pretty nice place with lots of drinks on tap and nice food. I’ve decided I quite like the Cascade soda drinks , especially the raspberry. The main problem was the the weather got a little cold and wet ( actually it was raining) after an hour so almost everybody was crowded inside rather than being spread out. I thus found it a bit noisy and crowded so I perhaps didn’t enjoy it as much, however nothing much you can do about the weather.

The keynote from Tom Limoncelli didn’t come across as well as I would have hoped. Tom’s books are really great but he was a little quiet as a speaker ( possibly caused by the sound system ). He was also talking from a Sysadmin angle which is good for me but didn’t reach 75% of the audience. I took away stuff especially the homework projects though so I was pretty happy.

The first talk I went to was on remotely managing ( across a 3G phone) a series of computers that were in a remote rural area. Really interesting, detailed and a great presentation from Thomas Sprinkmeier. Although he was a bit cagey on the purpose of the whole thing.

James Turnbull’s puppet talk was well attended and pretty good.

The Django tutorial was good although a bit over my head ( and I was feeling a little sleepy), the presenter answered a question I had though and provided a bit of insight on why a feature I was after wasn’t directly supported.

The kernel development talk from Jonathan Corbet was good. It both aimed to help potential developers get involved ( especially those from vendors ) and provided some insight for those generally interested. Comments from Linux during the talk were great as well.

I was really disappointed at the talk by Sarah Stokey and Jeff Waugh on the Crikey website and blogs. I have a rule of thumb that says if you spend more than a minute talking about your organisation (unless that is the point of the talk) then you are in trouble. However at least half the talk was on the history of the site, bios of the columnists/bloggers ( one after the other) and bits about how they got sued by such-and-such. This stuff was a total waste of the audience’s time, and nothing like what was promised in the topic description. I work on a media site and I was hoping for some actual nitty-gritty on the project, how they got some buy-in and the problems they encountered. Jeff did do a bit of this (after he spent 5 minutes making faces at the camera) and also broadly did some basic technical stuff about wordpress but I really wonder if a good talk missed out when this one was let in.

Zach has already documented the auction at the conference dinner, I’d like to add that I thought the dinner was quite nice. The food was great and overall pretty well organised.

2nd day of linux.conf.au 2009 I was again at the Sysadmin Miniconf all day.

The first talk was by Matt Moore on how he scaled a hosting solution for a charity ( $20 of donations over 6 weeks ) website and ensured ( in the face of some big problems and not huge budget ) reliability of the site ( where outages cost $1000 per minute during peak ).

Next there were a couple to IPv6 talks from Glen Turner and Angus Lees. Angus’ talk covered how he and a couple of other people were working towards getting a AAAA record for www.google.com . They have spent the last year or so working towards it. The most interesting bit was they were doing survey ( using a web-bug on 1/1000 user ) to see how many users of google could access a URL that had both a A and AAAA record. His stats found that when both records were presented 0.238% of users used the AAAA URL and 0.09% broke. The 0.238% was increasing during the period when he tested ( late 2008 ). He also found that access ( ie route path optimisation ) was around 150ms worse than the ipv4 path. Overall he seemed to think google could go live with ipv6 records in 1-2 years.

Glen’s talk was a bit more general with some intro to ipv6 and the way ipv4 addresses were running out. But he also addressed the concerns that in 3-5 years when ipv4 address run out ipv6 won’t be ready and ISPs will start implementing NAT solutions which will break the end-to-end connectivity of the Internet. He also thought that large access providers would try and use this to try and extract a greater control and share of the revenue from the Internet ( ie not just access but a share of the profits from the end sites in return for proving eyeballs).

After Lunch Devdas Bhagat covered database backed authoritative DNS servers. His company hosts a high number of domains and needs very high performance. He did a good comparison and work though of how they tried bind and eventually moved the PowerDNS ( pdns) . They paid a developer to enhance some of the paths in the system and that latest releases ( a RC with pdns although Devdas has been running it for months ) get some 40,000 q/s on standard hardware ( which exceeds the gold standard commercial product from nominum ( do do do do do ) ).

Steve Ellis then covered trac with some concentration on using trac ( and similar solutions ) better to tie in documentation, tickets and workflow better rather than just keeping them all separate.

The Lightning talks covered ( SAGE-AU , Time management, DKIM , Computer Room disaster stories and a quick plug from a linux based router vendor ). I especially liked the DKIM intro presentation which reminded me of the old format people used to sign Usenet Control messages.

The session covered Spam, Roland Turner from Boxsentry covered the product his company uses to reduce false positives. He also announced that his company was allowing people to freely talk to his companies system ( probably via a spamassian plugin ) up to 100,000 queries/day. See http://rex-tools.sf.net .

Peter Chubb then did some general spam filtering advice.

Last up we had a BOF talking about what software and hardware people were using on their site. A good chance for people to catch the names of tools others were using and ideas for what they could look at. One thing I was interested in was that only around 5 out of the people in the room were actually using puppet on their site.

So after all the preliminaries it was finally time for the actual Linux.conf.au conference.

Breakfast was a fairly common dinning hall setup and I walked down the hill to our venue. The Sysadmin Miniconf is in a 200-odd seat lecture theatre right next to the registration desk. We are  getting recorded so hopefully video’s of most talks will be available at some point. About the only problem with was that there was only a hand-held microphones available so we have problems when ( for instance ) people were trying to type and talk at the same time.

First talk was by Andew Bartlett on the directory ( as opposed to file ) service in Samba 4. He gave a basic overview followed by a lot of questions.

Rickard Keech then gave an overview on how he provisioned servers. It was a little different since it was a CD based method ( since many of his customers were fairly small and not well network connected). He is automatically building the kickstart files and testing and rebuilding them to ensure that the server is in sync with it’s deployed version ( much like maintaining a source tree and compiling it into a kickstart iso).

Several other presenters pointed out how sysadmin practice is 20 years behind programming ( no version control, testing, libraries, development environments etc).

Similarly Dedvas Bhgat talked on how his team was able to use configuration management ( puppet) to hugely reduce the load ( and stress ) on his team and to improve service. He is using puppet to look at ~300 servers.

The next two talk were a bit over my head. The first discussed a new snapshot implimentation which was much more efficient that the previous one while the second ( which was in the miniconf section ) demonstrated adding modules on a Xen server which was able to probe running Xen Child machine ( the extra shown was spotting signals being sent to the apache process but it looks like just about anything is available).

The lightning talks were next. Devdas talked about programming practices while I did my Mondorescue and PXE talk.

After lunch there was a talk about the setup (and large amounts of data and processing) at the NIWA ( New Zealand weather research agency )  and a talk on Performance Co-Pilot (PCP). I found PCP quite interesting, it seemed to be a fairly simple was to get date and states out of a server and access them ( realtime or later) for graphing, alarming etc.

Robert Postill then gave a talk on deploying Ruby on Rails with lots of experience and advice. Pretty good and applicable for similar systems like Django.

Last up there was an intro to Redhat Satellite and RHN.

For dinner we organised a little Sysadmin thing at the Bay Leaf Bistro on Sandy Bay Road. The place was really great, food was simple and great ( suburban ) while the service was excellent. They had a great system for taking orders and doing individual bills and overall were really friendly. Best restaurant experience I’ve had in ages.

During and after dinner I was chatting to various people so didn’t get to bed till a bit late. I probably need to get to bed earlier in the week or I’ll be tired out.

Today was my first full day in Hobart. After a good night’s sleep I went into town with a couple of
people to get some breakfast. Had a nice eggs benedict at a pub/cafe down by the waterfront in the  middle of town and then a couple of us walked back to the motel and caught a taxi up to venue.

The university accommodation is right at the top of the hill so but the rooms that about 2/3s of the people are in are pretty nice. They are 6 bedrooms with 2 toilets and a large common are ( kitchen, stoves, fridge, freezer, lounge and TV). It all feels only a few years only ( lots of pwer points for example).

After getting some problems sorted out with room access and finally geting my missing bag from Qantas ( and have a change and a shave) around 10 of us wandered over to the Sandy bay shops for dinner. We split up and 5 of us had some Indian curry (okay but a little expensive) and then headed back via the supermarket.

Looking at tomorrow I’ll be at the Sysadmin Miniconf pretty much all day. I have a lightning talk to deliver (some more practice tonight) there and the programme to organise. I’ll have a quick look at the other miniconfs and see if there is one I’m really keen on I can sneak away but probably not many.

Dinner will be the Sysadmin dinner which will be at a little place we’ll book tomorrow. Over things are looking pretty promising for LCA and organisation seems to be reasonable.