Category: Tech

I ended up staying up quite late on Wednesday night so I was a little zonked out on thursday morning.

Keynote – Glyn Moody

• Interviewed people for “rebel code” , found free software people “very nice” even compared to other people in computer industry
• arXiv.org setup week before Linux kernel first released (Aug 1991)
• Overview of public Library of science
• Human Gnome project – DNA inherently digital
• Bermuda Principles – finished annotated sequences submitted to public database
• Jim Kent published and got full human gnome into public domain a short time before Celera finished their work and could have patented everything.
• open data – data is not published just results – example of recent climate data being released, not a big problem if it had already been in public.
• open notebook , reqular updates on progress
• http://en.wikipedia.org/wiki/Open_Notebook_Science
• History of sharing art – Project Gutenbery 1971  .10 books 1991 , 1000 in 1997.
• Various free licenses slightly incompatible , hard to convert between, took several goes to get licences correct
• wikipedia – easy not programmer example of sharing tht people can understand – “open source is wikipedia for code”
• Open government is more “Shared Source Government” rather than “Open Source Government”
• Global economic crisis – tragedy of the commons
• At least the Financial crisis has some winners
• Very anti financial system, suggest more  “open source” options and commons
• “if you share stuff you are destrying property, you are taking jobs away from the poor people” – How the debate is being framed

It was noted by one person that this year’s keynotes are more “Freedom” and “High tech”.

Lindsay Holmwood – Flapjack and Monitoring

• Check – unit test – good bad ugly
• Monitoring system – monitors for failing checks
• 3 questions for monitoring systems – next check? , was check okay?, who do we notify? . Fetch , test , notify
• fetch – lookup
• test – execute , verify
• notify – decide , callout
• traditionally done in single process
• but it’s an embarrassingly parallel problem
• parts can be split. fetch+test fetch+notify – pass id/command between
• precompile checks – so fetch is less expensive
• transport between processes is the scheduler
• no data collection when testing (graph seperately)
• scheduler – workqueue – filled by populator, assigns stuff to notifier and workers
• Lots of workers can be created (to do test)
• flapjack – in ruby , talks to nagios plugin format
• beanstalk – ansyncrnise workqueue service – ubuntu/debian packages
• beanstalk – producer  puts jobs on beanstalk , consumer takes jobs off
• uses named tubes (queues) , multiple tubes per instance
• flapjack-worker – started up by flapjack-worker-manager starts multiple copies on machine. various control commands
• worker is simple so linear scaling, spread across multiple machines required
• flapjck-notifier – has manager to start it.
• notifier has recipients.conf file with list of people to notify
• notifier.conf – config for various notifiers (MAIL, SMS)
• APIs – notifiers, filters, systems
• notifier API – who , when and how sort of stuff.
• “how many here use puppet – about a dozen – How many use Chef? – none “thanks a shame” “no it’s not”
• persistence API – store stuff , mysql, couchdb whatever, standard way to store data.
• filter API – parent checks hierarchy (so don’t check ports if host down)
• flapjack-admin – pending – nodes , check templates , checks (check template + node ) , batches (group of checks)
• 3 types of checks
• Gaugaes – stuff within range – collectd ( point flapjack at collected output )
• Behavoural tests – cucumber-nagios
• Trending – reconoiter – growing area
• collectd – gets stats from anything – nagios bridge – collectd-nagios queries collectd data
• collectd client – gathers data from node and sends to collectd server
• collectd forwarding server – agregates, filters and forwards
• falapjack – crrently gems, soon to be real packages
• http://flapjack-project.com

Bob Edward – Yubikey authentication in a mid-sized organisation

• Reusable passwords are dead , hard to remeber, something you know which can be shared and discovered, captured, guessed
• Alternative – One time Passwords – doesn’t matter if captured.
• examples – RSA keys, SMS based systems, Yubikey, 2 factor authentication
• Created by Yubico in sweden, open-source
• Looks like a USB keyboard to a computer, generates a 44 character OTP each time button is pressed. No batteries, 2st 23 characters fixed for each key
• $12 each in volumn – $40 as one-off
• Based on secret AES 128-bit key
• Yubicoships yubikeys with pre-generated IDs and AES keys. Offer publicauthentication, they know secret 128-bit key, need to trust them
• secret-id+sess+timestamp+session+rand+CRC  string created by key , then encrypted and public ID prepended.
• Server decrypts , checks checksums and looks to make sure secret-id matches and session and timestamps are incrimented from previous values.
• Unless you trust and always want to use Yubicom’s servers you should reprogram you keys with your own keys and IDs. Can’t then be used against Yubicom’s server.
• weaknesses – requires computer with usb port that accepts usb keyboard – some bugs with 1st generation keys – unused generated keys remian live until the next valid key is used
• You can run your own server fairly easily – ykaserver – various interfaces, postgress database for storage – can also call out to PAM for two-factor authentication
• softykey – software Yubikey – can use to generate 1-time pad for stuff without usb keyboard interfaces
• Tested with ssh, VPNs , web logins – mostly use PAM or LDAP method
• See Linux Journal and yubico.com

vimperator – automatic launch prog for netbooks

Jan Schmidt – Towards GStreamer 1.0

• History of dev, faster bits during hackfests, when switched to git etc
• Overview of last year, switched to git, slowdown when people busyswitched to binary registry
• Support for various DVD playback  functions, special subtitles etc.
• I’m not really in this area so I was just listening to get an idea where things are going. A bit too much detail for me at times.

Adam Jackson – The rebirth of Xinerama

• Once again this was a bit over my head. It does look like the X guys spend a lot of time fighting assumptions built into the protocol and code 10 years ago however.

Stewart Smith et al – Building a Database kernel with Lego Like parts (Drizzle)

• What would you change about Mysql – Modular architecture
• Some crazy legacysuff in the Mysql code – good oppertunity to clean
• move alot of code out of core, especially option parts – understandable and to reduce load – don’t load if you don’t need
• more code coverage with tests
• plugin interfaces – protocols, replication , logging, etc
• modular replication system
• general refactoring of storage engines
• “If part of API sucks then fix API rather than work around it”
• New this week – rot13() powerful encryption
• Authentication plugins – auth_pam , auth_http
• Various Logging plugins – logging_query , logging_syslog
• Drizzle Community – All contributors equally – All project information public – No contributor license agreeements – Release early and often (~2 weeks ) – 100+ contributors , 500+ on mailing list
• Milestone releases
• When production release? – waiting to solidfy compatability – Sounds like a few months. – Reliable but still in flux
• Pacakages to be pushed out to dists once things stable

Afterwards I had some dinner and went to the Professional Deligates networking session.

Wednesday is the first day of Linux.conf.au proper. I thought that today I’d just keep my notes in a blog post to prevent doubling up.

The keynote was Benjamin Mako Hill talked about various things the most interesting bit was “antifeatures”. Things like DRM, crippling of products etc. The one of these I most hate right now is they way that cheap netbooks have fairly low specs (small resolutions, low RAM, slow CPUs ) partially because they have to keep the spec below a certain value in order to qualify for the really cheap Windows license.

The dreamwidth talk was quiet interesting (although the speakers pre-rehearsed banter between the speakers didn’t really work). Lots of practical examples , war stories and good sound advice.

Selena Deckelmann talked about choosing which open source database your should choose. The quick answer is “what problem are your trying to solve?”. She did a survey of the 50-odd databases out there and got 25 replies. Also did her own research and comparisons. Classified DBs into several categories (which I won’t list) such as

• General Model – Key-Value, OLTP.
• Distribution model (replication, partitioning, sharing).
• Memory vs disk (eg keegin g everything in memory only like memcached).
• HA options, Node failover.
• Code dev model – Core +modules , Monolithic , Infrastructure
• Community dev model – Dictator, Feature driven, Small group, A mix

Results at http://ossdbsurvey.org

• Databases implement each others protocols
• Need verification that protocols correctly implimented
• Need tools/test to check things like replication working
• More connections between projects/people (eg java seperate)

Ted Ts’o – Production-Ready filesystems

• Hard to make robust. Many different workloads, lots of state, very parallel
• Hard to balance getting it out with getting it stable enough to be fairly safe to use
• 75-100 persons-years for filesystem to be production ready.
• eg zfs around a dozen people , start 2001, announced 2005, shipped 2006, people confident with it around 2008-2009
• Ext4 renamed from ext4dev at end 2008
• Ext4 Shipping is some community distributions, soon in some enterprise distributions, widespread adoption 12+ months later
• Lots of bugfixes still in ext4, most not real-world and picked up by auto-tools or careful checks in weird conditions.
• Ted: “my other prefered term for Dbench is ‘random number generator’ “
• Paths like online resize, online defrag that are not regularly tested by users or testers so source of many bugs.
• Many bugs were in the recently subsystems and features
• Making General purpose file system takes longer and a lot more effort than you might expect. Labour of love, hard to justify from business perspective.
• Solid state drives with “flash translation layer” in place are fairly much the same as spinning disks. Extra optimizations for disks don’t help but they don’t hurt

Matthew Garrett on the Linux community

• Started by listing things he’s not talked about
• The Linux community is “Like the Koreas”
• To be a member of the Linux community “you just have to care, just have to turn”
• As community we are very hostile, it’s seen okay to flame and it is being rewarded still
• Should we stop just cause it’s a nice thing to do or because it’ll stop scaring people off?
• Ubuntu code of conduct has mean’t that users are consider part of the community more than in other distributions
• Code of Conduct must be enforced or it’s useless
• “We value code above all else… not a good thing” . We need people to feel that by using software they are part of something
• Communty entirely based on technical excellence or encompasing everybody who users, cares, contributes to projects
• Idea for positive examples Wiki with pointers to COPs and best practice examples
• Not gained behavior standards normally associated with grown communities

Sage Weil – ceph distributed file system

• How different
• scaleable to 1000s , grow from a few
• reliable, HA, replicated data, fast recovery
• snapshots, quota-like accounting
• Motivation – avoid bottlenecks and symetrical shared disks
• avoid manual workload partition, p3p-like protocols, intell storage agents
• POSIX file system , scaleable metadata server
• metadata (MDS) servers/clusters and object store boxes seperate
• CRUSH hash function used to distrubtute objects across devices, works as devices are added. Spread them out explicitly across infrastructure if required
• fast (no lookups), relieable, stable
• celp object storage daemon on each node
• talks to peers on other node: rep data, detect failures, migrate data
• hashing fuction means nodes don’t have to negotiate with each other, CRUSH says where data is going.
• monitor storage nodes, moves data around, make sure it’s in the right places, uptodate. fixes if required.
• raw storage API if you don’t need full filesystem fun (dirs etc)
• proxy that emulates s3 REST interface
• metadata cluster , uses object store for all long term storage, needs memory and fast network for performance.
• metadata streamed to journal. large journal (100s MB) flushed now and then
• snapshotting on per-directory basisi via simple mkdir
• snapshot leverages btrfs copy-on-write storage layer
• file systems client near-posix
• kernel client, FUSE, Hadoop clients
• stable but not production ready
• client should be in mainline kernel soon
• aim to work in multiple datacentre, across unrelieble links
• http://ceph.newdream.net/

Paul Fenwick – Worlds Worst Inventions

Not really a technical talk. More a few stories about funny inventions. Quiet amusing but I’m not sure it fits in with the rest of the conference.

First real day of Linux.conf.au is always full on anticipation. I woke up a little early and nibbled a small breakfast as I walked from ustay to the venue. After the crap weather on the weekend things were stating to look a bit better.

The signup are at the venue was fairly quite with people being processed quickly and many having been signed up for the weekend.

First up was the Welcome talk which had a few hitches. Due to illness it was being given by and understudy who was a little unpracticed with the delivery and had a problem when the overhead screen went blank for 5 minutes due to technical problems (not sure if it was the screen or the laptop’s fault). Highlights were a 42-below ad for Wellington and everyby singing Happy Birthday to Rusty.

I spent the first couple of sessions at the Haechsen/LinuxChix Miniconf since most of the topics were interesting and for various reasons (mumble mumble) talk times between miniconfs were not sync’d so it was hard to move between them.

It looks like this year the video situation is fairly good. All Miniconfs and main sessions are both being streamed live (although in wma format which caused some comment ) and being record for later download. Hopefully It’ll all work out.

Talks I attended:

• Version control for mere mortals by Emma Jane Hogbin was a good intro to VCS and practices including a bit aimed at sysadmins and content maintainers rather than just coders. She obviously likes Bazaar a lot more than git. Goods intro and once again I feel guilty about not using it more.
• Happy Hackers == Happy Code by Sara Falamaki was an overview of what makes programmers happy. Mostly concentrating on tools but with some other bits and pieces mentioned. Great, especially the bit where Sara started throwing (often wildly) lollies to members of the audience who made good suggestions.
• Through the Looking Glass by Elizabeth Garbee gave here perspective on using open source software and the high-school level. Interesting stuff on tools, and how other teens viewed open source and programming and the scary story about how her school had a rule that any student how bought a computer to school running Linux/Unix would be expelled!
• Creating Beautiful Documentation from Lana Brindley covered some high level bits of the process redhat uses to create documentation as well as a bit of an overview of what technical writers do and why their jobs rock 🙂
• Getting you feet wet for Angela Byron gave ways and advice for getting involved with Open source projects ( including the old “woman’s work” (my, not her term)) of documentation etc. Pretty good.

• Code of our own from Liz Henry was about the first feminist orientation talk of the day. Lots of stories and advice for women in open source as well as a few bits where she gave your low opinion of how well some ideas have worked in practice.

Overall fairly interesting sessions. I noticed that for most of the 2 session the majority of people in the room were male and quite a few of the audience questions/comments were from them. This didn’t really cause a problem for most talks which were on general topics but I noticed the “male perspective” was less useful/welcome for Liz Henry’s talk.

For Lunch I wandered around a little bit an eventually found a place called “The coffee club” where I had a soy milkshake and a pesto bruschetta. Very nice.

For the last session I went to “The business of Open Source” Miniconf and then “Libra Graphics”

• The 100 mile Client Roster from Emma Jane Hogbin was an interesting overview of the way her business and business model has evolved and where she thinks the next step is. Good talk and delivery although it’s a bit outside my area for me to give a good review of the content.
• Building a service business using open source software by Cameron Beattie didn’t really appear to me. The talk was a bit flat and delivery lacked much spark.
• Cheap Gimmicks to Make your designs ‘New’ by Andy Fitzsimon from suffered a bit from technical problems with delivery but looked like there was a good talk in there somewhere that just required a bit more prep.
• Dynamic PDF reports via XSL and Inkscape by Peter Lieverdink was cool but a little over my head.
• Inkscape: My Cheerleading Adventures by Donna Benjamin was a little sparse even for a 5 minutes talk

After the end of the day I went along to a Wikipedia Meetup at the Southern Cross Hotel. The Meetup was fairly small ( just 3 other people) but interesting people and several hours of discussion. Some talk about a NZ Wikimedia Chapter and also helping with the Wikimedia stand at the LCA open day.

Last up I grabbed a coffee and cake at Midnight Espresso.

Overall not a bad day, tomorrow will by Sysadmin Miniconf all day wih the Speakers Dinner in the evening.

A few things I’ve been looking at or intending to look at over the next few months.

• I’ve bought a new computer a couple of weeks ago for home. The computer is intended to replace the house server. The main functions will be as a file server and host for virtual machines. The big changes is that I’ll be switching from Xen to KVM as virtualisation technology.
• KVM + PXE + Kickstart + Ubuntu  – I really want to build my virtual machines automatically and at the same time to be using a more general machine building method . This page on the Ubuntu site looks like it is a good start and I’ll blog a bit when I get it all done.
• I need to do some work on Mondo Rescue , I have a bug I reported that is supposed to be fixed and I have to test.
• GlusterFS is a distributed network file system that looks really cool, I’m intending to play with this a bit.
• Once again we’ve applied to do a Sysadmin Miniconf at the 2010 Linux.conf.au conference. Once again we hope to have a really good miniconf. However no less that 32 miniconfs have applied for just 12 slots so not sure if we’ll get in. We were really popular last year but personally I’ve no idea what our chances are this year. Bit down about the thought of not getting but I guess whatever happens will happen.
• I keep getting good ideas for websites and products. Not programming and having poor time control means most of these ideas are probably not going anywhere. Maybe I’ll try a couple of them though. Also got some further ideas for technologies to play with but want to get the ones above sorted first.