Category: Linux.conf.au

Use the force Linus – Jon Oxer

• Usability, computing falacies
• The best UI is no UI
• Things should just work, you should have to do anything.
• eg your door should just unlock for you, not for others
• eg your TV should just play what you are watching, stop when you move away
• Story about reverse engineering of Kinect scanner for linux
• Demo of controlling helicopter via gesture
• Demo of controlling curtains

Kernel Development – How things go wrong – Jonathan Corbet

• Kernel dev a success – ~4.5 releases/year
• Linux soon to your toothbrush
• Why talk about failure? Gives kernel process a bad name, scares people off, bad press, failure can teach us
• Note: The kernel community does not lack for clowns – won’t be covering them, names named here are people he respects
• Example 1 – tux3 file systems by Daniel Phillips –
• 2008/07 announced , 2008/11 booting , 2009/08 last commit – project now dead
• Do NOT add stuff to out-of-tree project, merge early
• Lesson out of tree code – for users, few contributers, invisible, have to keep sync’d with mainline, harder work
• Lesson – get in mainline early
• Example 2- em28xx video4linux driver
• 2005/11 merged , 2008/01 last patch by ori author, 2009/08 author leaves completely
• “People submitting code have to be aware they will lose control”
• eg 2004/05 Hans Reiser tries to block updates to reiser3 fs , wanted to keep control and concentrate on reiser3
• maintainership does NOT mean ownsership
• Example 3 – 2.5.x IDE
• 2002/02 Martin Dalecki’s IDE cleanups , 2002/03 IDE18 subsystem takeover , 2002/08 IDE115 merged, 2002/08 Martin’s quits all IDE work reverted
• What happened? – “Breakage is the price you have to pay for advancements” – martin Daleski .
• Kernel intolerant towards regressions, listen to people who complain
• Example 4 – Deadline Scheduler
• 2007/03/ first post, 2 days later Linux likes, 2 weeks later Linus get irritated (due to regressions), 2007/04 Molnar posts CFS, 2007/07 CFS merged, 2007/07 Con leaves
• Con leaves unhappily
• Lessons – Improve for everybody, not just a small set, at least don’t make it worse
• Lessons – Some parts of the kernel are hard to change
• Lessons – Participate in the wider kernel mailing list discussion, not just list that is for your changes, subsystem
• Lessons: Aim for the solution to the problem, not for the inclusion of specific code
• Example 4 – reiser4
• 2002/10 First code post, 2003/07 merge request, 2004/08 added to mm, 2005/09 push 2.5.14 , 2006/07 push 2.6.19 , 2006/10 arrested
• Problems – Non-POSIX behavior , numerous technical difficulties , hard-to-reproduce benchmarks, antagonistic approach to others, memories of reiser3
• lessons – Linux is not a research system
• Lesson – visionary brilliance will not excuse pore implementation
• lessons – it’s best not to access others of conspiring against you
• lessons – community remembers the past and thinks far into the future
• Example 5 – systemtap
• 2003/11 Dtrace debuts, 2005/10 rhel4 introduces systemtap, 2008/07 Ftrace merged, 2009/06 perfevents merged, 2009-09 systemtap 1.0 released, Not yet merged possibly never
• 2008 kernel summit, 50% had used systemTap, 20% had succeeded
• bad sign if even kernel devs can’t make it work
• lesson- if kernel dev community doesn’t see value, it won’t go in
• Example 6 – Talpa – provide hooks for anti-virus
• posted in Aug 2008, never merged as such
• Problems – Kernel devs didn’t like, why broken security model, badly expressed requirements
• But – fanotify is accepted, same authors, similar code
• What changed – cleaned up file event notification (replace inotify and dnotify)
• What changed – Enable virus scanners to hook into file ops without using rootkit techniques
• lesson – sell to developers not to mangers or customers
• lesson – user-space API matters, hard to change later
• Example 6-15 posted
• Why bother? – “Why not hack on CMSses or something else where the standards are a little lower”
• Fun!
• Elite club
• you will get job offers
• Influence – how the kernel meets your needs
• VFS filesystem patches by Nick Hagen(sp) – worked – intrusive and tricky code – good code – listened to people – didn’t behavior for people, all behind the scen

Sendmail revisited – Eric Allman

• sendmail more than 30y old, survived well, changed the world
• Start 1980 at UC Berkeley, no support, before Internet existed, built as matter of nessesity
• Then: CPUs 16bit, <1MIPS  . Disks <<1GB ,  Memory < 1MB  , Network <56k/s  ,  DBs research only. Phones huge and uncommon
• Today: CPUs 64bit, >50k MIPS , Disks 2TB , Memory >100 GB , Network Megabit to ~160GB , NOSQL hot, RDMBS boring
• 9.6k link to arpa link, only 2 ports for everybody to share, $5k for 1 tty port
• Observed people just wanted email. Solution forward messages between networks
• design principles:Only 1 programmer, don’t redesign user agents, don’t change local mail store, make delivermail adapt to the world
• Problems: compiled in config, no address translate,
• Transittion to sendmail – UCB awarded contract , adding SMTP support, needed to be build in, forced introduction of queue, queues harder than look
• released 1982 with 4.1a BSD – SMTP, Queueing, runtime config
• sendmail picked up my most unix vendors. Unix wars raged on – vendors added value and incompatible added up.
• Returned to Berkley, sendmail rewrite -> sendmail8
• Added in extensions developed elsewhere. Revision of SMTP, DSNsm other systems, 8bit mail, new configuration package
• sendmail bat book dramatic increased uptake of sendmail8
• Commercial sendmail – Sendmail Inc in 1998. 1st commercial open source/commercial hybrid companies
• Added by commercial company – encryption & auth, milter, virtual hosting, LDAP, lots more checking
• Part 2 Lessons Learned
• Changes in requirements over time:
  
• Reliability – deliver or bounce
• Functionality and performance
• Then became about protection
• Next legal/regulatory issues
• Increasingly controlling costs
• Design decisions and evolution
• Overly general: can do anything, reality world was in flux, easier to build tool than solution, still true today to extent, retrospect would do again
• rewrite rules: good idea, using tabs as active character was wrong. concept right but syntax and control flow could have been better
• Message munging – essential for interoperability at the time, still used – to you pass through transh, reject or fix
• Syntax of config – flat (no nesting) , to much used of single characters, ugly but not fundimentally flawed, would use something like apache todday
• smtp and queuing embedded in sendmail – good idea
• structure of queue – two files per message, requires lots of scans, ascii simplified debugging, correct for time, wouldn’t do now
• Using m4 for config – painful syntax, damn “dnl” ‘s aren’t even necessary. Retrospec some tool was needed unclear if m4 was the right one.
• Extending vs changing features. Example masquerading. Wrong first time, reimplemented but added new feature instead of fixing. Didn’t want to break existing sites. retrospec wrong and big part of why sendmail hard to configure, should have provided upgrade tools
• Accepting and fixing bogus input. permits broken software to exist. right at the time unclear if it’s the right thing today.
• What do diff today –
• Fix problems asap, v7 mailbox format, tabs in config files, bogus features
• use modern tools
• more privilege separation
• create string abstraction
• create string separation
• separate mailbox names from unix user ids
• cleaner config file
• What would I do the same –
• use C as the implementation language
• Bit things off in smallish chunks
• syslog
• rewriting rules (without tab char)
• Net rely too heavily on outside tools
• Takeaways –
• KISS actually works ( sendmail was a “2nd System” )
• If you don’t know what you are doing advance designs don’t help much
• The world is a messy place
• Flexibility trumps performance
• Fix things early – if you succeed installed base only gets bigger
• ASCII is great for internal files and protocols
• documentation is key to broad acceptance
• the design space is always changing
• Q: if you were not Eric and you were starting today which mail server would you use? A: “postfix”

Open source Documentation –  Lana Brindley

• User Interface + documentation = can get into program
• People don’t think where manuals come from
• Like a black box
• At Redhat use 5 phase waterfall model
• Phase 1 – Information plan – work out dates, audience, sources
• Tools – wiki and ticketing system
• Phase 2 – Decide what goes in – look like, title, abstract, where info, ID Subject matter expert. Google, wikipedia
• Tools – wiki , bug system , master tracker bug
• Phase 3 – create 1st draft of document – reviewed by legal, writers, technical review
• bugs raise against doc, fixed, Quantity Engineering
• Tools – publican, draft to svn,
• Phase 4 – translation and publication
• Up to 26 languages
• Don’t publish dead-tree books, single page html, pdf, paged html, epub
• Phase 5 – Review on process
• Geographically spread out, , phone meetings
• Advice:
• Keep *some* documentation, blogs, wikis, comment code, that tech writers can use
• Get organised
• Choose your voice
• Say what you mean
• Use Short sentences. 15-20 words on paper, 10-15 on a screen
• You’re not at University – there is not an exact word limit
• edit with knife
• each editing pass should make your document 20% shorter
• read what you write, walk away from it so you read what it says not what you think it says. Read it aloud. Read it backwards (v short docs), or each sentence backwards.

The latest and coolest with HTML5 video – Silvia Pfeiffer

• htttp://caniuse.com
• Video Manipulation
• Not just a 2 new tags, combining technologies
• htmlvideoguide.net – updates from Silvia’s book
• new websites should probably user mp4 and webm
• CSS3 transitions
• Javascript API and Media
• SVG – not uniform implementation across the browsers
• SVG filters for video only in FF4 for now
• Canvas better implemented across browsers
• Very interesting demos
• face detection in 5 lines of a web worker
• shared video and audio playing
• Demo of implementing audio visualisation of audio samples in realtime directly in htnl
• Generate music directly ( or via samples) . Part of proposed audio API
• Media accessibility – captions, subtitles, audio descriptions, external text file or in-band
• WebVTT, <track> element , TextTrack API

Linux Graphics Directions – Keith Packard

• Room was so full Linux almost got kicked out
• Linux runs everywhere.
• X runs on desktops, laptops but only on some of the mobile devices
• Open source community not engaged in any graphics that is not X
• Dominates desktop market mainly because GTK and QT use them
• Otherwise nobody runs X
• need to engage these people
• X ended up doing just about all graphics tasks
• Why X? – Lots of mature code for mode setting, have to duplicate
• Why X? – Clipping
• Why X? – Video Memory management
• Window Manager + X Server + compositing manager are all separate, more complicated
• How have things changed? – shared libraries, huge data increase, http/html does remote access, theme-able UI’s , making things fast-enough is easier, screens have colour now
• Re-Integrating the Window System
• Simplify Architecture – integrate compositing, in-app window management, synchronous operations
• What about remote apps? – HTTP, VNC/Rdesktop, WiDi, X as peer application
• Providing broad system architecture support
• mode setting – KMS
• Input device – Evdev
• Rendering/memory management – GEM
• Missing pieces – key mapping – libxkbcommon creates sharable interface
• Missing – Input devices – evdev isn’t sufficient – need user-mode touch devices
• NMissing – Accessibility APIs – mouse keys, sticky keys
• Reducing X specific code apps have to depend on

Florian Haas & Tim Serong

• Not using slides, instead 1 talking and other drawing cartoons in real time
• Centralized backups and storage – snapshots
• over provision of resources, better utilisation
• pre-packages virtual images
• building blocks of a system:
• virtualisation
• central storage
• storage replication, increase reliability
• “If you live in Queensland you’ll agree not everything that comes out a cloud is necessary good”
• high availability
• Existing commercial solutions, vendor lock-in
• SAN stack: commodity boxes for storage -> drbd -> iscsi target -> pacemaker cluster
• Virt Stack: Commodity Box -> Open Iscsi -> KVM
• pacemake looks very nice interface for clustering, Will have to look at it.

Making great conference video on a budget – Ryan Verner and Ben Hutchens

• Guys from team doing lca2011 conference video
• BOF on this later in conference
• 2004 used tapes, hard to scale
• Volunteer AV very different from using pros, skillset limited
• common chellenges from conferences, not LCA specific
• each component relatively simple, very easy to underestimate, many components, many rooms
• incomplete AV workflow, focus primary on tech
• Volunteers – limited manpower, expeience, training
• unforseen tech issues – Small varience in hardware
• workflow – automate as much as possible
• workflow – manage schedule , feedback loop (recording sheets) , rapid post editing, distribute tasks, automate transcoding/.uploading
• mistakes – LCA 400 hours of video, too much to do ANY manual editing, combining vga,main,audio feeds
• vga capture – twinpact 100 – $600
• Basic mixer – mike, usb sound – $300
• Firewire camera – $300
• Laptops running linux, DVDSwitch software mixes into single DV file
• Audio quality – often neglected
• Get a VGA capture device
• 100% test befoe conference, impossible to solve once things start
• Train volenteers
• Managment important, clear roles, delegation of specific tasks
• Clear Documentation
• Examples http://pythin.mirocommunity.org
• http://videokollektiv.org – example of best practices
• DVswitch software – video mixing, recording, streaming
• Designed for free software confs, limited budget
• Ofter used, sometimes without streaming

• Like “test driven development”
• Initially unit test focussed
• business doesn’t care about underlying implementation, just the functional results
• “executable specification” – written in spoken language
• “infrastructure as code”
• code without tests is “bad”
• Taking BDD and adapting to infrastructure development
• Tools – cucumber – write specs, execute/test specs
• cucumber-nagios
• DEMO of cucumber and cucumber-nagios
• example – continuous server builds as you update you config manager
• migration to config management. – Use BDI to test existing, test CM env with same tests to make sure it replicates
• Monitoring System : notify -> test -> repeat
• Current monitoring systems miss many things
• cucumber provides a common specification format that dev and ops can share
• removes duplication of tests
• Libraries being built of commons test (some already in cucumber nagios)

Geoff Huston

• New technologies from the 70s – Unix and Packet switching (TCP/IP)
• Open technologies – anyone can implement
• TCP/IP not better than competing technologies but it’s openness greatly helped it win
• One thing to “be open” , another to “stay open”
• Useful technologies are rarely static
• Technology evolves, uses change (eg growth of wireless), exploitation models change
• Challenges – net neutrality, next generation networks, mobility and mobil service evolution, Triple/Quad play schemes
• The really important thing is “We are running out of addresses”
• 190 m addresses given out in 2009, 248m in 2010
• 300million new things on the network
• 9 million new addresses just in Australia
• 7 /8s left, rate of 1/month
• plan that IPv6 transition would happen before ipv4 ran out
• Only 0.3% hitting google IPv6
• IPv4 will run out during 2011/2012
• Need to transition to ipv6 in 200 days
• Won’t happen, have to muddle thing with ipv4
• NATs are an externalized problem
• ISP NATs, multi-level, within ISP network
• aperture through through which the Internet can be seen and used. Reduced port space
• transition to ipv6. Dual stack requires everybody to have a ipv4 address
• If you run 6to4 15% of connections don’t connect.
• Transition could take 5-40 years
• Making ipv4 addresses last longer, they will cost
• TCP/IP is the network monoculture
• Will openness be lost in the transition?
• Telcos being asking to make big investment in ipv6. No really in their interests to have an open network.
• Similar for large Internet companies like google and amazon
• Delays help the incumbents, open network infrastructure is at risk
• Need to figure out how to motivate big companies to goto ipv6 and open infrastructure