Category: Tech

AWS OpsWorks Orchestration War Stories – Andrew Boag

• Autoscaling too slow since running build-from-scratch every time
• Communications dependencies
• Full stack rebuild in 20-40 minutes to use data currently in production
• A bit longer in a different region
• Great for load testing
• If we ere doing again
  • AMI-based better
  • OPSWorks not suitable for all AWS stacks
  • Golden master for flexable
• Auto-Scaling
  • Not every AMI instance is Good to Go upon provisioning
  • Not a magic bullet, you can’t broadly under-provision
  • needs to be throughly load-tested
• Tips
  • Dual factor authentication
  • No single person / credentials should be able to delete all cloud-hosted copies of your data
• Looked at Cloudformation at start, seemed to be more work
• Fallen out of love with OpsWorks
• Nice distinction by Andrew Boag: he doesn’t talk about “lock-in” to cloud providers, but about “cost to exit”.   – Quote from Paul

Slim Application Containers from Source – Sven Dowideit

• Choose a base image and make a local version (so all your stuff uses the same one)
• I’d pick debian (a little smaller) unless you can make do with busybox or scratch
• Do I need these files? (check though the Dockerfile) eg remove docs files, manpages, timezones
• Then build, export, import and it comes all clean with just one layer.
• If all your images use same base, only on the disk once
• Use related images with all your tools, related to deployment image but with the extra dev, debug, network tools
• Version the dev images
• Minimise to 2 layers
  • look at docker-squash
  • Get rid of all the sourc code from your image, just end up with whats need, not junk hidden in layers
• Static micro-container nginx
  • Build as container
  • export as tar , reimport
  • It crashes 🙁
  • Use inotifywait to find what extra files (like shared libraries) it needs
  • Create new tarball with those extra files and “docker import” again
  • Just 21MB instead of 1.4GB with all the build fragments and random system stuff
  • Use docker build as last stage rather than docker import and you can run nginx from docker command line
  • Make 2 tar files, one for each image, one in libs/etc, second is nginx

Containers and PCP (Performance Co-Pilot) –  Nathan Scott

• Been around for 20+ years, 11 years open source, Not a big mindshare
• What is PCP?
  • Toolkit, System level analysis, live and historical, Extensible, distributed
  • pmcd daemon on each server, plus for various functions (bit of like collectd model)
  • pmlogger, pmchart, pmie, etc talk (pull or poll) to pmcd to get data
• With Containers
  • Use –container=  to grab info inside a container/namespace
  • Lots of work still needed. Metrics inside containers limited compared to native OS

The Challenges of Containerizing your Datacenter – Daniel Hall

• Goals at LIFX
  • Apps all stateless, easy to dockerize
  • Using mesos, zookeeper, marathon, chronos
  • Databases and other stuff outside that cloud
• Mesos slave launches docker containers
• Docker Security
  • chroot < Docker < KVM
  • Running untrusted Docket containers are a BAD IDEA
  • Don’t run apps as root inside container
  • Use a recent kernel
  • Run as little as possible in container
  • Single static app if possible
  • Run SELinux on the host
• Finding things
  • Lots of micoroservices, marathon/mesos moves things all over the place
  • Whole machines going up and down
  • Marathon comes with a tool that pushes it’s state into HAProxy, works fairly well, apps talk to localhost on each machines and haproxy forwards
  • Use custom script for this
• Collecting Logs
  • Not a good solution
  • can mount /dev/log but don’t restart syslog
  • Mesos collects stdout/stderror , hard to work with and no timestamps
  • Centralized logs
  • rsyslog log to 127.0.0.1 -> haproxy -> contral machine
  • Sometimes needs to queue/drop if things take a little while to start
  • rsyslog -> logstash
  • elasticsearch on mesos
  • nginx tasks running kibana
• Troubleshooting
  • Similar to service discover problem
  • Easier to get into a container than getting out
  • Find a container in marathon
  • Use docker exec to run a shell, doesn’t work so well on really thin containers
  • So debugging tolls can work from outside, pprof or jsonsole can connect to exposed port/pid of container

Clouds, Containers, and Orchestration Miniconf

Cloud Management and ManageIQ – John Mark Walker

• Who needs management – Needs something to tie it all together
• New Technology -> Adoption -> Proliferation -> chaos -> Control -> New Technology
• Many technologies follow this, flies under the radar, becomes a problem to control, management tools created, management tools follow the same pattern
• Large number of customers using hybrid cloud environment ( 70% )
• Huge potential complexity, lots of requirements, multiple vendors/systems to interact with
• ManageIQ
  • Many vendor managed open source products fail – open core, runt products
  • Better way – give more leeway to upstream developers
  • Article about taking it opensource on opensource.com. Took around a year from when decision was made
  • Lots of work to create a good open source project that will grow
  • Release named after Chess Grandmasters
  • Rails App

LXD: The Container-Based Hypervisor That Isn’t –  Tycho Andersen

• Part of Openstack
• Based on LXC , container based hypervisor
• Secure by default: user namespaces, cgroups, Apparmor, etc
• A EST API
• A daemon that doesn’t hypervisory things
• A framework for maintaining container based applications
• It Isn’t
  • No network configuration
  • No storage management – But storage aware
  • Not an application container tool
  • handwavy difference between it and docker, I’m sure it makes sense to some people. Something about running an init/systemd rather than the app directly.
• Features
  • Snapshoting – eg something that is slow to start, snapshot after just starts and deploy it in that state
  • Injection – add files into the container for app to work on.
  • Migration – designed to go fairly fast with low downtime
• Image
  • Public and private images
  • can be published
• Roadmap
  • MVP 0.1 released late January 2015
  • container management only

Rocket and the App Container Spec – Brandon Philips

• Single binary – rkt – runs everywhere, systemd not required
• rkt fetch – downloads and discovers images ( can run as non-root user )
• bash -> rkt -> application
• upstart -> rkt -> application
• rkt run coreos.com/etcd-v2.3.1
• multiple processes in container common. Multiple can be run from command line or specified in json file of spec.
• Steps in launch
  • stage 0 – downloads images, checks it
  • Stage 1 – Exec as root, setup namespaces and cgroups, run systemd container
  • Stage 2 – runs actual app in container. Things like policy to restart the app
  • rocket-gc garbage collects stuff , runs periodicly. no managmanent daemon
• App Container spec is work in progress
  • images, files, compressed, meta-data, dependencies on other images
  • runtime , restarts processes, run multiple processes, run extra procs under specified conditions
  • metadata server
  • Intended to be built with test suite to verify

Disclaimer: The below is my personal opinion and does not represent the views of the 2015 LCA organising committee. Some details have been left out, stuff may change, names may be wrong, may contain nuts, etc.

In January 2015 the Linux.conf.au conference will be held in Auckland, New Zealand. Each year the conference brings together 600 ( +-100 ) Linux developers and users for 5 days for talks, chat and social events. LCA 2015 will be the 12th Linux.conf.au I’ve attended (every year since 2004) and the first I’ve helped organise. It will be the 3rd time the conference has been held in New Zealand.

Each year’s LCA is held in a different city by a group who bid for and run it. The Auckland team consists of a “core team” of about 10 under the overall lead of Cherie Ellis, another dozen “supporters” (including me). Others volunteers  will be recruited closer to the time and there are also external groups like the papers committee and people from Auckland University doing various jobs.

The majority of the conference will be held in the Owen G Glenn Building at Auckland University. The is single big building with several large lecture theatres along with big central areas and smaller rooms. The currently plan is for just about the whole conference proper to happen there.

Over half the attendees with probably stay at nearby student accommodation, this is cheap, nearby and lets people mingle with other attendees after-hours. There will also be some planned social events (like the conference dinner) elsewhere in Auckland.

Since January 2014 when Auckland was announced as the winning bid for 2015 the pace has gradually been picking up. Over 30 main positions have been filled (most with both a main and backup person) and the core team is meeting (usually online) weekly and the second supporters meeting is coming up.

The amount of stuff to organise is pretty big. As well as the venues, there is food, travel, accommodation, swag, the programme, the websites, network, dinners, registration, etc etc. A huge amount of stuff which will take up many hours per week for the rest of 2015.

At the end of March there was a “Ghosts visit”, this is where half a dozen previous conference organisers ( “Ghosts of conferences past” ) come over for a weekend to look over the setup and talk to the group. The purpose is twofold, the Ghosts check that everything is on track and look for problems, while the 2015 organisers get to pick the Ghost’s brains

Even the Ghosts’ event itself is a small test of the organizers’ ability. They have  to fly, meeting, accommodate, hosts, feed and otherwise look after half a dozen people, a mini rehearsal  for the full conference.

Winner Rusty Wrench Award: Andrew Tridgell

Host of LCA2015: Auckland!!

Website: lcauckland.org.nz 

Lightning Talks part 2

• My toothbrush has a serial number
  • after sales support
  • can they find it for me?
  • In the post-Snowden world this should be investigated
• DIY Book Scanning for Fun
  • Scanned book useful for good reasons
  • diybookscanner.org
• Freedom Box project update
  • Almost ready for 0.2 release which will be pretty good
• OneRNG
  • Open Hardware, Random number generator
  • Trustable, see raw or AES whitened
  • trying various options
  • onerng.info
• Central Coast LUG
  • cclugtmp@gmail.com
• Bitcoin Myths
  • Anonymous – Nope, all transactions records
  • Bubble – nope, infrastaructer
  • Giant Ponzi scheme – Not sold as investment, no claims
• dlect – Lecture recording downloader
  • uqlectures.sf.net
  • Looking for help and the extend to other Universities
• Debian in Australia
  • Trying to get Debian Australia mailing list started
• Bitcoin architecture applied to capital markets
• Learning Opportunities in Rocketry Software
  • Maths makes by head hurt
• Electronic Frontiers Australia
  • Would like to invite you to volunteer and drink beer
• LA does other things
  • pycon AU in August
  • Drupal camps
  • Barcamp
  • Join a user group
  • hacker space
  • add your blog to our planet

Provisioning Bare Metal with OpenStack by Devananda van der Veen

• Tried to use the existing NOVA tool (which was used for VM provisioning) but all sorts of limitations
  • Hacked it a lot and then gave up
• Created a new project “Ironic” for bare-metal provisioning in May 2013
• Status
  • Being working on but lots of devs
  • Not in main release yet
  • Some push to simplify openstack installation – Triple-O (Openstack on Openstack)
  • “Openstack is not a virtualisation layer”
• Security not there yet
• Driver Interface
  • 3 classes of interfaces: core, common, vendor
  • core: power management, deploy,
  • common: console, rescue
  • vendor: fireware? boot-from-volume? something-else?
• Architecture
  • REST API
  • DB
  • conductor services
  • RPC
  • ( the slide makes more sense )
• Many conductors, Many drivers
• If the cluster changes
  • take-over hooks
  • consistent has updates
  • node(s) re-mapped to conductors
• Feature equiv to bare-metal in a few months
• Usable by that point, eventually to replace bare-metal, may require re-deployment

Talk only took 20 minutes. That was quick

D-Bus in the kernel by Lennart Poettering

• kdbus
• Most newer OS designs started around powerful IPC
• Linux only had IPC primitives ( sockets, fifos, shared memory)
• DBUS is powerful IPC
  • Method Call Transactions, Signals, Properties, OO, Broadcasting, Discovery, Introspection, Policy, etc,  etc etc
• D-Bus has it limitations
  • Suitable only for control, not payload
  • It’s inefficient ( 10 copies, 4 complete validations, 4 context switches per duplex method call transaction)
    • Can we optimise that rather than moving it into the kernel? – “We are, We are moving it into the kernel with kdbus”
  • Credentials one can send/recv are limited
  • No implicit timestamping
  • Not available in early boot, initrd, late boot
  • Hookup for security frameworks happens in userspace
  • Activatable bus services are independent from other system services
  • Codebase is a bit too baroque, XML
  • No race-free exit-on-idle bus activated services
• D-Bus is fantastic, solves real problems
  • Right approach: good concepts, generic, comprehensive, covers all areas
  • Established, it is the single most used local high-level IPC system on linux, bindings most languages
  • Used in init systems, desktop, etc
• kdbus
  • Suitable for large data (GiB), zero-copy, optionally reusable
  • Efficient ( 2 or fewer copies, 2 validations, 2 context switches, per duplex method call transaction)
  • Comprehensive credentials sent along ( uid, pid, gid, selinux, cgroup, caps, audit, etc etc)
  • Implicit timestamping
  • Always available from earliest boot to latest shutdown
  • Open for LSMs to hook into from the kernel side
  • Activation is identical to activation of other services
  • Userspace is much simpler, No XML
  • Rece-free exist-on-idle
• Overview
  • Intention to replace current D-BUS implimentation
  • Should be drop-in so program using bindings should be drop in without changing anything
  • No performance tests yet
  • Receiver buffers
  • Single copy to destinations(s)
  • method call Windows
  • Name registry
  • memfds
    • File descriptors for memory regions
    • Zero Copy
    • Sealing
    • At 512k and above zero copy is faster than single copy, below that the mapping overhead is too expensive
    • a bit like Android ashmem
  • Signal Broadcasting
    • Bloom filters
  • Userspace
    • proxy provides compatibility with dbus1 sockets
  • When?
    • Code all there, pretty much implemented, policy still to do
    • Compile-time switch in systemd
    • We hope to get in the kernel in 2014
    • gdbus support coming soon

VisualEditor: Wikipedia’s new editing system, and how you can use it too by James Forrester

• Why a visual editor
  • Drop off in new contributors is major issue for wikipedia
  • MediaWiki is based on the very rich wikitext syntax
  • Wikitext stops people becoming contributors or staying away
  • Not an appropriate way to ask people to contributive things in the 21st century
  • Prevents us from providing other helpful editing tools
• Flexibility and modularity
  • Modular
  • support by 300 languages
  • replace or extend existing tools
  • expand it for editing new content
• What?
  • Client-side JavaScript contentEditable HTML editor
  • Stand-alone, integrate into any platform
  • Mediawiki integration (parsoid converts HTML+RDFa to/from wikitext)
• 3 Main layers
  • Data Model
    • We can’t edit the input HTML+RDFa directly
    • Reliable convert from HTML to things we can edit and back without corruption
    • Synchronise with ContentEditable (both ways)
    • Needs to allow real-time collaborative editing
    • All sorts of problems with spliting characters in various languages
  • Browser contentEditable
    • Messes up HTML, inconsistent key handling
    • Native text insertion, spell-check, selection and cursor
    • Equal rendering
  • Programmatic cursor handling
    • Intercept the browser’s native cursor handling
    • allows us to skip over non-editable elements
    • Thought this will allow to avoid UX issues
    • but this breaks internationalisation
    • Move to “observe and fixup” of native browser action
  • Input Method Editors
    • Lots of languages reply on IMEs
    • Mainly East Asian and Indic languages
    • Each IME has it’s own unique way of changing content
    • But can change by OS, browser, IME version
  • SurfaceObserver
    • Poll DOM for changes and notify model
    • Lots of events emit no of few events (eg spellcheck)
    • Possibly use MutationObserver in the future
    • Re-render if really necessary
  • Actual rich editing of stuff
    • Annotation (bold, underline)
    • Generated content (references, tables )
    • Tool UX types – Dialogs
• Future
  • Patches welcome
  • Maybe can be integrated in wordpress or something ( but might be hard)

• How many people have wanted to work on something that went into space? – Everybody
• How many people have actually done that? – about a dozen
• How about if every kid could answer “tes” to that question
• Cubesats – Small standard way to luanch tidy stats – 10cm cubes
• ArduSat – Kickstart – June 2012
• Feb 2013 – Revises design of board
• Mar 2013 – New design built
• June 2013 – Hardware integrated and built – ArduSat-1 and ArduSat-X
• 3 Cubesats Launched to ISS include the 2 ArduSat’s
• Need to upgrade/update Experiments in flight
  • Browser based updates
  • Code sent via Internet to california, run on duplicate of satellite
  • Transmitted to satellite
  • Distributed to one of the nodes
• CubeSat System bus
  • Supervisor node controls smaller nodes
  • Stores stuff on Sd card
  • Allocates experiment to node, powers up the node
  • sends it via “serial port” to node
  • Supervisor controls Power and reset and serial for each node
  • node thinks it is just a stand-alone off the shelf Ardino
• Satellite
  • 4-6 month life
  • 12-15 experiments
  • If allocated in 1 week blocks we can sell those for $300 each
  • So much backlog and interest is plan to launch 30 ArduSats/year for next 5 years
• Make Satellite available
  • Flyable version costs approx $50k
  • Replace flying modules with open-source parts
• TODO
  • Release of designs without attracting the lack helicopters
  • Flight computer design
  • Terrestrial sensor suite design
  • Development of education material
• Wait there is more…
• Water powered cube-sat
  • 5kg total mass, 2.5kg of water propellant
  • On kickstarter

Disaster Recovery Lessons I Hoped I’d Never Have to Learn by Bdale Garbee

• Got backups?
  • Do you keep a copy of your essential data off-site
  • If someone called you and said your house was gone: completely devastating or just a really bad day?
• 11th June 2013
  • Got no warnings of fire, saw it one km away and then got instant evacuation order
  • Had 20m-1h to pack (unsure of time due to heat of situation)
  • Went to leave and fire was on driveway, unable to get car out.
  • Had to park car and firefighters walked them out.
• The fire
  • Started 1pm 11th of June 2013
  • 13,000 homes evacuated
  • 94,000 acres evacuated
  • 500 fire fighters involved
  • Destroyed – 14,290 acres ( 57.8 km²) , 511 homes, 2 people
  • Let back into property on 21 June
• Aftermath
  • House pretty destroyed
  • Normally 2000-2200 degrees, His house got to 3000 degrees plus Fahrenheit
  • Most metals melted
• Ready to go – what would you grab?
  • Usuaul pocket contents.. mobile phone, wallet, keys
  • Photo albums and scrape books
  • Jewelry
  • Computers
• Stuff you havn’t thought about
  • Mobile phone charger
  • Identity documents
  • Safe deposit box key
  • Account numbers and contact info
• Hows your insurance
  • Types and levels of coverage
    • Structure
    • Contents
    • Loss of use
    • Outbuildings
    • Landscaping
  • Replacement cost vs actual cash value
  • Personal vs Business Property – Have you even taken a business tax deduction for any of these items?
  • Costed out how much it would cost to rebuild old house, got that much money to build new cost (not the same)
• Rapidly recoving
  • Take care of human needs first
  • Find a place to live.. being homeless sucks
  • Minimise the amount of “throw away” stuff purchased ( don’t buy stuff you don’t want to keep, eg apartment-only stuff)
  • Pick one or two things to “put back to normal” ASAP
    • Mobile phone for my son
    • Big-screen TV with cable service
    • 3d printer
    • art supplies
  • Registries and wish lists
• Cleaning up
  • Lots of metal extracted from the ashes – copper, silver
  • Found very few intact items – a Japanese doll, a few other ceramics
  • Since house taken out by embers around 2/3s of trees intact
• Recovering Hobbies and Small Business
  • Back on the net quickly but not back in business
  • Commitment to AJ and Mike Beatie for rocket launch

A web page in seven syscalls by Tollef Fog Heen

• “Most of us, well at least the ones of us who have CS degrees”
• Modern Computers – Cores, Caches, 64 bit, Virtual memory
• Massively multi-threaded – usually 1000 threads or more
• Relative performance of CPU -> system calls -> Disk access
• Varnish from HTTP’s point of view is a origin server not a “http cache”
• Two processes
  • manager process – starts child, compiles config, watchdo on child and restarts if dies
  • Child process, handles all connections, handles storage
  • params shared via shared memory
• VCL
  • Domain specific language
  • Compiled into C when loaded
  • Can escape to C from within VCL
  • Can import plugins
• Optimisations
  • Avoid syscalls
  • Memory workspaces
  • Length-counted strings
  • Threads
  • Don’t fight the VM, use hints
  • Don’t copy data
  • LIFO schedule threads
  • Preallocate
  • Accept filters