Author: simon

• Everybody Sung Happy birthday to Baale
• Bdale said he has a new house and FreedomBox 0.3 release this week
• Rusty also on the panel
• Questions:
  • Why is Linus so mean
  • Unified Storage/Memory machines – from HP
  • Young people getting into community
  • systemd ( I asked this)
  • Year of the Linux Desktop
  • Documentation & training material
  • Predict the security problems in next 12 month
  • Does NZ and Australia need a joint space agency
  • Will you be remembered more for Linux or Git?

Drupal8 outta the box – Donna Benjamin

• I went to the first half of this but wanted to catch the talk below so I missed the 2nd part

 

Connecting Containers: Building a PaaS with Docker and Kubernetes – Katie Miller

• co-presented with Steve Pousty
• Plugs their OpenShift book, they are re-archetecturing the whole thing based on what in the book
• Platform as a service
  • dev tooling, runtime, OS , App server, middleware.
  • everything except the application itself
  • Openshift is an example
• Reasons to rebuild
  • New tech
  • Lessons learned from old deploy
• Stack
  • Atomic + docker + Kubeneties
• Atomic
  • Redhat’s answer of CoreOS
  • RPM-OSTree – atomic update to the OS
  • Minimal System
  • Fast boot, container mngt, Good Kernel
• Containers
  • Docker
  • Nice way of specifying everything
  • Pros – portable, easy to create, fast boot
  • Cons – host centric, no reporting
  • Wins – BYOP ( each container brings all it’s dependencies ) , Standard way to make containers , Big eco-system
• Kubernetes
  • system managing containerize maps across multiple hosts
  • declarative model
  • open source by google
  • pod + service + label + replication controller
  • cluster = N*nodes + master(s) + etcd
  • Wins: Runtime and operation management + management related containers as a unit, container communication, available, scalable, automated, across multiple hosts
• Rebuilding Openshift
  • Kubernetes provides container runtime
  • Openshift provides devops and team enviroment
• Concepts
  • application = multiple pods linked togeather (front + back + db ) managed as a unit, scald independantly
  • config
  • template
  • build config = source + build -> image
  • deployment = image and settings for it
• This is OpenShift v3 – things have been moving very fast so some docs are out of date
• Slides http://containers.codemiller.com

Tunnels and Bridges: A drive through OpenStack Networking – Mark McClain

• Challenges with the cloud
  • High density multi-tenancy
  • On demand provisioning
  • Need to place / move workloads
• SDN , L2 fabric, network virtualisation Overlay tunneling
• The Basics
  • The user sees the API, doesn’t matter too much what is behind
  • Neutron = Virtual subnet + L2 virtual network + virtual port
  • Nova = Server + interface on the server
• Design Goals
  • Unified API
  • Small Core. Networks + Subnets + Ports
  • Plugable open archetecture
• Features
  • Overlapping IPs
  • Configuration DHCP/Metadata
  • Floating IPs
  • Security Groups ( Like AWS style groups ) . Ingress/egress rules, IPv6 . VMs with multiple VIFS
• Deployment
  • Database + Neutron Server + Message Queue
  • L2 Agent , L3 agent + DHCP Agent
• Server
  • Core
  • Plugins types =  Proxy (proxy to backend) or direct control (login instide plugin)
  • ML2 – Modular Layer 2 plugin
• Plugin extensions
  • Add to REST API
  • dpch, l3, quota, security group, metering, allowed addresses
• L2 Agent
  • Runs on a hypervisor
  • Watch and notify when devices have been added/removed
• L3 agent – static routing only for now
• Load balancing as a service, based on haproxy
• VPN as a service , based on openswan, replicates AWS VPC.
• What is new in Juno?
  • IPv6
  • based on Radbd
  • Advised to go dual-stack
• Look ahead to Kilo
  • Paying down technical debt
  • IPv6 prefix delegation, metadata service
  • IPAM – hook into external systems
  • Facilitate dynamic routing
  • Enabling NFV Applications
• See Cloud Administrators Guide

 

Crypto Won’t Save You Either – Peter Gutmann

• US Govt has capabilities against common encryption protocols
• BULLRUN
• Example Games consoles
  • Signed executables
  • encrypted storage
  • Full media and memory encryption
  • All of these have been hacked
• Example – Replaced signature checking code
• Example – Hacked “secure” kernel to attack the application code
• Example – Modify firmware to load over the checking code
• Example – Recover key from firmware image
• Example – Spoof on-air update
• LOTS of examples
• Nobody noticed bunch of DKIM keys were bad, cause all attackers had bypassed encryption rather than trying to beat the crypto
• No. of times crypto broken: 0, bypassed: all the rest
• National Security Letters – The Legalised form of rubber-hose cryptanalysis
• Any well design crypto is NSA-proof
• The security holes are sitting right next to the crypto

 

8 writers in under 8 months: from zero to a docs team in no time flat – Lana Brindley

• Co Presenting with Alexandra Settle
• 8 months ago online 1 documentation person at rackspace
• Hired a couple people
• Horrible documentation suite
• Hired some more
• 4 in Australia, 4 in the US
• Building a team fast without a terrible culture
  • Management by MEME – everybody had a meme created for them when they started
  • Not all work and No play. But we still get a lot of work done
  • Use tech to overcome geography
  • Treat people as humans not robots
  • Always stay flexible. Couch time, Gym time
• Finding the right people
  • Work your network , job is probably not going to be advertise on linkedin, bad for diversity
  • Find great people, and work out how to hire them
  • If you do want a job, network
• Toolchains and Systems
  • Have a vision and work towards it
  • acknowledge imperfection. If you can’t fix, ack and just move forward anyway
• You can maintain crazy growth forever. You have to level off.
• Pair US person with AU person for projects
• Writers should attend Docs summit and encouraged to attend at least one Openstack summit

 

Cooper Lees – Facebook

• Open Source at facebook
• Increase in pull requests, not just pushing out stuff or throwing over the wall anymore
• Focussing on full life-cycle of opensource
• Big Projects: react , hhvm , asyncdisplaykit , presto
• Working on other projects and sending to upstream
• code.facebook.com  github.com/facebook
• Network Switches and Open Compute
  • Datacentre in NZ using open compute designs
• Open source Switch
  • Top of rack switch
  • Want to be the open compute of network switches
  • Installer, OS, API to talk to asic that runs ports
  • Switches = Servers. running chef
• Wedge
  • 16-32 of 40GE ports
  • Internal facebook design
  • 1st building block for disaggregated switching technology
  • Contributed to OCP project
  • Micro Server + Switchports

Carol Smith – Google

• Works in Google Open Source office
• Google Summer of code
  • Real world experience
  • Contacts and references
• 11th year of the program
• 8600 participated over last 10 years
• Not enough people in office to do southern hemisphere programme. There is “Google code-in” though

Mark McLoughlin – Red Hat

• Open Source and the datacenter
• iaas, paas, microservices, etc
• The big guys are leading (amazon, google). They are building on open source
• Telcos
  • Squeezed and scrambling
  • Not so “special” anymore
  • Need to be agile and responsive
  • Telecom datacentre – filled with big, expensive, proprietary boxes
  • opposite of agile
• OPNFV reference architecture
• OpenStack, Open vswitch, etc
• Why Open Source? – collaboration and coopetition , diversity drives innovation , sustainability

 

There was a Q&A. Mostly questions about diversity at the companies and grumps about having to move to US/Sydney for peopl eto work for them

 

• Clinton Roy + Tom Eastman – Python Conference Australia 2015 + Kiwi PyCon 2015
  • Brisbane , late July 2015
  • Similar Structure to LCA
  • Christchurch – Septemberish
  • kiwi.pycon.org
• Daniel Bryan – Comms for Camps
  • Detention camps for Australian boats people camps
  • Please contact if you can offer technical help
• Phil Ingram – Beernomics
  • Doing stuff for people in return for beer
  • Windows reinstall = a Keg
  • Beercoin
• Patrick Shuff – Open sourcing proxygen
  • C++ http framework. Built own webserver
  • Features they need, monitoring, fast, easy to add new features
  • github -> /facebook/progen
• Nicolás Erdödy – Multicore World 2015 & the SKA.
  • Multicore World – 17-18 Feb 2015 Wellington
• Paul Foxworthy – Open Source Industry Australia (OSIA)
  • Industry Body
  • Govt will consult with industry bodies but won’t listen to individual companies
  • Please join
• Francois Marier – apt-get remove –purge skype
  • Web RTC
  • Now usable to replace skype
  • Works in firefox and chrome. Click link, no account, video conversation
  • Firefox Hello
• Tobin Harding – Central Coast LUG
  • Update on Central Coast of NSW LUG
  • About 6 people regularly
• Mark Smith – Failing Gracefully At 10,000ft
  • Private pilot
  • Aircrafts have 400+ page handbooks
  • Things will fail…
  • Have procedures…
  • Before the engine is on fire
  • test
  • The most important task is to fly the plane
• Tim Serong – A very short song about memory management
  • 1 verson song
• Angela Brett – Working at CERN and why you should do it
  • Really Really awesome
  • Basic I applied, lots of fellowship
  • Meet someone famous
  • Lectures online from famous people
• Donna Benjamin – The D8 Chook Raffle
  • $125k fund to get Drupal8 out
  • Raffle. google it
• Matthew Cengia/maia sauren – What is the Open Knowledge Foundation?
  • au.okfn.org
  • Open govt/ data / tech / jouralism / etc
  • govHack
  • Open Knowledge Brisbane Meetup Govt
• Florian Forster – noping
  • Pretty graphs and output on command line ping
  • http://noping.cc
• Jan Schmidt – Supporting 3D movies in GStreamer
  • A brief overview of it all
• Justin Clacherty ORP – An open hardware, open software router
  • PowerPC 1-2G RAM
  • Package based updates
  • Signed packages
  • ORP1.com

EQNZ – crisis response, open source style – Brenda Wallace

• Started with a Trigger warning and “fucker”
• First thing posted – “I am okay” , one tweet, one facebook
• State of Scial Media
  • Social media not as common, SMS king, not many smartphones
  • Google Buzz, twitter, Facebook
  • Multiple hashtags
• Questions people asked on social media
• Official info was under strain, websites down due to bad generators
• Crisis Commons
• Skype
  • Free
  • Multi-platform
  • Txt based
  • Battery Drain very bad
  • Bad internet in Chc hard to use, no mobile, message reply for minutes on join
• Things pop up within an hour
  • Pirate Pad
  • Couch apps
  • Wikis
  • WordPress installs
• Short code 4000 for non-urgent help live by 5pm
  • Volenteers processing the queue
• All telcos agree to coordinate their social media effort
• Civil defence didn’t have site ready and refused offers, people decided to do independantly
• Ushahidi instance setup
  • Google setup people finder app
  • Moved into ec2 cluther
  • hackfest, including added mobile
  • Some other Ushidis, in the end newspaper sites enbedded
• Council
  • chc council wordpress for info
  • Very slow and bad UI
  • Hit very hard, old information from the previous earthquake
  • staff under extreme pressure
• Civil Defence
  • Official info only
  • Falls over
  • Caught by DDOS against another govt site
• Our reliability
  • Never wen tdown
  • contact and reassured some authorities
  • After 24h . 78k page impressions
• Skype
  • 100+ chatting. limitations
  • IRC used by some but many no common enough
  • Gap for something common. cross platform, easy to use
• Hashtag
  • twitter to SMS notifications to add stuff to website
• Maps were a new thing
  • None of the authorities knew them
• Council and DHB websites did not work on mobile and were not updating
• Government
  • Govt officers didn’t talk – except NZ Geospacial office
  • Meeting that some people attended
• Wrap up after 3 weeks
  • Redirected website
  • Anonymous copy of database
• Pragmatic
  • Used closed source where we had too (eg skype)
  • But easier with OS could quick to modify
  • Closed source people could install webserver, use git, etc. Hard to use contributions
• Burned Bridges
  • Better jobs with Gov agencies
• These days
  • Tablets
  • Would use EC2 again
  • phones have low power mode
  • more open street maps

 

collectd in dynamic environments – Florian Forster

• Started collectd in 2005
• Dynamic environments – Number and location of machines change frequently – VM or job management system
• NOTE: I use collectd so my notes are a little sparse here cause I knew most of it already
• Collects timeseries data, does one thing well. collectd.org
• agent runs on each host, plugins mostly in C for lots of things or exec plug to run random stuff.
• Read Plugins to get metrics from system metrics, applications, other weird stuff
• Write plugs – Graphite, RRD, Reimann, MongoDB
• Virtual machine Metrics
  • libvirt plugin
  • Various metrics, cpu, memory, swap, disk ops/bytes, network
  • GenericJMX plugin – connects to JVM. memory and garbage collection, threads
• Network plugin
  • sends and receives metric
  • Effecient binary protocol. 50-100 byte UDP multicast/unicast protocol
  • crypto available
  • send, receive, forward packets
• Aggregation
  • Often more useful for alerting
• Aggregation plugin
  • Subscribes to metric
  • aggregates and forwards
  • Limitation, no state, eg medium, mean are missing
  • only metrics with one value
  • can be aggregated at any level
  • eg instead of each CPU then total usage of all your CPUS
• Reimann
  • Lots of filters and functions
  • can aggregate, many otions
• Bosum
  • Monitoring and alert language
• Storage
  • Graphite
  • OpenTSDB based on hadoop
  • InfluxDB – understand collectd protocol native (and graphite).
  • Vaultaire ( no collectd integration but… )
• New Dishboard – facette.io

CoreOS: an introduction – Brandon Philips

• Reference to the “Datacenter as a Computer Paper“
• Intro to containers
• cAdvisor – API of what resources are used by a container
• Rocket
  • Multiple implementations of container spec , rocket is just one implementation
• Operating system is able to make less promises to applications
• Kernel API is really stable
• Making updates easy
  • Based on ChromeOS
  • Update one partition with OS version. Then flip over to that.
  • Keep another partition/version ready to fail back if needed
  • Safer to update the OS seperated from the app
  • Just around 100MB in size. Kernel, very base OS, systemd
• etcd
  • Key value store over http (see my notes from yesterday)
  • multiple, leader election etc
  • Individual server less critical since data across multiple hosts
• Scheduling stuff to servers
  • fleet – very simple, kinda systemd looking
  • fleetctl start foo.service   – sends it off to some machine
  • meso, kubernetes, swam other alternative scedulers
• Co-ordination
  • locksmith
• Service discover
  • skydns, discoverd, conf
  • Export location of application to DNS or http API
  • Need proxies to forward request to the right place (for apps not able to query service discovery directly)
• It is all pretty much a new way of thinking about problems

 

Why you should consider using btrfs, real COW snapshots and file level incremental server OS upgrades like Google does. – Marc Merlin

• Worked at netapp, hooked on snapshots, lvm snapshots never worked too well , also lvm partitions not too good
• Switched laptop to btrfs to 3 years ago
• Why you should consider btrfs
  • Copy on Write
  • Snapshots
  • cp -reflink=always
  • metadata is redundant and checksummed, data checksummed too
  • btrfs underlying filesystem [for now]
  • RAID 0, 1, 5, 6 built in
  • file compression is also built in
  • online background scrub (partial fsck)
  • block level filesystem diff backups(instead of a slow rsync)
  • convert difectly from ext3 (fails sometimes)
• Why not use ZFS instead
  • ZFS more mature than ZFS
  • Same features plus more
  • Bad license. Oracle not interested in relicensing. Either hard to do or prfer btrfs
  • Netapp sued sun for infringing patents with ZFS. Might be a factor
  • Hard to ship a project with it due to license condistions
• Is it safe now?
  • Use new kernels. 3.14.x works okay
  • You have to manually balance sometimes
  • snapshots, raid 0 , raid 1 mostly stable
  • Send/receive mostly works reliably
• Missing
  • btrfs incomplete, but mostly not needed
  • file encryption not supported yet
  • dedup experimental
• Who use it
  • openSUSE 13.2 ships with it by default
• File System recovery
  • Good entry on bfrfs wiki
  • btrfs scrub, run weekly
  • Plan for recovery though, keep backups, not as mature as ext4/ext3 yet, prepare beforehand
  • btrfs-tools are in the Ubuntu initrd
• Encryption
  • Recommends setup encryption on md raid device if using raid
• Partitions
  • Not needed anymore
  • Just create storage pools, under them create sub volumes which can be mounted
  • boot: root=/dev/sda1  rootflags=solvol=root
• Snapshots
  • Works using subvolumes
  • Read only or read-write
  • noatime is strongly recommended
  • Can sneakily fill up your disk “btrfs fi show” tells you real situation. Hard to tell what snapshots to delete to reclaim space
• Compression
  • Mount option
  • lzo fast, zlib slower but better
  • if change option then files changed from then on use new option
• Turn off COW for big files with lots of random rights in the middle. eg DBs and virtual disk images
• Send/receive
  • rsync very slow to scan many files before copy
  • initial copy, then only the diffs. diff is computed instantly
  • backup up ssd to hard drive hourly. very fast
• You can make metadata of file system at a different raid level than the the data
• Talk slides here. Lots of command examples

 

Bob Young

• Warns that some stories might not be 100% true
• ”  Liked about Early Linux – Nobody was very nice to each other but everybody was very respectful of the Intel Microprocessor “
• CEO of Redhat 1992 – 2000
• Various stories, hard to take notes from
• One person said they walked out of the Keynote when they heard the quote “it was a complete meritocracy” re the early days of Linux.
• Others didn’t other parts of the talk. General tone and some statements similar to the one above.
• “SuSe User Loser” proviked from laughs and a Suse Lizzard being thrown at the speaker
• Reasons the publishing industry rejects books: 1. no good; 2. market not big enough; 3. They already publish one on the subject.

Alerting Husbandry – Julien Goodwin

• Obsolete alerts
  • New staff members won’t have context to know was is obsolete and should have been removed (or ignorened)
• Unactionable alerts – It is managed by another team but thought you’d like to be woken up
• SLA Alerts – can I do something about that?
• Bad thresholds ( server with 32 cores had load of 4 , that is not load ), Disk space alerts either too much or not enough margin
• Thresholds only redo after complete monitoring rebuilds
• Hair trigger alerts ( once at 51ms not 50ms )
• Not impacting redundancy ( only one of 8 web servers is down )
• Spamming alerts, things is down for the 2925379857 time. Even if important you’ve stopped caring
• Alerts for something nobody cares about, eg test servers
• Most of earlier items end up in “don’t care” bucket
• Emails bad, within a few weeks the entire team will have a filter to ignore it.
• Undocumented alerts – If it is broken, what am I supposed to do about it?
• Document actions to take in  “playbook”
• Alert acceptance practice, only oncallers should e accepting alerts
• Need a way to silence it
• Production by Fiat

 

 

Managing microservices effectively – Daniel Hall

• Step one – write your own apps
• keep state outside apps
• not nanoservices, not milliservices
• Each should be replaceable, independantly deployable , have a single capability
• think about depandencies, especially circular
• Packaging
  • small
  • multiple versions on same machine
  • in dev and prod
  • maybe use docker, have local registry
  • Small performance hit compared to VMs
  • Docker is a little immature
• Step 3 deployment
  • Fast in and out
  • Minimal human interaction
  • Recovery from failures
  • Less overhead requires less overhead
  • We use Meso and marathon
  • Marathon handles switches from old app to new, task failure and recover
  •  Early on the Hype Cycle
• Extra Credit Sceduling
  • Chronos within Mesos
  • A bit newish

 

Corralling logs with ELK – Mark Walkom

• You don’t want to be your bosses grep
• Cluster Elastisearch, single master at any point
• Sizing best to determine with single machine, see how much it can hadle. Keep Java heap under 31GB
• Lots of plugins and clients
• APIs return json. ?pretty makes it looks nicer. The ” _cat/* ” api is more command line
• new node scales, auto balancers and grows automatic
• Logstash. lots of filters, handles just about any format, easy to setup.
• Kibana – graphical front end for elastisearch
• Curator, logstash-forwarder, grokdebugger

FAI — the universal deployment tool – Thomas Lange

• From power off to applications running
• It is all about installing software packages
• Central administration and control
• no master or golden image
• can be expanded by hooks
• plan your installation and FAI installs the plan
• Boot up diskless client via PXE/tftp
• creates partitions, file systems, installs, reboots
• groups hosts by classes, mutiple classes per host etc
• Classes can be executables, writeing to standard output, can be in shell, pass variables
• partitioning, can handle LVM, RAID
• Projected started in 1999
• Supports debian based distributions including ubuntu
• Supports bare metal, VM, chroot, LiveCD, Golden image

 

Documentation made complicated – Eric Burgueno

• Incomplete, out of date, inconsistent
• Tools – Word, LibreOffice  -> Sharepoint
• Sharepoint = lets put this stuff over here so nobody will read it ever again
• txt , markdown, html. Need to track changes
• Files can be put in version control.
• Mediawiki
• Wiki – uncontrolled proliferation of pages, duplicate pages
• Why can’t documentation be mixed in with the configuration management
• Documentation snippits
  • Same everywhere (mostly)
  • Reusable
• Transclusion in mediawiki (include one page install another)
• Modern version of mediawiki have parser functions. display different content depending on a condition
• awesomewiki.co