Linux.conf.au 2015 – Day 5 – Session 3

NoOps with Ansible and Puppet – Monty Taylor

  • NoOps
    • didn’t know it was a contentious term
    • “devs can code and let a service deploy, manage and scale their code”
    • I want to change the system by landing commits. don’t want to “do ops”
    • if I have to use my root access it is a bug
  • Cloud Native
    • Ephemeral Compute
    • Data services
    • Design your applications to be resilient via scale out
    • Cloud scale out, forget HA for one system, forget long-lived system, shared-nothing for everything. Cloud provides the hard scale-out/HA/9s stuff
    • Great for new applications
  • OpenStack Infra
    • Tooling, automation, and CI for the openstack project
    • 2000 devs
    • every commit is fully tested.
    • each test runs on a single use cloud slave
    • 1.7 million test jobs in the last 6 months. 18 TB of log data
    • all runs in HP and rackspace public clouds
  • Create Servers manually at 1st
  • Step 1 – Puppet
    • extra hipster because it is in ruby
    • If you like ruby it is awesome. If don’t is it less-awesome
    • collaboration from non-root users
    • code review
    • problem that it blows up when you try and install the same thing in two different places
    • 3 ways to run. masterless puppet apply. master + puppet agent daemon . master + puppet agent non-daemons
  • Secret stuff that you don’t want into you puppet git repo
    • hiera
  • Step 2 – Ansible for orchestration
    • Control the puppet agent so it runs it nicely and in schedule and on correct hosts first
    • Open source system management tool
    • Sequence of steps not description of state like puppet
    • ad-hoc operation. run random commands
    • easy to slowly grow over time till it takes over puppet
    • yaml syntax of config files
  • Step 3 – Ansible for cloud management
  • Ansible config currently mixed in with puppet under – http://git.openstack.org/cgit/openstack-infra/system-config/

 

Conference Closing

  • Steve Walsh wins Rusty Wrench award
  • Preview of Linux.conf.au 2016 in Geelong
    • Much flatter than Auckland
    • Deakin University – Waterfront Campus
    • Waurn Ponds student accomadation 15 minutes with shuttles
    • Feb 8th – 12th 2016
    • CFP 1st of June 2015
    • Theme “life is better with linux”
    • 4 keynotes confirmed or in final stages of discussion, 2 female, 2 male
    • NFS keytags
    • lcabythebay.org.au
  • Announcement for Linux.conf.au 2017 will be in Hobart

 

Linux.conf.au 2015 – Day 5 – Session 2

When Everything Falls Apart: Stories of Version Control System Scaling – Ben Kero

  • Sysadmin at Mozilla looking after VCS
  • Primarily covering mercurial
  • Background
    • Primarily mercurial
    • 3445 repos (1223 unique)
    • 32 million commits
    • 2TB+ transfer per day
    • 1000+ clones per day
    • Biggest customer = ourselves
    • tested platforms > 12
  • Also use  git (a lot) and a bit of:  subversion, CVS, Bazaar, RCS
  • 2 * ssh servers, 10 machines mirror http traffic behind load balancer
  • 1st story – know what you are hosting
    • Big git repo 1.7G somebody asked to move off github
    • Turned out to be mozilla git mirror, so important to move
    • plenty of spare resources
    • But high load straight away
    • turned out to be mercurial->git converter, huge load
    • Ran garbage collection – took several hours
    • tweaked some other settings
  • 2nd story
    • 2003 . “Try” CI system
    • Simple CI system (before the term existed or they were common)
    • flicks off to build server, sends status back to dev
    • mercurial had history being immutable up until v2.1 and mozilla was stuck on old version
    • ended up with 29,000 brashes in repo
    • Around 10,000 heads some operations just start to fail
    • Wait times for pushes over 45 minutes. Manual fixes for this
    • process was “hg serve” only just freezein gup, not any debug info
    • had to attached debugging. trying to update the cache.
    • cache got nuked by cached push, long process to rebuild it.
    • mercurial bug 4255 in process of being looked at, no fix yet
  • The new system
    • More web-scalable to replace old the system
    • Closer to the pull-request model
    • multi-homing
    • leverage mercurial bundles
    • stores bundles in scalable object store
    • hopefully minimal retooling from other groups (lots of weird systems supported)
  • Planet release engineering @ mozilla

SL[AUO]B: Kernel memory allocator design and philosophy – Christopher Lameter

  • NOTE: I don’t do kernel stuff so much of this is over my head.
  • Role of the allocator
    • page allocator only works in full page size (4k) and is fairly slow
    • slab allocator for smaller allocation
    • SLAB is one of the “slab allocators”
  • kmeme_cache , numa aware, etc
  • History
    • SLOB: K&R 1991-1999 . compact
    • SLAB: Solaris 199-2008 . cache friendly, benchmark friendly
    • SLUB: 2008-today , simple and instruction costs count, better debugging, defrag, execution time friendly
  • 2013 – work to split out common code for allocators
  • SOLB
    • manages list of free objects with the space of free objects
    • have to traverse list to find object of sufficient size
    • rapid fragmentation of memory
  • SLAB
    • queues per cpu and per node to track cache hotness
    • queues for each remote node
    • complete data structures
    • cold object expiration every 2 seconds on each CPU
    • large systems with LOTS of CPUs have huge amount of memory trapped, spending lots of time cleaning cache
  • SLUB
    • A lot less queuing
    • Pages associated with per-cpu. increased locality
    • page based policies and interleave
    • de-fragmentation on multiple levels
    • current default in the kernel
  • slabinfo tool for SLUB. tune, modify, query, control objects and settings
  • can be asked to go into debug mode even when debugging not enabled with rest of the kernel
  • Comparing
    • SLUB faster (SLAB good for benchmarks)
    • SLOB slow
    • SLOB less memory overhead for small/simple systems (only, doesn’t handle lots of reallocations that fragment)
  • Roadmap
    • More common framework
    • Various other speedups and features

 

Linux.conf.au 2015 – Day 5 – Session 1

How to get one of those Open Source jobs – Mark Atwood

  • Warns talk might still have some US-centric stuff still in it
  • “Open Source Job” – most important word is “Job”
    • The Open Source bit means you are a bit more transferable than a closed-source programmer
    • Don’t have to move to major tech city
  • Communication skills
    • Have to learn to Write clearly in English
    • Heave to learn how to speak, including in meetings and give some talks
    • Reachable – Have a public email address
    • Don’t be a jerk, reputation very important
  • Technical skills
    • Learn how to program
    • Start with python and javascript
    • Learn other languages eg scale, erlang, clojure, c, C++
    • How to use debugger and IDE
    • Learn to use git well
    • Learn how to code test (especially to work with CI testers like jenkins)
    • Idea: Do lots of simple practise problems in programming using specific technique or language
  • Relationships & Peers
    • Work with people remote and nearby
    • stackoverflow
    • Don’t be a jerk
  • Work
    • Have to “do the work” then “get the job”
    • Start by fixing bugs on a project
    • Your skills will improve and others will see you have those skills
  • Collaborate
    • Many projects use IRC
    • Most projects have bug tracker
    • Learn how to use the non-basic stuff in git
    • Peer programming
  • Reputation
    • Portfolio vs resume
    • github account is your portfolio
    • Need to be on social media, at least a little bit, most be reachable
  • Getting the Job
    • If you have a good enough a rep the jobs will seek you out
    • Keywords on github and linkedin will attract recruiters
    • People will suggest you that apply
    • Conferences like linux.conf.au
    • Remember to counter-offer the offer letter
    • Once you are working for them, work out what is job related an the company might have a claim on. make sure you list in your agreement any projects you are already working on
  • Health
    • Don’t work longer than 40h a week regularly
    • 60h weeks can only be sustained for a couple of weeks
    • Just eat junk-food
    • Don’t work for jerks
  • Money
    • Startups – bad for your health. Do not kill yourself for a nickle, have real equity
  • Keep Learning
  • 3 books to read
    • Oh the palces you will go – Dr Seuss
    • Getting things Done – David Allen
    • How to fail at almost everything and still win big – Scott Adams

 

Pettycoin: Towards 1.0 – Rusty Russell

  • Problem it bitcoining mining is expensive, places lower limit on transaction fees
  • Took 6 months of to mostly work on pettycoin
  • Petty coin
    • Simple
    • gateway to bitcoin
    • small amounts
    • partial knowledge, don’t need to know everything
    • fast block times
  • Altcoins – bitcoin like things that are not bitcoin
    • 2 million posts to altcoin announce forum
    • lots of noise to talk to people
  • review
    • Paper released saying how it should have been done
    • hash functions
    • bitcoin blocks
    • Bitcoin transactions
  • Sidechain
    • alternative chains that use real bitcoins
    • Lots of wasted work? – bitcoin miners can mine other chains at the same time
    • too fast to keep notes
    • Compact CVP Proofs (reduce length of block header to go all the way back )

 

Linux.conf.au 2015 – Day 5 – Keynote/Panel

  • Everybody Sung Happy birthday to Baale
  • Bdale said he has a new house and FreedomBox 0.3 release this week
  • Rusty also on the panel
  • Questions:
    • Why is Linus so mean
    • Unified Storage/Memory machines – from HP
    • Young people getting into community
    • systemd ( I asked this)
    • Year of the Linux Desktop
    • Documentation & training material
    • Predict the security problems in next 12 month
    • Does NZ and Australia need a joint space agency
    • Will you be remembered more for Linux or Git?

Linux.conf.ay 2015 – Day 4 – Session 3

Drupal8 outta the box – Donna Benjamin

  • I went to the first half of this but wanted to catch the talk below so I missed the 2nd part

 

Connecting Containers: Building a PaaS with Docker and Kubernetes – Katie Miller

  • co-presented with Steve Pousty
  • Plugs their OpenShift book, they are re-archetecturing the whole thing based on what in the book
  • Platform as a service
    • dev tooling, runtime, OS , App server, middleware.
    • everything except the application itself
    • Openshift is an example
  • Reasons to rebuild
    • New tech
    • Lessons learned from old deploy
  • Stack
    • Atomic + docker + Kubeneties
  • Atomic
    • Redhat’s answer of CoreOS
    • RPM-OSTree – atomic update to the OS
    • Minimal System
    • Fast boot, container mngt, Good Kernel
  • Containers
    • Docker
    • Nice way of specifying everything
    • Pros – portable, easy to create, fast boot
    • Cons – host centric, no reporting
    • Wins – BYOP ( each container brings all it’s dependencies ) , Standard way to make containers , Big eco-system
  • Kubernetes
    • system managing containerize maps across multiple hosts
    • declarative model
    • open source by google
    • pod + service + label + replication controller
    • cluster = N*nodes + master(s) + etcd
    • Wins: Runtime and operation management + management related containers as a unit, container communication, available, scalable, automated, across multiple hosts
  • Rebuilding Openshift
    • Kubernetes provides container runtime
    • Openshift provides devops and team enviroment
  • Concepts
    • application = multiple pods linked togeather (front + back + db ) managed as a unit, scald independantly
    • config
    • template
    • build config = source + build -> image
    • deployment = image and settings for it
  • This is OpenShift v3 – things have been moving very fast so some docs are out of date
  • Slides http://containers.codemiller.com

Linux.conf.au 2015 – Day 4 – Session 2

Tunnels and Bridges: A drive through OpenStack Networking – Mark McClain

  • Challenges with the cloud
    • High density multi-tenancy
    • On demand provisioning
    • Need to place / move workloads
  • SDN , L2 fabric, network virtualisation Overlay tunneling
  • The Basics
    • The user sees the API, doesn’t matter too much what is behind
    • Neutron = Virtual subnet + L2 virtual network + virtual port
    • Nova = Server + interface on the server
  • Design Goals
    • Unified API
    • Small Core. Networks + Subnets + Ports
    • Plugable open archetecture
  • Features
    • Overlapping IPs
    • Configuration DHCP/Metadata
    • Floating IPs
    • Security Groups ( Like AWS style groups ) . Ingress/egress rules, IPv6 . VMs with multiple VIFS
  • Deployment
    • Database + Neutron Server + Message Queue
    • L2 Agent , L3 agent + DHCP Agent
  • Server
    • Core
    • Plugins types =  Proxy (proxy to backend) or direct control (login instide plugin)
    • ML2 – Modular Layer 2 plugin
  • Plugin extensions
    • Add to REST API
    • dpch, l3, quota, security group, metering, allowed addresses
  • L2 Agent
    • Runs on a hypervisor
    • Watch and notify when devices have been added/removed
  • L3 agent – static routing only for now
  • Load balancing as a service, based on haproxy
  • VPN as a service , based on openswan, replicates AWS VPC.
  • What is new in Juno?
    • IPv6
    • based on Radbd
    • Advised to go dual-stack
  • Look ahead to Kilo
    • Paying down technical debt
    • IPv6 prefix delegation, metadata service
    • IPAM – hook into external systems
    • Facilitate dynamic routing
    • Enabling NFV Applications
  • See Cloud Administrators Guide

 

Crypto Won’t Save You Either – Peter Gutmann

  • US Govt has capabilities against common encryption protocols
  • BULLRUN
  • Example Games consoles
    • Signed executables
    • encrypted storage
    • Full media and memory encryption
    • All of these have been hacked
  • Example – Replaced signature checking code
  • Example – Hacked “secure” kernel to attack the application code
  • Example – Modify firmware to load over the checking code
  • Example – Recover key from firmware image
  • Example – Spoof on-air update
  • LOTS of examples
  • Nobody noticed bunch of DKIM keys were bad, cause all attackers had bypassed encryption rather than trying to beat the crypto
  • No. of times crypto broken: 0, bypassed: all the rest
  • National Security Letters – The Legalised form of rubber-hose cryptanalysis
  • Any well design crypto is NSA-proof
  • The security holes are sitting right next to the crypto

 

Linux.conf.au 2015 – Day 4 – Session 1

8 writers in under 8 months: from zero to a docs team in no time flat – Lana Brindley

  • Co Presenting with Alexandra Settle
  • 8 months ago online 1 documentation person at rackspace
  • Hired a couple people
  • Horrible documentation suite
  • Hired some more
  • 4 in Australia, 4 in the US
  • Building a team fast without a terrible culture
    • Management by MEME – everybody had a meme created for them when they started
    • Not all work and No play. But we still get a lot of work done
    • Use tech to overcome geography
    • Treat people as humans not robots
    • Always stay flexible. Couch time, Gym time
  • Finding the right people
    • Work your network , job is probably not going to be advertise on linkedin, bad for diversity
    • Find great people, and work out how to hire them
    • If you do want a job, network
  • Toolchains and Systems
    • Have a vision and work towards it
    • acknowledge imperfection. If you can’t fix, ack and just move forward anyway
  • You can maintain crazy growth forever. You have to level off.
  • Pair US person with AU person for projects
  • Writers should attend Docs summit and encouraged to attend at least one Openstack summit

 

Linux.conf.au 2015 – Day 4 – Keynotes

Cooper Lees – Facebook

  • Open Source at facebook
  • Increase in pull requests, not just pushing out stuff or throwing over the wall anymore
  • Focussing on full life-cycle of opensource
  • Big Projects: react , hhvm , asyncdisplaykit , presto
  • Working on other projects and sending to upstream
  • code.facebook.com  github.com/facebook
  • Network Switches and Open Compute
    • Datacentre in NZ using open compute designs
  • Open source Switch
    • Top of rack switch
    • Want to be the open compute of network switches
    • Installer, OS, API to talk to asic that runs ports
    • Switches = Servers. running chef
  • Wedge
    • 16-32 of 40GE ports
    • Internal facebook design
    • 1st building block for disaggregated switching technology
    • Contributed to OCP project
    • Micro Server + Switchports

Carol Smith – Google

  • Works in Google Open Source office
  • Google Summer of code
    • Real world experience
    • Contacts and references
  • 11th year of the program
  • 8600 participated over last 10 years
  • Not enough people in office to do southern hemisphere programme. There is “Google code-in” though

Mark McLoughlin – Red Hat

  • Open Source and the datacenter
  • iaas, paas, microservices, etc
  • The big guys are leading (amazon, google). They are building on open source
  • Telcos
    • Squeezed and scrambling
    • Not so “special” anymore
    • Need to be agile and responsive
    • Telecom datacentre – filled with big, expensive, proprietary boxes
    • opposite of agile
  • OPNFV reference architecture
  • OpenStack, Open vswitch, etc
  • Why Open Source? – collaboration and coopetition , diversity drives innovation , sustainability

 

There was a Q&A. Mostly questions about diversity at the companies and grumps about having to move to US/Sydney for peopl eto work for them

Linux.conf.au – Day 3 – Lightning talks

 

  • Clinton Roy + Tom Eastman – Python Conference Australia 2015 + Kiwi PyCon 2015
    • Brisbane , late July 2015
    • Similar Structure to LCA
    • Christchurch – Septemberish
    • kiwi.pycon.org
  • Daniel Bryan – Comms for Camps
    • Detention camps for Australian boats people camps
    • Please contact if you can offer technical help
  • Phil Ingram – Beernomics
    • Doing stuff for people in return for beer
    • Windows reinstall = a Keg
    • Beercoin
  • Patrick Shuff – Open sourcing proxygen
    • C++ http framework. Built own webserver
    • Features they need, monitoring, fast, easy to add new features
    • github -> /facebook/progen
  • Nicolás Erdödy – Multicore World 2015 & the SKA.
    • Multicore World – 17-18 Feb 2015 Wellington
  • Paul Foxworthy – Open Source Industry Australia (OSIA)
    • Industry Body
    • Govt will consult with industry bodies but won’t listen to individual companies
    • Please join
  • Francois Marier – apt-get remove –purge skype
    • Web RTC
    • Now usable to replace skype
    • Works in firefox and chrome. Click link, no account, video conversation
    • Firefox Hello
  • Tobin Harding – Central Coast LUG
    • Update on Central Coast of NSW LUG
    • About 6 people regularly
  • Mark Smith – Failing Gracefully At 10,000ft
    • Private pilot
    • Aircrafts have 400+ page handbooks
    • Things will fail…
    • Have procedures…
    • Before the engine is on fire
    • test
    • The most important task is to fly the plane
  • Tim Serong – A very short song about memory management
    • 1 verson song
  • Angela Brett – Working at CERN and why you should do it
    • Really Really awesome
    • Basic I applied, lots of fellowship
    • Meet someone famous
    • Lectures online from famous people
  • Donna Benjamin – The D8 Chook Raffle
    • $125k fund to get Drupal8 out
    • Raffle. google it
  • Matthew Cengia/maia sauren – What is the Open Knowledge Foundation?
    • au.okfn.org
    • Open govt/ data / tech / jouralism / etc
    • govHack
    • Open Knowledge Brisbane Meetup Govt
  • Florian Forster – noping
    • Pretty graphs and output on command line ping
    • http://noping.cc
  • Jan Schmidt – Supporting 3D movies in GStreamer
    • A brief overview of it all
  • Justin Clacherty ORP – An open hardware, open software router
    • PowerPC 1-2G RAM
    • Package based updates
    • Signed packages
    • ORP1.com

Linux.conf.au 2015 – Day 3 – Session 2

EQNZ – crisis response, open source style – Brenda Wallace

  • Started with a Trigger warning and “fucker”
  • First thing posted – “I am okay” , one tweet, one facebook
  • State of Scial Media
    • Social media not as common, SMS king, not many smartphones
    • Google Buzz, twitter, Facebook
    • Multiple hashtags
  • Questions people asked on social media
  • Official info was under strain, websites down due to bad generators
  • Crisis Commons
  • Skype
    • Free
    • Multi-platform
    • Txt based
    • Battery Drain very bad
    • Bad internet in Chc hard to use, no mobile, message reply for minutes on join
  • Things pop up within an hour
    • Pirate Pad
    • Couch apps
    • Wikis
    • WordPress installs
  • Short code 4000 for non-urgent help live by 5pm
    • Volenteers processing the queue
  • All telcos agree to coordinate their social media effort
  • Civil defence didn’t have site ready and refused offers, people decided to do independantly
  • Ushahidi instance setup
    • Google setup people finder app
    • Moved into ec2 cluther
    • hackfest, including added mobile
    • Some other Ushidis, in the end newspaper sites enbedded
  • Council
    • chc council wordpress for info
    • Very slow and bad UI
    • Hit very hard, old information from the previous earthquake
    • staff under extreme pressure
  • Civil Defence
    • Official info only
    • Falls over
    • Caught by DDOS against another govt site
  • Our reliability
    • Never wen tdown
    • contact and reassured some authorities
    • After 24h . 78k page impressions
  • Skype
    • 100+ chatting. limitations
    • IRC used by some but many no common enough
    • Gap for something common. cross platform, easy to use
  • Hashtag
    • twitter to SMS notifications to add stuff to website
  • Maps were a new thing
    • None of the authorities knew them
  • Council and DHB websites did not work on mobile and were not updating
  • Government
    • Govt officers didn’t talk – except NZ Geospacial office
    • Meeting that some people attended
  • Wrap up after 3 weeks
    • Redirected website
    • Anonymous copy of database
  • Pragmatic
    • Used closed source where we had too (eg skype)
    • But easier with OS could quick to modify
    • Closed source people could install webserver, use git, etc. Hard to use contributions
  • Burned Bridges
    • Better jobs with Gov agencies
  • These days
    • Tablets
    • Would use EC2 again
    • phones have low power mode
    • more open street maps

 

collectd in dynamic environments – Florian Forster

  • Started collectd in 2005
  • Dynamic environments – Number and location of machines change frequently – VM or job management system
  • NOTE: I use collectd so my notes are a little sparse here cause I knew most of it already
  • Collects timeseries data, does one thing well. collectd.org
  • agent runs on each host, plugins mostly in C for lots of things or exec plug to run random stuff.
  • Read Plugins to get metrics from system metrics, applications, other weird stuff
  • Write plugs – Graphite, RRD, Reimann, MongoDB
  • Virtual machine Metrics
    • libvirt plugin
    • Various metrics, cpu, memory, swap, disk ops/bytes, network
    • GenericJMX plugin – connects to JVM. memory and garbage collection, threads
  • Network plugin
    • sends and receives metric
    • Effecient binary protocol. 50-100 byte UDP multicast/unicast protocol
    • crypto available
    • send, receive, forward packets
  • Aggregation
    • Often more useful for alerting
  • Aggregation plugin
    • Subscribes to metric
    • aggregates and forwards
    • Limitation, no state, eg medium, mean are missing
    • only metrics with one value
    • can be aggregated at any level
    • eg instead of each CPU then total usage of all your CPUS
  • Reimann
    • Lots of filters and functions
    • can aggregate, many otions
  • Bosum
    • Monitoring and alert language
  • Storage
    • Graphite
    • OpenTSDB based on hadoop
    • InfluxDB – understand collectd protocol native (and graphite).
    • Vaultaire ( no collectd integration but… )
  • New Dishboard – facette.io