Category: Linux.conf.au

Cloud Anti-Patterns – Casey West

• The 5 stages of Cloud Native
• Deploying my apps to the cloud is painful – why?
• Denial
  • “Containers are like tiny VMs”
  • Anti-Pattern 1 – do not assume what you have now is what you want to put into the cloud or a container
  • “We don’t need to automate continuous delivery”
  • We shouldn’t automate what we have until it is perfect. Automate to make things consistent (not always perfect at least at the start)
• Anger
  • “works on my machine”
  • Dev is just push straight from dev boxes to production
  • Not about making worse code go to production faster
  • Aim to repeatable testable builds, just faster
• Bargaining
  • “We crammed the monolith into a container and called it a microservice”
  • Anti-Pattern: Critically think on what you need to re-factor (or “re-platforming” )
  • ” Bi-modal IT “
  • Some stuff on fast lane, some stuff on old-way slow lane
  • Anti-pattern: leagacy products put into slow lane, these are often the ones that really need to be fixed.
  • “Micros-services” talking to same data-source, not APIs
• Depression
  • “200 microservices but forgot to setup Jenkins”
  • “We have an automated build pipeline but online release twice per year”
• Acceptance
  • All software sucks, even the stuff we write
  • Respect CAP theorem
  • Respect Conway’s Law
  • Small batch sizes works for replatforming too
• Microservices architecture, Devops culture, Continuous delivery – Pick all three

Cloud Crafting – Public / Private / Hybrid  – Steven Ellis

• What does Hybrid mean to you?
• What is private Cloud (IAAS)
• Hybrid – communicate to public cloud and manage local stuff
• ManageIQ – single pain of glass for hardware, vms, clounds, containers
• What does it do?
  • Brownfields as well as Greenfields, gathers current setup
  • Discovery, API presentations, control and detect when env non-complient (eg not fully patched)
  • Premise or public cloud
  • Supplied as a virtual appliance, HA, scale out
  • Platform – Centos 7, rails, postgress, gui, some dashboards our of the box.
• Get involved
  • Online, roadmap is public
  • Various contributors
• DEMO
• Just put in credentials to allow access and then it can gather the data straiht away

Live Migration of Linux Containers by Tycho Andersen

• LXC / LXD
• LXD is a REST API that you use to control the container system
• tool -> RST -> Daemon -> lxc -> Kernel
• “lxc move host1:c1 host2: ” – Live migrations
  • Needs a bit of work since lots moving, lots of ways it could fail
  • 3 channels created, control, filesystem, container processes state
• CRIU
  • 5 years of check-pointing
  • Lots based off open-VZ initial work
  • All sorts of things need to support check-pointing and moving (eg selinux)
  • Iterative migration added
  • Lots of hooks needed for very privileged kernel features
• Filesystems
  • btrfs, lvm, zfs, (swift, nfs), have special support for migration that it hooks into
  • rsync between incompatable hosts
• Memory State
  • Stop the world and move it all
  • Iterative incremental transfer (via p.haul) being worked on.
• LXC + LXD 2.0 should be in Ubuntu 16.04 LTS
• Need to use latest versions and latest kernels for best results.

Open Cloud Miniconf – Continuous Delivery using blue-green deployments and immutable infrastructure by Ruben Rubio Rey

• Lots of things can go wrong in a deployment
• Often hard to do rollbacks once upgrade happens
• Blue-Green deployment is running several envs at the same time, each potentially with different versions
• Immutable infrastructure , split between data (which changes) and everything else only gets replaced fully by deployments, not changed
• When you use docker don’t store data in the container, makes it immutable. But containers are not required to do this.
• Rule 1 – Never modify the infrastructure
• Rule 2 – Instead of modifying – always create from ground up everything that is not data.
• Advantages
  • Rollbacks easy
  • Avoid Configuration drift
  • Updated and accurate infrastructure documentation
• Split things up
  • No State – LBs, Web servers, App Servers
  • Temp data , Volatile State – message queues, email servers
  • Persistent data – Databases, Filesystems, slow warming cache
• In case of temp data you have to be able to drain
• USe LBs and multiple servers to split up infrastructure, more bit give more room to split up the upgrades.
• If pending jobs require old/new version of app then route to servers that have/not been upgraded yet.
• Put toy rocket launcher in devs office, shoots person who broke the build.
• Need to “use activity script” to bleed traffic off section of the “temp data” layer of infrastructure, determine when it is empty and then re-create.

NoOps with Ansible and Puppet – Monty Taylor

• NoOps
  • didn’t know it was a contentious term
  • “devs can code and let a service deploy, manage and scale their code”
  • I want to change the system by landing commits. don’t want to “do ops”
  • if I have to use my root access it is a bug
• Cloud Native
  • Ephemeral Compute
  • Data services
  • Design your applications to be resilient via scale out
  • Cloud scale out, forget HA for one system, forget long-lived system, shared-nothing for everything. Cloud provides the hard scale-out/HA/9s stuff
  • Great for new applications
• OpenStack Infra
  • Tooling, automation, and CI for the openstack project
  • 2000 devs
  • every commit is fully tested.
  • each test runs on a single use cloud slave
  • 1.7 million test jobs in the last 6 months. 18 TB of log data
  • all runs in HP and rackspace public clouds
• Create Servers manually at 1st
• Step 1 – Puppet
  • extra hipster because it is in ruby
  • If you like ruby it is awesome. If don’t is it less-awesome
  • collaboration from non-root users
  • code review
  • problem that it blows up when you try and install the same thing in two different places
  • 3 ways to run. masterless puppet apply. master + puppet agent daemon . master + puppet agent non-daemons
• Secret stuff that you don’t want into you puppet git repo
  • hiera
• Step 2 – Ansible for orchestration
  • Control the puppet agent so it runs it nicely and in schedule and on correct hosts first
  • Open source system management tool
  • Sequence of steps not description of state like puppet
  • ad-hoc operation. run random commands
  • easy to slowly grow over time till it takes over puppet
  • yaml syntax of config files
• Step 3 – Ansible for cloud management
• Ansible config currently mixed in with puppet under – http://git.openstack.org/cgit/openstack-infra/system-config/

 

Conference Closing

• Steve Walsh wins Rusty Wrench award
• Preview of Linux.conf.au 2016 in Geelong
  • Much flatter than Auckland
  • Deakin University – Waterfront Campus
  • Waurn Ponds student accomadation 15 minutes with shuttles
  • Feb 8th – 12th 2016
  • CFP 1st of June 2015
  • Theme “life is better with linux”
  • 4 keynotes confirmed or in final stages of discussion, 2 female, 2 male
  • NFS keytags
  • lcabythebay.org.au
• Announcement for Linux.conf.au 2017 will be in Hobart

 

When Everything Falls Apart: Stories of Version Control System Scaling – Ben Kero

• Sysadmin at Mozilla looking after VCS
• Primarily covering mercurial
• Background
  • Primarily mercurial
  • 3445 repos (1223 unique)
  • 32 million commits
  • 2TB+ transfer per day
  • 1000+ clones per day
  • Biggest customer = ourselves
  • tested platforms > 12
• Also use  git (a lot) and a bit of:  subversion, CVS, Bazaar, RCS
• 2 * ssh servers, 10 machines mirror http traffic behind load balancer
• 1st story – know what you are hosting
  • Big git repo 1.7G somebody asked to move off github
  • Turned out to be mozilla git mirror, so important to move
  • plenty of spare resources
  • But high load straight away
  • turned out to be mercurial->git converter, huge load
  • Ran garbage collection – took several hours
  • tweaked some other settings
• 2nd story
  • 2003 . “Try” CI system
  • Simple CI system (before the term existed or they were common)
  • flicks off to build server, sends status back to dev
  • mercurial had history being immutable up until v2.1 and mozilla was stuck on old version
  • ended up with 29,000 brashes in repo
  • Around 10,000 heads some operations just start to fail
  • Wait times for pushes over 45 minutes. Manual fixes for this
  • process was “hg serve” only just freezein gup, not any debug info
  • had to attached debugging. trying to update the cache.
  • cache got nuked by cached push, long process to rebuild it.
  • mercurial bug 4255 in process of being looked at, no fix yet
• The new system
  • More web-scalable to replace old the system
  • Closer to the pull-request model
  • multi-homing
  • leverage mercurial bundles
  • stores bundles in scalable object store
  • hopefully minimal retooling from other groups (lots of weird systems supported)
• Planet release engineering @ mozilla

SL[AUO]B: Kernel memory allocator design and philosophy – Christopher Lameter

• NOTE: I don’t do kernel stuff so much of this is over my head.
• Role of the allocator
  • page allocator only works in full page size (4k) and is fairly slow
  • slab allocator for smaller allocation
  • SLAB is one of the “slab allocators”
• kmeme_cache , numa aware, etc
• History
  • SLOB: K&R 1991-1999 . compact
  • SLAB: Solaris 199-2008 . cache friendly, benchmark friendly
  • SLUB: 2008-today , simple and instruction costs count, better debugging, defrag, execution time friendly
• 2013 – work to split out common code for allocators
• SOLB
  • manages list of free objects with the space of free objects
  • have to traverse list to find object of sufficient size
  • rapid fragmentation of memory
• SLAB
  • queues per cpu and per node to track cache hotness
  • queues for each remote node
  • complete data structures
  • cold object expiration every 2 seconds on each CPU
  • large systems with LOTS of CPUs have huge amount of memory trapped, spending lots of time cleaning cache
• SLUB
  • A lot less queuing
  • Pages associated with per-cpu. increased locality
  • page based policies and interleave
  • de-fragmentation on multiple levels
  • current default in the kernel
• slabinfo tool for SLUB. tune, modify, query, control objects and settings
• can be asked to go into debug mode even when debugging not enabled with rest of the kernel
• Comparing
  • SLUB faster (SLAB good for benchmarks)
  • SLOB slow
  • SLOB less memory overhead for small/simple systems (only, doesn’t handle lots of reallocations that fragment)
• Roadmap
  • More common framework
  • Various other speedups and features

 

How to get one of those Open Source jobs – Mark Atwood

• Warns talk might still have some US-centric stuff still in it
• “Open Source Job” – most important word is “Job”
  • The Open Source bit means you are a bit more transferable than a closed-source programmer
  • Don’t have to move to major tech city
• Communication skills
  • Have to learn to Write clearly in English
  • Heave to learn how to speak, including in meetings and give some talks
  • Reachable – Have a public email address
  • Don’t be a jerk, reputation very important
• Technical skills
  • Learn how to program
  • Start with python and javascript
  • Learn other languages eg scale, erlang, clojure, c, C++
  • How to use debugger and IDE
  • Learn to use git well
  • Learn how to code test (especially to work with CI testers like jenkins)
  • Idea: Do lots of simple practise problems in programming using specific technique or language
• Relationships & Peers
  • Work with people remote and nearby
  • stackoverflow
  • Don’t be a jerk
• Work
  • Have to “do the work” then “get the job”
  • Start by fixing bugs on a project
  • Your skills will improve and others will see you have those skills
• Collaborate
  • Many projects use IRC
  • Most projects have bug tracker
  • Learn how to use the non-basic stuff in git
  • Peer programming
• Reputation
  • Portfolio vs resume
  • github account is your portfolio
  • Need to be on social media, at least a little bit, most be reachable
• Getting the Job
  • If you have a good enough a rep the jobs will seek you out
  • Keywords on github and linkedin will attract recruiters
  • People will suggest you that apply
  • Conferences like linux.conf.au
  • Remember to counter-offer the offer letter
  • Once you are working for them, work out what is job related an the company might have a claim on. make sure you list in your agreement any projects you are already working on
• Health
  • Don’t work longer than 40h a week regularly
  • 60h weeks can only be sustained for a couple of weeks
  • Just eat junk-food
  • Don’t work for jerks
• Money
  • Startups – bad for your health. Do not kill yourself for a nickle, have real equity
• Keep Learning
• 3 books to read
  • Oh the palces you will go – Dr Seuss
  • Getting things Done – David Allen
  • How to fail at almost everything and still win big – Scott Adams

 

Pettycoin: Towards 1.0 – Rusty Russell

• Problem it bitcoining mining is expensive, places lower limit on transaction fees
• Took 6 months of to mostly work on pettycoin
• Petty coin
  • Simple
  • gateway to bitcoin
  • small amounts
  • partial knowledge, don’t need to know everything
  • fast block times
• Altcoins – bitcoin like things that are not bitcoin
  • 2 million posts to altcoin announce forum
  • lots of noise to talk to people
• review
  • Paper released saying how it should have been done
  • hash functions
  • bitcoin blocks
  • Bitcoin transactions
• Sidechain
  • alternative chains that use real bitcoins
  • Lots of wasted work? – bitcoin miners can mine other chains at the same time
  • too fast to keep notes
  • Compact CVP Proofs (reduce length of block header to go all the way back )

 

• Everybody Sung Happy birthday to Baale
• Bdale said he has a new house and FreedomBox 0.3 release this week
• Rusty also on the panel
• Questions:
  • Why is Linus so mean
  • Unified Storage/Memory machines – from HP
  • Young people getting into community
  • systemd ( I asked this)
  • Year of the Linux Desktop
  • Documentation & training material
  • Predict the security problems in next 12 month
  • Does NZ and Australia need a joint space agency
  • Will you be remembered more for Linux or Git?

Drupal8 outta the box – Donna Benjamin

• I went to the first half of this but wanted to catch the talk below so I missed the 2nd part

 

Connecting Containers: Building a PaaS with Docker and Kubernetes – Katie Miller

• co-presented with Steve Pousty
• Plugs their OpenShift book, they are re-archetecturing the whole thing based on what in the book
• Platform as a service
  • dev tooling, runtime, OS , App server, middleware.
  • everything except the application itself
  • Openshift is an example
• Reasons to rebuild
  • New tech
  • Lessons learned from old deploy
• Stack
  • Atomic + docker + Kubeneties
• Atomic
  • Redhat’s answer of CoreOS
  • RPM-OSTree – atomic update to the OS
  • Minimal System
  • Fast boot, container mngt, Good Kernel
• Containers
  • Docker
  • Nice way of specifying everything
  • Pros – portable, easy to create, fast boot
  • Cons – host centric, no reporting
  • Wins – BYOP ( each container brings all it’s dependencies ) , Standard way to make containers , Big eco-system
• Kubernetes
  • system managing containerize maps across multiple hosts
  • declarative model
  • open source by google
  • pod + service + label + replication controller
  • cluster = N*nodes + master(s) + etcd
  • Wins: Runtime and operation management + management related containers as a unit, container communication, available, scalable, automated, across multiple hosts
• Rebuilding Openshift
  • Kubernetes provides container runtime
  • Openshift provides devops and team enviroment
• Concepts
  • application = multiple pods linked togeather (front + back + db ) managed as a unit, scald independantly
  • config
  • template
  • build config = source + build -> image
  • deployment = image and settings for it
• This is OpenShift v3 – things have been moving very fast so some docs are out of date
• Slides http://containers.codemiller.com

Tunnels and Bridges: A drive through OpenStack Networking – Mark McClain

• Challenges with the cloud
  • High density multi-tenancy
  • On demand provisioning
  • Need to place / move workloads
• SDN , L2 fabric, network virtualisation Overlay tunneling
• The Basics
  • The user sees the API, doesn’t matter too much what is behind
  • Neutron = Virtual subnet + L2 virtual network + virtual port
  • Nova = Server + interface on the server
• Design Goals
  • Unified API
  • Small Core. Networks + Subnets + Ports
  • Plugable open archetecture
• Features
  • Overlapping IPs
  • Configuration DHCP/Metadata
  • Floating IPs
  • Security Groups ( Like AWS style groups ) . Ingress/egress rules, IPv6 . VMs with multiple VIFS
• Deployment
  • Database + Neutron Server + Message Queue
  • L2 Agent , L3 agent + DHCP Agent
• Server
  • Core
  • Plugins types =  Proxy (proxy to backend) or direct control (login instide plugin)
  • ML2 – Modular Layer 2 plugin
• Plugin extensions
  • Add to REST API
  • dpch, l3, quota, security group, metering, allowed addresses
• L2 Agent
  • Runs on a hypervisor
  • Watch and notify when devices have been added/removed
• L3 agent – static routing only for now
• Load balancing as a service, based on haproxy
• VPN as a service , based on openswan, replicates AWS VPC.
• What is new in Juno?
  • IPv6
  • based on Radbd
  • Advised to go dual-stack
• Look ahead to Kilo
  • Paying down technical debt
  • IPv6 prefix delegation, metadata service
  • IPAM – hook into external systems
  • Facilitate dynamic routing
  • Enabling NFV Applications
• See Cloud Administrators Guide

 

Crypto Won’t Save You Either – Peter Gutmann

• US Govt has capabilities against common encryption protocols
• BULLRUN
• Example Games consoles
  • Signed executables
  • encrypted storage
  • Full media and memory encryption
  • All of these have been hacked
• Example – Replaced signature checking code
• Example – Hacked “secure” kernel to attack the application code
• Example – Modify firmware to load over the checking code
• Example – Recover key from firmware image
• Example – Spoof on-air update
• LOTS of examples
• Nobody noticed bunch of DKIM keys were bad, cause all attackers had bypassed encryption rather than trying to beat the crypto
• No. of times crypto broken: 0, bypassed: all the rest
• National Security Letters – The Legalised form of rubber-hose cryptanalysis
• Any well design crypto is NSA-proof
• The security holes are sitting right next to the crypto

 

8 writers in under 8 months: from zero to a docs team in no time flat – Lana Brindley

• Co Presenting with Alexandra Settle
• 8 months ago online 1 documentation person at rackspace
• Hired a couple people
• Horrible documentation suite
• Hired some more
• 4 in Australia, 4 in the US
• Building a team fast without a terrible culture
  • Management by MEME – everybody had a meme created for them when they started
  • Not all work and No play. But we still get a lot of work done
  • Use tech to overcome geography
  • Treat people as humans not robots
  • Always stay flexible. Couch time, Gym time
• Finding the right people
  • Work your network , job is probably not going to be advertise on linkedin, bad for diversity
  • Find great people, and work out how to hire them
  • If you do want a job, network
• Toolchains and Systems
  • Have a vision and work towards it
  • acknowledge imperfection. If you can’t fix, ack and just move forward anyway
• You can maintain crazy growth forever. You have to level off.
• Pair US person with AU person for projects
• Writers should attend Docs summit and encouraged to attend at least one Openstack summit

 

Cooper Lees – Facebook

• Open Source at facebook
• Increase in pull requests, not just pushing out stuff or throwing over the wall anymore
• Focussing on full life-cycle of opensource
• Big Projects: react , hhvm , asyncdisplaykit , presto
• Working on other projects and sending to upstream
• code.facebook.com  github.com/facebook
• Network Switches and Open Compute
  • Datacentre in NZ using open compute designs
• Open source Switch
  • Top of rack switch
  • Want to be the open compute of network switches
  • Installer, OS, API to talk to asic that runs ports
  • Switches = Servers. running chef
• Wedge
  • 16-32 of 40GE ports
  • Internal facebook design
  • 1st building block for disaggregated switching technology
  • Contributed to OCP project
  • Micro Server + Switchports

Carol Smith – Google

• Works in Google Open Source office
• Google Summer of code
  • Real world experience
  • Contacts and references
• 11th year of the program
• 8600 participated over last 10 years
• Not enough people in office to do southern hemisphere programme. There is “Google code-in” though

Mark McLoughlin – Red Hat

• Open Source and the datacenter
• iaas, paas, microservices, etc
• The big guys are leading (amazon, google). They are building on open source
• Telcos
  • Squeezed and scrambling
  • Not so “special” anymore
  • Need to be agile and responsive
  • Telecom datacentre – filled with big, expensive, proprietary boxes
  • opposite of agile
• OPNFV reference architecture
• OpenStack, Open vswitch, etc
• Why Open Source? – collaboration and coopetition , diversity drives innovation , sustainability

 

There was a Q&A. Mostly questions about diversity at the companies and grumps about having to move to US/Sydney for peopl eto work for them