Author: simon

Sanjeev Sharma – When DevOps met SRE: From Apollo 13 to Google SRE

• Author of Two DevOps Bookks
• Apollo 13
  • Who were the real heroes? The guys back at missing control. The Astronaunts just had to keep breathing and not die
• Best Practice for Incident management
  • Prioritize
  • Prepare
  • Trust
  • Introspec
  • Consider Alternatives
  • Practice
  • Change it around
• Big Hurdles to adoption of DevOps in Enterprise
  • Literature is Only looking at one delivery platform at a time
  • Big enterprise have hundreds of platforms with completely different technologies, maturity levels, speeds. All interdependent
  • He Divides
    • Industrialised Core – Value High, Risk Low, MTBF
    • Agile/Innovation Edge – Value Low, Risk High, Rapid change and delivery, MTTR
    • Need normal distribution curve of platforms across this range
    • Need to be able to maintain products at both ends in one IT organisation
• 6 capabilities needed in IT Organisation
  • Planning and architecture.
    • Your Delivery pipeline will be as fast as the slowest delivery pipeline it is dependent on
  • APIs
    • Modernizing to Microservices based architecture: Refactoring code and data and defining the APIs
  • Application Deployment Automation and Environment Orchestration
    • Devs are paid code, not maintain deployment and config scripts
    • Ops must provide env that requires devs to do zero setup scripts
  • Test Service and Environment Virtualisation
    • If you are doing 2week sprints, but it takes 3-weeks to get a test server, how long are your sprints
  • Release Management
    • No good if 99% of software works but last 1% is vital for the business function
  • Operational Readiness for SRE
    • Shift between MTBF to MTTR
    • MTTR  = Mean time to detect + Mean time to Triage + Mean time to restore
    • + Mean time to pass blame
  • Antifragile Systems
    • Things that neither are fragile or robust, but rather thrive on chaos
    • Cattle not pets
    • Servers may go red, but services are always green
  • DevOps: “Everybody is responsible for delivery to production”
  • SRE: “(Everybody) is responsible for delivering Continuous Business Value”

Marcus Bristol (Pushpay) – Moving fast without crashing

• Low tolerance for errors in production due to being in finance
• Deploy twice per day
• Just Culture – Balance safety and accountability
  • What rule?
  • Who did it?
  • How bad was the breach?
  • Who gets to decide?
• Example of Retributive Culture
  • KPIs reflect incidents.
  • If more than 10% deploys bad then affect bonus
  • Reduced number of deploys
• Restorative Culture
• Blameless post-mortem
  • Can give detailed account of what happened without fear or retribution
  • Happens after every incident or near-incident
  • Written Down in Wiki Page
  • So everybody has the chance to have a say
  • Summary, Timeline, impact assessment, discussion, Mitigations
  • Mitigations become highest-priority work items
• Our Process
  • Feature Flags
  • Science
  • Lots of small PRs
  • Code Review
  • Testers paired to devs so bugs can be fixed as soon as found
  • Automated tested
  • Pollination (reviews of code between teams)
  • Bots
    • Posts to Slack when feature flag has been changed
    • Nags about feature flags that seems to be hanging around in QA
    • Nags about Flags that have been good in prod for 30+ days
    • Every merge
    • PRs awaiting reviews for long time (days)
    • Missing postmortun migrations
    • Status of builds in build farm
    • When deploy has been made
    • Health of API
    • Answer queries on team member list
    • Create ship train of PRs into a build and user can tell bot to deploy to each environment

Michael Coté – Not actually a DevOps Talk

Digital Transformation

• Goal: deliver value, weekly reliably, with small patches
• Management must be the first to fail and transform
• Standardize on a platform: special snow flakes are slow, expensive and error prone (see his slide, good list of stuff that should be standardize)
• Ramping up: “Pilot low-risk apps, and ramp-up”
• Pair programming/working
  • Half the advantage is people speed less time on reddit “research”
• Don’t go to meetings
• Automate compliance, have what you do automatic get logged and create compliance docs rather than building manually.
• Crafting Your Cloud-Native Strategy

Sajeewa Dayaratne – DevOps in an Embedded World

• Challenges on Embedded
  • Hardware – resource constrinaed
  • Debugging – OS bugs, Hardware Bugs, UFO Bugs – Oscilloscopes and JTAG connectors are your friend.
  • Environment – Thermal, Moisture, Power consumption
  • Deploy to product – Multi-month cycle, hard of impossible to send updates to ships at sea.
• Principles of Devops , equally apply to embedded
  • High Frequency
  • Reduce overheads
  • Improve defect resolution
  • Automate
  • Reduce response times
• Navico
  • Small Sonar, Navigation for medium boats, Displays for sail (eg Americas cup). Navigation displays for large ships
  • Dev around world, factory in Mexico
• Codebase
  • 5 million lines of code
  • 61 Hardware Products supported – Increasing steadily, very long lifetimes for hardware
  • Complex network of products – lots of products on boat all connected, different versions of software and hardware on the same boat
• Architecture
  • Old codebase
  • Backward compatible with old hardware
  • Needs to support new hardware
  • Desire new features on all products
• What does this mean
  • Defects were found too late
  • Very high cost of bugs found late
  • Software stabilization taking longer
  • Manual test couldn’t keep up
  • Cost increasing , including opportunity cost
• Does CI/CD provide answer?
  • But will it work here?
  • Case Study from HP. Large-Scale Agile Development by Gary Gruver
• Our Plan
  • Improve tolls and archetecture
  • Build Speeds
  • Automated testing
  • Code quality control
• Previous VCS
  • Proprietary tool with limit support and upgrades
  • Limited integration
  • Lack of CI support
  • No code review capacity
• Move to git
  • Code reviews
  • Integrated CI
  • Supported by tools
• Archetecture
  • Had a configurable codebase already
  • Fairly common hardware platform (only 9 variations)
  • Had runtime feature flags
  • But
    • Cyclic dependancies – 1.5 years to clean these up
    • Singletons – cut down
    • Promote unit testability – worked on
    • Many branches – long lived – mega merges
• Went to a single Branch model, feature flags, smaller batch sizes, testing focused on single branch
• Improve build speed
  • Start 8 hours to build Linux platform, 2 hours for each app, 14+ hours to build and package a release
  • Options
    • Increase speed
    • Parallel Builds
  • What did
    • ccache.clcache
    • IncrediBuild
    • distcc
  • 4-5hs down to 1h
• Test automation
  • Existing was mock-ups of the hardware to not typical
  • Started with micro-test
    • Unit testing (simulator)
    • Unit testing (real hardware)
  • Build Tools
    • Software tools (n2k simulator, remote control)
    • Hardware tools ( Mimic real-world data, re purpose existing stuff)
  • UI Test Automation
    • Build or Buy
    • Functional testing vs API testing
    • HW Test tools
    • Took 6 hours to do full test on hardware.
• PipeLine
  • Commit -> pull request
  • Automated Build / Unit Tests
  • Daily QA Build
• Next?
  • Configuration as code
  • Code Quality tools
  • Simulate more hardware
  • Increase analytics and reporting
  • Fully simulated test env for dev (so the devs don’t need the hardware)
  • Scale – From internal infrastructure to the cloud
  • Grow the team
• Lessons Learnt
  • Culture!
  • Collect Data
  • Get Executive Buy in
  • Change your tolls and processes if needed
  • Test automation is the key
    • Invest in HW
    • Silulate
    • Virtualise
  • Focus on good software design for Everything

Mirror, mirror, on the wall: testing Conway’s Law in open source communities – Lindsay Holmwood

• The map between the technical organisation and the technical structure.
• Easy to find who owns something, don’t have to keep two maps in your head
• Needs flexibility of the organisation structure in order to support flexibility in a technical design
• Conway’s “Law” really just adage
• Complexity frequently takes the form of hierarchy
• Organisations that mirror perform badly in rapidly changing and innovative enviroments

Metrics that Matter – Alison Polton-Simon (Thoughtworks)

• Metrics Mania – Lots of focus on it everywhere ( fitbits, google analytics, etc)
• How to help teams improve CD process
• Define CD
  • Software consistently in a deployable state
  • Get fast, automated feedback
  • Do push-button deployments
• Identifying metrics that mattered
  • Talked to people
  • Contextual observation
  • Rapid prototyping
  • Pilot offering
• 4 big metrics
  • Deploy ready builds
  • Cycle time
  • Mean time between failures
  • Mean time to recover
• Number of Deploy-ready builds
  • How many builds are ready for production?
  • Routine commits
  • Testing you can trust
  • Product + Development collaboration
• Cycle Time
  • Time it takes to go from a commit to a deploy
  • Efficient testing (test subset first, faster testing)
  • Appropriate parallelization (lots of build agents)
  • Optimise build resources
• Case Study
  • Monolithic Codebase
  • Hand-rolled build system
  • Unreliable environments ( tests and builds fail at random )
  • Validating a Pull Request can take 8 hours
  • Coupled code: isolated teams
  • Wide range of maturity in testing (some no test, some 95% coverage)
  • No understanding of the build system
  • Releases routinely delay (10 months!) or done “under the radar”
• Focus in case study
  • Reducing cycle time, increasing reliability
  • Extracted services from monolith
  • Pipelines configured as code
  • Build infrastructure provisioned as docker and ansible
  • Results:
    • Cycle time for one team 4-5h -> 1:23
    • Deploy ready builds 1 per 3-8 weeks -> weekly
• Mean time between failures
  • Quick feedback early on
  • Robust validation
  • Strong local builds
  • Should not be done by reducing number of releases
• Mean time to recover
  • How long back to green?
  • Monitoring of production
  • Automated rollback process
  • Informative logging
• Case Study 2
  • 1.27 million lines of code
  • High cyclomatic complexity
  • Tightly coupled
  • Long-running but frequently failing testing
  • Isolated teams
  • Pipeline run duration 10h -> 15m
  • MTTR Never -> 50 hours
  • Cycle time 18d -> 10d
  • Created a dashboard for the metrics
• Meaningless Metrics
  • The company will build whatever the CEO decides to measure
  • Lines of code produced
  • Number of Bugs resolved. – real life duplicates Dilbert
  • Developers Hours / Story Points
  • Problems
    • Lack of team buy-in
    • Easy to agme
    • Unintended consiquences
    • Measuring inputs, not impacts
• Make your own metrics
  • Map your path to production
  • Highlights pain points
  • collaborate
  • Experiment

Using Bots to Scale incident Management – Anthony Angell (Xero)

• Who we are
  • Single Team
  • Just a platform Operations team
• SRE team is formed
  • Ops teams plus performance Engineering team
• Incident Management
  • In Bad old days – 600 people on a single chat channel
  • Created Framework
  • what do incidents look like, post mortems, best practices,
  • How to make incident management easy for others?
• ChatOps (Based on Hubot)
  • Automated tour guide
  • Multiple integrations – anything with Rest API
  • Reducing time to restore
  • Flexability
• Release register – API hook to when changes are made
• Issue report form
  • Summary
  • URL
  • User-ids
  • how many users & location
  • when started
  • anyone working on it already
  • Anything else to add.
• Chat Bot for incident
  • Populates for an pushes to production channel, creates pagerduty alert
  • Creates new slack channel for incident
  • Can automatically update status page from chat and page senior managers
  • Can Create “status updates” which record things (eg “restarted server”), or “yammer updates” which get pushed to social media team
  • Creates a task list automaticly for the incident
  • Page people from within chat
  • At the end: Gives time incident lasted, archives channel
  • Post Mortum
• More integrations
  • Report card
  • Change tracking
  • Incident / Alert portal
• High Availability – dockerisation
• Caching
  • Pageduty
  • AWS
  • Datadog

DevSecOps – Anthony Rees

“When Anthrax and Public Enemy came together, It was like Developers and Operations coming together”

• Everybody is trying to get things out fast, sometimes we forget about security
• Structural efficiency and optimised flow
• Compliance putting roadblock in flow of pipeline
  • Even worse scanning in production after deployment
• Compliance guys using Excel, Security using Shell-scripts, Develops and Operations using Code
• Chef security compliance language – InSpec
  • Insert Sales stuff here
• ispec.io
• Lots of pre-written configs available

Immutable SQL Server Clusters – John Bowker (from Xero)

• Problem
  • Pet Based infrastructure
  • Not in cloud, weeks to deploy new server
  • Hard to update base infrastructure code
• 110 Prod Servers (2 regions).
• 1.9PB of Disk
• Octopus Deploy: SQL Schemas, Also server configs
• Half of team in NZ, Half in Denver
  • Data Engineers, Infrastructure Engineers, Team Lead, Product Owner
• Where we were – The Burning Platform
  • Changed mid-Migration from dedicated instances to dedicated Hosts in AWS
  • Big saving on software licensing
• Advantages
  • Already had Clustered HA
  • Existing automation
  • 6 day team, 15 hours/day due to multiple locations of team
• Migration had to have no downtime
  • Went with node swaps in cluster
• Split team. Half doing migration, half creating code/system for the node swaps
• We learnt
  • Dedicated hosts are cheap
  • Dedicated host automation not so good for Windows
  • Discovery service not so good.
  • Syncing data took up to 24h due to large dataset
  • Powershell debugging is hard (moving away from powershell a bit, but powershell has lots of SQL server stuff built in)
  • AWS services can timeout, allow for this.
• Things we Built
  • Lots Step Templates in Octopus Deploy
  • Metadata Store for SQL servers – Dynamite (Python, Labda, Flask, DynamoDB) – Hope to Open source
  • Lots of PowerShell Modules
• Node Swaps going forward
  • Working towards making this completely automated
  • New AMI -> Node swap onto that
  • Avoid upgrade in place or running on old version