Simon Lyall's Blog

Jacob Ivester – Diagnose DevOps: The work behind the work

• Unhappy DevOps Family
  • Unsupport Software
  • Releases outside of primetime
  • etc
• Focus on Process as a common problem
  • Manage Change that Affects Multiple teams
  • Throughputs vs Outputs
• Repeatability
• Extensibility
• Visability
• Safety

Cameron Huysmans – Designing an Enterprise Secrets Management Service using HashiCorp Vault

• Australian based Bank
• Transition for last 30 years for a bank to a layered based security model (all the way down to the server in the datacentre)
• In 2017 moved to the cloud and infrastructure in the cloud
• What makes a bank – licensed to operate
  • Must demonstrate control of the process
  • Reports problems to regulator
  • Identifyable business Processes
  • All Humans
• If you use a pipeline there are no humans in the process. These machine process needs to conform to the same control
  • Archetecture naturally resistent to change. Change requires a complex process
  • ITIL
  • 2FA required for everything
  • Secrets everywhere
• Disruption
  • Dynamic Systems with constant updates
  • Immutable containers
  • Changes done via code
  • Live system changes
  • Code and automation drives things
  • Dynamic CMDB – High Levels of abstraction
  • But you still have a secrets problems
• Secrets Management
  • Not just a place to store passwords
  • But also a Chain of Trust
• If Pipelines make the change who owns it, who audits it?
• Vault becomes a bit of audit by saying who used something (person or process)
• Why another tool ?
• Created a pattered on how thing will be deployed. Got Security to okay it. Build it in a pipeline
• Vault placed in the highest security area
  • But less-secure areas needed to talk to it.
  • Lots of zones internally. Some in Cloud, DMZ
  • Some talk via API gateway to main vault
  • Had a Vault replica that had a copy of some secrets and could be used by those zones that were not allowed to to the secrets zone
• Learnings
  • This is hard, especially in the cloud
  • If Pipelines are doing the change, that must be kept secure. Attribution, notification and real-time analytics
  • Declarative manifests of change (code, scripts, tools) require more strict access controls
  • Avoid direct point-to-point connections

Cath Jones – The Myth of the Senior Engineer

• They won’t be able to hit the ground running on Day 1
  • Assume they know everything about how things work at your organisation that is organisation or industry-specific
  • If you don’t account for this you will see problems, stress, high turnover
• Example: Trail by Fire
  • You get shown the basic stuff and then given your first ticket
• How do you take organisation knowledge and empower people?
• Employee Socialisation
  • Helps mitigate problems and assumptions
  • Facilitates communication and networking
  • Allows people to begin contributing sooner
• Pre-Arrival Stage
  • Let people know what is expected
  • Let existing people kno who is thating and our expectations for them
  • Example: Automatic (wordpress)
    • Asked people in the final stages to complete some (paid) work.
    • Candiatites get better understanding of the company
• Preparing for Transition
  • Culture-shock
  • How are you like compared to where they came from?
  • The new role compared to their previous one?
  • Come from a place where they were an expert and had lots of domain-specific knowledge to being a newbie
• The Encounter Stage
  • Mentoring, Communication, Technical onboarding
  • Example: Cohorts of new hires
  • Mentoring: Proven way to socialise Senior engineers. Can be Labour intensive but helps when documentation lacking
  • Share Mentor-ship responsibilities: eg Technical and Organisational mentor seperate
  • Communication: Expectations that company places, how privledged and how transparent?
  • Authenticity: Can people be themselves. Reduces stress
• Technical onboarding: Needs to take time and do it properly. Allow new people to contribute back to it and make it better.
  • Pick out easy wins or low-hanging fruit so peopel can contribute sooner
  • Have Style Guides and good docs
• MetaMorphosis
  • Senior Engineers are fully Contributing

Katie McLaughlin – Being kind to 3am you

• Saw a previous version of the Talk at Linux.conf.au 2019. See my notes from there.

Gleidson Nascimento – Packaging OpenShift Origin Kubernetes Distribution (OKD)

• Centos SIG
• Based on latest upstream

Joshua King – Don’t Reinvent the Wheel, Just Realign It

• Project: Let notifications work for powershell users
• Then he found the UWP community toolkit
• Which had notifications built-in
• These days looks around first, asks for APIs rather than scraping
• Look around for open-source tools and give back
• Sometimes your implimentation might be fun or even better than the original

Srdan Dukic – Implicit trust agreement in Learning Organizations

• Sysadmin shell -> ansible -> APIs -> automate everything
• Programmers coded themselves out of a job
• Followup instructions or achieve results?
• A bit of both – tension between the two
• Money today or Money tomorrow?
• Employee – Expected to make things better
• Employer – Support things getting better, not fire people when they automate themselves out of a job

Julie Gunderson – You Can’t Buy DevOps

• Lots of companies talking about DevOps are trying to sell you a solution
• What doesn’t makes you a devops company
  • Be in the Cloud
  • Have a DevOps team
  • Get rid of the Ops Team
  • A checklist you can tick off
  • Easy
• Westrum 3 Cultures Model
• We want the generative model
• Keeping information flowing between teams is prerequisite for high performance teams
• Psychological Safety to make decisions. Lets employees focus on problems and getting work done rather than politics
• Practices
  • Configuration management
  • CICD Pipelines
  • Work in small batches
  • Test every commit and everything else (look at Chaos engineering)
• Tools
  • Let the teams who are using the tools decide on what tools they will use
  • XebiaLabs Periodic table of DevOps tools
• Getting there
  • Start with one team and a POC

Allen Geer, Michael Harrod – Kiwi Ingenuity – Kiwi’s can Overcome Tough Problems In DevOps

• Contrast – US vs NZ
  • In the US companies are bigger, lots more people, lots more money to throw at problems.
  • Contrast with Arial Topdressing pioneered in NZ using surplus WW2 aircraft
  • Since the problems are up to 100x bigger in the US the tools are designed for that scale. ROI might not not be there for smaller companies.
  • Dealing with Scale
    • Avoid “Shinny new thing” syndrom, plan for keeping things for at least 5 years.
    • Ramp up slowly with the tool, push it into other areas.
    • Avoid Single Person Silo.
    • Bring up some Kiwi Inginuity (Look at Open source, Use the Free Tiers or Cheap Tiers).
    • Out-Innovate the US companies rather than trying to out-scale
  • Infrastructure: Monetization of Toil
    • Spending time and money on stuff you can automate
    • Lots of manual creating of infrastructure, servers, firewalls.
    • Lack of incentive for providers who charge for changes to automate stuff
    • Other Providers will automate (especially overseas ones that will come into NZ)
    • People take risks (eg no DR) in order to save money.
    • Innovator’s Dilemma
  • Solutions
    • More vocal customers
    • Providers should provider a platform, lots more self-service. Ahnd-holding for the hard stuff not the day-to-day
    • Charge for outcomes not person-hours
    • Begin Small
    • It’s an experiment – Freedom to Fail
  • Inattentive Customer Service
    • Overseas companies have a lot more forums, helpdesks, quick responses.
    • “Kiwis reluctant to make a fuss” , Companies not used to people making a fuss
    • Apply “American Ingenuity” – Striving focus to increase customer satisfaction.
    • Build a healthy community (eg online forums) around your service.
    • Gather insights from customers
    • Bezos – “When a customer contacts us, we see this as a defect” . Focus on the source of problems
  • Evolving Kiwi Workforce
    • NZ has older and aging workforce. 2nd oldest in the OECD
    • Slightly Fewer peoples with degrees
    • 11% of workforce 65+ by 2038
  • Learning in the workplace
    • Leverage senior Knowledge
    • Telco – Older customers didn’t want to approach young workers in mall. Brought in retired engineers to work in stores.
    • Mentoring and reverse-mentoring. Mentor learns insights from mentoree too (eg about younger people’s habits)
  • Introducing people to DevOps
    • Kiwi DevOps models

Craig Box – Teaching Old Servers New Tricks: extending the service mesh outside the cluster

• Service Mesh
  • Managing a service is hard
  • metrics, monitoring, logging, traceing
  • AAA encryption, certs
  • load balancing, routing, network policy
  • quota
  • Failure handling, fault inject
• Microservices
  • Not just for hipsters
  • Works best at scale. Lots of devs
• Now introduce a network in between everything. Lots of hard dtuff, distributed systems are hard
• Leaky abstractions
  • Have to build stuff into microservice to deal with problems of the network
  • In multiple libraries and languages
  • Can we fix it?
• Sidecar Pattern
  • The sidecar does all the hard stuff instead of making the microservice itself do it.
  • Talks TCP. Able to work with all languages
• Proxies as sidecars
  • SPOF
  • Sidecar is attached to each MS
• Flexability and Power
  • Single place where we can do everything
  • Traffic going in: TLS termination, metrics, quota
  • Traffic out of workloads: Authentication, TLS connections
• Istio
  • Open platform
  • Not always microservices
  • Uniform observability
  • Operational Agility
  • Policy driven Security
• How istio works
  • Proxies + control plane
  • Pilot in control plane pushes config to proxies, keeps track of them, looks up stuff in k8s cluster
  • Mixer – policy check and telemetry
  • Citidel – cert authority to proxies
  • Control plane has to run on k8s
  • Proxies run using envoy
  • Zipkin built in
  • All done automatically for kubernetes environments ( admission controller adds sidecar )
• Adding a VM to a service mesh
  • Enable the mesh expansion, connect the networks
  • Add the gateway IP to the VM
  • Get a cert and copy to the VM
  • Install proxy and node agent
  • Traffic from cluster -> VM .
    • Add the service to DNS in the cluster,
    • Create a ServiceEntry on the cluster
  • Traffic from VM -> Cluster service
    • Add Service and IP to /etc/host on the VM
• Sample Application – Hipster Shop
  • productcatalogservice is outside of kubernetes
  • headless service in kubernetes
  • manually created service entry in k8s
  • Experimental istio commands to simplify process to single command

Brooke Treadgold – Back to Basics

• Transformation Lead ANZ Bank
• Not originally from a Tech background
• Tech has a lot of buzzwords and acronyms that make it an exclusive club. Improvements relay on people from other parts of the business that aren’t in that club
• These people have to care about it and understand it.
• Had to use terms that everybody in the business understood and related to.
• Case for change – What top orgs do:
  • 208 times more frequent deployments
  • 2604 times faster to recover from incidents
  • 7 times lower change failure rate
• What you need
  • High Priority -> Access to people to do the work
  • Needed tangible goal (weekly releases) to get people to focus (and pay)
• Making change a reality
  • Risk Management
    • You can just stop doing the reports
    • You need to gain their trust in order to get influence
    • Have to take them along the way with the changes
  • Empathy
  • Influence
• History at ANZ
  • First pipeline replace just one document
    • Explained to change managment team how the pipeline could replace the traditional plan
  • Rethink of Change Plan and Outcome Reports
    • Other teams needed these for confidence in the change
    • Found out what people actually cared about, found better ways to provide that information (confidence) it an automated way
  • Security Assessment
    • Traditionally required a big document filled in and signed off
    • Found that this was only required for “Significant” changes
    • Got a definition of what significant means so didn’t need to do this.
  • High Risk Change Records
    • Lots of paperwork for High Risk changes
    • Decided that these are not high risk changes so lots less work
    • Templated them so a lot easier to do

Charles Korn – Dockerised local build and testing environments made easy

• Go Script – Single script that a consistence place in all you repos that does the basic function. install, help, run, deploy
• batect – tool he wrote
  • dockerized dev environment plus a Go Script
• Dev environment
  • Build env: code to an artifact
  • Testing Environments. Fake stuff, lots of different levels
• Build Environment
  • Container with the build tools. Mount our code directory into this
  • Isolation brings consistency and repeatability. No more “works on my machine”
  • Clean container every single time we run a build
  • CI agents just need docker since teams will provide the container
  • Ease of Onboarding. Just get git and docker installed
  • Ease of change. Environment and tasks defined in yaml and versioned like everything else. New version downloaded. Kept in sync with actual code
• Test Environments
  • You can run local tests
  • Consistently runs test on CI
  • Have to launch multiple containers for more complex tests, using built in docker definitions and health checks and networking
• Path to Production
  • If deploying docker then can use same image
  • But works with stuff that isn’t deployed as docker too
• What about docker compose?
  • Better performance
  • Model – tasks are a first class citizen – Doesn’t feel like you are fighting the too.
  • Better UI and developer experience. Updates managed automatically
  • Cleans up better after each run
  • It just works. Works with proxies better. Works with file permissions better.
• How to get started?
  • start small, work incrementally
  • Start with the build enviroment
  • With the Test env work though one piece at a time.
  • Reuse components
  • Take advantage for other people’s images. Lots of mocks for cloud services.
  • Docker has library of health check scripts
  • Bunch of sample scripts for batect
• github.com/charleskorn/batect