Category: Tech

Owen Evans – DevOps is Dead, long live DevOps

Theory: Devops is role that never existed.

In the old days
– Shipping used to be hard and expensive, eg on physical media
– High cost of release
– but everybody else was the same.
– Lots of QA and red tape, no second chances

Then we got the Internet
– Speed became everything
– You just shipped enough

But Hardware still was a limiting factor
– Virtual machines
– IaaS
– Containers

This led to complacency
– Still had a physical server under it all

Birth of devops
– Software got faster but still had to have hardware under their somewhere
– Disparity between operations cadence and devs cadence
– things got better
– But we didn’t free ourselves from hardware
– Now everything is much more complex

Developers are now divorced from the platform
– Everything is abstracted
– It is leaky buckets all the way down

Solutions
– Education of developers as to what happens below the hood
– Stop reinventing the where
– Harmony is much more productive
– Lots of tools means that you don’t have enough expertise on each
– Reduce fiefdoms
– Push responsibility but not ownership (you own it but the devs makes some of the changes)
– Live with the code
– Pit of success, easy ways to fail that don’t break stuff (eg test environments, by default it will do the right thing)
– Be Happy. Everybody needs to be a bit devops and know a bit of everything.

Martina Iglesias – Automatic Discovery of Service metadata for systems at scale

Backend developer at Spotify

Spotify Scale
– 100m active users
– 800+ tech employees
– 120 teams
– Microservices architecture

Walk though Sample artist’s page
– each component ( playlist, play count, discgraphy) is a seperate service
– Aggregated to send result back to client

Hard to co-ordinate between services as scale grows
– 1000+ services
– Each need to use each others APIs
– Dev teams all around the world

Previous Solution
– Teams had docs in different places
– Some in Wiki, Readme, markdown, all different

Current Solution – System Z
– Centralise in one place, as automated as possible
– Internal application
– Web app, catalog of all systems and its parts
– Well integrated with Apollo service

Web Page for each service
– Various tabs
– Configuration (showing versions of build and uptimes)
– API – list of all endpoints for service, scheme, errors codes, etc (automatically populated)
– System tab – Overview on how service is connected to other services, dependencies (generated automatically)

Registration
– System Z gets information from Apollo and prod servers about each service that has been registered

Apollo
– Java libs for writing microservices
– Open source

Apollo-meta
– Metadata module
– Exposes endpoint with metadata for each service
– Exposes
– instance info – versions, uptime
– configuration – currently loaded config of the service
– endpoints –
– call information – monitors service and learns and returns what incoming and outgoing services the service actually does and to/from what other services.
– Automatically builds dependencies

Situation Now
– Quicker access to relevant information
– Automated boring stuff
– All in one place

Learnings
– Think about growth and scaling at the start of the project

Documentation generators
-Apollo
– Swagger.io
– ralm.org

Blog: labs.spotify.com
Jobs: spotify.com/jobs

Q: How to handle breaking APIs
A: We create new version of API endpoint and encourage people to move over.

Bridget Cowie – The story of a performance outage, and how we could have prevented it

– Works for Datacom
– Consultant in Application performance management team

Story from Start of 2015

– Friday night phone calls from your boss are never good.
– Dropped in application monitoring tools (Dynatrace) on Friday night, watch over weekend
– Prev team pretty sure problem is a memory leak but had not been able to find it (for two weeks)
– If somebody tells you they know what is wrong but can’t find it, give details or fix it then be suspicious

Book: Java Enterprise performance

– Monday prod load goes up and app starts crashing
– Told ops team but since crash wasn’t visable yet, was not believed. waited

Tech Stack
– Java App, Jboss on Linux
– Multiple JVMs
– Oracle DBs, Mulesoft ESB, ActiveMQ, HornetQ

Ah Ha moment
– Had a look at import process
– 2.3 million DB queries per half hour
– With max of 260 users, seems way more than what is needed
– Happens even when nobody is logged in

Tip: Typically 80% of all issues can be detected in dev or test if you look for them.

Where did this code come from?
– Process to import a csv into the database
– 1 call mule -> 12 calls to AMQ -> 12 calls to App -> 102 db queries
– Passes all the tests… But
– Still shows huge growth in queries as we go through layers
– DB queries grow bigger with each run

Tip: Know how your code behaves and track how this behavour changes with each code change (or even with no code change)

Q: Why Dynatrace?
A: Quick to deploy, useful info back in only a couple of hours

Ken Mugrage – What we’re learning from burnout and how DevOps culture can help

Originally in the Marines, environment where burnout not tolerated
Works for Thoughtworks – not a mental health professional

Devops could make this worse
Some clichéd places say: “Teach the devs puppet and fire all the Ops people”

Why should we address burnout?
– Google found psychological safety was the number 1 indicator of an effective team
– Not just a negative, people do better job when feeling good.

What is burnout
– The Truth about burnout – Maslach and Leiter
– The Dimensions of Burnout
– Exhaustion
– Cynicism
– Mismatch between work and the person
– Work overload
– Lack of control
– Insufficient reward
– Breakdown of communication

Work overload
– Various prioritisation methods
– More load sharing
– Less deploy marathons
– Some orgs see devops as a cost saving
– There is no such thing as a full stack engineer
– team has skills, not a person

Lack of Control
– Team is ultimately for the decissions
– Use the right technolgy and tools for the team
– This doesnt mean a “Devops team” contolling what others do

Insufficient Reward
– Actually not a great motivator

Breakdown in communication
– Walls between teams are bad
– Everybody involved with product should be on the same team
– 2 pizza team
– Pairs with different skill sets are common
– Swarming can be done when required ( one on keyboard, everybody else watching and talking and helping on big screen)
– Blameless retrospectives are held
– No “Devops team”, creating a silo is not a solution for silos

Absence of Fairness
– You build it, you run it
– Everybody is responsible for quality
– Everybody is measured in the same way
– example Expedia – *everything* deployed has A/B tesing
– everybody goes to release party

Conflicting Values
– In the broadest possible sense
– eg Company industry and values should match your own

Reminder: it is about you and how you fit in with the above

Pay attention to how you feel
– Increase your self awareness
– Maslach Burnout inventory
– Try not to focus on the negative.

Pay attention to work/life balance
– Ask for it, company might not know your needs
– If you can’t get it then quit

Talk to somebody
– Professional help is the best
– Trained to identify cause and effect
– can recommend treatment
– You’d call them if you broke your arm

Friends and family
– People who care, that you haven’t even meet
– Empathy is great , but you aren’t a professional
– Don’t guess cause and effect
– Don’t recommend treatment if not a professional

Q: Is it Gender specific for men (since IT is male dominated) ?
– The “absence of fairness” problem is huge for women in IT

Q: How to promote Psychological safety?
– Blameless post-mortems

 

Damian Brady – Just let me do my job

After working in govt, went to work for new company and hoped to get stuff done

But whole dev team was unhappy
– Random work assigned
– All deadlines missed
– Lots of waste of time meetings

But 2 years later
– Hitting all deadlines
– Useful meetings

What changes were made?

New boss, protect devs for MUD ( Meetings, uncertainty, distractions )

Meetings
– In board sense, 1-1, all hands, normal meetings
– People are averaging 7.5 hours/week in meetings
– On average 37% of meeting time is not relevant to person ( ~ $8,000 / year )
– Do meetings have goals and do they achieve those goals?
– 38% without goals
– only half of remaining meet those goals
– around 40% of meetings have and achieve goals
– Might not be wasted. Look at “What has changed as result of this meeting?”

Meetings fixes
– New Boss went to meetings for us (didn’t need everybody) as a representative
– Set a clear goal and agenda
– Avoid gimmicks
– don’t default to 30min or 1h

Distractions
– 60% of people interrupted 10 or more times per day
– Good to stay in a “flow state”
– 40% people say they are regularly focussed in their work. but all are sometimes
– 35% of time loss focus when interrupted
– Study shows people can take up to 23mins to get focus back after interruption
– $25,000/year wasting according to interruptions

Distraction Fixes
– Allowing headphones, rule not to interrupt people wearing headphones
– “Do not disturb” times
– Little Signs
– Had “the finger” so that you could tell somebody your were busy right now and would come back to them
– Let devs go to meeting rooms or cafes to hide from interruptions
– All “go dark” where email and chat turned off

Uncertainty
– 82% in survey were clear
– nearly 60% of people their top priority changes before they can finish it.
– Autonomy, mastery, purpose

Uncertainty Fixes
– Tried to let people get clear runs at work
– Helped people acknowledge the unexpected work, add to Sprint board
– Established a gate – Business person would have to go through the manager
– Make the requester responsible – made the requester decide what stuff didn’t get done by physically removing stuff from the sprint board to add their own

I’ve been playing with Prometheus monitoring lately. It is fairly new software that is getting popular. Prometheus works using a pull architecture. A central server connects to each thing you want to monitor every few seconds and grabs stats from it.

In the simplest case you run the node_exporter on each machine which gathers about 600-800 (!) metrics such as load, disk space and interface stats. This exporter listens on port 9100 and effectively works as an http server that responds to “GET /metrics HTTP/1.1” and spits several hundred lines of:

node_forks 7916
node_intr 3.8090539e+07
node_load1 0.47
node_load15 0.21
node_load5 0.31
node_memory_Active 6.23935488e+08

Other exporters listen on different ports and export stats for apache or mysql while more complicated ones will act as proxies for outgoing tests (via snmp, icmp, http). The full list of them is on the Prometheus website.

So my problem was that I wanted to check my virtual machine that is on Linode. The machine only has a public IP and I didn’t want to:

1. Allow random people to check my servers stats
2. Have to setup some sort of VPN.

So I decided that the best way was to just use put a user/password on the exporter.

However the node_exporter does not  implement authentication itself since the authors wanted the avoid maintaining lots of security code. So I decided to put it behind a reverse proxy using apache mod_proxy.

Step 1 – Install node_exporter

Node_exporter is a single binary that I started via an upstart script. As part of the upstart script I told it to listen on localhost port 19100 instead of port 9100 on all interfaces

# cat /etc/init/prometheus_node_exporter.conf
description "Prometheus Node Exporter"

start on startup

chdir /home/prometheus/

script
/home/prometheus/node_exporter -web.listen-address 127.0.0.1:19100
end script

Once I start the exporter a simple “curl 127.0.0.1:19100/metrics” makes sure it is working and returning data.

Step 2 – Add Apache proxy entry

First make sure apache is listening on port 9100 . On Ubuntu edit the /etc/apache2/ports.conf file and add the line:

Listen 9100

Next create a simple apache proxy without authentication (don’t forget to enable mod_proxy too):

# more /etc/apache2/sites-available/prometheus.conf 
<VirtualHost *:9100>
 ServerName prometheus

CustomLog /var/log/apache2/prometheus_access.log combined
 ErrorLog /var/log/apache2/prometheus_error.log

ProxyRequests Off
 <Proxy *>
Allow from all
 </Proxy>

ProxyErrorOverride On
 ProxyPass / http://127.0.0.1:19100/
 ProxyPassReverse / http://127.0.0.1:19100/

</VirtualHost>

This simply takes requests on port 9100 and forwards them to localhost port 19100 . Now reload apache and test via curl to port 9100. You can also use netstat to see what is listening on which ports:

Proto Recv-Q Send-Q Local Address   Foreign Address State  PID/Program name
tcp   0      0      127.0.0.1:19100 0.0.0.0:*       LISTEN 8416/node_exporter
tcp6  0      0      :::9100         :::*            LISTEN 8725/apache2

 

Step 3 – Get Prometheus working

I’ll assume at this point you have other servers working. What you need to do now is add the following entries for you server in you prometheus.yml file.

First add basic_auth into your scape config for “node” and then add your servers, eg:

- job_name: 'node'

  scrape_interval: 15s

  basic_auth: 
    username: prom
    password: mypassword

  static_configs:
    - targets: ['myserver.example.com:9100']
      labels: 
         group: 'servers'
         alias: 'myserver'

Now restart Prometheus and make sure it is working. You should see the following lines in your apache logs plus stats for the server should start appearing:

10.212.62.207 - - [31/Jul/2016:11:31:38 +0000] "GET /metrics HTTP/1.1" 200 11377 "-" "Go-http-client/1.1"
10.212.62.207 - - [31/Jul/2016:11:31:53 +0000] "GET /metrics HTTP/1.1" 200 11398 "-" "Go-http-client/1.1"
10.212.62.207 - - [31/Jul/2016:11:32:08 +0000] "GET /metrics HTTP/1.1" 200 11377 "-" "Go-http-client/1.1"

Notice that connections are 15 seconds apart, get http code 200 and are 11k in size. The Prometheus server is using Authentication but apache doesn’t need it yet.

Step 4 – Enable Authentication.

Now create an apache password file:

htpasswd -cb /home/prometheus/passwd prom mypassword

and update your apache entry to the followign to enable authentication:

# more /etc/apache2/sites-available/prometheus.conf
 <VirtualHost *:9100>
 ServerName prometheus

 CustomLog /var/log/apache2/prometheus_access.log combined
 ErrorLog /var/log/apache2/prometheus_error.log

 ProxyRequests Off
 <Proxy *>
 Order deny,allow
 Allow from all
 #
 AuthType Basic
 AuthName "Password Required"
 AuthBasicProvider file
 AuthUserFile "/home/prometheus/passwd"
 Require valid-user
 </Proxy>

 ProxyErrorOverride On
 ProxyPass / http://127.0.0.1:19100/
 ProxyPassReverse / http://127.0.0.1:19100/
 </VirtualHost>

After you reload apache you should see the following:

10.212.56.135 - prom [01/Aug/2016:04:42:08 +0000] "GET /metrics HTTP/1.1" 200 11394 "-" "Go-http-client/1.1"
10.212.56.135 - prom [01/Aug/2016:04:42:23 +0000] "GET /metrics HTTP/1.1" 200 11392 "-" "Go-http-client/1.1"
10.212.56.135 - prom [01/Aug/2016:04:42:38 +0000] "GET /metrics HTTP/1.1" 200 11391 "-" "Go-http-client/1.1"

Note that the “prom” in field 3 indicates that we are logging in for each connection. If you try to connect to the port without authentication you will get:

Unauthorized
This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.

That is pretty much it. Note that will need to add additional Virtualhost entries for more ports if you run other exporters on the server.

 

The Gathering

Chloe Swarbrick

• Whose responsibility is it to disrupt the system?
• Maybe try and engage with the system we have for a start before writing it off.
• You disrupt the system yourself or you hold the system accountable

Nick McFarlane

• He wrote a book
• Rock Stars are dicks to work with

So you want to Start a Business

• Hosted by Reuben and Justin (the accountant)
• Things you need to know in your first year of business
• How serious is the business, what sort of structure
  • If you are serious, you have to do things properly
  • Have you got paying customers yet
  • Could just be an idea or a hobby
• Sole Trader vs Incorporated company vs Trust vs Partnership
• Incorperated
  • Directors and Shareholders needed to be decided on
  • Can take just half an hour
• when to get a GST number?
  • If over $60k turnover a year
  • If you have lots of stuff you plan to claim back.
• Have an accounting System from Day 1 – Xero Pretty good
• Get an advisor or mentor that is not emotionally invested in your company
• If partnership then split up responsibilities so you can hold each other accountable for specific items
• If you are using Xero then your accountant should be using Xero directly not copying it into a different system.
• Remuneration
  • Should have a shareholders agreement
  • PAYE possibility from drawings or put 30% aside
  • Even if only a small hobby company you will need to declare income to IRD especially non-trivial level.
• What Level to start at Xero?
  • Probably from the start if the business is intended to be serious
  • A bit of pain to switch over later
• Don’t forget about ACC
• Remember you are due provisional tax once you get over the the $2500 for the previous year.
• Home Office expense claim – claim percentage of home rent, power etc
• Get in professionals to help

Diversity in Tech

• Diversity is important
  • Why is it important?
  • Does it mean the same for everyone
• Have people with different “ways of thinking” then we will have a diverse views then wider and better solutions
• example “Polish engineer could analysis a Polish specific character input error”
• example “Controlling a robot in Samoan”, robots are not just in english
• Stereotypes for some groups to specific jobs, eg “Indians in tech support”
• Example: All hires went though University of Auckland so had done the same courses etc
• How do you fix it when people innocently hire everyone from the same background? How do you break the pattern? No be the first different-hire represent everybody in that group?
• I didn’t want to be a trail-blazer
• Wow’ed out at “Women in tech” event, first time saw “majority of people are like me” in a bar.
• “If he is a white male and I’m going to hire him on the team that is already full of white men he better be exception”
• Worried about implication that “diversity” vs “Meritocracy” and that diverse candidates are not as good
• Usual over-representation of white-males in the discussion even in topics like this.
• Notion that somebody was only hired to represent diversity is very harmful especially for that person
• If you are hiring for a tech position then 90% of your candidates will be white-males, try place your diversity in getting more diverse group applying for the jobs not tilt in the actual hiring.
• Even in maker spaces where anyone is welcome, there are a lot fewer women. Blames mens mags having things unfinished, women’s mags everything is perfect so women don’t want to show off something that is unfinished.
• Need to make the workforce diverse now to match the younger people coming into it
• Need to cover “power income” people who are not exposed to tech
• Even a small number are role models for the future for the young people today
• Also need to address the problem of women dropping out of tech in the 30s and 40s. We can’t push girls into an “environment filled with acid”
• Example taking out “cocky arrogant males” from classes into “advanced stream” and the remaining class saw women graduating and staying in at a much higher rate.

Podcasting

• Paul Spain from Podcast New Zealand organising
• Easiest to listen to when doing manual stuff or in car or bus
• Need to avoid overload of commercials, eg interview people from the company about the topic of interest rather than about their product
• Big firms putting money into podcasting
• In the US 21% of the market are listening every single month. In NZ perhaps more like 5% since not a lot of awareness or local content
• Some radios shows are re-cutting and publishing them
• Not a good directory of NZ podcasts
• Advise people use proper equipment if possible if more than a once-off. Bad sound quality is very noticeable.
• One person: 5 part series on immigration and immigrants in NZ
• Making the charts is a big exposure
• Apples “new and noteworthy” list
• Domination by traditional personalities and existing broadcasters at present. But that only helps traction within New Zealand

 

 

At the Gather Conference again for about the 6th time. It is a 1-day tech-orientated unconference held in Auckland every year.

The day is split into seven streamed sessions each 40 minutes long (of about 8 parallel rooms of events that are each scheduled and run by attendees) plus and opening and a keynote session.

How to Steer your own career – Shirley Tricker

• Asked people hands up on their current job situation, FT vs PT, sinmgle v multiple jobs
• Alternatives to traditional careers of work. possible to craft your career
• Recommended Blog – Free Range Humans
• Job vs Career
  • Job – something you do for somebody else
  • Career – Uniqie to you, your life’s work
  • Career – What you do to make a contribution
• Predicted that a greater number of people will not stay with one (or even 2 or 3) employers through their career
• Success – defined by your goals, lifestyle wishes
• What are your strengths – Know how you are valuable, what you can offer people/employers, ways you can branch out
• Hard and Soft Skills (soft skills defined broadly, things outside a regular job description)
• Develop soft skills
  • List skills and review ways to develop and improve them
  • Look at people you admire and copy them
  • Look at job desctions
• Skills you might need for a portfilio career
  • Good at organising, marketing, networking
  • flexible, work alone, negotiation
  • Financial literacy (handle your accounts)
• Getting started
  • Start small ( don’t give up your day job overnight)
  • Get training via work or independently
  • Develop you strengths
  • Fix weaknesses
  • Small experiments
  • cheap and fast (start a blog)
  • Don’t have to start out as an expert, you can learn as you go
• Just because you are in control doesn’t make it easy
• Resources
  • Careers.govt.nz
  • Seth Goden
  • Tim Ferris
  • eg outsources her writing.
• Tools
  • Xero
  • WordPress
  • Canva for images
  • Meetup
  • Odesk and other freelance websites
• Feedback from Audience
  • Have somebody to report to, eg meet with friend/adviser monthly to chat and bounce stuff off
  • Cultivate Women’s mentoring group
  • This doesn’t seem to filter through to young people, they feel they have to pick a career at 18 and go to university to prep for that.
  • Give advice to people and this helps you define
  • Try and make the world a better place: enjoy the work you are doing, be happy and proud of the outcome of what you are doing and be happy that it is making the world a bit better
  • How to I “motivate myself” without a push from your employer?
    • Do something that you really want to do so you won’t need external motivation
    • Find someone who is doing something write and see what they did
    • Awesome for introverts
  • If you want to start a startup then work for one to see what it is like and learn skills
  • You don’t have to have a startup in your 20s, you can learn your skills first.
  • Sometimes you have to do a crappy job at the start to get onto the cool stuff later. You have to look at the goal or path sometimes

Books and Podcasts – Tanya Johnson

Stuff people recommend

• Intelligent disobedience – Ira
• Hamilton the revolution – based on the musical
• Never Split the difference – Chris Voss (ex hostage negotiator)
• The Three Body Problem – Lia CiXin – Sci Fi series
• Lucky Peach – Food and fiction
• Unlimited Memory
• The Black Swan and Fooled by Randomness
• The Setup (usesthis.com) website
• Tim Ferris Podcast
• Freakonomics Podcast
• Moonwalking with Einstein
• Clothes, Music, Boy – Viv Albertine
• TIP: Amazon Whispersync for Kindle App (audiobook across various platforms)
• TIP: Blinkist – 15 minute summaries of books
• An Intimate History of Humanity – Theodore Zenden
• How to Live – Sarah Bakewell
• TIP: Pocketcasts is a good podcast app for Android.
• Tested Podcast from Mythbusters people
• Trumpcast podcast from Slate
• A Fighting Chance – Elizabeth Warren
• The Choice – Og Mandino
• The Good life project Podcast
• The Ted Radio Hour Podcast (on 1.5 speed)
• This American Life
• How to be a Woman by Caitlin Moran
• The Hard thing about Hard things books
• Flashboys
• The Changelog Podcast – Interview people doing Open Source software
• The Art of Oppertunity Roseland Zander
• Red Rising Trilogy by Piers Brown
• On the Rag podcast by the Spinoff
• Hamish and Andy podcast
• Radiolab podcast
• Hardcore History podcast
• Car Talk podcast
• Ametora – Story of Japanese menswear since WW2
• .net rocks podcast
• How not to be wrong
• Savage Love Podcast
• Friday Night Comedy from the BBC (especially the News Quiz)
• Answer me this Podcast
• Back to work podcast
• Reply All podcast
• The Moth
• Serial
• American Blood
• The Productivity podcast
• Keeping it 1600
• Ruby Rogues Podcast
• Game Change – John Heilemann
• The Road less Travelled – M Scott Peck
• The Power of Now
• Snow Crash – Neil Stevensen

My Journey to becoming a Change Agent – Suki Xiao

• Start of 2015 was a policy adviser at Ministry
• Didn’t feel connected to job and people making policies for
• Outside of work was a Youthline counsellor
• Wanted to make a difference, organised some internal talks
• Wanted to make changes, got told had to be a manager to make changes (10 years away)
• Found out about R9 accelerator. Startup accelerator looking at Govt/Business interaction and pain points
• Get seconded to it
• First month was very hard.
• Speed of change was difficult, “Lean into the discomfort” – Team motto
• Be married to the problem
  • Specific problem was making sure enough seasonal workers, came up with solution but customers didn’t like it. Was not solving the actual problem customers had.
  • Team was married to the problem, not the married to the solution
• When went back to old job, found slower pace hard to adjust back
• Got offered a job back at the accelerator, coaching up to 7 teams.
  • Very hard work, lots of work, burnt out
  • 50% pay cut
  • Worked out wasn’t “Agile” herself
  • Started doing personal Kanban boards
  • Cut back number of teams coaching, higher quality
• Spring Board
  • Place can work at sustainable pace
  • Working at Nomad 8 as an independent Agile consultant
  • Work on separate companies but some support from colleges
• Find my place
  • Joined Xero as a Agile Team Facilitator
• Takeaways
  • Anybody can be a change agent
  • An environment that supports and empowers
  • Look for support
• Conversation on how you overcome the “Everest” big huge goal
  • Hard to get past the first step for some – speaker found she tended to do first think later. Others over-thought beforehand
  • It seems hard but think of the hard things you have done in your life and it is usually not as bad
  • Motivate yourself by having no money and having no choice
  • Point all the bad things out in the open, visualise them all and feel better cause they will rarely happen
  • Learn to recognise your bad patterns of thoughts
  • “The Way of Art” Steven Pressfield (skip the Angels chapter)
• Are places Serious about Agile instead of just placing lip-service?
  • Questioner was older and found places wanted younger Agile coaches
  • Companies had to completely change into organisation, eg replace project managers
  • eg CEO is still waterfall but people lower down are into Agile. Not enough management buy-in.
  • Speaker left on client that wasn’t serious about changing
• Went though an Agile process, made “Putting Agile into the Org” as the product
• Show customers what the value is
• Certification advice, all sorts of options. Nomad8 course is recomended