Linux.conf.au 2020 – Tuesday – Session 3 – Container Miniconf

Unsafe Defaults: Deploying Kubernetes Safer(ish) – James

Overview of Kubernetes

  • A compromised container is very close to being a compromised host
  • While you shouldn’t curl|bash the attacker can do it to get the latest exploits.

Three Quick things for some easy wins

  • The Kubernetes API is completely open from localhost. This is no longer required but old clusters and some upgraded clusters may still have it.
  • Put a Valid certificate on the cluster or at least one you can keep track of.
  • Get rid of unauthenticated user roles as much as possible.
  • Check you don’t still have “forever tokens”
  • A Good idea not to give service tokens to most pods.
    automountServiceAccountToken: false

PodsecurityPolicy

  • Keep an eye on
  • New
  • You need good RBAC
  • Have a look at k-rail

etcd

  • Can turn on authentication
  • Can turn on TLS between peers and clients
  • Can encrypt on disk
  • Can restrict it with a firewall

Every Image Has A Purpose by Allen Shone

Docker Images

  • What are they anyway
  • A base definition to prepare a filesystem for execution as a container
  • Caching mechanism
  • Reproduceable
  • Great way to share runtime circumstances
  • A comprehensive environment structure

Layers

  • image is a series of layers
  • Minimizing layers makes things better
  • Structure the image build process to get the best set of images

Basic Uses

  • Use the most appropriate image
  • A small fix can add up

Images in Production / Customers facing envs

  • When deploying containers, be precise as possible.
  • The image should be ready to go without further work
  • Keep image and small and simple as possible
  • “FROM: golang:alpine” in testing
  • “FROM: scratch” in production
  • Two images but they serve different purposes

Development

  • Possible to use the same image as previously
  • Bring in some extra debug tools etc, mocks for other services

Trimming the final image to be very specific

  • Start with the production image and add extra layers of stuff

Deployed Considerations

  • Some things only come into consideration once they are deployed
  • Instead of creating a big general container, create two containers in a pod that share a file system
  • Configuration should be injeted, as an env-specific setup
  • Images should be agnostic

Extras

  • Look at using the .dockerignore file
  • Use image scannign tools ( Diive and Clair)
  • A little preparation up front can prevent a lot of headache later
Share

Linux.conf.au 2020 – Tuesday – Session 2 – Security, Identity, Privacy Miniconf

Privacy and Transparency in the VPN industry by Ruben Rubio Rey

We are at an “Oh Noe!” Moment in the VPN Industry

VPN Advantages

  • Protect your privacy
  • Bypass Geo-Restrictions
  • Beat Censorship
  • Save money on Hotels and Flights
  • Download torrents anonymously
  • Bypass ISP speed regulations
  • Secures Public WIFI

What Can be intercepted?
– Without Encryption: Any Data
– With Encryption: IP and Port

But HTTPS only works of client and server configured correctly
Client: Rough root certificate
Servers: CORS, insecure SSL version

Protect Your Privacy

  • Many Countries Systematicly collecting data about citizens
  • ISP collect data, must keep for two years and accessabil to agencies
  • USA ISP’s can sell information
  • Others Countries tried to put in MITM Certs

So Private companies have incentives to protect my data?

The Reality of Private VPN providers

  • Several examples of collecting Data
  • Several examples of them releasing data to agencies
  • Random security and implementation problems
  • Exaggerations in sales pitches
  • Installs Rouge Roots Cert on user machine

Conflict of Interest, what is a business model of the providers?

Stats

  • 59% of Free VPNs in play store had hidden Chinese ownership
  • 86% had privacy policy flaw
  • 85% asked for excessive permissions

Are VPN Companies Needed?

People with non-technical skills need an option

How to Improve the VPN Market?

  • Privacy and Transparency go hand and hand
  • Open Source Provides Transparency
  • End to End open source VPN Company
  • theVPNcompany.com.au

Install you own VPN

Algo and Streisand

Create your own VPN Company using the base for “The VPN Company”

https://thevpncompany.com.au/

Authentication Afterlife: the dark side of making lost password recovery harder by Ewen McNeill

Twitter Account “badthingsdaily” . Fictional Scenarios that might happen to security people. Inspired this talk.

Scenario 1

  • A Big fire took out your main computer
  • You done have the computer and you don’t know all your passwords

Recovery Traditional

  • You get email somewhere else. On your phone
  • Click on Forgot my password
  • Repeat until all accoutns recoveryed

Scenario 2

  • You need to login to your account on a new device
  • All account secured with 2FA
  • Your 2FA isn’t working

Recovery

  • Recovery Tokens
  • Alternative 2FA Solution

Scenario 3

  • Your bad was stolen
  • It had computer, phone and 2FA
  • Can bad guy impersonate you?
  • Can you recovery faster than the other guy (or at all?)

Recovery

  • Does you 2FA pop up on your lock screen?
  • So anybody with your computer is able to get this?
  • Race to reset passwords and invalidate your login tokens
  • Maybe you remember your passwords but not you 2FA
  • Recovery questions “Mother’s maiden name”
  • Can be easy to discover, but if it is something random then you have to be able to find it (ie on the password store you just lost)

Multiple alternate authentication methods

  • Primary you use every day
  • One or more backups

If resetting your password every time is easier than remembering your password people will do that.

Attackers will use the easiest authentication method. Eg Contacting the Helpdesk or going into a bank branch office.

But if recovery is too hard you can end up losing access to your account permanently

Recommend: GitHub’s 2FA recovery guide

Scenario 4

You startups founder has left. He has wipped out all his computer. Now your Cloudprovider is threatening to lock you out unless you authenticate using 2FA

  • Hopefully in the password store
  • Or perhaps they no longer work
  • Contact Helpdesk, Account Manager, Lawyer, Social Media (usually the bigger you are and the more you pay the better you chance)
  • Sore everything centrally. How do you audit that? , regularly?

Scenario 5

A relative dies. You first step is to login to all their accounts work out what should be kept.

This will take months not years. Sometimes you will only find out the account exists when they email you that your account is about to expire.

Personal Observations

  • You will not have access to their cellphone
  • or probably not past the lock screen
  • Anything they told you that was obvious you will forget
  • You will not have access to the password store
  • You may have access to saved passwords in browser
  • Maybe you need to optimise for family can access stuff not complete lockdown.
  • Physical notebook with passwords
  • Consider in advance how you will recover if your 2FA device breaks
  • How will you convince a helpdesk person that you are you?

Personal Mitigations

  • Kawaiicon 2019 ” How can I help you” Talk by Laura Bell

You Shall Not Pass by Peter Burnett

Moodle is an open souce Learning Management System.

  • Legacy System
  • First developed in 1997
  • Open Sourced in 2001
  • New Code is good quality, older stuff not as much

Efforts to improve password policy

  • Password policy was a bit antiquated
  • Best policies come from NIST, 2018 version is good.
  • Don’t force a pattern, Check for compromised passwords, Check for dictionary based and identifying passwords
  • Look at the “Have I been Pwned” API – takes first 5 characters of the sha of the password.
  • Dictionary checks – Top 10,000 English words might be enough
  • Indentifying information – Birthdays, names, cities are things to watch for. Name of the company.

Released as an open source plugin for Moodle

A look at the Authentication Flow

  • Natively supported LDAP etc.
  • Lots of extra plugins impliment other methods
  • Had to put MFA in when people using plugins. Difficult to mix
  • Added extra hook on “account related” actions, they would check for MFA etc.
  • Required a bit of work to get merged in.

Implementing MFA

  • MFA is a superset of 2FA implimentations
  • Had to do extensible platform
  • Traditional: TOTP, Email
  • Non-Traditional: IP verification, Authentication type (might already have MFA)
  • Design considerations – Keep secure but impact people as little as possible.
  • Different users: Not required, Optional, Forced Upon . So built in the ability for a range of use across platform.
  • Learnings
    • Anything can be used as a factor
    • delicate balance between secure and usable
    • When designing, paranoid is the right mindset
    • Give the least information possible to allow a legit user to authenticate
    • What can the attacker do if this factor is compromised?

Final Thoughts

  • Long way to go
  • Security is a shifting goalpost
  • Keep on top of new developments
Share

Linux.conf.au 2020 – Tuesday – Session 1 – Security, Identity, Privacy Miniconf

Facebook, Dynamite, Uber, Bombs, and You – Lana Brindley

  • Herman Hollerith
    • Created the punch card, introduced for the 1890 US Census
    • Hollerith leased companies to other people
  • Hollerith machines and infrastructure used by many Census in Europe.
    • Countries with better census infrastructure using Hollerith machines tended to use have higher deather rate in The Holocaust
  • Alfred Nobel
    • Invented Dynamite and ran weapons company
  • Otto Hahn
    • Invented Nuclear Fission
  • Eugenics
    • 33 US states have sterilization programmes in place
    • 65,000 Americans sterilized as part of programmes
    • WHO was created as a result.
  • Thalidomide
    • Over-the-counter morning sickness treatment
    • Caused birth defeats
    • FDA strengthened

Unintended consequences of technology, result was stronger regulation

Volkswagen emission and Uber created Greyball
– Volkswagen engineers went to jail, Uber engineers didn’t

Here are some IT innovations that didn’t lead to real change

  • Medical Devices
    • Therac-25 was a 1980s machine used for treating cancer with radiation
    • Control software had race condition that gave people huge radiation overloads
  • Drive by Wire for Cars
    • Luxus ES350 sudden acceleration
    • Toyota replaced floor mats, not software
    • Car accelerator stuck at full speed and brakes not working
    • No single cause ever identified
  • Deep Fake Videos
  • Killer Robots
    • South Korean Universities came under pressure to stop research, said they had stopped but not confirmed.
  • Chinese Surveillance
    • Checkpoints all though the city, average citizen goes though them many times per day and have phoned scanned, other checks.
    • Cameras with facial recognition everywhere
  • Western Surveillance – Palantir and other companies installing elsewhere
  • Boeing Software – 373 Max

Bad technology should have consequences and until it does people have to avoid things themselves as much as possible and put pressure on governments and companies

The Internet: Protecting Our Democratic Lifeline by Brett Sheffield

Lost of ways technology can protect us (Tor etc) and at the same time plenty of ways technology works against our prevacy.

The UN Declaration of Human Rights
Australia is the only major country without a bill of rights.

Ways to contribute
– They Work for you type websites
– Protesting
– Whistleblowers

Democracy Under Threat
– Governments blocking the Internet
– Netblocks.org
– Police harrass journalists (AFC raids ABC in Aus)
– Censorship

Large Companies
– Gather huge amounts of information
– Aim for personalisation and monotisation
– Leads to centralisation

Rebuilding the Internet with Multicast
– Scalable
– Happens at the network layer
– Needs to be enabled on all routers in each hop
– Currently off by default

Libracast
– Aims to get multicast in the hands of developers
– Tunnels though non-multicast enabled devices
– Messaging Library
– Transitional tunneling
– Improved routing protocol
– Try to enable in other FOSS projects
– Ensure new standards ( WebRTC, QUIC) support multicast



Share

Linux.conf.au 2020 – Tuesday – KeyNote: Sean Brady

Keynote: Drop Your Tools – Does Expertise have a Dark Side? by Dr Sean Brady

Harford Convention Center

Engineers ignored warnings of problems, kept saying calculations were good. Structure collasped under light snow load

People are involved with engineering, therefore it is a people problem

What it possessing expertise has a dark side? Danger isn’t ignnorance it is the illusion of knowledge.

Mann Gulch fire

Why did the firefighters not drop their tools?
Why did they not get in the Escape Fire?

Priming – You get information that primes you to think a certain way.

What if Expertise priming somebody?
– Baseball experts primed to go down the wrong path, couldn’t even stop when explicitly told about the trick.

Firefighters explicitly trained that they are faster runners with tools.

Creative Desperation – Mentally drop your existing tools.



Share

Donations 2019

Each year I do the majority of my Charity donations in early December (just after my birthday) spread over a few days (so as not to get my credit card suspended). I’m a little late this year due to a new credit card and other stuff distracting me.

I also blog about it to hopefully inspire others. See: 2018, 2017, 2016, 2015

All amounts this year are in $US unless otherwise stated

My main donations was to Givewell (to allocate to projects as they prioritize). Once again I’m happy that Givewell make efficient use of money donated.

I donated $50 each to groups providing infrastructure and advocacy. Wikipedia only got $NZ 50 since they converted to my local currency and I didn’t notice until afterwards

Some Software Projects. Software in the Public Interest provides admin support for many Open Source projects. Mozilla does the Firefox Browser and other stuff. Syncthing is an Open Source Project that works like Dropbox

Finally I’m still listening to Corey Olsen’s Exploring the Lord of the Rings series (3 years in and about 20% of the way though) plus his other material

Share

DevOpsDays NZ 2019 – Day 2 – Session 3

Everett Toews – Is GitOps worthy of the [BuzzWord]Ops moniker?

  • Usual Git workflow
  • But it takes some action
  • Applying desired state from Git
  • Example: Infrastructure as code
    • DNS
    • Onboarding and offboarding
  • Git is now a SPOF
  • Change Management Dept is now a barrier
  • Integrate with ITSM
  • Benefits: Self-service, Compiience

Joel Wirāmu Pauling – Why Bare Metal still maters

  • Cloud Native Dev doesn’t exist as a closer system
  • IoT is all hardware
  • AI/ML is using special hardware
  • Networks is all hardware offloads
  • FPGAs and ASICS need more standard open way to access
  • You’ll always have weird stuffs on your network
  • Virtualization has abstracted away the real
  • We care able vendor lockin with cloud APIs and Aus electricity isn’t all that green

Steven Ensslen – Do you have a data quality problem?

  • What is data ops and why do we want it?
  • People think they have a data quality problem but they don’t actually measure it to see how bad.
  • Causes all sorts of problems.
  • 3 Easy steps to fix data quaility
  • 1 – Document data charactersistics and train people to know them
  • 2 – Monitor data as if it is infrastructure
    • Test data like it is code
  • 3 – Professionalize your support of data professionals
    • Bring in the spreadsheet experts
    • Support reporting and analytics people too

Mandi Buswell – What are Kubernetes Operators and Why do I care

  • Like an App Store on your kubernetes cluster
  • Like a like Kubernetes robot doing that hard work for you. Lifecycle management
  • Operators run as microservices on the kubernetes cluster
  • operatorhub.io
  • Work on any kubernetes cluster
  • You can even write your own

Laura Bell – Securing the systems of the future

  • Fear and Lothing
    • It is an old problem because “People are Jerks”
  • All organization try either Fight, Flight, Freeze
  • Trying to protect: Confidentiality, Integrity, Availbality
  • Protect, Detect, Respond
  • Monolith
    • A big wall around
    • Layered defense is better but not the final solution
    • Defensive software architecture is not just prevention
    • Castles had lots of layers of defenses. Some prevention, Some Detection, Some response
  • MIcroservices
    • Look at something in the middle of a star and erase it
    • Push malicious code into deployment pipelines
  • Avoid scar tissue, stuff put in just to avoid specific previous problems. Make you feel safe but without any real evidence.
  • Fearless security patterns and approaches
  • Technology is changing but the basics are still the same
  • Lots of techniques in computer security.
  • Prevention and Detection are interchangeable
  • Batman vs Meercat model
  • Be Aware and challenge your own bubble
  • Supply Chains are vulnerable: Integrations, dependencies, Data Sources
  • Determinate threat vs Dynamic Threat
    • Can’t predicts which steps in which order are going to get the result
    • Comprimise the data then the engine will return bad results
  • Plug for opensecurity.nz

Share

DevOpsDays NZ 2019 – Day 2 – Session 2

Jacob Ivester – Diagnose DevOps: The work behind the work

  • Unhappy DevOps Family
    • Unsupport Software
    • Releases outside of primetime
    • etc
  • Focus on Process as a common problem
    • Manage Change that Affects Multiple teams
    • Throughputs vs Outputs
  • Repeatability
  • Extensibility
  • Visability
  • Safety

Cameron Huysmans – Designing an Enterprise Secrets Management Service using HashiCorp Vault

  • Australian based Bank
  • Transition for last 30 years for a bank to a layered based security model (all the way down to the server in the datacentre)
  • In 2017 moved to the cloud and infrastructure in the cloud
  • What makes a bank – licensed to operate
    • Must demonstrate control of the process
    • Reports problems to regulator
    • Identifyable business Processes
    • All Humans
  • If you use a pipeline there are no humans in the process. These machine process needs to conform to the same control
    • Archetecture naturally resistent to change. Change requires a complex process
    • ITIL
    • 2FA required for everything
    • Secrets everywhere
  • Disruption
    • Dynamic Systems with constant updates
    • Immutable containers
    • Changes done via code
    • Live system changes
    • Code and automation drives things
    • Dynamic CMDB – High Levels of abstraction
    • But you still have a secrets problems
  • Secrets Management
    • Not just a place to store passwords
    • But also a Chain of Trust
  • If Pipelines make the change who owns it, who audits it?
  • Vault becomes a bit of audit by saying who used something (person or process)
  • Why another tool ?
  • Created a pattered on how thing will be deployed. Got Security to okay it. Build it in a pipeline
  • Vault placed in the highest security area
    • But less-secure areas needed to talk to it.
    • Lots of zones internally. Some in Cloud, DMZ
    • Some talk via API gateway to main vault
    • Had a Vault replica that had a copy of some secrets and could be used by those zones that were not allowed to to the secrets zone
  • Learnings
    • This is hard, especially in the cloud
    • If Pipelines are doing the change, that must be kept secure. Attribution, notification and real-time analytics
    • Declarative manifests of change (code, scripts, tools) require more strict access controls
    • Avoid direct point-to-point connections

Share

DevOpsDays NZ 2019 – Day 2 – Session 1

Cath Jones – The Myth of the Senior Engineer

  • They won’t be able to hit the ground running on Day 1
    • Assume they know everything about how things work at your organisation that is organisation or industry-specific
    • If you don’t account for this you will see problems, stress, high turnover
  • Example: Trail by Fire
    • You get shown the basic stuff and then given your first ticket
  • How do you take organisation knowledge and empower people?
  • Employee Socialisation
    • Helps mitigate problems and assumptions
    • Facilitates communication and networking
    • Allows people to begin contributing sooner
  • Pre-Arrival Stage
    • Let people know what is expected
    • Let existing people kno who is thating and our expectations for them
    • Example: Automatic (wordpress)
      • Asked people in the final stages to complete some (paid) work.
      • Candiatites get better understanding of the company
  • Preparing for Transition
    • Culture-shock
    • How are you like compared to where they came from?
    • The new role compared to their previous one?
    • Come from a place where they were an expert and had lots of domain-specific knowledge to being a newbie
  • The Encounter Stage
    • Mentoring, Communication, Technical onboarding
    • Example: Cohorts of new hires
    • Mentoring: Proven way to socialise Senior engineers. Can be Labour intensive but helps when documentation lacking
    • Share Mentor-ship responsibilities: eg Technical and Organisational mentor seperate
    • Communication: Expectations that company places, how privledged and how transparent?
    • Authenticity: Can people be themselves. Reduces stress
  • Technical onboarding: Needs to take time and do it properly. Allow new people to contribute back to it and make it better.
    • Pick out easy wins or low-hanging fruit so peopel can contribute sooner
    • Have Style Guides and good docs
  • MetaMorphosis
    • Senior Engineers are fully Contributing

Katie McLaughlin – Being kind to 3am you

Share

DevOpsDays NZ 2019 – Day 1 – Session 3

Gleidson Nascimento – Packaging OpenShift Origin Kubernetes Distribution (OKD)

  • Centos SIG
  • Based on latest upstream

Joshua King – Don’t Reinvent the Wheel, Just Realign It

  • Project: Let notifications work for powershell users
  • Then he found the UWP community toolkit
  • Which had notifications built-in
  • These days looks around first, asks for APIs rather than scraping
  • Look around for open-source tools and give back
  • Sometimes your implimentation might be fun or even better than the original

Srdan Dukic – Implicit trust agreement in Learning Organizations

  • Sysadmin shell -> ansible -> APIs -> automate everything
  • Programmers coded themselves out of a job
  • Followup instructions or achieve results?
  • A bit of both – tension between the two
  • Money today or Money tomorrow?
  • Employee – Expected to make things better
  • Employer – Support things getting better, not fire people when they automate themselves out of a job

Julie Gunderson – You Can’t Buy DevOps

  • Lots of companies talking about DevOps are trying to sell you a solution
  • What doesn’t makes you a devops company
    • Be in the Cloud
    • Have a DevOps team
    • Get rid of the Ops Team
    • A checklist you can tick off
    • Easy
  • Westrum 3 Cultures Model
  • We want the generative model
  • Keeping information flowing between teams is prerequisite for high performance teams
  • Psychological Safety to make decisions. Lets employees focus on problems and getting work done rather than politics
  • Practices
    • Configuration management
    • CICD Pipelines
    • Work in small batches
    • Test every commit and everything else (look at Chaos engineering)
  • Tools
    • Let the teams who are using the tools decide on what tools they will use
    • XebiaLabs Periodic table of DevOps tools
  • Getting there
    • Start with one team and a POC
Share