Category: Linux.conf.au

Lightning Talk

• Usability Fails
• Etching
• Diverse Events
• Kids Space – fairly unstructured and self organising
• Opening up LandSat imagery – NBAR-T available on NCI
• Project Nacho – HTML -> VPN/RDP gateway . Apache Guacomle
• Vocaloids
• Blockchain
• Using j2 to create C++ code
• Memory model code update
• CLIs are user interface too
• Complicated git things
• Mollygive -matching donations
• Abusing Docker

Closing

• LCA 2019 will be in Christchurch, New Zealand – http://lca2019.linux.org.au
• 700 Attendees at 2018
• 400 talk and 36 Miniconf submissions

QUIC: Replacing TCP for the Web Jana Iyengar

• History
  • Protocol for http transport
  • Deployed Inside Google 2014 and Chrome / mobile apps
  • Improved performance: Youtube rebuffers 15-18% , Google search latency 3.6 – 8 %
  • 35% of Google’s egree traffic (7% of Internet)
  • Working group started in 2016 to standardized QUIC
  • Turned off at the start of 2016 due to security problem
  • Doubled in Sept 2016 due turned on for the youtube app
• Technology
  • Previously – ip _> TCP -> TLS -> HTTP/2
  • QUIC -> udp -> QUIC -> http over QUIC
  • Includes crypto and tcp handshake
  • congestion control
  • loss recovery
  • TLS 1.3 has some of the same features that QUIC pioneered, being updated to take account
• HTTP/1
  • 1 trip for TCP
  • 2 trips for TLS
  • Single connection – Head Of Line blocking
  • Multiple TCP connections workaround.
• HTTP/2
  • Streams within a single transport connection
  • Packet loss will stall the TCP layer
  • Unresolved problems
    • Connection setup latency
    • Middlebox interference with TCP – makes it hard to change TCP
    • Head of line blocking within TCP
• QUIC
  • Connection setup
    • 0 round trips, handshake packet followed directly by data packet
    • 1 round-trips if crypto keys are not new
    • 2 round trips if QUIC version needs renegotiation
  • Streams
    • http/2 streams are sent as quic streams
• Aspirations of protocol
  • Deployable and evolveable
  • Low latency connection establishment
  • Stream multiplexing
  • Better loss recovery and flexible congestion control
    • richer signalling (unique packet number)
    • better RTT estimates
  • Resilience to NAT-rebinding ( UDP Nat-mapping changes often, maybe every few seconds)
• UDP is not a transport, you put something in top of UDP to build a transport
• Why not a new protocol instead of UDP? Almost impossible to get a new protocol in middle boxes around the Internet.
• Metrics
  • Search Latency (see paper for other metrics)
  • Enter search term > entire page is loaded
  • Mean: desktop improve 8% , mobile 3.6 %
  • Low latency: Desktop 1% , Mobile none
  • Highest Latency 90-99% of users: Desktop & mobile 15-16%
  • Video similar
  • Big gain is from 0 RTT handshake
• QUIC – Search Latency Improvements by Country
  • South Korea – 38ms RTT – 1% improvement
  • USA – 50ms – 2 – 3.5 %
  • India – 188ms – 5 – 13%
• Middlebox ossification
  • Vendor ossified first byte of QUIC packet – flags byte
  • since it seemed to be the same on all QUIC packets
  • broke QUIC deployment when a flag was fixed
  • Encryption is the only way to protect against network ossification
  • “Greasing” by randomly changing options is also an option.
• Other Protocols over QUIC?
  • Concentrating on http/2
  • Looking at Web RPC

Remote Work: My first decade working from the far end of the earth John Dalton

• “Remote work has given me a fulfilling technical career while still being able to raise my family in Tasmania”
• First son both in 2015, wanted to start in Tasmania with family to raise them, rather than moving to a tech hub.
• 2017 working with High Performance Computing at University Tasmania
• If everything is going to be outsourced, I want to be the one they outsourced to.
• Wanted to do big web stuff, nobody in Tasmania doing that.
• Was a user at LibraryThing
  • They were searching for Sysadmin/DBA in Portland, Maine
  • Knew he could do the job even though was on other side of the world
  • Negotiated into it over a couple of months
  • Knew could do the work, but not sure how the position would work out

Challenges

• Discipline
  • Feels he is not organised. Doesn’t keep planner uptodate or todo lists etc
  • “You can spend a lot of time reading about time management without actually doing it”
  • Do you need to have the minimum level
• Isolation
  • Lives 20 minutes out of Hobart
  • In semi-rural area for days at a time, doesn’t leave house all week except to ferry kids on weekends.
  • “Never considered myself an extrovert, but I do enjoy talking to people at least weekly”
  • Need to work to hook in with Hobart tech community, Goes to meetups. Plays D&D with friends.
  • Considering going to coworking space. sometimes goes to Cafes etc
• Setting Boundries
  • Hard to Leave work.
  • Have a dedicated work space.
• Internet Access
  • Prioritise Coverage over cost these days for mobile.
  • Sometimes fixed provider go down, need to have a backup
• Communication
  • Less random communicated with other employees
  • Cannot assume any particular knowledge when talking with other people
  • Aware of particular cultural differences
  • Multiple chance of a miscommunication

Opportunities

• Access to companies and jobs and technologies that could get locally
• Access to people with a wider range of experiences and backgrounds

Finding remote work

• Talk your way into it
• Networking
• Job Bof
• stackoverflow.com/jobs can filter
• weworkremotely.com

Making it work

• Be Visable
• Go home at the end of the day
• Remember real people are at the end of the email

Self-Documenting Coders: Writing Workshop for Devs Heidi Waterhouse

History of Technical documentation

• Linear Writing
  • On Paper, usually books
  • Emphasis on understanding and doing
• Task-based writing
  • Early 90s
  • DITA
  • Concept, Procedure, Reference
• Object-orientated writing
  • High art for of tech writers
  • Content as code
  • Only works when compiled
  • Favoured by tech writers, translated. Up to $2000 per seat
• Guerilla Writing
  • Stack Overflow
  • Wikis
  • YouTube
  • frustrated non-writers trying to help peers
• Search-first writing
  • Every page is page one
  • Search-index driven

Writing Words

• 5 W’s of journalism.
• Documentation needs to be tested
• Audiences
  • eg Users, future-self, Sysadmins, experts, End users, installers
• Writing Basics
  • Sentences short
  • Graphics for concepts
  • Avoid screencaps (too easily outdated)
  • User style guides and linters
  • Accessibility is a real thing
• Words with pictures
  • Never include settings only in an image ( “set your screen to look like this” is bad)
  • Use images for concepts not instructions
• Not all your users are readers
  • Can’t see well
  • Can’t parse easily
  • Some have terrible equipment
  • Some of the “some people” is us
  • Accessibility is not a checklist, although that helps, it is us
• Using templates to write
  • Organising your thoughts and avoid forgetting parts
  • Add a standard look at low mental cost
• Search-first writing – page one
  • If you didn’t answer the question or point to the answer you failed
  • answer “How do I?”
• Indexing and search
  • All the words present are indexed
  • No false pointers
  • Use words people use and search for, Don’t use just your internal names for things
• Semantic tagging and reuse
  • Semantic text splits form and content
  • Semantic tagging allows reuse
  • Reuse saves duplication
  • Reuse requires compiling
• Sorting topics into buckets
  • Even with search you need some organisation
  • Group items by how they get used not by how they get prammed
  • Grouping similar items allows serendipity
• Links, menus and flow
  • give people a next step
  • Provide related info on same page
  • show location
  • offer a chance to see the document structure

Distributing Words

• Static Sites
• Hosted Sites
• Baked into the product
  • Only available to customers
  • only updates with the product
  • Hard to encourage average user to input
• Knowledge based / CMS
  • Useful to community that known what it wants
  • Prone to aging and rot
  • Sometimes diverges from published docs or company message
• Professional Writing Tools
  • Shiny and powerful
  • Learning Cliff
  • IDE
  • Super features
  • Not going to happen again
• Paper-ish things
  • Essential for some topics
  • Reassuring to many people
  • touch is a sense we can bond with
  • Need to understand if people using docs will be online or offline when they want them.
• Using templates to publish
  • Unified look and feel
  • Consistency and not missing things
  • Built-in checklist

Collaborating on Words

• One weird trick, write it up as your best guess and let them correct it
• Have a hack day
  • Ste a goal of things to delete
  • Set a goal of things to fix
  • Keep track of debt you can’t handle today
  • team-building doesn’t have to be about activities

Deleting Words

• What needs to go
  • Old stuff that is wrong and terrible
  • Wrong stuff that hides right stuff
• What to delete
  • Anything wrong
  • Anything dangerious
  • Anything used of updated in year
• How
  • Delete temporarily (put aside for a while)
  • Based on analytics
  • Ruthlessly
  • Delete or update

Documentation Must be

• True
• Timely
• Testable
• Tuned

Documentation Components

• Who is reading and why
  • Assuming no one likes reading docs
  • What is driving them to be here
• Pre Requisites
  • What does a user need to succeed
  • Can I change the product to reduce documentation
  • Is there any hazard in this process
• How do I do this task
  • Steps
  • Results
  • Next steps
• Test – How do I know that it worked
  • If you can’t test i, it is not a procedure
  • What will the system do, how does the state change
• Reference
  • What other stuff that affects this
  • What are the optionsal settings
  • What are the related things
• Code and code samples
  • Best: code you can modify and run in the docs
  • 2nd Best: Code you can copy easily
  • Worst: retyping code
• Option
  • Why did we build it this way
  • What else might you want to know
  • Have other people done this
  • Lifecycle

Documentation Types

• Instructions
• Ideas (arch, problem space,discarded options, process)
• Action required (release notes, updates, deprecation)
• Historical (roads maps, projects plans, retrospective documents)
• Invisible docs (user experience, microinteractions, error messages)
  • Error messages – Unique ID, what caused, What mitigation, optional: Link to report

Keynote: Containers aka crazy user space fun

• Work at Microsoft on Open Source and containers, specifically on kubernetes
• Containers vs Zones vs Jails vs VMs
• Containers are not a first class concept in the kernel.
  • Namespaces
  • Cgroups
  • AppArmour in LSM (prevent mounting, writing to /proc etc) (or SELinux)
  • Seccomp (syscall filters, which allowed or denied) – Prevent 150 other syscalls which are uncommon or dangerous.
    • Got list from testing all of dockerhub
    • eg CLONE, UNSHARE
    • NoNewPrivs (exposed as “AllowPrivilegeEsculation” in K8s)
    • rkt and systemd-nspawn don’t 100% follow
• Intel Clear containers are really VMs

History of Containers

• OpenVZ – released 2005
• Linux-Vserver (2008)
• LXC ( 2008)
• Docker ( 2013)
  • Initially used LXC as a backend
  • Switched to libcontainer in v0.7
• lmctfy (2013)
  • By Google
• rkt (2014)
• runc (2015)
  • Part of Open container Initiative
• Container runtimes are like the new Javascript frameworks

Are Containers Secure

• Yes
• and I can prove it
• VMs / Zones and Jails are like all the Lego pieces are already glued togeather
• Containers you have the parts seperate
  • You can turn on and off certain namespaces
  • You can share namespaces between containers
  • Every container in k8s shares PID and NET namespaces
  • Docker has sane defaults
  • You can sandbox apps every further though
• https://contained.af/
  • No one has managed to break out of the container
  • Has a very strict seccomp profile applied
  • You’d be better off attacking the app, but you are still running a containers default seccomp filters

Containerizing the Desktop

• Switched to runc from docker (had to convert stuff)
• rootless containers
• Runc hook “netns” to do networking
• Sandboxed desktop apps, running in containers
• Switch from Debian to CoreOS Container Linux as base OS
  • Verify the integrity of the OS
  • Just had to add graphics drivers
  • Based on gentoo, emerge all the way down

What if we applied the the same defaults to programming languages?

• Generate seccomp filters at build-time
  • Previously tried at run time, doesn’t work that well, something always missed
  • At build time we can ensure all code is included in the filter
  • The go compiler writes the assembly for all the syscalls, you can hijack and grab the list of these, create a seccomp filter
  • No quite that simply
    • plugins
    • exec external stuff
    • can directly exec a syscall in go code, the name passed in via arguments at runtime
• metaparticle.io
  • Library for cloud-native applications

Linux Containers in secure enclaves (SCONE)

• Currently Slow
• Lots of tradeoffs or what executes where (trusted area or untrsuted area)

Soft multi-tenancy

• Reduced threat model, users not actively malicious
• Hard Multi-tenancy would have potentially malicious containers running next to others
• Host OS – eg CoreOs
• Container Runtime – Look at glasshouse VMs
• Network – Lots to do, default deny in k8s is a good start
• DNS – Needs to be namespaced properly or turned off. option: kube-dns as a sidecar
• Authentication and Authorisation – rbac
• Isolation of master and System nodes from nodes running containers
• Restricting access to host resources (k8s hostpath for volumes, pod security policy)
• making sure everything else is “very dumb” to it’s surroundings

Insights – solving every problem for good Paul Wayper

Sysadmins

• Too much to check, too little time
• What does this message mean again
• Too reactive

How Sysadmins fix problems

• Read text files and command output
• Look at them for information
• Check this information against the knowlede
• Decide on appobiate solution

Insites

• Reads test files and outputs
• Process them into information
• Use information in rules
• Rules provide information about Solution

Examples

• Simple rule – check “localhost” is in /etc/hosts
• Rule 2 – chronyd refuses to fix server’s time since is out by more than 1000s
  • Checks /var/log/message for error message from chrony
• Insites rolls up all the checks against messages, so only down once
• Rule 3 – rsyslog dropping messages

Website

http://red.ht/demo_rules

Personalisation at Scale: A “Cookie Cutter” Approach Jim O’Halloran

• Impact on site performance on conversion is huge
• Magento
  • LAMP stack + Redis or memcached
  • Generally App is CPI bound
  • Routing / Rendering still time consuming
• Varnish full page caching (FPC)
• But what about personalised content?
• Edge Side Includes (ESIs)
  • But ESIs run in series, is slllow when you have many
  • Content is nont cacheable, expensive to calculate, significant render time
  • ESI therefore undermines much advantage of FPC
• Ajax
  • Make ajax request and fetch personalised content
  • Still load on backend
  • ESI limitations plus added network latency
• Cookie Cutter
  • When an event occurs that modifies personalisation state, send a cookies containing the required data with the response.
  • In the browser, use the content of that cookie to update the page

Example

• Goto www.example.com
  • Probably cached in varnish
  • I don’t have a cookie
  • If I login, uncachable request, I am changing login state
  • Response includes Set-Cookie header creating a personalised cookie
• Advantages
  • No backend requests
  • Page data served is cached always
• How big can cookies be?
  • RFC 6265 has limits but in reality
  • Actual limit ~4096 bytes per cookie
  • Some older browsers also limit to ~4096 bytes total per domain

Potential issues

• Request Size
  • Keep cookies small
    • Store small values only, No pre-rendered markup, No larger data structures
  • Serve static assets via CDN
  • Lot of stuff in cart can get huge
• Information leakage
  • Final URLs leaked to unlogged in users
• Large Scale changes
  • Page needs to look completely different to different users
  • Vary headers might be an option
• Formkeys
  • XSRF protection workarounds
• What about cache misses
  • Megento assembles all it’s pages from a series of blocks
  • Most parts of page are relatively static (block cache)
  • Aligent_CacheObserver – Megento extension that adds cache tags to blocks that should be cached but were not picked up as cachable by default
  • Aoe_TemplateHints – Visibility into Block cache
  • Cacheing != Performance Optimisation – Aoe_Profiler

Availability

• Plugin availbale for Megento 1
  • Varnish CookieCutter
• For Magento 2 has native varnish
  • But has limitations
  • Maybe some off CookieCutter stuff could improve

Future

• localStorage instead of cookies

Panel: Meltdown, Spectre, and the free-software community Jonathan Corbet, Andrew ‘bunnie’ Huang, Benno Rice, Jess Frazelle, Katie McLaughlin, Kees Cook

• FreeBSD only heard 11 days beforehand. Would have liked more notice
• Got people involved from the Kernel Summit in Oct
• Hosting company only heard once it went official, been busy patching since
• Likely to be class-action lawsuit for $billions. That might make chip makers more paranoid about documentation and disclosure.
• Thoughts in embargo
  • People noticed strange patches going in beforehand.
  • Only broke 6 days early, had been going for 6 months
  • “Linus is happy with this, something is terribly wrong”
  • Sad that the 2nd-tier cloud providers didn’t know. Exclusive club and lines as to who got informed were not clear
  • Projects that don’t have explicit relationship with Intel didn’t get informed
• Thoughts on other vendors
  • This class of bugs could affect anybody, open hardware would probably not fix
  • More open hardware could enable people to review the processors and find these from the design rather than poking around
  • Hard to guarantee the shipped hardware matches the design
  • Software people can build everything at home and check. FABs don’t work at home.
• Speculative execution warned about years ago. Danger ignored. How to make sure the next one isn’t ignored?
  • We always have to do some risky stuff
  • The research on this built up slowly over the years
  • Even if you have only found impractical attacks against something doesn’t mean the practical one doesn’t exist.
• What criteria do we use to decide who is in?
  • Mechanisms do exist, they were mainly not used. Perhaps because they were for software vulnerabilities
• Did people move providers?
  • No but Containers made things easier to reboot stuff and shuffle
• Are there similar vulnerabilities ( similar or general hardware ) coming along?
  • The Kernel page-table patches were fairly general, should cover many similar ones
  • All these performance optimising bit of your CPU are now attack surfaces
  • What are people going to do if this slows down hardware too much?
• How do we explain problems like these to politicians etc
  • Legos
  • We still have kernel devs getting their laptops
• Can be use CPUs that don’t have speculative execution?
  • Not really. Back to 486s
• Who are we protesting against with the embargo?
  • Everybody
  • The longer period let better fixes get in
  • The meltdown fix could be done in semi-public so had better quality

What is the most common street name in Australia? Rachel Bunder

• Why?
  • Saw a map with most common name by US street
• Just looking at name, not end bit “park” , “road”
• Data
  • PSMA Geocoded national address file – Great but came out after project
  • Use Open Street Maps
• Started with Common Name in Sydney
  • Used Metro Extracts – site closing down soon
  • Format is geojson
  • Road files separately provided
• Procedure
  • Used python, R also has good features and libaraies
  • geopandas
  • Had some paths with no names
  • What is a road? – “Something with a name I can drive a car on”
• Sydney
  • Full street name
    • Victoria Road
    • Pacific Highway
    • oops like like names are being counted twice
  • Tried merging them together
  • Roads don’t 100% match ends. Added function to fuzzy merge the roads that are 100m apart
  • Still some weird ones but probably won’t affect top
  • Second attempt
    • Short st, George st, William st, John st, Church st
• Now with just the “name bit”
  • Tried taking out just the last name. ended up with “the” as most common.
  • Started with “The” = whole name
  • Single word = whole name
  • name – descriptor – suffex
  • lots of weird names
  • name list – Park, Victoria, Railway, William, Short
• Wouldn’t work in many other counties
• Now for all over Australia
  • overpass data
  • Downloaded in 50kmx50x squares
• Lessons
  • Start small
  • Choose something familiar
  • Check you bias (different naming conventions)
  • Constance vigerlence
  • Know your problem
• Common plant names
  • Wattle – 15th – 385
• Other name
  • “The Esplanade” more common than “The Avenue”
• Top names
  • 5th – Victoria
  • 4th – Church – 497
  • 3rd – George –  551
  • 2nd – Railway
  • 1st – Park – 693
• By State
  • WA – Forest
  • SA – Railway
  • Vic – Park
  • Tas – Esplanade
  • NT – Smith/Stuart
  • NSW – Park

Wandering through the Commons

Reflections on Free and Open Source Software/Hardware in Australia, New Zealand and beyond

• Past Linux.conf.au’s reviewed
• FOSS in Aus and NZ
  • Above per capita
• List of Aus / NZ people and their contributions
  • John Lions , Lions book on Unix
  • Pia Andrews/Waugh/Smith – Open Government, GovHack, Linux Australia, Open Data
  • Vik Oliver – 3D Printing
  • Clare Cuuran – Open Government in NZ
  • plus a bunch of others

Working in Free Software and Open Hardware

• The basics
  • Be visable in projects of relevance
    • You will be typed into Google, looked at in GitHub
  • Be yourself
    • But be business Friendly
  • Linkedin is a thing, really
  • Need a accurate basic presence
• Finding a new job
  • Networks
  • Local user groups
  • Conferences
  • The projects you work on
• Application and negotiation
  • Be professional, courteous
  • Do homework about company and culture
  • Talk to people that work there
  • Spend time on interview prep
    • Know your stuff, if you don’t know, say so
  • Think about Salary expectations and stick to them
    • Val Aurora’s page on this is excellent
  • Ask to keep copyright on your code
    • Should be a no-brainer for a FOSS.OH company
• In the Job
  • Takes time to get into groove, don’t sweat it
  • Get out every now and then, particularly if working from home
  • Work/life balance
  • Know when to jump
    • Poisonous workplaces
  • An aside to People’s managers
    • Bring your best or don’t be a people manager
    • Take your reports welfare seriously

Looking after You

• Ours is in the main a sedentary and solitary pursuit
  • exercise
• Sitting and standing in front of a desk all day is bad
  • takes breaks
• Depression is a real thing
• Eat more vegetables
• Find friends/colleagues to exercise with

Working if FOSS / OH – Staying Current

• Look over a colleagues shoulder
• Do something that is not part of your regular job
  • low level programming
  • Karger systems, Openstack
• Stay uptodate with Security Blogs and the like
  • Many of the attack vectors have generic relevance
• Take the lid off, tinker with hardware
  • Lots of videos online to help or just watch

Make Hay while the Sun Shines

• Save some money for rainy day
• Keep networks Open
• Even when you have a job

You’re fired … Now What? – In a moment

• Don’t panic
  • Going out in a twitter storm won’t help anyone
• It’s not personal
  • It is the position that is no longer needed, not you
• If you think it an unfair dismissal, seek legal advice before signing anything
• It is normal to feel rubbish
• Beware of imposter syndrome
• Try to keep 2-3 opportunities in the pipeline
• Don’t assume people will remember you
  • It’s not personal, everyone gets busy
  • It’s okay to (politely naturally) follow up periodically
• Keep search a little narrow for the first week or two
  • The expand widely
• Balance take “something/everything” as better than waiting for your dream job

Dream Job

• Power 9 CPU
  • 14nm process
  • 4GHz, 24 cores
  • 25km of wires
  • 8 billion transisters
  • 3900 official chips pins
  • ~19,000 connections from die to the pin

Conclusions

• Part of a vibrant FOSS/OH community both hear and abroad
• We have accomplished much
• The most exciting (in both senses) things lie before us
• We need all of you to be part at every level of the stack
• Look forward to working with you…

Securing the Linux boot process Matthew Garrett

• Without boot security there is no other security
• MBR Attacks – previously common, still work sometimes
• Bootloader attacks – Seen in the wild
• Malicious initrd attacks
  • RAM disk, does stuff like decrypt hard drive
  • Attack captures disk pass-shrase when typed in
• How do we fix these?
  • UEFI Secure boot
  • Microsoft required in machines shipped after mid-2012
  • sign objects, firmware trusts some certs, boots things correctly signed
  • Problem solved! Nope
  • initrds are not signed
• initrds
  • contain local changes
  • do a lot of security stuff
• TPMs
  • devices on system motherboards
  • slow but inexpensive
  • Not under control of the CPU
  • Set of registers “platform configuration registers”, list of hashes of objects booted in boot process. Measurements
  • PCR can enforce things, stop boots if stuff doesn’t match
  • But stuff changes all the time, eg update firmware . Can brick machine
• Microsoft to the resuce
  • Tie Secure boot into measured boot
  • Measure signing keys rather than the actual files themselves
  • But initrds are not signed
• Systemd to the resuce
  • systemd boot stub (not the systemd boot loader)
  • Embed initrd and the kernel into a single image with a single signature
  • But initrds contain local information
  • End users should not be signing stuff
• Kernel can be handed multiple initranfs images (via cpio)
  • each unpacked in turn
  • Each will over-write the previous one
  • configuration can over-written but the signed image, perhaps safely so that if config is changed, stuff fails
  • unpack config first, code second
• Kernel command line is also security sensative
  • eg turn off iommu and dump RAM to extract keys
  • Have a secure command line turning on all security features, append on the what user sends
• Proof of device state
  • Can show you are number after boot based on TPM. Can compare to 2FA device to make sure it is securely booted. Safe to type in passwords
• Secure Provision of secrets
  • Know a remote machine is booted safely and not been subverted before sending it secret stuff.