Simon Lyall's Blog

Why linux.conf.au continues to amaze 20 years on!
Steven Hanley, Angus Lees, Hugh Blemings

• Three people who have attended every Linux.conf.au
• What has happened to keep it relivant and how it has evolved
• Focus on Open Source Software , These days hardware too
• Pre-history
  • Australia big in early Linux
  • late 90s Linux and LUGS growing in Aus
• CALU
  • In 1998 Rusty toured LUGS and organised CALU in July 99 in Monash
  • 9 July – 11 July
  • 6 tutorials and 16 talks
• Post-CALU
  • Lots of excitement
  • Height of dotcom boom, Big Linuxcare Expo in Darling Harbour in 2000. Very commercial
  • Push for another one
• 2001
  • UNSW interested in Lions memorial conference
  • CALU type conference would be good
  • Sydney big due to Olympics
  • Domain picked
• Challenge of early events
  • Everythign was new
  • CFP wasn’t formalised
  • Linux Aus still in infancy
  • Open Source was new and Few conferences
  • Few people worked fulltime in FOSS
  • sceduled over a weekend cause people didn’t get holidays
• Early objectives we hold on to
  • Community rather than commercial, modest size
  • Easier for students and hobbists, low price
  • tech/non-tech balance to encource interesting delegates
  • Miniconfs
  • Speakers treated well
  • Timeing to make conference attractive (Summer!)
  • University venue, dorms, communal accom, Holidays
• Miniconfs
  • First in Brs in 2002. more in 03, formalised in 2004
  • Open to all delegates
  • Incubate possible future conferences
  • Fill the week without adding more effort
  • Try out nice topics, extended BOFS
  • Practice ground for new speakers
• Growth and Roaming
  • Change of location helps
  • New team help avoids burnout and bring fresh ideas
  • Allows more people to contribute
  • Repeat city visits with new people involved
  • Allows people who are less-technical to help out
  • Bid process introduced, overseen by LA
  • Specifics of venue and location help teams structure their event
  • New locations add tourism aspect, encourages aspect
  • Positive experience shared by work-of-mounth
• Expansion
  • Will anyone go to Perth? NZ?
  • Infinite growth is not a goal (complexity, conference atmosphere)
  • New activities and events are continuously evaluated
• Call for papers
  • Originally ad-hoc
  • Seperate Panel since 2005
  • See Mary’s blog post in 2006
  • 4-5 times proposals to slots
  • Process
    • Actively solicit hot speakers / topics
    • Review submissions individually
    • face-to-face to reach final as group
  • Conference organisers set overall theme, choose keynote speakers
• Conference MNGT software
  • Much NIH
  • New software in 2004, 2005, 2007, 2017
  • Mainly for CFP has hooks to other conference components (scheduling, badges, website)
• Ghosts of conferences Past
  • Mailing list to ask
  • Visit conference organisers
  • Some people have remained over multiple years
• Giving Back
  • Donation to a charity, action off a shirt singned by speakers
• Event Style and Flow
  • The current event is typical
  • Moved from Weekend to week as people get paid by employer
  • Less Beer, More Food
• Giving back to the technical commons
  • Regional delegate program ( 2004 for a few years )
  • Stories of laptops being fixed by the guy who dev’d it
  • git is an outcome of “no more bitkeeper” in 2005
  • Encouraging work done once talk has been accepted
  • The Debian couch had no back so he had to sit around and support each other
• Sponsorship
  • Some Very long term
  • Some years had to educate sponsors on how to participate
  • Local vendors and sponsors
  • Nurturing sponsosrs takes consious effort
• Learning lessons
  • Floods (Brisbane 2011)
  • Budget issues (where LA comes into play)
  • CFP feedback loop has sought to tweak technical/non-tech focus
  • Code of conduct has made conf better
  • People step up to make things happen
• Resources
  • Mirror
  • Mary Gardiner’s post on getting a talk accepted
  • Simon Lyall’s guide

Right to Not Broadcast – Karen Sandler

• Thought the problem with the propitiatory device in her heart was about transparency, now feels it is more about control
• Got a new device where the programmer (controller) only worked when it was touching, rather than over a distance.
• Research team bought a device on ebay that had patients data still in it.
• On the other hand the Keynote Speaker this morning had to hack into her own device to get the info.
• Sleep Apnea machines transmitting data to both doctors and insurance companies
• Smart TVs
  • Listening for wake words all the time
  • Sending viewing data to 3rd parties
• Various Legislation
  • HIPAA , NZ Health Information Privacy Code, Aus Privacy Act, GDPR
  • GDPR – Europe’s gift to the rest of the world
• ” Incorporating connectivity means we can never be totally in control of our critical information “
• The environment/reason we provide the data in now may change
• Often the non-connected option is marginal or doesn’t even exist.
• [ Laptop ran out of battery here ]
• Things need to be worked on
  • Raising the awareness of the non-networking, privacy-first issue
  • Even among the LCA-type crowd

Around the world in 80 Microamps: ESP32 and LoRa for low-power IoT – Christopher Biggs

• Promise of IOT
  • Control everything
  • Sensors everywhere
  • Reduce cognitive load
• Problem
  • Computers everywhere = wires everywhere
  • Can’t be done every time
  • But em if you got em
  • eg Power over ethernet, Ethernet over power, Ethernet over coax
  • Wifi is great for connections. But what about power?
• Batteries are bad
  • Lead acid – obsolete just about everywhere
  • Single-use dry cells – leak
  • Nickel Metal-hydride – Some stuff
  • Lithium – Everything else
  • Sample battery
    
    • Chap-stick battery = 2 Amp/hours
    • 3.7 volts
    • Labels on batteries often lie – you need to always verify
    • Energy capacity is quoted for 20h discharge, not linear relationship
  • But they are geting better due to phones, scooters, drones pushing
• Off the shelf solutions for packs
  • Smart ones may turn themselves off if draw very low
  • Cell plus simple system works
  • Cell with “Battery Managmnet system” is a bit more complex
  • Solar panels are useless, needs to be a4, a4, a5 size at least
  • Linux systems too much draw for non-wired sensors, need to be used as hubs
• Computers can spend most of time asleep
  • Config one or more wake-interupts
  • Arduino deep sleep – Sleep consumption as low as 6 micro amps. ( 38 years with Chapstick battery)
  • Watch out for stock voltage regulators (eats 10 mA)
• ESP 8266 Sleep modes
  • Several levels off sleep modes
  • Wake up every 5 minutes = 1 week battery life
• ESP 32
  • Sleepier modes
  • Complex sleep patterns.
  • Ultra-low coprocessor
    • 4 register, 10 instructions, 16-bit, special slow memory
    • Can be configed to wake up at intervals
    • Can go back to sleep or wake the main cores
  • ULP in practice
    • Write code, load into the ULP processor
    • Enough code to decide to go back to sleep or wake up main processes
• Aim for efficiency
  • Sleep as much as possible
  • Use interrupts not polling where possible
• Nasty Surprises
  • Simple resistors ladders leak power
  • Linear regulators leak power
  • Poor antennas cost watts
  • Beware: USB programming bridges that are always on
  • Almost all the off-the-shelf IoT boards are no good for permanent installation
• Solutions
  • Turn of everything you are not using
  • radio turn off when not in use, receiver turn it on now and then. Do store-and-forward
  • slow down the cpu, turn off bluetooth
  • Reduce brightness of lights
  • BE careful about cutting out safety features
• Case Study – Smart water meter in multi-tenant building
  • Existing meter has a physical rotating dial, can count rotations
  • In cellar with no power
  • Create own
    • ESP-32
    • Wifi for setup or maint
    • LoRA for comms every 15 minutes
    • ULP monitors 4 sensors
    • ULP wakes CPU after number of elasped minutes and/or pulse
    • Transmits to Linux-based hub covers building
    • 150mA WiFi
    • 100mA over LoRA
    • 50mA when idle with radio on
    • 40mA when idle with radio off
    • 80uA in deep sleep
    • Average under 1mA , lifetime = 1-5 years
• Recap
  • Wires are hard
  • Measure and understand usage
  • The basics off deep-sleep
  • ESP32 Ultra-low-Power co-processor
  • Design your own battery-friendly systems (see Arts Miniconf presentation)
  • Project and monitor your battery lifetime
  • Website

Deep Learning, Not Deep Creepy – Jack Moffitt

• What is machine learning
  • Make decisions based on statistics
• How is deep learning different?
  • Many layers of neurons each learning more sophisticated representations of features in the data
  • Transfer learning – reuse N-network for similar task where less training data
  • Generative Adversarial networks
• The dark side of deep learning
  • Works better with more data. Incentive for companies to get a huge amount of data
  • Computationally very expensive – Creates incentive to move things to large clouds
  • Inaccessible to smaller players
  • Hard to debug, black boxes.
  • Amplify biases in training data, somemays to fix but not generally fixable
  • Data may be low quality
• Machine learning @mozilla
  • Deepspeech and common voice
  • Deepspeech – state-of-art speech detection
    • Existing solutions owned by big companies. Costs $ and in cloud
    • Opening up models and train data will allow innovation
    • Based on baidu’s deep speech paper
    • pre-trained models for english
    • runs real-time on mobile
    • word error rate of 6.48% on librivox
    • streaming support
  • Common voice
    • Crowd source voice data for new applications
    • 20 languages launched
    • 1800 hours collected so far
  • Deepproof – spelling and grammar checker
    • Existing one is basically a keylogger ( Grammarly )
    • Needs to be small enough to run on device
    • Learn by example, rather than few rules
      • Less language-specific tuning
      • More scaleable
    • Local interface to avoid sending private text to several
    • 12 million 300-character chunks from wikipedia
    • Inject plausible mistakes
    • Real-life data
    • Maybe improve with federated learning without disclosing text
  • Lpcnet
    • lots of test-to-speech are end-to-end
    • a separate network converts spectrograms to audio
    • GRiffin-Lim sounds bad
    • WaveNet / WaveRNN needs 10s GFLOPS
    • needs something for efficient for on-device
    • Currently 1.5-6 GFLOPS
    • real-time on mobile
    • Works okay
    • Other applications
      • Speech compression
      • noise compression
• Questions:
  • Does the audio slow-down work on non-speech? Not really
  • How do you deal with region variations of speech and grammar – Common voice is collecting

Filesender: Sending large files across facilities – Ben Martin

• 10 Year old project
• Web based File Sharing
  • Quick 1-to-few file sharing between people
  • Files go away after a month by default
  • Simpler to run than anon-ftp etc
  • Stats of downloads available to sharer
  • User only needs web browser
  • Upload resume, important with TB sized files
  • Notifications
  • share with explicit groups
  • Browser-to-browser encryption of data AES-256
  • SAML for auth scale
  • GDPR by default, about privacy page
• Overview
  • Server side is PHP
  • Client side JS with light widgets
  • MariaDB
• Server Storage
  
  • Chunked 5MB files
  • Cept used at aarnet
• Downloading
  • On the fly zip64 archive creation
  • One of more files listed per transfer
  • Links for console download if needed
• Dragons
  • Auto Downloads and fast uploads cross browser is HARD
  • Mixed browsers
  • Long uploads can exceed auth sessions times
  • Web crypto support w3c
  • People use ancient versions of databases
• Lots of details on the Database and Encryption. Sounds like both have improved to a good state
• Future
  • UI refresh
  • Mobile App
  • E2E Encryption
  • Docker image for easy setup
  • More SAML info, apache config?
  • Integration of Endpoints ( auto youtube etc )
  • Session Clone to investigate problems (but privacy?)
  • Run the whole thing in the cloud
• Questions
  • Command line? – REST API (php and client)
  • RSYNC for slightly changing files – Being investigated

Hot Potato – James Forman and Callum Dickinson

• What is Hot potato
  • Not a monitoring System
  • monitoring System -> Hot potato -> On-call person
  • Web app in python and flask
  • Tells you things and stays out of the way
• Why ?
  • Spark shutdown paging Network
  • Needed quick version
• Goals
  • Don’t get in the way
  • Alert reduction
  • Highly available
  • Support any System – Nagios family now, Prometheus later
  • Support methods – Pushover, SMS, Paging
• What else can it do
  • Failure notifications when contacts are not working
  • Heartbeats so know when monitoring system is down
• Planning stuff to add to it
  • Teams – put everyone on call
  • Team escalations
  • Planned work ( go to person working before oncall, extend windows )
  • Support Hotline integration
  • Mobile App
  • Adding German and Italian
• How it works
  • Flask App
  • RabbitMQ
  • Database (cockroachDB)
  • Apps talk via the Databases
  • Alert -> Object in DB -> Put on Queue -> Worker
  • Worker -> Get details to send to -> Try to send -> Store result in DB
  • If Failure fails then work it’s way though the list.
• Questions:
  • ACK can use pushover so don’t have to login to app
  • Looking at teams functions
  • CockroachDB picked since it seems very reliable
  • Not sure about restoring/calendaring features going in since need to make it generic?
  • Endpoints fairly modular so should be extendable to new ones.

#WeAreNotWaiting: how open source is changing healthcare
– Dana Lewis

• Getting diagnosed with a chronic disease is like being struck by lighting.
• Insulin takes a while to kick in
• Manual diabetes
  • Have to check level over and over again
  • Have to judge trend and decide more insulin, food, exercise, etc
  • Constantly
• Device
  • Windows-only interface software to access
  • Alarm not great, various other limitations
• Idea
  • Pull data off the device, create smarter app
  • Hard to do
• First version
  • Device -> WinPC -> dropbox -> app -> pushover -> phone
• 2nd Version
  • Press button to indicate what she is doing (eating, sleeping) when get alert
• 3rd version
  • Hook into Insulin pump to do it automatically
  • Replace the human do they same thing over-and-over again in the loop.
  • all portable
  • Person doesn’t have to wake up to adjust, petter sleep
• OpenAPS
  • Open Source
  • Created a list of ways it could fail (battery fails, wires come out)
  • Focusing on safety
  • Limiting dosing ability in hardware and software
  • Failing back safely to standard device operation
• Approx 1000 users
  • 9 million+ hours of DIY closed loop experience
  • Anonymized dataset available
• Sample child user
  • Before: 4.5 manual interventions per day by parents
  • After: 0.7 per day
• School Child ( 5 vs 6th Grade )
  • 420 visits to school nurse ( 2.3 /day ) – 66 visits for events
  • 5 visits with OpenAPS – 3 visits (Gym related)
• Intel Edison Platform
  • Smaller than Raspberry Pi
  • But discontinued (looking for old ones to replace)
• Going back to Pi, now smaller
  • Has built in display with “Explorer Hat”
• Outsiders are building stuff because they can and the traditional companies are not meeting the need. Innovate with small solution and build it up
• Ask what little things you can do for people with similar problems. This started with just “Make a louder alarm”.

How to Avoid Meetings – Maia Sauren

• What Distributed/International Teams involves
  • Lots of late meetings
  • Not up on the inclusive languages
  • We’d language quirks from ESL speakers
  • Taxonomy changes between fields
  • Stereotypes are incomplete
  • Ask Culture vs Guess Culture
  • Micro-cultures , down to schools, extra years.
  • When you make a private joke without context, someone is left out
  • What governance model wins (who do they raise barriers for)
  • It’s harder to change a relationship over the phone than maintain one
  • A relationship with a CoC is less fragile (shouldn’t be figured out in the fly during conflict)
  • “How do you want to have arguments?”
  • Set standards early and resolve swiftly
  • Normalise conflict resolution
  • Adulting: it’s for people who don’t want to cry even more later
  • What have you done this week you are proud of?

Disaster Recovery Book – Svetlana Marina

• Software Development Process
  • Speaning all time “firefighting” so no time to improve processes
• Operation Support Model
• Incident Model
  • Alert Raise
  • Initial Response
    • Runbook should have: Alert type -> Investigation help, exact queries into log search, links
  • Impact, escalations, SLA
    • Send email to stakeholders, keep their trust
    • Include “communication required” in runbook
  • Investigation and Damage control
    • Do what is required to fix problem, not fix the root cause
    • Message to stakeholders again
  • Through Investigation (depends on situation)
  • Fix root cause
  • Post Incident Review

The Bus Plan: Junior Staff Training – Andrew Jeffree

• Staff coming into industries with lots of automation, but what when automation fails?
• Staff only know about automation, can only run a playbook, don’t understand what it does
• Disclaimer: Automation isn’t bad.
  • Config Managment, Custom tooling, CICD, Infrastructure as code
• Buttons
  • Need to understand what they do
  • Sometimes they are not there
  • Can be Hard to implement a button that doesn’t exist
• Do it the manual way
• Training
  • Do something manually (install wordpress) and document
  • Expand what you have done
  • Tweak it
  • Break and fix it
  • Don’t be afraid to change something manually
• Questions
  • Split for new people – 30% training , 70% work
  • Manual -> ansible -> puppet
  • Don’t want them to be stuck in the mindset that everthing we do is the best way

When Agile Doesn’t Work Anymore: Managing a Large Documentation Project – Lana Brindley

• We all age
  • Sometimes old docs should just be binned
• Old docs base, massive changes needed, but wanted to save it
  • Changes to toolchain too
• Proof of concept
  • Define how big the job will be, what is involved
  • Get buy-in
• Plan, Plan, Plan
  • Create a real timeline
  • Advertise your plan, tell everybody about it
  • Do a presentation to team for each phase
• Research
  • Who is your audience?
  • No, really, who is your audience? – eg Sales and support may use docs more than customers
  • Lets the customers know that “somebody cares about these docs”
• User/task analysis
  • Where do we need to focus work
  • What are the big tasks
• Do the thing
  • You have to sit down and write it
  • You are breaking the agile system
  • You need to track your work and who doing what
  • eg this case, moving from big chapters to small topic-based docs.
  • Agree ahead of time on process, have buy-in
• Review
  • What went on?
  • WHo not to stuff it up next time
  • Try not to blame people too much
• Agile vs … something else
  • Sometimes Agile is not the best model
• Tips for working outside Agile
  • Outreach
    • Especially product owners.
    • Get people on board
  • Track your work
    • Create Epics of sprints
    • Don’t go overboard
  • Should about it
    • outreach never stops
    • Present at sprint reviews
    • Brownbags

Finding Your Tribe: Choosing open source Communities – Cintia Del Rio

• When started out she couldn’t find info on picked what project to work on. Crowdsourced some opinions from others
• You need to work out why you want to volenteer for a project.
• 3 types of code on Github
  • Source Available
    • Backed by companies (without open source as their business model)
    • Core devs from company, roadmap controlled by them
    • Limited influence from externals
    • Most communication not on public channels
    • You will be seen as guest/outsider
  • On Person Band
    • Single core maintainer, working in spare time
    • Common even for very popular libraries and tools (lots of examples from node and java ecosystem)
    • Few resource
    • Conflicts might not be handled well
  • Communities
    • Communication Channells – forums, mailing list, chat
    • Multiple core devs
    • Github org
    • Code of conduct
• Some things to check beforehand
  
  • Is it dead yet?
    • Communication channels, Commits, issues, pull requests – how old, recent updates
  • How aggressive is the community?
    • Look at how a clueless user is handled
    • Declined pull requests. HOW did they handle the decline
    • ” Is the mailing list/icr/slack, is it a trash fire? ”
    • “Can you please rule” – add “Can you please” in front of a comment and does it sound nice or still mean/sarcasm?
  • Is non-coding work valued
  • Communities with translations?
  • Look at photos from the conferences
  • Grammar mistakes and typos – how are the handled?
  • Newbie tags and Getting started docs
  • Cool languages tend to attract toxic people
  • Look for Jerks in leadership
  • Ask around
• Does it spark joy? – If not let it go.

Intro to the Open Source Voice Stack By Kathy Reid

• In the past we have taught spreadsheets etc. We need to teach the latest thing and that is now voice interfaces
• Worked for Mycroft, mainly using that for demo
• Voice stack
  • Wake Word
  • Speech 2 Text (utterance)
  • Action ( command )
  • Text to Speech (dialogue )
• Request response life-Cycle
  • Wake word
  • Intent parser over utterance
• When Kathy just started she was first woman and first Australian so few/no samples in database and had problems understanding her.

• Text to Speech
  • Needs to have a well speaker speaking for 40-60 hours
  • Mimic Recording Studio – List of Phrases that people need to speak to train the output

Being Kind to 3am You – Katie McLaughlin

• Not productive and operating at her best at 3am
• But 3am’s will happen and they probably will be important
• Essentials
  • You should have documentation, don’t keep it in your head since people are not available at 3am
  • Full doc management system might take a while
  • Must be Editable
    • Must be updateable at any time
  • Searchable
  • Have browser keywords that search confluence or github
  • Secure but discoverable by co-workers
• Your Tools
  • Easy cache commands to use
  • Not dangerous
• Stepping Up
  • Integrate your docs so it’ll be available and visible when people need it.
  • Alerts could link to docs for service
• Post Mortem
  • List of commands you typed to fix it
• Reoccurring Issues
  • Sometimes the quick fix is all you can do or is good enough. You can get back to sleep.
  • Maybe just log rotate to clean the disk. Or restart process once a week
  • Make you fix an ansible playbook you can just click
• Learning
  • Learn new stuff so when you have chance you can do it write
• Flag Changes
  • Handover changes to over to everyone else
• So Empathy towards the other people (and they may show it back)
• Audience
  • One guy gave anyone who go paged overnight $100 bill on their desk next day ( although he charged customers $150 )
  • From Fire Depts – Label everything, Have the docs come with the alert. Practice during the daylight.
  • Project IPXE – every single error message is a link to wiki page
  • Advice: Write down every command, everything you did, every output you saw. So useful for next day.

Making youself Redundant on Day One – Alexandra Perkins

• Experiences
  • All Docs as facebook posts
  • All docs as comment codes
  • Word documents hidden in folders
• Why you should document in your first weeks
  • Could you know the what the relevant questions for new people
  • You won’t remember it the first time you hear it
  • Easier for the next person
  • Inclusive and diverse workplace
• What should you document
  • Document the stuff you find hard
  • Think about who else can use your docs
  • Stuff like: How to book leave, Who to ask about what topics, Info on workplace social events. Where lunch is.
• How to document from the start
  • In wiki or Sharepoint
  • Word docs locally and copy it the official place once you have access
  • Saved support tickets
  • Notes to yourself on slack
  • Screenshots of slack conversations
  • Keep it simple, informal content is your friend. All the Memes!
  • Example Tutorial: “Send yourself an email and trace it though the logs”
• Future Proofing
  • Create or Improve the place for Internal documentation
  • Everyone should be able to access and edit (regardless of technical expertise)
  • Must be searchable and editable so can be updated
  • Transfer all docs you did on your personal PC to company-wide documentation
  • Make others aware of the work you have done
  • Foster a culture of strong documentation.
    • Policy to document all newly announced changes
    • Have a rotation for the documentation person
  • Quality Internal Docs should be
    • Accessible
    • Editable
    • Searchable
    • Peer Reviewed

JIT Learning: It’s great until it isn’t – Tessa Bradbury

• What should we learn?
  • There is a lot to learn
• What is JIT Learning?
  • Write Code -> Hit an issue -> Define Problem -> Find a Solution
• Assumption – You will ask the required questions (hit the issue)
  • Counter example: accessibility, you might not hit the problem yourself
• Assumption – You can figure out the problem
  • Sometimes you can’t easily, you might not have the expereince and/or training
• Assumption – You can find the solution
  • Sometimes you can’t find the solution on google
  • Sometimes you are not in Open Source, you can’t just read the code and the docs may be lacking
• Assumption – You might not be sure the best way to write your fix
  • Best way to implement the code, if you should fix it in code
  • Or if your code has actually fixed the whole problem
• Assumption – The benefit of getting it done now outweighs the cost of getting it wrong
  • Counter example – Security