Putting Prometheus node_exporter behind apache proxy

I’ve been playing with Prometheus monitoring lately. It is fairly new software that is getting popular. Prometheus works using a pull architecture. A central server connects to each thing you want to monitor every few seconds and grabs stats from it.

In the simplest case you run the node_exporter on each machine which gathers about 600-800 (!) metrics such as load, disk space and interface stats. This exporter listens on port 9100 and effectively works as an http server that responds to “GET /metrics HTTP/1.1” and spits several hundred lines of:

node_forks 7916
node_intr 3.8090539e+07
node_load1 0.47
node_load15 0.21
node_load5 0.31
node_memory_Active 6.23935488e+08

Other exporters listen on different ports and export stats for apache or mysql while more complicated ones will act as proxies for outgoing tests (via snmp, icmp, http). The full list of them is on the Prometheus website.

So my problem was that I wanted to check my virtual machine that is on Linode. The machine only has a public IP and I didn’t want to:

  1. Allow random people to check my servers stats
  2. Have to setup some sort of VPN.

So I decided that the best way was to just use put a user/password on the exporter.

However the node_exporter does not  implement authentication itself since the authors wanted the avoid maintaining lots of security code. So I decided to put it behind a reverse proxy using apache mod_proxy.

Step 1 – Install node_exporter

Node_exporter is a single binary that I started via an upstart script. As part of the upstart script I told it to listen on localhost port 19100 instead of port 9100 on all interfaces

# cat /etc/init/prometheus_node_exporter.conf
description "Prometheus Node Exporter"

start on startup

chdir /home/prometheus/

/home/prometheus/node_exporter -web.listen-address
end script

Once I start the exporter a simple “curl” makes sure it is working and returning data.

Step 2 – Add Apache proxy entry

First make sure apache is listening on port 9100 . On Ubuntu edit the /etc/apache2/ports.conf file and add the line:

Listen 9100

Next create a simple apache proxy without authentication (don’t forget to enable mod_proxy too):

# more /etc/apache2/sites-available/prometheus.conf 
<VirtualHost *:9100>
 ServerName prometheus

CustomLog /var/log/apache2/prometheus_access.log combined
 ErrorLog /var/log/apache2/prometheus_error.log

ProxyRequests Off
 <Proxy *>
Allow from all

ProxyErrorOverride On
 ProxyPass /
 ProxyPassReverse /


This simply takes requests on port 9100 and forwards them to localhost port 19100 . Now reload apache and test via curl to port 9100. You can also use netstat to see what is listening on which ports:

Proto Recv-Q Send-Q Local Address   Foreign Address State  PID/Program name
tcp   0      0*       LISTEN 8416/node_exporter
tcp6  0      0      :::9100         :::*            LISTEN 8725/apache2


Step 3 – Get Prometheus working

I’ll assume at this point you have other servers working. What you need to do now is add the following entries for you server in you prometheus.yml file.

First add basic_auth into your scape config for “node” and then add your servers, eg:

- job_name: 'node'

  scrape_interval: 15s

    username: prom
    password: mypassword

    - targets: ['myserver.example.com:9100']
         group: 'servers'
         alias: 'myserver'

Now restart Prometheus and make sure it is working. You should see the following lines in your apache logs plus stats for the server should start appearing: - - [31/Jul/2016:11:31:38 +0000] "GET /metrics HTTP/1.1" 200 11377 "-" "Go-http-client/1.1" - - [31/Jul/2016:11:31:53 +0000] "GET /metrics HTTP/1.1" 200 11398 "-" "Go-http-client/1.1" - - [31/Jul/2016:11:32:08 +0000] "GET /metrics HTTP/1.1" 200 11377 "-" "Go-http-client/1.1"

Notice that connections are 15 seconds apart, get http code 200 and are 11k in size. The Prometheus server is using Authentication but apache doesn’t need it yet.

Step 4 – Enable Authentication.

Now create an apache password file:

htpasswd -cb /home/prometheus/passwd prom mypassword

and update your apache entry to the followign to enable authentication:

# more /etc/apache2/sites-available/prometheus.conf
 <VirtualHost *:9100>
 ServerName prometheus

 CustomLog /var/log/apache2/prometheus_access.log combined
 ErrorLog /var/log/apache2/prometheus_error.log

 ProxyRequests Off
 <Proxy *>
 Order deny,allow
 Allow from all
 AuthType Basic
 AuthName "Password Required"
 AuthBasicProvider file
 AuthUserFile "/home/prometheus/passwd"
 Require valid-user

 ProxyErrorOverride On
 ProxyPass /
 ProxyPassReverse /

After you reload apache you should see the following: - prom [01/Aug/2016:04:42:08 +0000] "GET /metrics HTTP/1.1" 200 11394 "-" "Go-http-client/1.1" - prom [01/Aug/2016:04:42:23 +0000] "GET /metrics HTTP/1.1" 200 11392 "-" "Go-http-client/1.1" - prom [01/Aug/2016:04:42:38 +0000] "GET /metrics HTTP/1.1" 200 11391 "-" "Go-http-client/1.1"

Note that the “prom” in field 3 indicates that we are logging in for each connection. If you try to connect to the port without authentication you will get:

This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.

That is pretty much it. Note that will need to add additional Virtualhost entries for more ports if you run other exporters on the server.


Gather Conference 2016 – Afternoon

The Gathering

Chloe Swarbrick

  • Whose responsibility is it to disrupt the system?
  • Maybe try and engage with the system we have for a start before writing it off.
  • You disrupt the system yourself or you hold the system accountable

Nick McFarlane

  • He wrote a book
  • Rock Stars are dicks to work with

So you want to Start a Business

  • Hosted by Reuben and Justin (the accountant)
  • Things you need to know in your first year of business
  • How serious is the business, what sort of structure
    • If you are serious, you have to do things properly
    • Have you got paying customers yet
    • Could just be an idea or a hobby
  • Sole Trader vs Incorporated company vs Trust vs Partnership
  • Incorperated
    • Directors and Shareholders needed to be decided on
    • Can take just half an hour
  • when to get a GST number?
    • If over $60k turnover a year
    • If you have lots of stuff you plan to claim back.
  • Have an accounting System from Day 1 – Xero Pretty good
  • Get an advisor or mentor that is not emotionally invested in your company
  • If partnership then split up responsibilities so you can hold each other accountable for specific items
  • If you are using Xero then your accountant should be using Xero directly not copying it into a different system.
  • Remuneration
    • Should have a shareholders agreement
    • PAYE possibility from drawings or put 30% aside
    • Even if only a small hobby company you will need to declare income to IRD especially non-trivial level.
  • What Level to start at Xero?
    • Probably from the start if the business is intended to be serious
    • A bit of pain to switch over later
  • Don’t forget about ACC
  • Remember you are due provisional tax once you get over the the $2500 for the previous year.
  • Home Office expense claim – claim percentage of home rent, power etc
  • Get in professionals to help

Diversity in Tech

  • Diversity is important
    • Why is it important?
    • Does it mean the same for everyone
  • Have people with different “ways of thinking” then we will have a diverse views then wider and better solutions
  • example “Polish engineer could analysis a Polish specific character input error”
  • example “Controlling a robot in Samoan”, robots are not just in english
  • Stereotypes for some groups to specific jobs, eg “Indians in tech support”
  • Example: All hires went though University of Auckland so had done the same courses etc
  • How do you fix it when people innocently hire everyone from the same background? How do you break the pattern? No be the first different-hire represent everybody in that group?
  • I didn’t want to be a trail-blazer
  • Wow’ed out at “Women in tech” event, first time saw “majority of people are like me” in a bar.
  • “If he is a white male and I’m going to hire him on the team that is already full of white men he better be exception”
  • Worried about implication that “diversity” vs “Meritocracy” and that diverse candidates are not as good
  • Usual over-representation of white-males in the discussion even in topics like this.
  • Notion that somebody was only hired to represent diversity is very harmful especially for that person
  • If you are hiring for a tech position then 90% of your candidates will be white-males, try place your diversity in getting more diverse group applying for the jobs not tilt in the actual hiring.
  • Even in maker spaces where anyone is welcome, there are a lot fewer women. Blames mens mags having things unfinished, women’s mags everything is perfect so women don’t want to show off something that is unfinished.
  • Need to make the workforce diverse now to match the younger people coming into it
  • Need to cover “power income” people who are not exposed to tech
  • Even a small number are role models for the future for the young people today
  • Also need to address the problem of women dropping out of tech in the 30s and 40s. We can’t push girls into an “environment filled with acid”
  • Example taking out “cocky arrogant males” from classes into “advanced stream” and the remaining class saw women graduating and staying in at a much higher rate.


  • Paul Spain from Podcast New Zealand organising
  • Easiest to listen to when doing manual stuff or in car or bus
  • Need to avoid overload of commercials, eg interview people from the company about the topic of interest rather than about their product
  • Big firms putting money into podcasting
  • In the US 21% of the market are listening every single month. In NZ perhaps more like 5% since not a lot of awareness or local content
  • Some radios shows are re-cutting and publishing them
  • Not a good directory of NZ podcasts
  • Advise people use proper equipment if possible if more than a once-off. Bad sound quality is very noticeable.
  • One person: 5 part series on immigration and immigrants in NZ
  • Making the charts is a big exposure
  • Apples “new and noteworthy” list
  • Domination by traditional personalities and existing broadcasters at present. But that only helps traction within New Zealand



Gather Conference 2016 – Morning

At the Gather Conference again for about the 6th time. It is a 1-day tech-orientated unconference held in Auckland every year.

The day is split into seven streamed sessions each 40 minutes long (of about 8 parallel rooms of events that are each scheduled and run by attendees) plus and opening and a keynote session.

How to Steer your own career – Shirley Tricker

  • Asked people hands up on their current job situation, FT vs PT, sinmgle v multiple jobs
  • Alternatives to traditional careers of work. possible to craft your career
  • Recommended Blog – Free Range Humans
  • Job vs Career
    • Job – something you do for somebody else
    • Career – Uniqie to you, your life’s work
    • Career – What you do to make a contribution
  • Predicted that a greater number of people will not stay with one (or even 2 or 3) employers through their career
  • Success – defined by your goals, lifestyle wishes
  • What are your strengths – Know how you are valuable, what you can offer people/employers, ways you can branch out
  • Hard and Soft Skills (soft skills defined broadly, things outside a regular job description)
  • Develop soft skills
    • List skills and review ways to develop and improve them
    • Look at people you admire and copy them
    • Look at job desctions
  • Skills you might need for a portfilio career
    • Good at organising, marketing, networking
    • flexible, work alone, negotiation
    • Financial literacy (handle your accounts)
  • Getting started
    • Start small ( don’t give up your day job overnight)
    • Get training via work or independently
    • Develop you strengths
    • Fix weaknesses
    • Small experiments
    • cheap and fast (start a blog)
    • Don’t have to start out as an expert, you can learn as you go
  • Just because you are in control doesn’t make it easy
  • Resources
    • Careers.govt.nz
    • Seth Goden
    • Tim Ferris
    • eg outsources her writing.
  • Tools
    • Xero
    • WordPress
    • Canva for images
    • Meetup
    • Odesk and other freelance websites
  • Feedback from Audience
    • Have somebody to report to, eg meet with friend/adviser monthly to chat and bounce stuff off
    • Cultivate Women’s mentoring group
    • This doesn’t seem to filter through to young people, they feel they have to pick a career at 18 and go to university to prep for that.
    • Give advice to people and this helps you define
    • Try and make the world a better place: enjoy the work you are doing, be happy and proud of the outcome of what you are doing and be happy that it is making the world a bit better
    • How to I “motivate myself” without a push from your employer?
      • Do something that you really want to do so you won’t need external motivation
      • Find someone who is doing something write and see what they did
      • Awesome for introverts
    • If you want to start a startup then work for one to see what it is like and learn skills
    • You don’t have to have a startup in your 20s, you can learn your skills first.
    • Sometimes you have to do a crappy job at the start to get onto the cool stuff later. You have to look at the goal or path sometimes

Books and Podcasts – Tanya Johnson

Stuff people recommend

  • Intelligent disobedience – Ira
  • Hamilton the revolution – based on the musical
  • Never Split the difference – Chris Voss (ex hostage negotiator)
  • The Three Body Problem – Lia CiXin – Sci Fi series
  • Lucky Peach – Food and fiction
  • Unlimited Memory
  • The Black Swan and Fooled by Randomness
  • The Setup (usesthis.com) website
  • Tim Ferris Podcast
  • Freakonomics Podcast
  • Moonwalking with Einstein
  • Clothes, Music, Boy – Viv Albertine
  • TIP: Amazon Whispersync for Kindle App (audiobook across various platforms)
  • TIP: Blinkist – 15 minute summaries of books
  • An Intimate History of Humanity – Theodore Zenden
  • How to Live – Sarah Bakewell
  • TIP: Pocketcasts is a good podcast app for Android.
  • Tested Podcast from Mythbusters people
  • Trumpcast podcast from Slate
  • A Fighting Chance – Elizabeth Warren
  • The Choice – Og Mandino
  • The Good life project Podcast
  • The Ted Radio Hour Podcast (on 1.5 speed)
  • This American Life
  • How to be a Woman by Caitlin Moran
  • The Hard thing about Hard things books
  • Flashboys
  • The Changelog Podcast – Interview people doing Open Source software
  • The Art of Oppertunity Roseland Zander
  • Red Rising Trilogy by Piers Brown
  • On the Rag podcast by the Spinoff
  • Hamish and Andy podcast
  • Radiolab podcast
  • Hardcore History podcast
  • Car Talk podcast
  • Ametora – Story of Japanese menswear since WW2
  • .net rocks podcast
  • How not to be wrong
  • Savage Love Podcast
  • Friday Night Comedy from the BBC (especially the News Quiz)
  • Answer me this Podcast
  • Back to work podcast
  • Reply All podcast
  • The Moth
  • Serial
  • American Blood
  • The Productivity podcast
  • Keeping it 1600
  • Ruby Rogues Podcast
  • Game Change – John Heilemann
  • The Road less Travelled – M Scott Peck
  • The Power of Now
  • Snow Crash – Neil Stevensen

My Journey to becoming a Change Agent – Suki Xiao

  • Start of 2015 was a policy adviser at Ministry
  • Didn’t feel connected to job and people making policies for
  • Outside of work was a Youthline counsellor
  • Wanted to make a difference, organised some internal talks
  • Wanted to make changes, got told had to be a manager to make changes (10 years away)
  • Found out about R9 accelerator. Startup accelerator looking at Govt/Business interaction and pain points
  • Get seconded to it
  • First month was very hard.
  • Speed of change was difficult, “Lean into the discomfort” – Team motto
  • Be married to the problem
    • Specific problem was making sure enough seasonal workers, came up with solution but customers didn’t like it. Was not solving the actual problem customers had.
    • Team was married to the problem, not the married to the solution
  • When went back to old job, found slower pace hard to adjust back
  • Got offered a job back at the accelerator, coaching up to 7 teams.
    • Very hard work, lots of work, burnt out
    • 50% pay cut
    • Worked out wasn’t “Agile” herself
    • Started doing personal Kanban boards
    • Cut back number of teams coaching, higher quality
  • Spring Board
    • Place can work at sustainable pace
    • Working at Nomad 8 as an independent Agile consultant
    • Work on separate companies but some support from colleges
  • Find my place
    • Joined Xero as a Agile Team Facilitator
  • Takeaways
    • Anybody can be a change agent
    • An environment that supports and empowers
    • Look for support
  • Conversation on how you overcome the “Everest” big huge goal
    • Hard to get past the first step for some – speaker found she tended to do first think later. Others over-thought beforehand
    • It seems hard but think of the hard things you have done in your life and it is usually not as bad
    • Motivate yourself by having no money and having no choice
    • Point all the bad things out in the open, visualise them all and feel better cause they will rarely happen
    • Learn to recognise your bad patterns of thoughts
    • “The Way of Art” Steven Pressfield (skip the Angels chapter)
  • Are places Serious about Agile instead of just placing lip-service?
    • Questioner was older and found places wanted younger Agile coaches
    • Companies had to completely change into organisation, eg replace project managers
    • eg CEO is still waterfall but people lower down are into Agile. Not enough management buy-in.
    • Speaker left on client that wasn’t serious about changing
  • Went though an Agile process, made “Putting Agile into the Org” as the product
  • Show customers what the value is
  • Certification advice, all sorts of options. Nomad8 course is recomended


Wellington Open 2016

Over Easter 2016 (March 25th – 27th) I played in the Wellington Open Chess Tournament. I play in the tournament about half of the time. This year it was again being played at the CQ Hotel in Cuba street so I was able to stay at the venue and also visit my favorite Wellington cafes.

There were 43 players entered (the highest for several years) with around 9 coming down from Auckland. I was ranked 16th with a rating of 1988 and the top 4 Wellington players ( Dive, Wastney, Ker & Croad) who are all ranked in the Top 10 in NZ were playing.

See the Tournament’s page for details and downloads for the games. Photos by Lin Nah and me are also up on Flickr for Days one, two and three.

Round 1 – White vs Dominic Leman (unrated) – Result win

This game was over fairly quickly after my opponents 5th Move (Nf6) which let me win a free Bishop after ( 5.. Nf6 6.Nxc6 bxc6 7.Bxc5 ) and then they played (7.. Nxe4) to take the pawn which loses the Night since I just pin it again the King with Qe2 and pick it up a move or two later.



Round 2 – Black vs Michael Steadman ( 2338) – Result lose

Mike plays at my club and is rated well above me. However I put on a pretty poor show and made a mistake early in the Opening (which was one of my lines rather than something Mike usually plays). Error on move 5 lost me a pawn and left my position poor. I failed to improve and resigned on move 21.

Round 3 – White vs Kate Song (1701) – Result win

After 6. ..a5

I was very keen on beating Kate. While she is rated almost 200 points lower than me she improving faster and beat me in the last round of the Major Open at the NZ Champs at the start of this year.

We were the same colours as our game in January so I spent some time prepping the opening to avoid my previous mistakes.

In that game Black played 6.. a5  (see diagram) and I replied with the inaccurate Be2 and got tied into knots on the Queen side. This time I played 7. Bd3 which is a better line. However after 7. ..Nh6 8. dxc5 Bxc5 9. O-O black plays Ng4 which gives me some problems. After some back and forth Black ended up with a bit of a mid-game advantage with a developed bishop pair. and control of the open C file.


27. Bg5 and I offer a draw

However on move 27 after the rooks had been swapped I was able to play Bg5 which threaten to swap Black’s good Bishop or push it backwards. I offered a draw.

Luckily for me Kate picked to swap the Bishops and Queens with 27. ..Bxg5 28.Nxg5 Qd1+ 29.Qxd1 Bxd1 which left me with almost all my pawns on black squares and pretty safe from her white squared bishop. I then was able to march my King over to the Queenside while my Kingside was safe from the Bishop. After picking up a the a-pawn when the Knight and Bishops swapped I was left with a King plus A&B pawns vs King an b-pawn with around 3 tempo in reserve for pushing back the Black king.

Round 3 – Michael Nyberg vs Leighton Nicholls

Position after 71. Kxg4

Another game during round 3 went very long. This was the position after move 71 , White has just taken blacks last pawn. The game kept going till move 125! White kept try to force black to the edge of the board while black kept his king close to the centre and the Knight nearby (keeping the king away with checks and fork threats).

At move 125 Black (Nicholls) claimed a draw under the 50-move rule at which point Michael Nyberg asked “are you sure” and “are you prepared for any penalties?”. After Leighton confirmed he wanted to go ahead with the claim Michael claimed that the draw rules were changed a couple of years ago and that King+Rook vs King+Knight was allowed 75 moves. And that since the draw claim was incorrect Leighton should lose.

However a check of the Official FIDE rules online showed that there was no such special limited for the material, the rule is always 50 moves (Rule 9.3) . The penalty for incorrectly claiming a draw would also have been 2 minutes added to Michael’s time not Leighton losing the game (Rule 9.5b).

The Arbiter checked the rules and declared the game a draw while Michael grumbled about appealing it (which did not happen). Not a good way to end the game since I thought Leighton defended very well. Especially the way Michael was very aggressive while being completely in the wrong.

There have been exceptions to the 50-move draw rule in the past but it has been a flat 50 moves since at least 2001 since while some positions take longer in theory no human would actually be able to play them perfectly.

Round 4 – Black vs David Paul – Result win

Another game against somebody close to my rating but a little below. So while I should win it could be hard. I didn’t play the opening right however and ended up in a slightly poor position a couple of tempo down.

After 32 Re4 draw offered

After some maneuvering (and the odd missed move by both sizes) white offered a draw after move 32. I decided to press on with f6 and was rewarded when after 32. ..f6 33.Kf2 Kf7 White played 34.b4? which allowed me to play Nc3 and bounce my Night to b5 and then take the Bishop on d6 along with an extra pawn.


After 44. ..Kd6

A few moves later I’m a pawn up and with a clear path to the win although I made a mistake at the ended it wasn’t bad enough to be fatal.




Round 5 – White vs Russell Dive – Game lost

After getting onto 3 points after 6 rounds I was rewarded with playing the top seed. As often happens with stronger players he just seemed to make 2 threats with every move and my position slowly (well not that slowly) got worse and worse as I couldn’t counter them all (let alone make my own threats).

Eventually I resigned 3 pawns down with no play (computer assessed my position as -5.0)

Round 6 – Black vs Brian Nijman – Game Lost

Last round was once again against a higher rater play but one I had a reasonable chance against.

After 10. ..Bg6

I prepped a bit of the opening but he played something different and we anded up in a messy position with White better developed but not a huge advantage.

We both had bishops cutting though the position and Queens stuck to the side but it would be hard for me to develop my pieces. I was goign to have to work hard at getting them out into good positions


After 23. d5

After some swaps white ended up charging though my centre and with lots of threats. I spent a lot of time looking at this position workign out what to do.

White has the Bishop ready to take the pawn on b5 and offer check, possibly grab the Knight or pin the rook. While th Knight can also attack the rook. and the pawns can even promote.

I ended up giving up the exchange for a pawn but promptly lost a pawn when white castled and took on f7.

After 32. Ne2

I decided to push forward hoping to generate some threats and managed to when I threated to mate with two Knights or win a rook after 32. Ne2

34.Rxc5+ Kxc5 35.Be1 Rd8 36.Rc7+ followed but I played 36. ..Kd4 and blocked by Rook rather than Kb6 giving myself a tempo to move my rook to d1. This would have probably picked up another exchange and should have been enough for the win.


After 47. g6

And then I found another win. All I had to do was push the pawn. On move 47 I just have to put a piece on f2 to block the bishop from taking my pawn on g1. If 47. ..Nf2 48. Bxf2 Rxf2 49. g1=Q leaves me a Queen vs a rook and I can take the pawn on g6 straight away.

But instead I got Chess Blindness and just  swapped the pawn for the Bishop. I then tried to mate (or perpetual check) the King instead of trying to stop the pawns (the computer says 50. ..Nf4 is just in time). A few moves later I ran out of King-chasing moves and resigned. At which point everybody told me the move I missed 🙂

So I ended up with 3/6 or 50% in the tournament. I Losts to the players better than me and beat the lower rated ones. I’m a little disappointed with the last game and the games against Russell Dive and Mike Steadman but happy with the others. Definitely need to keep working on things though.

Linux.conf.au 2016 – Friday – Session 3

Lighting talks

  • New Zealand Open Source Society
    • nzoss.org.nz
  • LCA 2015 give-aways of ARM chromebooks
    • Linux on ARM chellenge
    • github/steven-ellis
  • Call to Arms
    • x86 != Linux
    • Please consider other archetectures
  • StackPtr
    • Open Source GPS and MAP sharing
    • Android client and IOS to come
    • Create a group, Add placemaps, Share location with a group
    • Also run OpenStreetmaps tileserver
    • stackptr.com/registration  – Invite code LCA2016
  • Hat Rack
    • code is in githug, but what about everything else?
    • How to ack stuff that isn’t code?
    • bit.do/LABHR    #LABHR
    • Recommend people, especially people not like you
    • github.com/LABHR/octohatrack
  • Pycon
    • Melbourne 12-16 August
    • DjangoCon Au, Science and Data Miniconf, Python in Education plus more on 1st day
    • CPF open in mid-March
    • Financial assistence programme
    • pycon-au.org
  • Kiwi PyCon
    • 2016 in dunedin
    • Town Hall
    • 9-11 September
    • kiwi.pycon.org
  • GovHack
    • Have fun
    • Open up the government data
    • 29-31 July across Aus and NZ
  • JMAP: a better way to email
    • Lots of email standards, all aweful
    • $Company API
    • json over https
    • Single API for email/cal/contacts
    • Mobile/battery/network friendly
    • Working now at fastmail
    • Support friendly (only uses http, just one port for everything).
    • Batches commands, uses OOB notification
    • Effecient
    • Upgrade path – JMAP proxy
    • http://jmap.io  , https://proxy.jmap.io/
  • Tools
    • “Devops is just a name for a Sysadmin without any experience”
    • Lets get back to unix principals with tools
  • Machine Learning Demo
  • Filk of technical – Lied about being technical/gadget type.
  • ChaosKey
    • Randomness at 1MB/s
    • Copied from OneRNG
    • 4x4mm QFN package attached to USB key
    • Driver in Linux 4.1 (good in 4.3)
    • Just works!
    • Building up smaller batches to test
    • Hoping around $30


  • Thanks to Speakers
  • Clarification about the Speaker Gifts
  • Thanks to Sponsors
  • Raffle – $9680 raised
  • SFC donations with “lcabythebay” in the comment field will be matched (twice) in next week or two.
  • Thanks to Main Organisers from LCA President
  • Linux.conf.au 2017
    • Hobart
    • January 16th-20th 2017
    • At the Wrest Point casino convention centre. Accommodation on site and at Student accommodation
    • hobart.lca2017.org
  • Thanks to various people
  • hdmi2usb.tv is the video setup

Linux.conf.au 2016 – Friday – Session 2

Free as in cheap gadgets: the ESP8266 by Angus Gratton

  • I missed the start of the talk but he was giving a history of the release and getting software support for it.
  • Arduino for ESP8266 very popular
  • 2015-2016 maturing
  • Lots of development boards
    • Sparkfun ESP8266 thing, Adafruid Hazaah, WeMOS D1
  • Common Projects
    • Lots of lighting projects, addressable LED strips
    • Wireless power monitoing projects
    • Copy of common projects. Smoke alarm project
    • ESPlant – speakers project built in Open Hardware Miniconf – solar powered gardening sensor
    • Moodlight kickstarter
  • Shortcomings
    • Not a lot of documentation compared to other micro-controllers. 1/10 that of similar products
    • Weird hardware behaviour. Unusual output
    • Default baud rate 74880 bps
    • Bad TLS – TLS v1.0, 1.1 only , RSA 512/1024 . 2048 might work
    • Other examples
  • FOSS in ESP8266
    • GCC , Lua , Arduino, Micro Python
    • axTLS , LWIP, max80211, wpa_supplicant
    • Wrapped APIs, almost no source, mostly missing attribution
    • Weird licenses on stuff
  • Does this source matter?
    • Anecdote: TLS random key same every time due to bad random function (later fixed). But still didn’t initially use the built-in random number generator.
  • Reverse Engineering
    • Wiki , Tools: foogod/xtobjdis , ScratchABit , radara2 (soon)
    • esp-open-rtos – based on the old version that was under MIT
    • mbedTLS – TLS 1.2 (and older) , RSA to 4096 and other stuff. Audited and maintained
    • Working on a testing setup for regression tests
  • For beginners
    • Start with Ardino
    • Look at dev board
  • Future
    • Hopefully other companies will see success and will bring their own products out
    • but with a more open licenses
    • ESP32 is coming, probably 1y away from being good and ready

secretd – another take on securely storing credentials by Tollef Fog Heen

  • Works for fastly
  • What is the problem?
    • Code can be secret
    • Configuration can be secret
    • Credentials are secret
  • Secrets start in the following and move to the next..
    • directly code
    • then a configuration file
    • then an pre-encrypted store
    • then an online store
  • Problems with stores
    • Complex or insecure
    • Manual work to re-encrypt
    • Updating is hard
    • Not support for dev/prod split
  • Requirements for a fix
    • Dynamic environment support
    • Central storage
    • Policy based access controls, live
    • APIs for updating
  • Use Case
    • Hardware (re)bootstrapping
    • Hands-of/live handling
    • PCI: auditing
    • Machine might have no persistent storage
  • Options
    • pwstore – pre-encrypted
    • chef-vault – pre-encrypted
    • Hashicorp Vault – distributed, complex, TTL on secrets
    • etcd – x509
  • Secretd
    • go
    • SQL
    • ssh
    • tree structure, keys are just strings
    • positive ACLs
    • PostgressSQL backend
    • Apache Licensed
  • Client -> json over ssh -> secret-shell -> unix socket ->  secretd -> postgressSQL
  • Missing
    • Encrypting secrets on disk
    • Admin tools/other UIs
    • Auditing
    • Tool integration
    • Enrolment key support
  • Demo
  • Questions:
    • Why not sqlite? – Cause  I wanted at database. Postgres more directly supported the data structure I wanted, also type support
    • Why do just use built-in postgress security stuff? – Features didn’t exist a year ago, also requires all users must exist as DB users.


Linux.conf.au 2016 – Friday – Session 1

Keynote – Genevieve Bell

  • Building the Future
  • Lots of rolls as an Anthropologist at Intel over last 15 years or so
  • Vision of future from 1957 shows what the problems are in 1957 that the future would solve
  • Visions of the future seem very clean and linear, in reality it is messy and myriad.
  • ATM machine told her “Happy Birthday”
  • Imagining “Have you tried turning it off and on again?” at smart city scale is kind of terrifying.
  • Connectivity
    • Many people function well when they are offline, some people used to holiday in places with no cell reception
    • Social structures like Sabbath to give people time offline, but devices want us to be always online
    • Don’t want to always have seamless between devices, context matters. Want work/home/etc split
  • IOT
    • Technology lays bare domestic habits that were previously hidden
    • Who is else knows what you household habits are -> Gossip
  • Big Data
    • Messy , incomplete, inaccurate
    • Average human tells 6-200 lies per day
    • 100% of Americans lie in online profiles
      • Men lie about height, Women lie about weight
    • More data does not equal more truth. More data just means more data
  • Algorithms
    • My optimise for the wrong things (from the user’s point of view)
  • Security and Privacy
    • Conversation entwined with conversation about National Security
    • Concepts different from around the world
    • What is it like to release data under one circumstance and then to realise you have released it under several others
  • Memory
    • Cost of memory down to zero, we should just store everything
    • What are the usage models
    • What if everything you ever did and said was just there, what if you can never get away from it. There are mental illnesses based on this problem
  • Innovation
    • What is changing? to whose advantage and disadvantage? what does this mean to related areas?
    • Our solutions need to be human
    • We are the architects of our future
  • Question
    • Explain engineers to the world? – Treated first year at Intel like it was Anthropology fieldwork. Disconnect between what people imagine technologists think/do and what they really do. Need to explain what we do better

Helicopters and rocket-planes by Andrew Tridgell

  • The wonderful and crazy world of Open Autopilots
  • Outback Challenge
    • 90km/h for 45 minutes
    • Search pattern for a lost bushwalker with UAV
    • Drop them a rescue package
    • 2016 is much harder VTOL, get blood sample. Most do takeoff and landing remotely (30km from team).
    • “Not allowed to get blood sample using a propeller”
  • VTOL solutions – Helicopters and Quadplanes – tried both solutions
    • Communication 15km away, 2nd aircraft as a relay
    • Pure electric doesn’t have range. 100km/h for 1h
  • Helicopters
    • “Flying vibration generators with rotating swords at the top”
    • Hard to scale up which is needed in this case. 15cc motor, 2m blades, 12-14kg loaded
    • Petrol engines efficient VTOL and high energy density
    • Very precise control, good in high wind (competition can have ground wind up to 25 knots)
    • Normal stable flight vibrates at 6G , show example where in a couple of seconds flight goes bad and starts vibrating at 30+ G in a few seconds due to control problem (when pitch controller was adjusted and then started feedback loop)
  • Quadplanes
    • Normal Plane with wings but 4 virtually pointing propellers added
    • Long range, less vibration
    • initially two autopilots plus one more co-ordinating
    • electric for takeoff, petrol engine for for long range forward flight.
    • Hard to scale
    • crashed
  • Quadplane v2
    • Single auto-pilot
    • avoid turning off quad motors before enough speed from forward motor
    • Pure electric for all motors
    • Forward flight with wings much more efficient.
    • Options with scale-up to have forward motor as petrol
  • Rockets
    • Lohan rocket plane – Offshoot of The Register website
    • Mission hasn’t happened yet
    • Balloon takes plane to 20km, drops rocket and goes to Mach 2 in 8 seconds. Rocket glides back to each under autopilot and lands at SpacePort USA
    • 3d printed rocket. Needs to wiggle controls during ascent to stop them freezing up.
    • This will be it’s first flight so has autotune mode to hopefully learn how to fly for the first time on the way down
  • Hardware running Ardupilot
    • Bebop drone and 3DR solo runs open autopilot software
    • BBBmini fully open source kit
    • Qualcom flight more locked down
    • PXFMini for smaller ones
  • Sites
    • ardupilot.com
    • dronecode.org
    • canberrauav.org.au

The world of 100G networking by Christopher Lameter

  • Why not?
    • Capacity needed
    • Machines are pushing 100G to memory
    • Everything reqires more Bandwidth
  • Technologies
    • Was 10 * 10G standards CFP Cxx
    • New standard is 4 * 28Gs QSFP28 . compact and designed to replace 10G and 40G networking
    • Inifiband (EDR)
      • Most mature to date, switches and NICs available
    • Ethernet
      • Hopefully available in 2016
      • NICS under dev, can reuse EDR adapter
    • OmniPath
      • Redesigned to try replace infiband
    • Comparison connectors
      • QSFP28 smaller
    • QSFP idea with spliter into 4 * 25G links for some places
      • Standard complete in 2016 , 50G out there but standard doesn’t exist yet.
      • QSFP is 4 cables
  • 100G switches
    • 100G x 32 or 50G x64 or 25G x 128
    • Models being released this year, hopefully
    • Keeping up
  • 100G is just 0.01ns per bit , 150ns for 1500MTU packet, 100M packets/second, 50 packets per 10 us
  • Hardware distributed packets between cores. will need 60 cores to handle 100G in CPU, need to offload
  • Having multiple servers (say 4) sharing a Nic using PCIe!
  • How do you interface with these?
    • Socket API
  • Looking Ahead
    • 100G is going to be a major link speed in data centers soon
    • Software needs to mature especially the OS stack to handle bottlenecks


Linux.conf.au 2016 – Thursday – Session 3

Law and technology: impedance mismatch by Michael Cordover

  • IP lawyer
  • Known as the EasyCount guy
  • Lawyers and Politicians don’t get it
    • Governing behaviour that is not well understood (especially by lawyers) is hard
    • Some laws are passed under assumption that they won’t always be enforced (eg Jaywalking, Speeding limits). Pervasive monitoring may make this assumption obsolete
  • Technology people don’t get the law either
    • Good reasons for complexity of the law
    • Technology isn’t neutral
  • Legal detailed programmatic specifically
    • Construction
    • Food
    • Civil aviation
    • Broadcasting
  • Anonymous Data
    • Personal information – info from which id can be worked out
  • 100s of examples where law is vague and doesn’t well map to technology
    • Encryption
    • Unauthorised access
    • Copyright
    • Evidence
  • The obvious, easy solution:
    • Everybody must know about technology
    • NEVER going to happen
  • Just make a lot of contracts
    • Copyright – works fairly well, eg copyleft
    • TOS – works to restrict liability of service providers so services can actually be safely provided
    • EULAs
    • P3P – Privacy protection protocol
    • But doesn’t work well in multiple jurisdictions, small ppl against big companies, etc
  • Laws that are fit for purpose
    • An ISP is not an IRC server
    • VOIP isn’t PSTN
    • Focus on the outcome, sometimes
  • A somewhat radical shift in legal approach
    • It turns out the Internet is (sometimes) different
    • United States vs Causby – 1946 case that said people don’t work air above their property to infinity. Airplanes could fly above it.
  • You can help
    • Don’t ignore they law
    • Don’t be too technical
    • Don’t expect a technical solution
    • Think about policy solutions
    • Talk to everybody


Linux.conf.au 2016 – Thursday – Session 2

Machine Ethics and Emerging Technologies by Paul ‘@pjf’ Fenwick

  • Arrived late
  • Autonomous cars
    • Little private ownership of autonomous vehicles
    • 250k driving Taxis
    • 3.5 million truck drivers + plus more that depend on them
    • Most of the cost is the end-to-end on a highway. Humans could do the hard last-mile
  • Industrial revolution
    • Lots of people put out of jobs
    • Capital offence to harm machines
    • We still have tailors
    • But some jobs have been eliminated – eg Water bearer in cities
  • Replacing humans with small amounts of code
  • White collar jobs now being replaced
  • If more and more people are getting put out of jobs and we live in a society that expects people to have jobs what can we do?
    • Education to retrain
  • We *are* working less 1870=70h work week , 1988=40h work week
  • Leisure has much increased 44k hours -> 122k hours (shorter week + live longer)
  • What do people do with more leisure?
    • Pictures of cats!
    • Increase in innovation
  • How would the future work if machines are doing the vast majority of jobs?
    • Technological dividend
    • Basic income
  • Drones
    • “Drones have really taken off in the last few years”
    • Delivery drones
    • Disaster relief
    • Military drones – If autonomous then radio silent
    • Solar powered drones with multi-day/week duration
      • Good for environmental monitoring
      • Have anonymous warfare, somebody launches it, and it kills some people, but you don’t know who to blame
  • Machine Intelligence
    • Watson getting better at cancer diagnosis and treatments plan than many doctors
  • Questions:
    • Please focus on the upsides of lethal autonomous robots – Okay with robots, less happy with taking the machine out of the loop.
    • Why work week at 40 hours – Conjecture by Paul – Culture says humans must work and work gives you value and part time work is seen as much less important

Open Source Tools for Distributed Systems Administration by Elizabeth K. Joseph

  • Tools that enable distributed teams to work
  • Works day to day on Openstack
  • How most projects do infrastructure
    • Team or company manges do it or they just use github
    • Requests via mailing list or bug/ticketing system
    • Priority determined by the core team
  • Is there a better way – How Openstack is different – Openstack infrastructure team
    • Host own git, wiki, ircbots, mailing lists, web servers and run them themselves
    • All configs are open source and tracked in git
    • Anyone can submit changes to our project.
    • We all work remotely
  • Openstack CI system
    • 800+ projects
    • All projects must work togeather
    • changes can’t break master branch
    • code must be clean
    • testing must be completely automated
  • Tools for CI (* is they own tools)
    • Launchpad for Auth
    • git
    • gerrit
    • zuul* – gatekeep
    • Geaman
    • jenkins
    • nodepool*
  • Automated Test for infrastructure
    • flake8
    • puppet parser validate, puppet lint, puppet application tests
    • XML checkers
    • Alphabetized files ( cause people forget the alphabet)
    • Permissions on IRC channels
  • Peer review means
    • Multiple eyes on changes prior to merging
    • Good infrastructure for developing new solutions
    • No special process to go through commit access
    • Trains us to be collaborative by default
    • Since anyone can contribute, anyone can devote resources to it
  • Gerrit in-line comments
  • Automated deployments. Either puppet directly or via vcsrepo
  • Can you really manage infrastructure via git commits
    • Cacti – cacti.openstack.org
      • Cacti are public so anybody can check them
      • No active monitoring
    • Puppetboard
      • so you can watch changes happening
      • Had to change a little so secret stuff not public
    • Documentation
      • Fairly good since distributed team
    • Not quiet everything
      • Need to look at logs
      • Some stuff is manual
      • Passwords need to be privately managed (but in private git repo)
      • Some complicated migrations are manual
  • Maintenance collaboration on Etherpad
  • Collaboration
    • Via IRC various channels
    • main + incident + sprint + weekly meetings
    • channel/meeting logs
    • pastebin
    • In-person collaboration at Openstack design summit every 6 months
  • And then there are timezones
    • The first/root member in a particular region struggles to feel cohesion with the team
    • Increased reluctance to land changes into production
    • makes slower on-boarding
    • Only solved by increasing coverage in that time-zone so they’re not alone
  • Questions
    • Reason why no audio/video? – Not recorded or even hard to access if they are
    • How to dev “write documentation” culture – Make that person responsible to write docs so others can still handle it. Helps if it it really easy to do. Wikis never seem to work in practice, goes though same process as everything else (common workflow)
    • Task visibility – was bugzilla + launchpad – trying storyboard but not working well.

Linux.conf.au 2016 – Thursday – Session 1

Jono Bacon Keynote

  • Community 1.0 (ca 1998)
    • Observational – Now book on how to do it
    • Organic – people just created them
    • Technical Enviroment – Had to know C (or LaTex)
  • Community 2.0 (ca 2004, 2005)
    • Wikipedia, Redhat, Openstack, Github
    • Renaissance – Stuff got written down on how to do it
    • Self Organising groups – Gnome, Kde, Apache foundation – push creation of tech and community
    • Diversity – including of skills , non-technical people had a seat at the table and a role.
    • Company Engagement – Starting hiring community managers, sometimes didn’t work very well
  • Community 3.0 ?
  • Why?
    • “Thoughtful and productive communities make us as species better
  • Access and power is growing exponentally
  • But stuff around is changing
    • Cellphones are access method for most
    • Cloud computering
    • CD-printers, drones, cloud, crowdfunding, Ardinino
    • Lots for channels to get things to everybody and everybody can participate
  • “We need to empower diversity of both people and talent”
  • Human brain has not had a upgrade in a long time
  • Bold and Audacious Goals
    • Openness is at the heart of all of these
    • Open source in the middle of many
  • Eg Drone
    • Runs linux
    • Open API
  • “Open Source is where Society innovates”
  • “Need to make great community leadership accessible to everybody”
  • “Predictable collaboration – an aspirational goal where we won’t *need* community managers”
  • Not just about technology
    • We are all human.
  • Tangible value vs Intangible value
    • Tangible can be measured and driven to fix the numbers
    • Intangible – trust, dignety
  • System 1 thinking vs System 2 thinking
    • Instant vs considered
  • SCARF Model of thinking
    • Status – clarity of relative importance, need people to be able to flow between them
    • Certainty – Security and predictability
    • Autonomy – People really want choices
    • R – I got distracted by twitter, I’m sure it was important
    • Fair – fairness
  • Two Golden Rules
    • We accomplish our goals indirectly
    • We influence behaviour with small actions
  • We need to concentrate to building an experience for people to who join the community
  • Community Workflow
    • Communication – formal, inclormal? Coc? Tech to use?
    • Release sceduled, support?
    • How to participate, tech, hackthons
    • Government structure
  • Paths for different people
    • New developers
    • Core Developers
    • Consumers
    • Downstream Cosutomers
    • Organizations
  • Opportunity vs Belonging
  • Questions
    • Increasing Signal to Noise ratio – Trolls are easy[er], harder for people who are just no deft in communication. Mentorship can help
    • Destructive communities (like 4chan) , how can technology be used to work against these – Leaders need to set examples. Make clear abusive behavour towards others. Won’t be able to build tools that will completely remove bad behaviour. Had to tell destructive vs direct automatically but they can augmented.
    • What about Linus type people? – View is that even though it works for him and it is okay with people he knows. Viewed inwards by others it sets a bad example.

Using Persistent Memory for Fun and Profit by Matthew Wilcox

  • What is it?
    • Retains data without power
    • NV-DIMMs available – often copy DRAM to flash when power lost
    • Intel 3D X-point shipping in 2017. will become more a standard feature
  • How do we could use it
    • Total System persistence
      • But the CPU cache is not backed up, so pending writes vanish
    • Application level persistence
      • Boot new kernel be keep the running apps
      • CPU cache still
    • Completely redesigned operating system to use
      • But we want to use in 2017
    • A special purpose filesystem
      • Implementation not that great
    • A very fast block device
      • Usaged as very fast cache for apps really need it. Not really general purpose
    • Small modifications to existing file systems
      • On top of ext2 (xip)
      • DAX
  • How do we actually use it
    • New CPU instructions ( mostly to make sure encourage that things are flushed from the CPU cache)
    • Special purpose programming language shouldn’t be needed for interpreted languages. But for compiled code libraries might be needed
  • NVML library
  • Stuff built on NVML library so far.
    • Red-Black tree, B-tree, other data-structures
    • Key-value store
    • Fuse file system
    • Example MySQL storage engine
  • Resources
  • Questions
    • In 2017 will we have mix of persistent and non-persistent RAM? – Yes . New Layer in the storage hierarchy
    • Performance of 3d will be slower a little slow than DRAM but within ballpark, various trade-offs with other characteristics
    • Probably won’t have native crypto

Dropbox Database Infrastructure by Tammy Butow

  • Dropbox for last 4 months, previously Digital Ocean, prev National Australia Bank
  • Using MySQL for last 10 years. Now doing it FT.
  • 400 Million customers
  • Petabytes of data across thousands of servers
  • In 2012 Dropbox just had 1 DBA, but was huge then.
  • In 2016 it has grown to 9 people
  • 6000 DB servers -> DB Proxy -> DB as a service (edgestore) -> memcache -> Web Servers (nginx)
  • Talk – Go at Dropbox, Zviad Metreveli on Youtube
  • Applications talk directly to edgestore not directly to database
  • vitess is mysql proxy (by youtube) similar to what dropbox wrote. Might move to that
  • Details
    • Percona 5.6
    • Constantly upgrading (4 times in last year)
    • DBmanager – service we manage mysql via
  • Each Cluster is proiamry + 2 replicas
  • Use xtrabackup ( to hdfs locally and s3)
  • Tools
    • Tasks grow and take time
    • DBmanager
      • Automating DB operations
      • Web interface with standard operations and status of servers
      • Cloning Screen
      • Promotion Screen
      • Create and restore backups
      • WebUI gives you feedback and you can see how things are going. Don’t need magic command lines. Good for other teams to see stuff and do stuff (options right in front of them).
      • Benchmarking
      • Database job scheduling and prioritization. Promotion will take priority over anything else.
      • Common logging, centralized server and nice gui that everyone can see
    • HERMES
      • Availbale on dropbox github
      • Visable all quests and actions that need to be done by the team
    • Monitoring
      • Grafana
  • Performance
    • Improving backup and restore speed.
      • LZOP
      • xtrabackup
  • Auto-remediation (naoru) – up on github at some point
  • Inventory Management
    • Machine Database (MDB)
    • Has tags for things like kernel versions
  • Diognostics
    • Automated periodic tcpdump
    • Tools to kill long running transactions
    • List current queries running
    • atop
  • The Future
    • Reliabilty, performance and cost improvements
    • Config management
    • Love the “Go Programming Language” by Kernighan
    • List of Papers they love
  • Questions
    • Using percona not mariadb. They also shard not cluster DBs
    • Big Culture change from Back to Dropbox – At Bank tried to decom old systems, reduce risk. At Dropbox everyone is very Brave and pushing boundarys
    • machine database automatically built largely
    • Predictive Analysis on hardware – Do some , lots of dashboards for hardware team, lifecycle management of hardware. Don’t hug servers. Hug the hardware class instead.
    • Rollbacks are okay and should be easy. Always be able to rollback a change to get to back to a good stack.