Audiobooks – April 2020

Cockpit Confidential: Everything You Need to Know About Air Travel: Questions, Answers, and Reflections by Patrick Smith

Lots of “you always wanted to know” & “this is how it really is” bits about commercial flying. Good fun 4/5

The Day of the Jackal by Frederick Forsyth

A very tightly written thriller about a fictional 1963 plot to assassinate Frnch President Charles de Gaulle. Fast moving, detailed and captivating 5/5

Topgun: An American Story by Dan Pedersen

Memoir from the first officer in charge of the US Navy’s Top Gun school. A mix of his life & career, the school and US Navy air history (especially during Vietnam). Excellent 4/5

Radicalized: Four Tales of Our Present Moment
by Cory Doctorow

4 short stories set in more-or-less the present day. They all work fairly well. Worth a read. Spoilers in the link. 3/5

On the Banks of Plum Creek: Little House Series, Book 4 by Laura Ingalls Wilder

The family settle in Minnesota and build a new farm. Various major and minor adventures. I’m struck how few possessions people had back then. 3/5

My Father’s Business: The Small-Town Values That Built Dollar General into a Billion-Dollar Company by Cal Turner Jr.

A mix of personal and company history. I found the early story of the company and personal stuff the most interesting. 3/5

You Can’t Fall Off the Floor: And Other Lessons from a Life in Hollywood by Harris and Nick Katleman

Memoir by a former studio exec and head. Lots of funny and interesting stories from his career, featuring plenty of famous names. 4/5

The Wave: In Pursuit of the Rogues, Freaks and Giants of the Ocean by Susan Casey

75% about Big-wave Tow-Surfers with chapters on Scientists and Shipping industry people mixed in. Competent but author’s heart seemed mostly in the surfing. 3/5

Share

YouTube Channels I subscribe to in April 2020

I did a big twitter thread of the YouTube channels I am following. Below is a copy of the tweets. They are a quick description of the channel and a link to a sample video.

Lots of pop-Science and TV/Movie analysis channels plus a few on other topics.

I should mention that I watch the majority of YouTube videos at speed 1.5x since they usually speak quite slowly. To Speed up videos click on the settings “cog” and then select “Playback Speed” . YouTube lets you go up to 2x

Image

Chris Stuckmann reviews movies. During normal times he does a couple per week. Mostly currently releases with some old ones. His reviews are low-spoiler although sometimes he’ll do an extra “Spoiler Review”. Usually around 6 minutes long.
Star Wars: The Rise of Skywalker – Movie Review

Wendover Productions does explainer videos. Air & Sea travel are quite common topics. Usually a bit better researched than some of the other channels and a little longer at around 12 minutes. Around 1 video per week.
The Logistics of the US Census

City Beautiful is a channel about cities and City planning. 1-2 videos per month. Usually around 10 minutes. Pitched for the amateur city and planning enthusiast
Where did the rules of the road come from?

PBS Eons does videos about the history of life on Earth. Lots of Dinosaurs, early humans and the like. Run and advised by experts so info is great quality. Links to refs! Accessible but dives into the detail. Around 1 video/week. About 10 minutes each.
How the Egg Came First

Pitch Meetings are a writer pitching a real (usually recent) movie or show to a studio exec. Both a played by Ryan George. Very funny. Part of the Screen Rant channel but I don’t watch their other stuff
Playlist
Netflix’s Tiger King Pitch Meeting

MrMobile [Michael Fisher] reviews Phones, Laptops, Smart Watches & other tech gadgets. Usually about one video/week. I like the descriptive style and good production values, Not too much spec flooding.
A Stunning Smartwatch With A Familiar Failing – New Moto 360 Review

Verge Science does professional level stories about a range of Science topics. They usually are out in the field with Engineers and scientists.
Why urban coyote sightings are on the rise

Alt Shift X do detailed explainer videos about Books & TV Shows like Game of Thrones, Watchmen & Westworld. Huge amounts of detail and a great style with a wall of pictures. Weekly videos when shows are on plus subscriber extras.
Watchmen Explained (original comic)

The B1M talks about building and construction projects. Many videos are done with cooperation of the architects or building companies so a bit fluffy at times. But good production values and interesting topics.
The World’s Tallest Modular Hotel

CineFix doesn’t a variety of Movie-related videos. Over the last year only putting about one or two per month and mostly high quality. A few years ago they were at higher volume and had more throw-aways
Jojo Rabbit – What’s the Difference?

Marques Brownlee (MKBHD) does tech reviews. Mainly phones but also other gear and the odd special. His videos are extremely high quality and well researched. Averaging 2 videos per week.
Samsung Galaxy S20 Ultra Review: Attack of the Numbers!

How it Should have Ended does cartoons of funny alternative endings for movies. Plus some other long running series. Usually only a few minutes long.
Avengers Endgame Alternate HISHE

Power Play Chess is a Chess channel from Daniel King. He usually covers 1 round/day from major tournaments as well as reviewing older games and other videos.
World Champion tastes the bullet | Firouzja vs Carlsen | Lichess Bullet match 2020

Tom Scott makes explainer videos mostly about science, technology and geography. Often filmed on site rather than being talks over pictures like other channels.
Inside The Billion-Euro Nuclear Reactor That Was Never Switched On

Screen Junkies does stuff about movies. I mostly watch their “Honest Trailers” but they sometimes do ‘Serious Questions” which are good too.
Honest Trailers | Terminator: Dark Fate

Half as Interesting is an offshoot of Wendover Productions (see above). It does shorter 3-5 minutes weekly videos on a quick amusing fact or happening (that doesn’t justify a longer video)
United Airlines’ Men-Only Flights

Red Team Review is another movie and TV review channel. I was mostly watching them when Game of Thrones was on and since then they have had a bit less content. They are making some Game of Thrones videos narrated by the TV actors though
Game of Thrones Histories & Lore – The Rains of Castamere

Signum University do online classes about Fantasy (especially Tolkien) and related literature. Their channel features their classes and related videos. I mainly follow “Exploring The Lord of the Rings”. Often sounds better at 2x or 3x speed.
A Wizard of Earthsea: Session 01 – Mageborn

The Nerdwriter does approx monthly videos. Usually about a specific type of art, a painting or film making technique. Very high quality
How Walter Murch Worldized Film Sound

Real Life Lore does infotainment videos. “Answers to questions that you’ve never asked. Mostly over topics like history, geography, economics and science”.
This Was the World’s Most Dangerous Amusement Park

Janice Fung is a Sydney based youtuber who makes videos mostly about food and travel. She puts out 2 videos most weeks.
I Made the Viral Tik Tok Frothy DALGONA COFFEE! (Whipped Coffee Without Mixer!!)

Real Engineering is a bit more technical than the average popsci channel. The especially like doing videos covering flight dynamics. but they cover lots of other topics
How The Ford Model T Took Over The World

Just Write by Sage Hyden puts out a video roughly once a month. They are essays usually about writing and usually tied into a recently movie or show.
A Disney Monopoly Is A Problem (According To Disney’s Recess)

CGP Grey makes high quality explainer videos. Around one every month. High quality and usually with lots of animation.
The Trouble With Tumbleweed

Lessons from the Screenplay are “videos that analyze movie scripts to examine exactly how and why they are so good at telling their stories”
Casino Royale — How Action Reveals Character

HaxDogma is another TV Show review/analysis channel. I started watching him for his Watchmen Series videos and now watch his Westworld ones.
Official Westworld Trailer Breakdown + 3 Hidden Trailers

Lindsay Ellis does videos mostly about pop culture, Usually movies. These days she only does a few a year but they are usually 20+ minutes.
The Hobbit: A Long-Expected Autopsy (Part 1/2)

A bonus couple of recommended Courses on ‘Crash Course
Crash Course Astronomy with Phil Plait
Crash Course Computer Science by Carrie Anne Philbin

Share

Audiobooks – March 2020

My rating for books I read. Note that I’m perfectly happy with anything scoring 3 or better.

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recommend
  • 3/5 = Average. in the middle 70% of books I read
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

The World As It is: Inside the Obama White House by Ben Rhodes

A memoir of a senior White House staffer, Speechwriter & Presidential adviser. Lots of interesting accounts with and behind the scenes information. 4/5

Redshirts by John Scalzi

A Star Trek parody from the POV of five ensigns who realise something is very strange on their ship. Plot moves steadily and the humour and action mostly work. 3/5

Little House on the Prairie by Laura Ingalls Wilder

The book covers less than a year as the Ingalls family build a cabin in Indian territory on the Kansas Prairie. Dangerous incidents and adventures throughout. 3/5

Wheels Stop: The Tragedies and Triumphs of the Space Shuttle Program, 1986-2011 by Rich Houston

A book about the post-Challenger Shuttle missions. An overview of most of the missions and the astronauts on them. Lots of quotes mainly from the astronauts. Good for Spaceflight fans. 3/5

The Optimist’s Telescope: Thinking Ahead in a Reckless Age by Bina Venkataraman

Ways that people, organisations and governments can start looking ahead at the long term rather than just the short and why they don’t already. Some good stuff 4/5

Share

Audiobooks – February 2020

A Reminder of my rating System

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recommend
  • 3/5 = Average. in the middle 70%
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

Abraham Lincoln: A Life (Volume Two) by Michael Burlingame

2nd volume covering Lincoln’s time as president. Lots of quotes from contemporary sources. Fairly good coverage of just about everything. 3/5

Capital City: Gentrification and the Real Estate State
by Samuel Stein

Some interesting insights although everything being about New York and very left-wing politics of the author muddle the message. Worth a read if you are into the topic. 3/5

Young Men and Fire by Norman Maclean

The story of the 1949 Mann Gulch fire that killed 13 smoke jumpers. Misses a point due to lots of talking to maps/photographs but still a gripping story. 3/5

The Walls Have Ears: The Greatest Intelligence Operation of World War II by Helen Fry

The secret British operation to bug German POWs to obtain military intelligence. Only declassified in the late 1990s so very few personal recollections, but an interesting story. 3/5

Share

Audiobooks – January 2020

I’ve decided to change my rating system

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recomend
  • 3/5 = Average. in the middle 70%
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

Far Futures edited by Gregory Benford

5 Hard SF stories set it the distant (10,000 years+) future. I thought they were all pretty good. Would recommend 4/5

Farmer Boy: Little House Series, Book 2 by Laura Ingalls Wilder

A year in a life of a 9 year old boy on a farm in 1860s New Year State. Lots of hard work and chores. His family is richer than Laura’s from the previous book. 3/5

Astrophysics for People in a Hurry
by Neil DeGrasse Tyson

A quick (4h) overview and introduction of our current understanding of the universe. A nice little introduction to the big stuff. 3/5

The Pioneers: The Heroic Story of the Settlers Who Brought the American Ideal West by David McCullough

The Story of five of the first settlers of Marietta, Ohio from 1788 and the early history of the town. Not a big book or wide scope but works okay within it’s limits. 4/5

1971, Never a Dull Moment: Rock’s Golden Year by David Hepworth

A month by month walk though musical (and some other) history for 1971. Lots of gossip, backstories and history changing (or not) moments. 4/5

Digital Minimalism: Choosing a Focused Life in a Noisy World by Cal Newport

A guide to cutting down electronic distrations (especially social media) to those that make your life better and help towards your goals. 3/5

Share

Linux.conf.au 2020 – Friday – Lightning Talks and Close

Steve

  • Less opportunity for Intern type stuff
  • Trying to build team with young people
  • Internships
  • They Need opportunities
  • Think about giving a chance

Martin

  • Secure Scuttlebutt
  • p2p social web
  • more like just a protocol
  • scuttlebutt.nz
  • Protocol used for other stuff.

Emma

  • LCA from my perspective

Mike Bailey

  • Pipe-skimming
  • Enahncing UI of CLI tools
  • take first arg in pipe and sends to the next tool

Aleks

  • YOGA Book c930
  • Laptop with e-ink display for keyboard
  • Used wireshark to look at USB under Windows
  • Created a device driver based on packets windows was sending
  • Linux recognised it as a USB Keyboard and just works
  • Added new feature and
  • github.com/aleksb

Evan

  • Two factor authentication
  • It’s hard

Keith

  • Snekboard
  • Crowdsourced hardware project
  • crowdsupply.com/keith-packard/snekboard
  • $79 campaign, ends 1 March

Adam and Ben

  • idntfrs
  • bytes are not expensive any more

William

  • Root cause of swiss cheese

Colin

  • OWASP
  • Every person they taught about a vulnerbility 2 people appeared to write vulnerable code
  • WebGoat
  • Hold you hand though OWASP vulnerability list. Exploit and fix
  • teaching, playing to break, go back and fix
  • Forks in various languages

Leigh

  • Masculinity
  • Leave it better than you found it

David

  • Fixing NAT
  • with more NAT

Caitlin

  • Glitter!
  • conferences should be playful
  • meetups can be friendly
  • Ways to introduce job
  • Stickers

Miles

  • Lies, Damn lies and data science
  • Hipster statistics
  • LCA 2021 is in Canberra

Share

Linux.conf.au 2020 – Friday – Session 1 – Protocols / LumoSQL

The Fight to Keep the Watchers at Bay – Mark Nottingham

Disclaimer: I am not a security person, But in some sense we are all security people.

Why Secure the Internet

  • In the beginning it was just researchers and a Academics
  • Snowden was a watershed moment
  • STRINT Workshop in 2014
  • It’s not just your website, it’s the Javascript that somebody in injecting in front of it.

What has happened so far?

  • http -> https
    • In 2010 even major services, demo of firesheep program to grab cookies and auth off Wifi
    • Injecting cookies in http flows
    • Needed to shift needle to https
    • http/2 big push to make encrypted-only , isn’t actually though browsers only support https.
    • “Secure Contexts” cool features only https
  • Problem: Mixed Content
    • “Upgrading Insecure Requests” allow ad-hoc by pages
    • HTTPs is slow – istlsfastyet.com
    • Improvement in speed of implimentations
    • Let’s Encrypt
  • Around 85-90% https as of Early 2020
  • Some people were unhappy
    • Slow Satellite internet said they needed middle boxes to optimise http over slow links
    • People who did http shared caching
  • TLS 1.2 -> TLS 1.3
    • Complex old protocol
    • Implementation monculture
    • Outdated Crypto
    • TLS 1.3
      • Simplify where possible
      • encrypt most of handshake
      • get good review of protocol
      • At around 30%
      • Lots of implementations
    • Some unhappy. Financial institutions needed to sniff secure transactions (and had bought expensive appliances to do this)
      • They ended up forkign their own protocol
  • TCP -> QUIC
    • TCP is unencrypted, lots of leaks and room for in-betweens to play around
    • QUIC – all encrypted
    • Spin Bit – single bit of data can be used by providers to estimate packet loss and delay.
  • DNS -> DOH
    • Lots of click data sold by ISPs
    • Countries hijacking DNS by countries to block stuff
    • DNS over https co be co-located by a popular website
    • Some were unhappy
      • Lots of pushback from governments and big companies
      • Industry unhappy about concentration of DNS handling
      • Have to decide who to trust
  • SNI -> Encrypted SNI
    • Working progress, very complex
    • South Korea unhappy, was using it to block people
  • Traffic Analysis
    • Packet length, frequency, destinations
    • TOR hard to tell. Looking at using multiplexing and fix-length records
  • But the ends
    • Customer compromised or provider compromised (or otherwise sharing data)
  • Observations
    • Cost and Control
      • Cost: Big technology spends no obsolete
      • Control: some people want to do stuff on the network
    • We have to design tthe Internet to the pessimistic case
    • You can’t expose application data to the path anymore
    • Well-defined interfaces and counterbalanced roles
    • Technology and Policy need to work togeather and keep each other in check
    • Making some people unhappy means you need some guiding principles

LumoSQL – updating SQLite for the modern age – Dan Shearer

LumoSQL = SQLite + LMDB – WAL

SQLite

  • ” Is a replacement for fopen() “
  • Key/Value stores.
    • Everyone used Sleepycat BDB – bought be Oracle and licensed changed
    • Many switched to LMDB (approx 2010)
  • Howard Chu 2013 SQLightning faster than SLQite but changes not adopted into SQLite

LumoSQL

  • Funded by NLNet Foundation
  • Dan Shearer and Keith Maxwell

What isn’t working with SQLite ?

  • Inappropriate/unsupported use cases
  • Speed
  • Corruption
  • Encryption

What hasn’t been done so far

  • Located code, started on github.com/LumoSQL
  • Benchmarking tool for versions matrix
  • Mapped out how the keywords store works
    • So different backend can be dropped in.
  • Fixed bugs with the port and with lmdb

What’s Next

  • First Release Feb 2020
  • Add Multiple backends
  • Implement two database advances
Share

Linux.conf.au 2020 – Thursday – Session 3 – Software Freedom lost / Stream Processing

Open Source Won, but Software Freedom Hasn’t Yet: A Guide & Commiseration Session for FOSS activists by Bradley M. Kuhn, Karen Sandler

Larger Events elsewhere tend to be corperate sponsored so probably wouldn’t accept a talk like this

Free Software Purists

  • About 2/3s of audenience spent some time going out of their way using free software
  • A few years ago you could only use free software
  • To watch TV. I can use DRM or I can pirate. Both are problems.
  • The web is a very effecient way to install proprietary software (javascript) on your browser
  • Most people don’t even see that or think about it

Laptops

  • 2010-era Laptops are some of the last that are fully free-software
  • Later have firmware and other stuff that is all closed.
  • HTC Dream – some firmware on phone bit but rest was free software

Electronic Coupons

  • Coupons are all Digital. You need to run an app that tracks all you processors
  • “As a Karen I sometimes ask the store to just ket me have the coupon, even though it is expired”
  • Couldn’t install Disneyland App on older phones. So unable to bypass lines etc.

Proprietary dumping ground

  • Bradly had a device. Installed all the proprietary apps on it rather than his main phone
  • But it’s a bad idea since all the tracking stuff can talk to each other.

Hypocrisy of tradition free software advocacy

  • Do not criticise people for use Proprietary software
  • It is it is almost impossible to live your life without use it
  • It should be an aspirational goal
  • Person should not be seen as a failure if they use it
  • Asking others to use it instead is worse than using it yourself
  • Karen’s Laptop: It runs Debian but it is only “98% free”

Paradox: There more FOSS there is, the less software freedom we actually have in our technology

  • But there is less software freedom than there is in 2006
  • Because everything is computerized, a lot more than 15 years ago.
  • More things in Linux that Big companies want in datacentres rather than tinkerers in their homes want.

What are the right choices?

  • Be mindful
  • Try when you can to use free software. Make small choices that support software freedom
  • Shine a light on the problem
  • Don’t let the shame you feel about using proprietary software paralyze you
  • and don’t let the problems we face overwhelm you into inaction
  • Re-prioritize your FOSS development time.
    • Is it going to give more people freedom in the world?
    • Maybe try to do a bit in your free time.
  • Support each other
  • FAIF.us podcast

Advanced Stream Processing on the Edge by Eduardo Silva

Data is everywhere. We need to be able to extract value from it

  • Put it all in a database to extract value
  • Challenge: Data comes from all sorts of places
    • More data -> more bandwidth -> more resource required
    • Delays as more data ingested
  • Challenge: lots of different formats

Ideal Tool

  • Collect from different sources
  • convert unstructured to structured
  • enrichment and filtering
  • multiple destinations like database or cloud services

Fluentbit

  • Started in 2015
  • Origins lightweight log processor for embedded space
  • Ended up being used in cloud space
  • Written in C
  • Low mem and CPU
  • Plugable arch
  • input -> parser -> filter -> buffer -> routing -> output

Structure Messages

  • Unstructured to structured
  • Metadata
  • Can add tags to date on input, use it later for routing

Stream processing

  • Perform processing while the data is still in motion
  • Faster data processing
  • in Memory
  • No tables
  • No indexing
  • Receive structured data, expose a query language
  • Nomally done centrally

Doing this on the edge

  • Offload computation from servers to data collectors
  • Only sends required data to the cloud
  • Use a SQL-like language to write the queries
  • Integrated with fluent core

Functions

  • Aggregation functions
  • Time funtiocs
  • Timeseries functions
  • You can also write functions in Lua

Also exposed prometheus-type metrics

Share

Linux.conf.au 2020 – Thursday – Session 2 – Origins of X / Aerial Photography

The History of X: Lessons for Software Freedom – Keith Packard

1984 – The Origins of X

  • Everything proprietary
  • Brian Reid and Paul Asente: V Kernel -> VGTS -> W window system
    • Ported to VAXstation 100 at Stanford
    • 68k processor, 128k of VRAM
    • B&W
  • Bob Scheifler started hacking W -> X
  • Ported to Unix , made more Unix Friendly (async) renamed X

Unix Workstation Market

  • Unix was closed source
  • Vendor Unix based on BSD 4.x
  • Sun, HP, Digital, Apollo, Tektronix, IBM
  • this was when the configure program happened
  • VAXstation II
    • Color graphics 8bit accelerated
  • Sun 3/60
    • CPU drew everything on the screen

Early Unix Window System – 85-86

  • SunView dominates (actual commerical apps, Ddesktop widgets)
  • Digital VMS/US
  • Apollo had Domain
  • Tektronix demonstrated SmallTalk
  • all only ran on their own hardware

X1 – X6

  • non-free software
  • Used Internally at MIT
  • Shared with friends informally

X10 – approx 1986

  • Almost usable
  • Ported to various workstations
  • Distribution was not all free software (had bin blobs)
    • Sun port relied on SunView kernel API
    • Digital provided binary rendering code
    • IBM PC/RT Support completed in source form

Why X11 ?

  • X10 had warts
  • rendering model was pretty terrible
  • External Windows manager without borders
  • Other vendors wanted to get involved
    • Jim Gettys and Smokey Wallace
    • Write X11, release under liberal terms
    • Working against Sun
    • Displace Sunview
    • “Reset the market”
    • Digital management agreed

X11 Development 1986-87

  • Protocol designed as croos-org team
  • Sample implementation done mostly at DEC WRL, collaboration with people at MIT
  • Internet not functional enough to property collaborate, done via mail
    • Thus most of it happened at MIT

MIT X Consortium

  • Hired dev team at MIT
  • Funded by consortium
  • Members also voted on standards
    • Members stopped their on develoment
    • Stopped collaboration with non-members
  • We knew Richard too well – The GPL’s worst sponsor
  • Corp sponsors dedicated to non-free software

X Consortium Standards

  • XIE – X Imaging Extensions
  • PIX – Phigs Extension for X
  • LBX – Low Bandwidth X
  • Xinput (version 1)

The workstation vendors were trying to differentiate. They wanted a minimal base to built their stuff on. Standard was frozen for around 15 years. That is why X fell behind other envs as hardware changed.

X11 , NeWs and Postscript

  • NeWS – Very slow but cool
  • Adobe adapted PostScript interpreter for windows systems – Closed Source
  • Merged X11/NeWS server – Closed Source

The Free Unix Desktop

  • All the toolkits were closed source
  • Sunview -> XView
  • OpenView – Xt based toolkit

X Stagnates – ~1992

  • Core protocol not allowed to change
  • non-members pushed out
  • market fragments

Collapse of Unix

  • The Decade of Windows

Opening a treasure trove: The Historical Aerial Photography project by Paul Haesler

  • Geoscience Australia has inherated an extensive archive of hisorical photography
  • 1.2 million images from 1920 – 1990s
  • Full coverage of Aus and more (some places more than others)

Historical Archive Projects

  • Canonical source of truth is pieces of paper
  • Multiple attempts at scanning/transscription. Duplication and compounding of errors
  • Some errors in original data
  • “Historian” role to sift through and collate into a machine-readable form – usually spreadsheets
  • Data Model typically evolves over time – implementation must be flexible and open-minded

What we get

  • Flight Line Diagrams (metadata)
  • Imagery (data)
  • Lots scanned in early 1990s, but low resolution and missing data, some missed

Digitization Pipeline

  • Flight line diagram pipeline
    • High resolution scans
    • Georeferences
  • Film pipeline
    • Filmstock
    • High Resolution scans
    • Georeference images
    • Georectified images
    • Stitched mosaics + Elevation models

Only about 20% of film scanned. Lacking funding and film deteriorating

Other states have similar smaller archives (and other countries)

  • Many significantly more mature but may be locked in propitiatory platforms

Stack

  • Open Data ( Cc by 4.0)
  • Open Standards (TESTful, GeoJSON, STAC)
  • Open Source
  • PostGreSQL/PostGIS
  • Python3: Django REST Framework
  • Current Status: API Only. Alpha/proof-of-concept

API

  • Search for Flight runs
  • Output is GeoJSON

Coming Next

  • Scanning and georeferencing (need $$$)
  • Data entry/management tools – no spreadsheets
  • Refs to other archives, federated search
  • Integration with TerriaJS/National Map
  • Full STAC once standardized
Share

Linux.conf.au 2020 – Thursday – Session 1 – .NET to Linux / Collecting information

Engineer tested, manager approved: Migrating Windows/.NET services to Linux – Katie Bell

Works at Campaign Monitor

  • sends email spam
  • Company around since 2004

Software product generations

  • Originally a monolith
  • Windows, C# .net framework, IIS, Monolithic SQLServer
  • Went to microservices (called Reckless Microservices)
  • Windows, C# .net , OWIN Hosting / Nancy , Modular databases

Gen 2 – “Reckless” Microservice

  • Easy to create a new microservices
  • and deploy etc
  • Runs in ec2

Wanted to go to a tools like dockers, kubernetes that were not well supported by microsoft tools

Gen 3 – Docker Services

  • Linux
  • Java / Go

Lots of ways to do stuff

  • 3 different ways of doing everything
  • Confusing and big tax on developers
  • Losing knowledge about how the older Reckless stuff worked

A Crazy Idea

  • Run all the Reckless services in docker
  • Get rid of one whole generation

What does it take?

  • Move from .NET Framework to .NET Core
  • Framework very Windows specific – runtime installed at OS level
  • Core more open and cross-platform – self contained executable apps
  • But what about Mono? (Open Source .NET Framework) .
    • Probably not worth the effort since Framework is the way forward
  • But a lot of .NET Framework APIs not ported over to .NET Core. Some replaced by new APIs
  • .Net Standard libraries support on both though, which is lots of them

What Doesn’t port to Core?

  • Libraries moved/renamed
  • Some libs dropped
  • IIS, ASP.NET replaced with ASP.NET Core + MVC
  • WCF Server communication
  • Old unmaintained libraries

Luckily Reckless not using ASP.NET so shouldn’t to too hard to do. Maybe not sure a crazy idea.

But most companies don’t let people spend lots of time on Tech Debt.

Asked for something small – 2 weeks of 3 people.

  • 1 week: Hacky proof of concept (getting 1 service to run in .NET Core)
  • 2nd week: Document and investigate what full project would require and have to do
  • Last Day: Time estimates
  • Found that Windows ec2 instance were 45%
  • Cost saving alone of moving from Windows to Linux justied the project
  • Pitching:
    • Demo
    • Detailed time estimates
    • Proposal with multiple options
    • Concrete benifits, cost savings, problems with rusty old infra
  • Microsoft Portability Analyzer
    • Just run across app and gives very detailed output
  • icanhasdot.net
    • Good for external dependencies

Web Hosting differences

  • OWIN Hosting vs Kestrel
  • ASP.NET Core DI

Libraries that Do support .NET Standard

  • Had to upgrade all our code to support the new versions
  • Major changes in places

OS Differences

  • case-sensitive filenames
  • Windows services, event logging

Libararies that did not support .net Standard

  • Magnum – unmaintained
  • Topshelf

.NET Framework Libraries can be run under .NET Core using compatibility shim. Sometimes works but not really a good idea. Use with extreme caution

Overall Result

  • Took 6-8 months of 2-3 people
  • Everything migrated over.
  • Around 100 services
  • 78 actually running
  • 43 really needed to be migrated
  • 31 actually needed in the end
  • Estimated old hosting cost $145k/year
  • Estimated new hosting costing $70k/year
  • Actual hosting cost $15k/year
  • Got rid of almost all the extra infrastructure that was used to support reckless. another $25k/year saved

Advice for cleanup projects

  • Ask for something small
  • Test the idea
  • Demonstrate the business case
  • Build detailed time estimates

Collecting information with care by Opel Symes

The Problem

  • People build systems for people without checking our assumptions about people are valid
  • Be aware of my assumptions, this doesn’t cover all areas

Names

  • Form “First Name” and “Last Name” -> “Dear John Smith”
  • Fields Required – should be optional
  • Should not do character checks ( blocking accents etc )
  • Check production support emoji.. everywhere
  • MySQL Character Encodings. Only since 5.5 , default in MySQL 8
  • Every Database, table and text cloumn and defaults need to be changed to the new character set. Set connection options so things don’t get lost in transfer.
  • Personal Names around the world
  • Chinese names
  • Names can be long
  • Recommendation
    • Ask for “Full name” (where a legal name is required) and “Greeting”
    • Unicode all the way down – test with emoji
    • No Length limits

Email

  • Email addresses are quite complex
  • Does it have an “@”
  • Checked it is not a simple typo of a well-known email down
  • Will it be accepted by the email sender?
  • Look for an MX record
  • Ask the SMTP server if this username is valid
  • Simple checks for common errors
  • Don’t roll your own checking, use you own mail server or the mail library that you will using to send.

Gender

  • Transgender vs Cisgender
  • Non-binary – Gender that isn’t male or female
  • Don’t just give the two options
  • A 3rd “other” option isn’t ideal
  • A freeform field is good.
  • Gender Alternative from Nikki Stevens
  • Instead ask if people make up an “under representated community”

Pronouns

  • What pronounces should we use to refer to you? ( he , she, they )
  • Works okay in English but may not in other languages
  • Some lanugages lack gender-nutral pronoun
  • Some languages lack gender pronouns
  • pronoun.is

Titles

  • Ask for “None” but don’t actually print it “Dear None Smith”
  • Ask for Mx
  • Have a freeform field ( Dr, Count )
  • Maybe avoid titles if possible
  • Don’t show people according to gender, ask specifically.

Gender – WGEA

  • The Act defines gender as male or female.
  • Others are not reported.
  • Have an explanation for people who don’t fit in the above

Data Retention

  • Make it simple to change
  • Give users options if it isn’t (eg show preferred name)

Changing Username

  • Usernames are often options
  • Changing them comes with some caveats
  • Using UUIDS to links to users rather than usernames

Changing Emails

  • There are security implications

Deleting Data

  • Make it possible and no to hard
Share