Linux.conf.au 2019 – Thursday – Session 3

Open Source Tools for Publishing and Processing Earth Observation Imagery – Paul Haesler

Paul Haesler
  • Golden age of satelite imagery
  • Geostationary – One area – Good for weather
  • Circum-polar orbits – all over earth every 10-16 days
  • Data processing Chain
    • Level 0 – Raw
    • Level 1 – Geo rectify – Measure surface radience
    • Level 2 – Curroect for sun, sat angle, atmostphere – ARD – Records surface reflectance
  • Landsat-8 (25-30m , 8-16 day cycle, data since 1982 ) Sentinel-2 ( better, 5 day cycle, 10m resolution)
  • Digital Earth Australia
  • The Problem
    • Open Data Cube
    • Python, based on xarry
    • Postgres for metadata
    • Actual satellite sata from local or network repo (transparently)
    • GUI maintained by CEOS
  • Nationalmap.govt.au
    • Aims to publish all Aus OpenData that can be mapped
    • Based on TerriaJS
    • Some DEA data was already being publish but need for additional stuff
  • COGs – Cloud Optimised GeoTIFFs
  • DataCube_ows
    • Lightweight web application server
    • Developed by Datda61 for GA
    • WMS
      • OGC Web Map service
      • Good for general-usage web apps
      • Returns standard images (eg png)
      • Support 1.3 well, works with most clients
      • Styles for band-mapping
      • on-the-fly solar angle correction
    • WCS
      • Version 1 supported
      • Works well with TerriaJS , works okay with QGIS or ArcGIS
  • Next Steps
    • WPS for on-the-fly processing is regularly discussed
    • Better ingegration with datacube-core
    • More recent WCS versions inc WCS-2EO
    • Sparse Data problems

The Tragedy of systemd – Benno Rice

  • References to Contempt Culture
  • Ancestry of Systemd
    • Unix: Happy accident, place, time, reaction to the previous thing
    • housekeeping functions – “mounting filesystems and starting daemons”
    • inetd – Super Daemon for all sockets – “worked well until The Internet Happened”
  • Then the Internet happened
    • forking a process per connection doesn’t scale
    • Lots of persistent state for things like databases
    • Service
      • Might be a bunch of processors
      • Init starts but doesn’t manage
      • initab can restart things in SystemV
  • System Config vs Service bootstrap
    • Mixed in togeather
    • Service management needs more
  • Windows NT
    • Service model there from beginning
  • MacOs
    • Application Model means lot richer interaction with the host
    • Application delegate
    • launchd
  • The Idea of Systemd
    • launchd
      • Service handling in MacOS
      • Took over init, inetd, cron
      • Can listen on ports. Start stuff. doesn’t need to start on boot, boot gets faster, power reduced, security improved
      • Move system services to daemons, then start daemons as needed
    • From Launchd to systemd
      • upstart
        • event driven
        • shell based
      • Rethinking PID 1 – Lennart
      • “Start less” , “Start more in parallel” , “listen to hardware and software changes”
      • cites launchd
  • System management
    • Everything is a lot more dynamic
    • Hotplug , DHCP , etc
    • Don’t install 15 different packages that all behave differently
    • But systemd will have to do things in a different way to those 15 other things
  • The reality of systemd
    • Widely adopted ( 2011 – 2015 )
    • Arguments
      • Violates the unix philosphy – actually systemd actually is many binaries
      • It is bloated and monolithic – Well it does do a lot of thing
      • It is buggy – So is all software, actually a good failure mode
      • I can’t stand Lennart Poettering – He’s delivered. “I won’t defend his community interaction”
      • It is not portable – UNIX is dead – Posix isn’t really a thing anymore, there are not a bunch of crazy Linux variations. “These days you have Linux and some rounding errors”
    • cgroups
    • User-Level units
  • Change – System is a lot of disruptive change
  • The Tragedy of Change
    • Nerds love change as long as we are the ones doing it
    • System boot ups using shell script interaction is like the old blanky we should of got rid of 20 years ago
    • The Knee-jerk – Abuse is not Cool
  • The Next Generation
    • They See a lot more APIs
    • Thinking in Containers is different from thinking in not-containers
  • What does Systemd have that FreeBSD (or even future Linux) could use, or could do better
    • Message Transport
    • RPC Framework
    • Kernel and Use-space services should look similarly to the services above them
    • Service Lifescycle
    • Automation via API – Easier for vendors to write appliances
    • Containers
    • The System Layer
      • Doesn’t have to be the only implementation of theis
    • Consistent Device Naming
    • Better Log/Event/Audit Handling
    • A new model of an application ( a bunch of things managed as a Unit, See the MacOS model)
  • Questions
    • Launchd option – Too MacOS specific
    • Dynamic Libraries = DLL Hell – Containers avoid, different problems
    • Is reaction to systemd scaring other big changes off – Possible, hard to write, very hard to handle the social issues to push though
    • Where is FreeBSD at? – A long way away, no consensus this sort of change needed
    • Should everything have been swallowed up – Thought experiment, If systemd had instead defined an API for separate projects instead of writing them itself, would that have worked? And now we do know what is needed could we switch to a separate model with APIs?
    • Enbeded Devices need systemd – Anything Dynamic needs it
    • What Push back from FreeBSD – Something like that but not systemd. Some like launchd
    • What needs to change in community and systemd team to make things better – See Adam Harvey’s talk on language changes. Hard since everythign is asking for different stuff, systemd people.
    • What should systemd go further into – Messageing and RPC stuff more pervasive and more thought about. Something into the kernel.

Share