Linux.conf.au 2014 – Day 2 – Session 3 – Astronomy Miniconf

Supercomputing and Data Storage Design for the SKA – Stephen Ord

  • Still in design process but not everything finalised
  • Square Kilometre Array
    • Dense core, spread out arms
    • Across Aus and South Africa
    • different antenna cover different frequencies
    • Main goals of array:
      • Galaxy Evolution, Cosmology and Dark energy
      • The first billion years of the Universe
      • Strong field tests of gravity using pulsars and black holes
      • Cosmic Magnetism
      • Cradle of life – complex molecules around distant stars
  • Interferometer – Very heavy proccessing required
  • Lots of data coming in, lots to be archived
  • MWA ( approx 1% of the size of the SKA)
    • 7km of trenching
    • 10kn of electrical cable
    • 16km of fibre
    • 2048 dual pole antennas
    • 42 km of coax cable
  • Way too dense information for me to summarise
  • Next Generation Archiving System
    • Distrubuted, forwards data to various clusters around the world
    • Users can subscribe to tags

 

Desert Fireball Network – with Linux under the bonnet – Martin Cupák

  • Meteoroid (in space), Meteor (in atmosphere) , Meteorite (on earth)
    • Observed from 2 or more locations. 120-30km above surface. Freefall below 30-40km
    • Triangulation, analysis, weather model (can move up to 10km sidewise)
    • Search trip
    • Meteorite!
  • Types: Stoney, Achondrite, stony-Iron, Iron
  • History
    • Manually operated film cameras (since 1959)
    • Automated film cameras (first in 1997, since 2001)
    • Digital cameras (ideas and dev since 2010, operation since 2013)
  • First Meteorite observed, triangulated and found in Czech republic in April 1959
  • Since 2009 15 automated film cameras across 13 stations in Europe
  • Nullarbor network of automated film cameras – 3 stations 2005, 4 stations since 2007
    • 150km apart
  • Based on Industrial PC
    • Old ones P1 , newer P3
    • Microphone picks up sound from meteor going through atmosphere
  • Software
    • Using Redhat 7.3 (released 2002)
    • ntpd sync’d
    • New System using Centos 5.2
  • Digital Cameras
    • Triggered system
      • sequence of hi-speed HR images fired when event detected
      • 11 images/second
      • Complicated design so put aside for now
    • Long exposure system
      • MK-I – 2 built as testbed
        • eBox Pc, 933Mhz, 3xUSB, 1TB HDD, DSLR taking 30s exposures, leo stick uController for power, GPS and camera control, 3G modem, 2x2TB NAS
        • 3G modem lockups
        • Slow CPU
        • SD card unrelaible
        • External HDD a worry
      • MK-II – 10 built, 5 deployed, 60 planned across Australia + USA
        • Industrial PC
        • Video Camera
        • Atom N2600 dual core 1.6GHz
        • 480G SSD
        • 2 X 4TB external HDs
    • Kit camera system for enthusiasts
      • Similar to long-exposure system, possibly lower resolution
  • Goal 60 Cameras, cover 1/3 of Australia. See 15-20 meteorites / year in searchable areas, 500TB data per year
  • Data available for other uses?
Share

Linux.conf.au 2014 – Day 2 – Session 2 – Open Programming Miniconf

Coming home: a return to PHP’s roots by Adam Harvey

  • I arrived late for this but he looked like he was having fun
  • He wrote a PHP micro-framework live in the talk
  • Cute

Get your PaaS into gear by Katie Miller

  • An introduction to OpenShift
  • Platform As A Service, include web servers, databases, storage.
  • Iaas -> Paas -> Saas
  • Why?
    • Focus on code, not config
    • Speed of deployment
    • Convenience
    • Scalability
    • effeciency
  • Flavours
    • Origin – Source can be downloaded
    • online – Hosted in Amazon
    • Enterprise – Run you own supported
  • Runs on RHEL
    • Runs in containers
    • Cartridges (programming environment) within the container
  • Tissues!

The Cobblers Children Have No Shoes: Development Tools and the Unix Philosophy by Russell Keith-Magee

  • Django Guy
  • Over last 30 years – My tools have got appreciable worse
    • Libraries, languages and computers better
    • Programming tools have not
    • Some have got worse
    • Not language specific
  • Debugging
    • Borland C – GUI – see where in code you are
    • 1994 gdb
    • 2013 gdb
  • Why?
    • The Unix philosphy – small, seperate tools
    • Cargo Cult
      • Powerful tools only need a CLI
  • Fully featured IDE?
    • Only one tool the GUI
    • Maybe a plugin interface, if you are lucky
  • Keep good bit of Unix philosphy
    • Different Window dressing
  • BeeWare
    • Common problem with bad interface – testing
    • Current test suite outputs little to screen until it finishes.
    • Cricket – runs test suites and lets you look at output while it is running, see progress, what is passing/failing, can browse errors, does just one thing
      • cross platform – uses what Python provides out of the box, uses tk
      • Zero configuration – uses pip and run from command line. Nothing set in tool
      • Integration
    • Duvet – Test coverage tool
    • Why not a web interface
      • Needs to have a server
      • Live updates are still ard
      • Don’t believe web browsers will be superior to native tools and time soon
  • pybee.org

 

A Pseudo-Random Talk on Entropy by Jim Cheetham

  • Random vs Pseudo-Random vs Cryptographically Secure Pseudo-Random
  • Some Pseudo-Random generators not as random as was expected
  • Good PRNG – Mersenne Twister,
  • Blum Blum Shub – Crypto good if you pick right numbers
  • Diehard test spot bad random numbers
  • Random numbers for science – use publish “table” of pre-generated ones
  • /dev/random very random enviromental noise , /dev/urandom not crypto-good
  • /dev/random should block if not enough random bits available
  • RdRand – Source of random data built into chip, measures physical noise, runs through AES, can’t acess original data though
  • Sources from various hardware generators
  • Turbit – reads noise from soundcards
  • OneRNG – Speakers project, Open etc
Share

Linux.conf.au 2014 – Day 2 – Session 1 – Open Programming Miniconf

Keeping Current: Maintaining an internal fork by Benno Rice

  • FreeBSD contributor, works for EMC isilon division
  • OneFS file system runs on appliances FreeBSD
  • OneFS = FreeBSD + stuff
  • Various add-one. Pretty interfaces, file system, etc
  • Some changes upstreamed but not others
  • We maintain a divergent fork of FreeBSD

Strategy:

  • Don’t maintain a fork
    • Version is old. 7.3 from 2010 plus backports
  • Don’t get behind
    • Need to catchup and stay caught up
    • Track FreeBSD-current
    • Do our own releases
  • Don’t track branches
    • upstream as much as possible to minimise diffs
  • Upstream as much as possible
    • Code checkout 5GB
    • Pile it all up approach approach – bulk merge and fix things – no timeline – no way to test
    • Pretend we use git approach
      • merge changes from bsd and apply
      • subversion couldn’t do directly, fixed with script
  • If your tools don’t do what you need fix them
    • worlds ugliest rebase – merge in all changes since 7.3
    • eventually got there
    • merge 20 changesets and then test a bit. bisect changes
    • 9 weeks of work to get to FSB v8
  • Don’t Stop
    • Got to keep tracking it forever
    • need to work out when to do releases

 

USB and Android by Joel Stanley

  • Last 3 years making consumer electronic devices, all talk to USB
  • Android as a tool
    • cheap, portable, lots of connectivity
    • No licensing worries vs laptops
    • Lots of features
    • Works out of the box
  • USB APIs as of Andriod 4.0
  • Normally runs in peripheral mode – we are interested in host mode
    • Most hardware requires a dongle
    • ADB over TCP to debug
    • Asus Transformer
  • Architecture
    • libusb  – std Linux
    • app.so – Same as linux version
    • C wrapper – app code created and mait by C, simplyfyin Java
    • JNI
    • App Classes – calls into C code
    • User Interface
  • Java Naive Interface – JNI
    • C can call Java
    • Jave can call purpose-written C-code
    • Hard to get started with, look at others people code off github
  • Native Development Kit
    • C/C++ cross compiler for targeting Android
    • Provided by Google
    • Used it to compile shared libs
  • Android permissions
    • Limits access to device files
    • No auto access to hardware, can’t hard-code since USB is dynamic
    • Get permission in Java code and then change permission so C code can use
  • https://github.com/shenki/FreeDV-Android

 

Developing OQGRAPH, a tool for graph based traversal of SQL data in MariaDB by Andrew McDonnell

  • Graphs = nodes + edges/links
  • Graph alogrithms – Breadth first, depth first, shortest path
  • Graphs in the “real world” – maps (how does GPS work), relationships (linkedin, facebook), trees
  • MariaDB is fork of MySQL
    • Being adopted a various distributios
    • Used by big companies
  • Graphs in a SQL RDBMS
    • store a graph as a list of edges
  • OQGraph
    • Creates “latch” column that contains a special command executed by the storage engine
    • where latch=’breadth_first”
    • Project hosted on lauchpad

 

 

Share

Linux.conf.au 2014 – Day 2 – Keynote

Kate Chapman– OpenStreetmap for disasters

  • Background
    • Most maps people used are not free
    • “The Wikipedia of Maps”
    • Goal: A free map of the whole world
    • ODbL – Like CC-BY-SA
  • Humanitarian Openstreetmap Projects (HOT)
  • 1st official activation in 2009 for Gaza
  • Activatation is a response from the community to a callout
  • Port-au-prince Jan 12 2010
    • Tracing existing satellite photos
    • Being used by people on the ground within a week
    • Geographic agency build collapsed and many staff killed in earthquake
    • Data released by agencies used for maps
    • Also covered Northern Haiti which wasn’t as affected by quake
    • Created NPO to organise
  • Project to map Indonesian ahead of time since disaster-prone country
    • Map once, use for many things
    • Pilot project – West Nusa Tenggara
      • Worked with community to collect data, including demographics data
      • Some existing maps were hand drawn or used “dumb” tools like coral-draw
      • ASM tools distributed
      • Agreement from Govt, NGOs, Companies to make best mapped province in Indonesia
    • Mapping Jakarta
      • 267 Urban villages
      • Talked to each head, get person to map data, University students to input the data
      • Varying levels of data quality, in-putted by students
      • Flooding in Jan 2013 – Map uses as basis of maps to show areas affected
      • Creating a training program (training the trainers)
    • openstreetmap.or.id
    • Data collected (often by Scouts), printed out maps, got people to walk around fill in gap son paper. Add data to online version
    • Used to create impact models for future floods
    • 3 part training manual created (English and Indonesian)
  • Eurosha
    • Training people to go work for 6 months in various African countries
  • Other projects to map 200 asian cities that are prone to disasters
  • Senegal – training and workshops
  • Project in Japan in Tsunami affected areas
  • Mapmill –
    • Civil Air Patrol takes GPS tagged photos of disaster areas
    • People sorted images by damage None/some/bad
  • American Red Cross
    • Fire risk map in Uganda
  •   Philippines – Typhoon Haiyan hit Nov 2013
    • Mapped 10,000 buildings in Tacloban before Typhoon hit
    • Website allows people to login and see various tasks that can be done
    • Post-event Satellite Imagery needed
    • US State Dept released satellite imagery for 30 days
    • Able to use photos to do before-vs-after maps on damage
  • github.com/hotosm
  • digitalhumanitarians.com  – various groups
  • Volunteer in Person for a Project – See website for lists
  • hot.openstreetmap.org
Share

Linux.conf.au 2014 – Day 1 – Keynote

Keynote – Dr Suelette Dreyfus

 The Surveillance state

  • Our state is out of balance, to much state power
  • Evolution to the Post-Snowden World
  • Era of the Whistleblower – Reveals information from inside an organisation
    • Doesn’t have to be a traditional employee (eg contractor)
    • about serious wrongdoing
    • to someone they believe can do something
  • Today’s public perception of whistleblowers
    • Most people think “normal” or “heroes”
    • http://whistleblowingsurvey.org
    • About 50% of population too much secrecy within organisations
    • 80% support people should be supported revealing inside information
    • 87% support someone should be able to go to the media
  • Why the change from traditional views which were much more anti-whistleblower?
    • Whistle-blowers, publishers, journalists, academics are harassed, stopped at borders. Live in exile from UK, USA, Australia. Live in Russia

What has the state been doing

  • Reached a state of security saturation
    • So much money being spent it is being wasted on WoW and stupid places
    • Extra money being spent produces no additional useful data

Case Study – The inside platform

  • Cradle to graduation state tracking of children in Victoria – she used “our children”
  • Move assesment to one-stop-shop
  • Track individual progress of children, to provide picture of child, tailor teaching
  • Learning disabilities, behavioural problems, all in one place
  • What about religious schools, will church attendance and attitude be recorded?
  • Who has access, how long retained, who will it be shared with? Kept overseas?
  • Lots of rules around heath data but none in this tender (hard to tell since tender not publicly available)

Sample of Hardware exploit hardware that can be installed on Dell servers

  • Examples for Dell servers
  • Multiple vendors targeted
  • Hacks for Phones and SIM cards

Status of The Surveillance State

  • Bulk Collection – Met, still to be cross linked
  • Ongoing Storage – Mostly Met
  • Capture of all state data – Not yet fully linked
  • Co-option of big-data players – Being worked on, some push back
  • Legal – Met
  • One-View of citizen across all interactions across lifetime – Getting closer

What you can do

  • “In a time of universal deceit, telling the truth is a revolutionary acts” – George Orwell
  • Get Political
  • Write privacy-enhancing software, write defensive/detection software for ordinary people
  • Get involved in NFPs that give tech support to journalists, NGOs and average citizens
  • If you work for the Govt, use your voice
Share

Wikipedia vs Mana Party

Local elections are happening in New Zealand in a few weeks. As part of it people from the parties are active online.

On September 6th a wikipedia account called “manamangerecreated a Wikipedia page for James Papali’i who is standing for Auckland city council seats on a Mana-linked ticket (Mangere is the suburb of Auckland near the Airport).

However a regular New Zealand Wikipedia editor made this edit to the article who linked back to a 2006 conviction that Papali’i had received for fraud.

“manamangere” is now asking that the article be removed since if it includes the fraud conviction it does more harm than good. In reality it will probably be removed since Papali’iis below the level of fame that usually gets an article.

 

Share

Newsblur RSS reader review

On March 13th 2013 Google announced that they were shutting down Google Reader from July 1st. As a big RSS user I started looking around for an alternative a few days later and settled on using the web-based Newsblur. Now I’ve been using it for close to a month I thought I’d give a quick review.

How and Why RSS?

Before I do the actual review I thought I’d give a quick overview of Why I use a RSS reader. I think Marco Arment put it well when he wrote:

RSS is best for following a large number of infrequently updated sites: sites that you’d never remember to check every day because they only post occasionally, and that your social-network friends won’t reliably find or link to.

Which is very much the case with me. I read relatively obscure but interesting blogs that are not massively retweeted or linked to. I read enough of them that it isn’t practical to visit them every day to check for updates (which may only happen every few weeks) and I follow enough people on twitter (145) that I will sometimes skip a few hundred tweets that happened overnight so I could miss a tweet announcing a blog post (assuming I want to follow that persons tweets).

A specific use case I saw is for web comics. Instead of me checked the half-dozen web-comics I follow every day for a new stripe they all appear automatically in my reader.

Getting Started with Newsblur

To switch from Google reader to Newsblur was fairly simple. From Google reader I went to “settings” and then “Import/Export” and exported my information via Google “takeout”. Google then gave my a zip file which contained a subscriptions.xml file and some .json files.

I then setup a Newsblur account. Currently Newsblur offers free accounts and paid accounts. Free accounts are limited to just 64 feeds being active and update less often. Paid accounts cost $US 24/year and have unlimited sites, more updates and some sharing options. Once you create your account you can import you subscriptions.xml (call and OPML file) and you will see you previous subscriptions. After a couple of days using it I signed up to a one year subscription.

If you don’t have an old RSS reader you can just grab RSS feeds from sites you read. Look our for the RSS symbol or use your browser to find the RSS URL of the sites you want to follow.

Newsblur’s Interface

I found Newsblur’s interface to a little clunky but usable.

Newsblur Interface

The left hand sidebar has my various feeds (the number is how many unread articles are in each) . The main pane on the right has the article I’m reading and the bottom right pane has the list of articles in my current feed (just the one I am reading in this case). The go to feeds you click on the name of the feed on the left and it loads in the right-hand panes.

The interface above shows i follow 141 sites and I have 225 unread articles. The “Ars technica” site (2nd from the top) is an example of a site I’ve used the built-in filtering function with. There are 37 articles I’ve filtered to “good”, 46 filtered to neutral and another 77 completely hidden ones I’ve filtered out. The filtering is fairly simple and works on keywords, authors or titles of articles. Sites which have good tagging are easiest to filter while those that don’t (eg slate.com) are a lot harder. This is a feature that Google reader never implemented and is very useful for readers sites where their are whole groups of posts you want to exclude (eg gaming articles on Ars Technica).

One of the bugs with the interface is that their are a lot of different option menus ( the “cog” at the bottom left, the “cog” in the feed pane next to the feed’s name, right clicking on  a feed name) which are a bit confusing. There is even a dev.newsblur.com which has a preview of the upcoming interface . At least once I’ve switched the the “dev” site to fix a setting I couldn’t change in production.

Overall Impression

I’m fairly happy with Newsblur as a replacement for Google reader.

  • Immediately after Reader announced their shutdown the site was overwhelmed with new customers but over the lat month they have expanded capacity and are now pretty stable and fast
  • The site owner (1 person does everything) is fairly responsive to bug reports and help requests
  • The interface is improving and being worked on
  • The company appears to have a sustainable business model
  • There are a good set of short-key keys to speed up reading.
  • The filtering is very useful.

A few rough areas

  • The interface is still a bit of a mess and has some bugs, hopefully the new release will improve
  • I’ve found a few feeds are completely broken on Newsblur (eg James Hamilton’s Perspectives) but seem to work okay when I test them directly. Unfortunately Newsblur probably doesn’t have the manpower to fix these one by one.
  • If you are a big using of the sharing and highlighting functions of Google Reader then Newsblur’s equivalent may be short of what you need. I don’t use these however.
  • There is no “sort by magic” function that Google Reader had, but this filtering is a good replacement in most cases.

But I think the site is more than usable and getting better and I definitely recommend it to others.

Share

Links: Spinning Levers, Redis, College a bad investment, Playing the World Champ

Share

lca2013 – Day 4 – Session 2

Git For Ages 4 And Up by Michael Schwern

It is not you, is really is complicated

It is easier to understand git from inside out, cause the interface is so far…

Getting started – init and clone

  • Entire repo in .git directory after “git init”

Getting stuff done add and commit

  • add writes to the repo
  • commit creates a commit object and puts labels on it
  • take a new copy this file.
  • Add creates a new node, commit attaches a commit-object to it and moves the head and master

Branching

  • git branch feature – added feature label to current node
  • git checkout feature – next commit will move feature label
  • git commit – moves the feature label and the head
  • head always points to what you have checked out

Commits

  • Every one is unique and has unique id
  • They never change
  • content, author, date

Staging area / Index / Cache

  • Place to build up stuff before a commit
  • eg what is in “add”

Workflow

  • Isolate -> work <-> update -> share
  • git checkout master – move back to master branch, can’t see stuff in branch
  • get merge feature – merge in the branch “feature”
  • git log –graph –decorate all
  • reset – “arbitary move labels around command”
  • Merge master back into feature
    • git checkout feature
    • git merge master
  • Unlike cvs “commit does not meet share”

Working with others

  • git clone remote work – clones a remote into work folder
  • Normally “remote” will be the URL of the remote repo
  • origin/master – label to commit where git thinks remote currently is.
  • pull == fetch + merge
  • git checkout -b bugfix  – checkout and branch
  • git push origin bugfix – push to the branch “bugfix” on the remote “origin”
  • git push -u origin bufix – the -u sets the default branch to push to on the remote “origin” for next time

Tags

  • git tag v1.0 b08234
  • Add any old random tag to a commit, defaults to current commit you are on
  • cannot move tags except with force
  • can branch from tags

Rebase

  • git rebase -i  – interactive mode
  • squash
  • creates a new commit ( hanging off parent of commit your are rebasing ) and move branch to it, leaves previous branch hanging detached
  • Don’t rebase after you have pushed
Share

lca2013 – Day 4 – Keynote

Bunnie – Linux in the Flesh: Adventures Embedding Linux in Hardware 

As CPU speeds growth has stall mobile CPUs have caught up with Desktop CPUs

Cost of Mobile CPUs $20 on intro vs $X00 on intro for desktop CPUs

Time spend making a product

  • 11% on Hardware design – schematic and PCB layout
  • 20% on product and software dev
  • 33% in mass production ramp

Giving Linux a body

  • Consumers don’t like bare boards
  • Comprehensive 3D models
  • Build injection molds – Steel cut, EDM tidy, Hand polish
  • Around $20k to create tool
  • People like Apple will start multiple tool runs and keep the best
  • General rule – $99 product you want to sell wholesale for $30
  • Magic prices products sell at $99, $149, $199
  • 90% of hardware sales between Black Friday and Christmas. Can’t ship just after Christmas

Chellenges of Embedding Linux

  • Embedded CPUs have lots of weird bugs
  • Bootloaders
    • CPU-Specific media loaders
    • Firmware has raw Iron – NO MMU, No DRAM, etc
  • Small memory footprint
    • Linux runs just a few MB of memory
    • Every $ counts, lots of time fixing code
  • Flash File Systems
    • Few 1000 read/write cycles
    • No write g’teed to succeed
    • Power-down corruption is a fact of life, need backup or read-only copies
  • Power Management
    • Reduce frequency and voltage when CPU is lightly loaded
    • Need to adjust for high and low load
    • Rouge processes can chew CPU/Power
    • Lots of auditing needed
    • User apps are hard
  • Suspended / Sleep hard
    • Drivers might be written with this in mind
  • Huge number of devices that can be plugged in, lots of drivers to verify
  • Dev community
    • Smaller pop that desktop
  • Updates and patches and rollout and distribution hard to do, must do yourself
  • Software is the major bottleneck – cause it is easy, you can do anything
  • System architect must create a useful cage for the software dev

Case Study: Robot Vision

  • Kovan
  • Needs to do image processing
  • Lots of choices
    • Using FPGA on PCIe uses up all of memory bandwidth in small systems
    • Connect via USB means it is just a “camera driver”
    • Connect between Soc and the Screen
    • Or between SoC and the DRAM
  • Ended up between SoC and LCD
  • Ends up as /dev/fpga
  • Handle multiple types by probing

Case: NeTV

  • Goes between device and screen
  • Overlay stuff over video
  • But signal is all encrypted
  • Sync Pixels and replace one by one using Man-in-the-middle attack on encrypted signal!!!
  • Very hard to get timing, drift of 100s of pixels per frame
  • Source clock from the incoming signal
  • A few other tricks

Dev Enviroment

  • Created Amazon AMI to help people get up and going
  • But hardware devs not used to using these
  • Not a 100% fit to normal dev model
Share