Linux.conf.au 2014 – Day 4 – Session 1

Programming Diversity by Ashe Dryden

  • What is diversity
  • More than gender
    • backgrounds experiences and lifestyles
    • not always visable
    • sexuality, age, language, class, race, ability
  • Terms
    • Intersectionality
      • The interaction of traits (race, sex, etc) and how people treat you beacuse of that
    • Privilege
      • unearned advantages a person gets for a perceived trait
      • Education, access to technology, higher pay, assumed competency, quality of network
      • Seen as a skill-set instead of traits
      • Easily fit/identify with subculture
    • Stereotype Threat
      • Worry you will confirm the stereotype that is applied to you
      • Lots of pressure
    • Imposter Syndrome
      • Unable to internalise their accomplishments
      • almost anyone can suffer
      • less likely to apply for jobs, apply to talk at conferences or even attend conferences
    • Marginalised
      • Doesn’t fit into the default groups
      • their needs or desires being ignored
      • Even marginalised groups marginalise others, nobody is trait blind
  • Women are about 20% of tech
  • Maybe Women aren’t into programming
    • Women like Grace Hopper prominat in field early
    • No physical of biological difference in race or gender affecting programming ability
  • Bulgaria
    • 73% of CS Students are women
    • teach children in schools that STEM is important to everybody, push everybody towards it
  • Diversity matters –
    • Companies that are more diverse tend to have better sales, profits, etc
    • Diverse teams:
      • solve complex problems faster
      • more creative and stimulated
      • get better decisions and generate for something
      • financial viability and success
  • Why lack of diversity?
    • Pipeline
      • Difference in toys and games for boys and girls
      • no famous role models that represent them
      • Access to technology. On average Boys get first computer at age 11, girls at age 14. Early teens great best age to learn and retain skills
    • Geek stereotypes
      • people who don’t identify and aren’t represented by the geek stereotype are turned off by those who do
    • Attrition
      • 56% of women leave tech in 10 years
      • twice the rate of men
      • our Grandmothers more likely to be programmers than our granddaughters are
    • Why attrition?
      • Harassment
      • People in marginalised groups twice as likely to report being harassed or mistreated
      • men 2.7 more likely to be promoted to higher ranking positions
    • Why can I do about this stuff?
      • Change starts with us
      • educate people who don’t understand this problem
      • Get to know people different from us – talk to people wearing a specific color that day
      • Follow people on twitter that are different from you
      • bias & discrimination are often subtle
      • learn to apologize
      • Talk about these issues openly ” That’s not cool πŸ™ “
      • increase education and access
      • Facilitate event for marginalised groups
      • work with colleges and universities to remove bias
      • “have you programmed before?”
      • Thinks about what the “about” page of your website looks like
      • Think about the company culture
      • Job listing language and requirements – joblint.org
      • Interviewing
      • equal pay
      • mentoring and career goal attainment

 

From Kookaburra to the Cloud: where to now for copyright in Australia by Ben Powell

  • Several recent cases
  • Australian law deals by exception, under copyright except where “fair dealing” , “fair use” etc allowed specicly by law
  • ALRC Review
    • More exceptions or general “fair use”
    • Report not yet tabled, but interim discussion paper released
  • Kookaburra
    • Song from 1932
    • “Down under” 1981
    • Nobody noticed till 2007 when on TV Quiz show
    • Court decided infringing
    • Two culturally significant songs
  • Fair Use vs Fair dealing
    • Fair dealing has specific exceptions
    • Things are not fair dealing
      • Sampling
      • non commercial use of incidental music
      • memes
      • commercial services to allow recording in the cloud
      • stoarage of copyright material
      • copying DVD to other devices
      • search engines (thumbnails)
      • digital archiving
    • More exceptions?
      • Quotations
        • in the Berne Convention
        • anachronistic term
        • doesn’t cover transformation, implies verbatim use
      • Transformation
        • not a substitute for the original work
        • low threshold – undermines creators rights
        • high threshold – confusing, how much change needed
        • How does commercial use fit?
        • Hard for court to decide
      • Private and Domestic use
        • Format shifting and time shifting exists already (VHS only, not DVD)
        • doesn’t cover the cloud
        • not technology neutral
        • Canadian more technology neutral but “non-commercial” bit heard to define
    • Fair Use
      • See US Copyright Act
      • Fair Use in Australia
        • Fairness facter
        • illustrative uses (non-exhaustive)
        • flexible defence, weighing up the factors
      • Advantages
        • Balance
        • Flexible
        • aligns with community expectations
      • Against Fair Use
        • Uncertainly ( parliament vs law)
        • Requires litigation
        • Originated from different legal enviroment
      • The reply to objections
        • Uncertainty – See normal consumer law with terms like “unfair contracts” , “misleading and deceptive conduct”
        • Different legal env – same common law roots, AUSFTA meant to “harmonise” copyright law.
        • International Law , 3 step test – The US gets away with it, never chellenged
  • Govt unlikely to go forward with fair use based on their leanings
  • The introduction of a Fair Use defence would encourage Australian innovation
  • “General the US likes to export the ‘bad parts’ of it’s copyright law, not the ‘good bits’Β  “

 

Share

Linux.conf.au 2014 – Day 4 – Keynote – Matthew Garrett

Matthew Garrett

Security in the modern world

  • 2013 was an interesting year
    • UEFI Secure boot was deployed to the masses. On most PCs by default
    • ..and vendor implementations promptly broken
    • Snowden revelations
    • First highly visible hypervisor related compromise?
    • ..no it turns out
  • Who are we protecting against?
    • The NSA?
    • Our Hosting providers?
    • Opportunistic attackers?
  • Imperfect security is better than no security
  • NSA
    • Leaked material is from 2007-2008 so don’t know how is advanced
    • No evidence that the entire stack has been subverted
    • Leaked material describes model-specific (rather than vendor-specific) exploits
    • Plausible that the vendors aren’t actively involved
      • although passive involvement is likely
    • Would it be in anyone’s interest to have a generic exploit?
  • Intelligence agencies are probably not your biggest concern
  • Most security compromises are either political or profit driven
    • But that doesn’t make the user feel better
  • What can we do to protect users
  • Protecting the entire chain
    • Boot verification is an absolute requirement
      • OS’s are just too big to be perfect
      • Persistent infections (of boot process) make recovery impractical
    • …but so is user freedom
      • Stopping users from building/booting their own kernels is not a good long term situation
      • …ideally including the firmware
  • Where do we stand
    • UEFI Secure boot on x86 systems
      • Guaranteed that user can replace keys
      • No guaranteed that the user can replace the fireware
    • Andriod
      • Primary compute device for many
      • Some permit the user to replace the OS
      • No ability to replace keys or fireware – cannot boot your own signed kernels
      • Need to push vendors to provide replacement of OS and keys
    • Apple
      • No ability to replace OS, keys or fireware
  • How much can I trust my system
    • OS backdoors
      • Doesn’t really seem necessary, too many holes already
    • Firmware backdoors
      • Why has nobody audited the Jetway (leaked BIO and fireware) leak?
    • Lower Level?
      • AMT, CPU microcode
      • AMT has a lot of access to running and even turned off systems, Intel would be greatly embarrassed
      • CPU Microcode – could be updates by OS-level exploit
    • It’s fine, all my data is in the cloud
      • What even *is* the cloud
      • If you are giving you data to someone else you are trusting them not to lose it or steal it
      • …History suggest this is not a good idea
      • But this is still a spectrum
        • Running you server means you trust all your software
        • Running a VM means you need to trust the hypervisor and other guests
        • ..do you trusts those guests .. do you trusts those guest will be unable to compromise the hypervisor
      • Questions to ask you cloud providers
        • What secuity to isolates guests? selinux over kvm perhaps?
        • How do you manage hypervisor updates in response to security issues?
        • my mechanisms do you have to detect compromises to the hypervisor?
        • what is your response to to finding a compromised device?
      • Can you trust them at all?
        • Introspection of the bare metal is hard
        • Introspection of VMs is trivial
        • Virtualisation requires different security considerations than bare metal requirements, more attacks
  • Security in 2014
    • Be more agressive about securing every layer of systems
    • .. but do so in a way that ensures users don’t have to choose between freedom and security
    • Start asking cloud vendors hard questions
    • … and their customers, too
    • Security and free are two sides of the same coin
    • Don’t buy into any narrative that asks you you to give up one for the other

 

Share

Linux.conf.au 2014 – Day 3 – Session 3

Continuous Integration for your database migrations by Michael Still

  • Running unit and integration test on all patches
  • Terminology
    • sqlalchemy – The database ORM the Openstack nova uses
    • Schema version: a single database schema, represented by a number
    • Database migration: the process of moving between schema versions
  • Motivation
    • Test weren’t testing upgrades on real large production data
    • We found the following things
      • Schema drift – some deployments had schemas that wenr’t possible to upgrade because they didn’t match current tools
      • Performance issues – Some upgrades took too long
      • Broken downgrades – didn’t work for non-trivial downgrades
    • Are downgrades important?
  • Turbo-hipster is a test runner
    • A series of test plugins
    • Register with Zuul
    • Runs task plugins when requested, return results
  • Task Plugin
    • Python Plugin
  • The DB upgrade plugin
    • Upgrade to tunk
    • Upgrade to the patch
    • Downgrade to the 1st migration in the release
    • Upgrade again
    • Pass / fail based on analysis of the logs from the shell script
  • Lets go made with plugins
    • Email people when code they worked on is changed by others
    • Cause doc bugs to be created
    • Cause dependant patches when a patch requres changes to “flow down” repos.
    • Much already in Gerrit but does it need to be?
  • OMG Security
    • This is a bit scary
    • We’re running code on our workers provided by 3rd parties
    • Mitigation
      • Limited access to nodes
      • untrusted code tested with network turned off
      • checks logs for suspicious data
      • We’re working on dataset anonymousation
  • Running a process with networking turned off
    • Explored LXC (containers)
    • netns is much simpler
  • Interesting Bugs
    • Slow upgrade -> Dev iterated his code multiple times ran against the test until was fast enough
  • Would be happy to do this with Postgres if Postgres community wants to help get it going

 

Live upgrading many thousands of servers from an ancient RedHat 7.1 to a 10 year newer Debian based one by Marc Merlin

  • Longer version http://marc.merlins.org/linux/talks/ProdNG-LCA2014/
  • Google Started with a Linux CD (in our case Red Hat 6.2)
  • Then kickstart
  • updates had ssh loops to connect to machines and upgrade
  • Any push based method is doomed
  • Running from cron will break eventually
  • Across thousands of machines a percentage will fail and have to br fixed by hand
  • File Level syncing
    • makes all you servers the same
    • Exclude a few files (resolv.conf, syslog)
    • Doesn’t scale well but he can have rsync-like software that doesn’t something similar
  • All servers are the same
    • for the root partition yes
    • per-machine software outside root parition
    • static links for libraries
    • hundreds of different apps with own dependencies
  • How to upgarde root partition
    • just security upgrades mainly
    • running Redhat 7.1 for a long time
  • How to upgrade base packages
    • upgrade packages, create and test new master image, slowly push to live
    • only two images in prod, current and the old one
  • How about pre/post installs?
    • removed most of them
    • sync daemon has a watch on some files and does something when that file changed
  • How did running 7.1 work out?
    • It works a long time but not forever
    • Very scary
    • Oh and preferable not reboot the machines if at all possible
  • What new distribution?
    • Workstations already moved to debian from redhat
    • Debian has more packages
    • Ubuntu is better than debain so started with Ubuntu Dapper
  • Init System choice
    • Boot time not a big decided
    • Consistent Boot order very useful
    • systemd a lot of work to convert, upstart a lot too
    • systemd option for future
  • ProdNG
    • self hosting
    • Entirely rebuilt from source
    • Remove unneeded dependencies
    • end distribution 150MB (without google custom bits)
    • No complivated upstart, dbus, plymouth
    • Small is quicker to sync
    • Newer packages not always better, sometimes old is good, new stuff as lots of extra stuff you might not need
  • How to push it
    • 20k+ files changed
    • How to convince people it will work, how to test?
    • push hard to do slowly, have to maintain 2 very different systems in prod
  • Turned into many smaller jumps
    • Take debian packages into rpms and install on existing server one at a time
  • Cruft Removal
    • Get rid of junks, like X fonts, X server, random locales
    • Old libs nothing is using
    • No C++ left so libstdc++ removed
  • One at time
    • Upgrade libc from 2.2.2 to 2.3.6
    • Upgrade small packages and work up
    • 150 packages upgraded a few at a time. took just over 2 years
  • Convert rpms to debs
    • Same packages on both images
    • Had to convert internal packages from rpms to debs
    • used alien and custom scrip to convert.
    • changelogs have more fixed format in debs than rpms
  • Switch live base packages everything back to debs
    • Only one major bug
  • Lessons learned
    • If you are maintain a lot of machines if you have your own fork you can remove all the bits you don’t need
    • Forcing server uses to use an API you provide and not to write to the root FS
    • File level sync recovers from any state and is more reliable than most other methods
    • You can do crazy things like distribution switches
    • Don’t blindly install upstream updates
    • If you don’t need it remove it
    • You probably don’t want to run the latest thing, more trouble than it is worth
    • Smaller jumps is easier
    • the best way to do a huge upgrade is a few packages at a time

 

 

Share

Linux.conf.au 2014 – Day 3 – Session 2

HTTP/2.0 And You by Mark Nottingham

  • Why?
    • Web heavier and more interactive than it used to be
    • Average size and elements on each page have doubled over last 2 years
    • People have found bad performance affects attention from users
    • Latency is bad on mobile and mobile is growing
  • Current Techniques
    • Spriting images
    • Inlining – encode images directly in CSS
    • Sharding – multiple hostnames
    • Concatenation – jam all js or css into one file/download
    • HACKS!
  • “Eliminate Requests”
  • Why are HTTP request so expensive
    • HTTP/1 uses TCP poorly
    • Head of Line blocking – requests/responses have to be ordered within each TCP connection. Slow one blocks others
    • HTTP request short and bursty, TCP was built for long lived flows
    • TCP slow start
  • Therefore HTTP uses multiple connections
    • Increases congestion events
    • resource intensive on server
    • all of this is really tricky for clients (which request to which connection in which order)
  • Http headers are verbose
    • Same data sent multiple times from request to request
    • Large request headers split across multiple packets
    • 7-8 round trips just to get page loaded
  • SPDY was the starting point
    • Previous efforts ( Opera Turbo, HTTP-NG, Waka )
  • GOAL: One TCP connection for a page load
    • longer lived
    • less resource intensive
    • more fair
  • Protocol
    • Frames – settings, header, data
      • Multiple stream IDs so data and headers from different request can be mixed on same connection
    • Prioritisation and flow control
      • Priority field
      • session level and stream level control
      • WINDOW_UPDATE frame
    • Header compression
      • 1st proposal gzip
        • One compression context for all headers in each direction (don’t redo dictionaries)
        • Very efficient, easy to impliment
        • Some memory overhead
        • But CRIME attack allowed attacker to inject data
      • HPACK instead
        • Coarse-grained delta-coding
    • Server Push
      • Push URLs straight to client in anticipation it will be needed (eg CSS, js after page requested)
  • About a dozen implimentations
  • How will it affect me?
    • 25% page size saved
    • Multiplexing allows a lot better use of network
    • HTTP semantics won’t change, but leaked abstractions will
    • Less “Best Practises” for Perf
    • Rethink connection handling – load balances
    • Now a binary format
    • TLS compulsory (effectively since major browsers will make it) πŸ™
    • Getting most out of protocol will take effort
    • RESTful HTTP APIs , lower request overhead, BATCH operations no needed
  • TLS still being looked at for small/medium operators
  • http://github.com/http2/

 

Reverse engineering vendor firmware drivers for little fun and no profit by Matthew Garrett

  • Deploying Servers is tedious. Fireware config often needs keyboard/screen needs to be connected
  • Automated server deployment is awesome
  • Different mechanism
    • Serial console (can be automated, but very hard)
    • Web services console
    • Vendor-specific method
  • The vendor tool
    • Spits out XML file
    • You Modify file in your editor etc
    • Reads modified file
    • 250KB binary
    • 32-bit only
  • Matthew’s company didn’t have 32-bit libraries
  • strace to the rescue
    • Assumed it was going to use /dev/ipmi – but it didn’t
    • No /sys/bus/pci access
  • MMIOtrace to the rescue
    • No access to PCI BARS either
  • This tool does not use any kernel-provided hardware access
  • strace logs show oipl()
    • this should only be for very simple stuff. app should not access hardware bypassing the kernel
  • gdb
    • find inb/outb functions
    • set breakpoints
  • Was accessing the 0xcf8 / oxcfc PCI configuration register
  • Not just PCI config space, was also doing CMOS access
  • But wait there is more!
    • Some options didn’t trigged the breakpoints above
    • Step though gdb
    • Wait, hang on that is not my address space
    • /dev/mem being opened and mmap()ed
    • Executing BIOS code in process context
  • LD_PRELOAD
    • Trap iopl()
    • Install segfault handler
    • Wait for trap
    • decode instructions around instruction pointer
    • raise privs, execcute, drop priv
    • increment ip
    • continue
  • What is it doing
    • Accessing io ports on the IPMI controller
    • Mirror of the PCI config space registers
  • So where does the XML come from
    • Staring at the output of “strings”
    • Undocumented debug flag
    • When you set the program printed out everything it did.
  • DMI
    • DMI table contains pointer to memory
    • map that memory
    • find another table
    • parse that table
  • Summary
    • Tool access PCI config space in racy manner
    • Tool access CMOS space in a racy manner
    • Tools executes BIOS code from userspace
  • A shockingly happy ending
    • Communication with vendor
    • Work on improving this in future
    • Victory!

 

 

Share

Linux.conf.au 2014 – Day 3 – Session 1

Systems Administration in the Open by Elizabeth Krumbach Joseph

  • Works for HP , paid to work on Openstack project
  • Normally infrasture is maintained by a team
    • You interact with them via tickets and email
    • Priority determined by the team
  • Infrastructure team at open stack
    • CI system
    • Wiki, website, IRC bots
  • Everything is in a public git repo
    • Puppet modules
    • git.openstack.org
    • Anyone can submit patches
  • Openstack code review and CI chelleganes
    • Lots of individual projects (infrastructure is just another one)
    • Syntax checks, testing automated, changes should never break the master
  • Using
    • launchpad
    • git
    • gerrit
    • zuul
    • gearman
    • jenkins
    • nodepool
  • Probably don’t need that much at most places, but already used by other projects
  • Anyone on the Internet can look at our changes and/or do code reviews
  • checks:
    • puppet pareser validate
    • puppet-lint
    • XML
    • Alphabetized project files
  • Peer review
    • Multiple eyes on changes prior to merging
    • Good infrastructure for developing new solutions (particularly for distributed teams)
    • Trains us to be collaborative by default
    • No special process to go through commit access
  • Changes get checked in
    • Either puppet master gets updated and applies change
    • Or vcsrepo module in puppet pulls in latest version of the project
  • Can you really manage via git commits
    • Cacti to keep eye on server usage
    • Puppet dashboards so you can watch your changes get applie(or not)
    • Thorough, specific documentation at http://ci.openstack.org
  • Sometimes you need to login to a server
    • More difficult for complicated migrations, upgrades
    • Passwords need to be more privately managed
    • Other stuff kept in hiera and maintained out of band
  • Conclusion
    • You have the tools you need to figure out and patch the infrastructure
    • Priority is largely determined by the patch submitter
    • No need to wait for the infrastructure team to figure out and write your change

 

Linux Filesystems: Where did they come from? by Dave Chinner

  • Motivation
    • Based on previous study that reviewed commits on file systems from git tree
    • Focus primary on bug fixes
    • Ignored feature work, didn’t look at the people
    • Only considered 2.6.x kernels
    • Listed Patch type, type of bug fixed
    • Only listed by number of commits, not the number of lines
    • Listed bugs fixed per release
    • But didn’t list “why” some of these things happened
  • Fire System Technologies
    • 1970s
      • Record based
      • small file name sizes
      • File control blocks rather than inodes
      • No Hierarchical namespaces – multiple “user regions” to segregate files
    • early 1980s
      • Sector based – complex Cylinder/head/sector mappings and optimisations
      • inode tables
      • bitmaps for free space
      • Resource forks
      • Hierarchical directory structure
        • simple and limited in size and depths
    • late 1980s
      • Extents and btrees in research
      • journalling first used
      • full 32 bit address space in use
      • Cylinder roup seek optimisation lead to multiple inode/data groups per file system
    • Early 1990s
      • maximising IO, > 4GB file systems. minimising seeks in reasearch.
      • RAID
      • log structured filesystems and copy-on-write
      • 5-10 year gap from research to production
    • late 1990s
      • soft updates to replace journaling
      • data transformations (compression, encryption)
      • wandering logs
      • async journalling
    • Current tech 00s
      • transperant error correction
      • direct device management (raid itself)
      • reference counted copy-on-write btrees
      • log structured merge trees
  • Linux file Systems
    • Minux FS – 16 bit, 64MB
    • 1992 ext File System – 2GB MAx, 255 char file names
    • 1993 – ext2 – 4TB max size, a/c/mtime support , group based bitmap allocator, extensible
    • 1998 – journaling proposed for ext2
    • 1999 – IBM release JFS under GPL
    • 2000 – SGI release XFS under the GPL
    • 2001 – Ext3 merged, reiser3 merged, JFFS2 merged, First JFS and XFS releases
    • 2002 – JFS and XFS merged
    • 2004 – Reiser4 released
    • 2005 – NILFS released
    • 2006 – ext4 first proposed, created
    • 2007 – BTRFS concenved
    • 2008 – ext4 declared stale, tux3 design first published
    • 2009 – BTRFS merged, NILFS2 merged
    • 2010 – LogFS merged in 2.6.34
    • 2013 – F2FS merged in 3.8
  • Linux history in git trees
    • Complete from 2.4.0
    • Older releases mostly intact
    • commit date issues, time based search fail
  • XFS History in a git tree
    • Complete from initial commit in 1993
    • Older commits mostly intact, some issues
  • Looking at extN / XFS / btrfs
    • When – diffstat
    • what – difss and commit messages
    • who – commits, mailing lists
  • EXT file system
    • 2500 lines added in 1991 to get it working
    • Removed in 2.1.21
  • ext2
    • 1994 – first commit
    • 1998 – Steven Tweedy did lots of work
    • 2002 – various stuff done
    • 2003 – extended attributes originally from XFS added by XFS team
    • 2008 – reservation based block allocation backported from ext3
  • ext3
    • 2002 – created
    • 2003 – journalling added , extended attributes
    • nothing much since then except maintenance
  • ext4
    • 2007- create
    • features steadily being added 2007 till now
  • btrfs
    • 2x the amount of code than ext4
    • Steady add of features since 2008
    • Stuff being added not removed
    • Sync’d to Linux merge and fix windows every 3 months
  • xfs
    • 1995 release
    • 2000 removed a tonne of stuff due to licenses pre-GPL (and it still worked)
    • Several other code removals
    • Code removals often mechanism being replaced by something better
  • Compare
    • xfs is as old as all others and has had more work done on it most years except recently with btrfs
    • btrfs due to overtake xfs as largest filesystem in 2014

 

Share

Linux.conf.au 2014 – Day 3 – Welcome + Lightning talks

Lightning talks

  • Storing Passwords
    • Salt per per password
    • See Aus Govt standards
    • Use bcrypt or approved standards
    • Don’t write your own
  • Pyladies Australia
    • Setting up in Aus
    • Local groups
    • Various ways you can help
    • australia.pyladies.com
    • @pyladiesAU on twitter
  • Crowd-funding Free Software
    • eg kickstarter
    • wish was a feed of free software crowdfunding campaigns
    • cffsw.modernthings.org
    • @crowdfundfloss
    • Keep record of old and list existing and upcoming funding rounds
  • slidelint
    • everybody makes same mistakes in slides
    • like code lint but for you slides
    • eg text to small, too close to edge, low contrast, spelling
    • Website version coming
    • runs against pdf
    • All in python, pluggable, docs
    • github.com/mithro/slidelint
  • Raising Geek Girls
  • Blue Hackers
    • Blue Hackers area near rego desk
    • Blue Hacker BOF happened on Tuesday
    • Psychologist on campus on Thursday and Friday. Signup in area near desk (anonymous method)
  • File System for Raspberry Pi
  • PyCon Australia 2014
    • www.pycon-au.org
    • Brisbane 1st to 5th August 2014
  • RapRap based curriculum
    • Newer designs are better, can be built in 3 hours
    • Idea that High Schools could build various classes around it: art, maths, physics, chemistry
    • Idea to create a book around this idea that schools could use
  • Kiwi PyCon 2014
    • Wellington somewhere
    • September
    • kiwi.pycon.org

 

Share

Linux.conf.au 2014 – Day 2 – Session 3 – Astronomy Miniconf

Supercomputing and Data Storage Design for the SKA – Stephen Ord

  • Still in design process but not everything finalised
  • Square Kilometre Array
    • Dense core, spread out arms
    • Across Aus and South Africa
    • different antenna cover different frequencies
    • Main goals of array:
      • Galaxy Evolution, Cosmology and Dark energy
      • The first billion years of the Universe
      • Strong field tests of gravity using pulsars and black holes
      • Cosmic Magnetism
      • Cradle of life – complex molecules around distant stars
  • Interferometer – Very heavy proccessing required
  • Lots of data coming in, lots to be archived
  • MWA ( approx 1% of the size of the SKA)
    • 7km of trenching
    • 10kn of electrical cable
    • 16km of fibre
    • 2048 dual pole antennas
    • 42 km of coax cable
  • Way too dense information for me to summarise
  • Next Generation Archiving System
    • Distrubuted, forwards data to various clusters around the world
    • Users can subscribe to tags

 

Desert Fireball Network – with Linux under the bonnet – Martin CupΓ‘k

  • Meteoroid (in space), Meteor (in atmosphere) , Meteorite (on earth)
    • Observed from 2 or more locations. 120-30km above surface. Freefall below 30-40km
    • Triangulation, analysis, weather model (can move up to 10km sidewise)
    • Search trip
    • Meteorite!
  • Types: Stoney, Achondrite, stony-Iron, Iron
  • History
    • Manually operated film cameras (since 1959)
    • Automated film cameras (first in 1997, since 2001)
    • Digital cameras (ideas and dev since 2010, operation since 2013)
  • First Meteorite observed, triangulated and found in Czech republic in April 1959
  • Since 2009 15 automated film cameras across 13 stations in Europe
  • Nullarbor network of automated film cameras – 3 stations 2005, 4 stations since 2007
    • 150km apart
  • Based on Industrial PC
    • Old ones P1 , newer P3
    • Microphone picks up sound from meteor going through atmosphere
  • Software
    • Using Redhat 7.3 (released 2002)
    • ntpd sync’d
    • New System using Centos 5.2
  • Digital Cameras
    • Triggered system
      • sequence of hi-speed HR images fired when event detected
      • 11 images/second
      • Complicated design so put aside for now
    • Long exposure system
      • MK-I – 2 built as testbed
        • eBox Pc, 933Mhz, 3xUSB, 1TB HDD, DSLR taking 30s exposures, leo stick uController for power, GPS and camera control, 3G modem, 2x2TB NAS
        • 3G modem lockups
        • Slow CPU
        • SD card unrelaible
        • External HDD a worry
      • MK-II – 10 built, 5 deployed, 60 planned across Australia + USA
        • Industrial PC
        • Video Camera
        • Atom N2600 dual core 1.6GHz
        • 480G SSD
        • 2 X 4TB external HDs
    • Kit camera system for enthusiasts
      • Similar to long-exposure system, possibly lower resolution
  • Goal 60 Cameras, cover 1/3 of Australia. See 15-20 meteorites / year in searchable areas, 500TB data per year
  • Data available for other uses?
Share

Linux.conf.au 2014 – Day 2 – Session 2 – Open Programming Miniconf

Coming home: a return to PHP’s roots by Adam Harvey

  • I arrived late for this but he looked like he was having fun
  • He wrote a PHP micro-framework live in the talk
  • Cute

Get your PaaS into gear by Katie Miller

  • An introduction to OpenShift
  • Platform As A Service, include web servers, databases, storage.
  • Iaas -> Paas -> Saas
  • Why?
    • Focus on code, not config
    • Speed of deployment
    • Convenience
    • Scalability
    • effeciency
  • Flavours
    • Origin – Source can be downloaded
    • online – Hosted in Amazon
    • Enterprise – Run you own supported
  • Runs on RHEL
    • Runs in containers
    • Cartridges (programming environment) within the container
  • Tissues!

The Cobblers Children Have No Shoes: Development Tools and the Unix Philosophy by Russell Keith-Magee

  • Django Guy
  • Over last 30 years – My tools have got appreciable worse
    • Libraries, languages and computers better
    • Programming tools have not
    • Some have got worse
    • Not language specific
  • Debugging
    • Borland C – GUI – see where in code you are
    • 1994 gdb
    • 2013 gdb
  • Why?
    • The Unix philosphy – small, seperate tools
    • Cargo Cult
      • Powerful tools only need a CLI
  • Fully featured IDE?
    • Only one tool the GUI
    • Maybe a plugin interface, if you are lucky
  • Keep good bit of Unix philosphy
    • Different Window dressing
  • BeeWare
    • Common problem with bad interface – testing
    • Current test suite outputs little to screen until it finishes.
    • Cricket – runs test suites and lets you look at output while it is running, see progress, what is passing/failing, can browse errors, does just one thing
      • cross platform – uses what Python provides out of the box, uses tk
      • Zero configuration – uses pip and run from command line. Nothing set in tool
      • Integration
    • Duvet – Test coverage tool
    • Why not a web interface
      • Needs to have a server
      • Live updates are still ard
      • Don’t believe web browsers will be superior to native tools and time soon
  • pybee.org

 

A Pseudo-Random Talk on Entropy by Jim Cheetham

  • Random vs Pseudo-Random vs Cryptographically Secure Pseudo-Random
  • Some Pseudo-Random generators not as random as was expected
  • Good PRNG – Mersenne Twister,
  • Blum Blum Shub – Crypto good if you pick right numbers
  • Diehard test spot bad random numbers
  • Random numbers for science – use publish “table” of pre-generated ones
  • /dev/random very random enviromental noise , /dev/urandom not crypto-good
  • /dev/random should block if not enough random bits available
  • RdRand – Source of random data built into chip, measures physical noise, runs through AES, can’t acess original data though
  • Sources from various hardware generators
  • Turbit – reads noise from soundcards
  • OneRNG – Speakers project, Open etc
Share

Linux.conf.au 2014 – Day 2 – Session 1 – Open Programming Miniconf

Keeping Current: Maintaining an internal fork by Benno Rice

  • FreeBSD contributor, works for EMC isilon division
  • OneFS file system runs on appliances FreeBSD
  • OneFS = FreeBSD + stuff
  • Various add-one. Pretty interfaces, file system, etc
  • Some changes upstreamed but not others
  • We maintain a divergent fork of FreeBSD

Strategy:

  • Don’t maintain a fork
    • Version is old. 7.3 from 2010 plus backports
  • Don’t get behind
    • Need to catchup and stay caught up
    • Track FreeBSD-current
    • Do our own releases
  • Don’t track branches
    • upstream as much as possible to minimise diffs
  • Upstream as much as possible
    • Code checkout 5GB
    • Pile it all up approach approach – bulk merge and fix things – no timeline – no way to test
    • Pretend we use git approach
      • merge changes from bsd and apply
      • subversion couldn’t do directly, fixed with script
  • If your tools don’t do what you need fix them
    • worlds ugliest rebase – merge in all changes since 7.3
    • eventually got there
    • merge 20 changesets and then test a bit. bisect changes
    • 9 weeks of work to get to FSB v8
  • Don’t Stop
    • Got to keep tracking it forever
    • need to work out when to do releases

 

USB and Android by Joel Stanley

  • Last 3 years making consumer electronic devices, all talk to USB
  • Android as a tool
    • cheap, portable, lots of connectivity
    • No licensing worries vs laptops
    • Lots of features
    • Works out of the box
  • USB APIs as of Andriod 4.0
  • Normally runs in peripheral mode – we are interested in host mode
    • Most hardware requires a dongle
    • ADB over TCP to debug
    • Asus Transformer
  • Architecture
    • libusbΒ  – std Linux
    • app.so – Same as linux version
    • C wrapper – app code created and mait by C, simplyfyin Java
    • JNI
    • App Classes – calls into C code
    • User Interface
  • Java Naive Interface – JNI
    • C can call Java
    • Jave can call purpose-written C-code
    • Hard to get started with, look at others people code off github
  • Native Development Kit
    • C/C++ cross compiler for targeting Android
    • Provided by Google
    • Used it to compile shared libs
  • Android permissions
    • Limits access to device files
    • No auto access to hardware, can’t hard-code since USB is dynamic
    • Get permission in Java code and then change permission so C code can use
  • https://github.com/shenki/FreeDV-Android

 

Developing OQGRAPH, a tool for graph based traversal of SQL data in MariaDB by Andrew McDonnell

  • Graphs = nodes + edges/links
  • Graph alogrithms – Breadth first, depth first, shortest path
  • Graphs in the “real world” – maps (how does GPS work), relationships (linkedin, facebook), trees
  • MariaDB is fork of MySQL
    • Being adopted a various distributios
    • Used by big companies
  • Graphs in a SQL RDBMS
    • store a graph as a list of edges
  • OQGraph
    • Creates “latch” column that contains a special command executed by the storage engine
    • where latch=’breadth_first”
    • Project hosted on lauchpad

 

 

Share

Linux.conf.au 2014 – Day 2 – Keynote

Kate Chapman– OpenStreetmap for disasters

  • Background
    • Most maps people used are not free
    • “The Wikipedia of Maps”
    • Goal: A free map of the whole world
    • ODbL – Like CC-BY-SA
  • Humanitarian Openstreetmap Projects (HOT)
  • 1st official activation in 2009 for Gaza
  • Activatation is a response from the community to a callout
  • Port-au-prince Jan 12 2010
    • Tracing existing satellite photos
    • Being used by people on the ground within a week
    • Geographic agency build collapsed and many staff killed in earthquake
    • Data released by agencies used for maps
    • Also covered Northern Haiti which wasn’t as affected by quake
    • Created NPO to organise
  • Project to map Indonesian ahead of time since disaster-prone country
    • Map once, use for many things
    • Pilot project – West Nusa Tenggara
      • Worked with community to collect data, including demographics data
      • Some existing maps were hand drawn or used “dumb” tools like coral-draw
      • ASM tools distributed
      • Agreement from Govt, NGOs, Companies to make best mapped province in Indonesia
    • Mapping Jakarta
      • 267 Urban villages
      • Talked to each head, get person to map data, University students to input the data
      • Varying levels of data quality, in-putted by students
      • Flooding in Jan 2013 – Map uses as basis of maps to show areas affected
      • Creating a training program (training the trainers)
    • openstreetmap.or.id
    • Data collected (often by Scouts), printed out maps, got people to walk around fill in gap son paper. Add data to online version
    • Used to create impact models for future floods
    • 3 part training manual created (English and Indonesian)
  • Eurosha
    • Training people to go work for 6 months in various African countries
  • Other projects to map 200 asian cities that are prone to disasters
  • Senegal – training and workshops
  • Project in Japan in Tsunami affected areas
  • Mapmill –
    • Civil Air Patrol takes GPS tagged photos of disaster areas
    • People sorted images by damage None/some/bad
  • American Red Cross
    • Fire risk map in Uganda
  • Β  Philippines – Typhoon Haiyan hit Nov 2013
    • Mapped 10,000 buildings in Tacloban before Typhoon hit
    • Website allows people to login and see various tasks that can be done
    • Post-event Satellite Imagery needed
    • US State Dept released satellite imagery for 30 days
    • Able to use photos to do before-vs-after maps on damage
  • github.com/hotosm
  • digitalhumanitarians.comΒ  – various groups
  • Volunteer in Person for a Project – See website for lists
  • hot.openstreetmap.org
Share