Simon Lyall's Blog

Lots of momentium in last year

MPs expenses
Couple of conferences

Open govt conference – went fairly well

Data.govt.nz

Grabbed Linz data and chatham islands in openstreet maps built using it. Rolling out to mainland next 6+ months

Igovt login being rolled out across government sites

NZ transport has open API – infoconnect
Real time bus infomation often sold to or maintained by 3rd parties

Govt depts tidying up contract to clarify ownership of data

See cat.open.org.nz

NZ book cover images provided by national library

Often councils make very little per year for data they are charging for

Mytours – apps for walks and museum tours – 4 people

Workshop enterprises – 1+ person

What is startup?

• high growth
• not cash-flow positive
• less defined business model
• aggressive targets
• pre-ipo
• high-risk profile
• some industries ( airlines ) startup on completely different scale and timeline

How Auckland different

• in SanFran networking for webapps with diggs etc easier
• in NZ easier to meet retailers and customers
• different reports on how easy to get matching funds from development
• easy to start, hard to scale
• maybe launch globally rather than NZ first and then expand
• lot easier with beta customers you talk meet & talk to in person
• CARE really good source for contacts

Second session at barcamp Auckland. Trying to do less and automate stuff . Avoid cut and paste

Examples

• Xero has bank rules which automate creation of invoices
• mail chimp easy mailing lists, free for small groups, pay pretty good too
• CapsuleCRM – list of tasks for customers
• use the capsule API he adds new customers when signed to their own system
• capistrano for automated deploy
• textmate bundles
• gmail labs pre-canned resposes
• hoptoad – monitors logs
• xero – re-occuring in voices
• accounting system checks prod system and looks for unbilled accounts or locks people who don’t pay
• single password systems
• firefox sync, dropbox
• greasemonkey
• insta-paper
• google calendar, trippit
• quicklaunches
• lighthouse, activecolab, atlaysan
• run lint or jslint at commit time

First talk at barcamp auckland 4 is Ben from NZ Post

Existing site hard to find stuff, often things listed by product name rather than what people wanted to do.

Around 3000 pages. Huge job to rewrite many of them, many docs written by insiders for insiders.

Key customer profiles for each product , narrowed to 8 customer types. Products matched to some of these types.

New platform is Drupal. Was going to broad RFP but decided to narrow things first. Down to 4 systems Drupal, Plone, Silverstripe, and ruby-on-rails. Lots of alternate vendors for that platform.

Site being built by catalyst. Platform is Ubuntu .

Some product rationalization and presentation changes.

Planned launch in a couple of months

A few things that have caught my eye over the last week or two.

• I’m finally getting around to implementing puppet at work and after a couple of false starts I found example42.com with examples and a huge amount of sample infrastructure to base things on (and in some cases recycle).
• Slides and Video’s from the OReilly Velocity conference ( Web performance and Operations) are being posted. They are still catching up so new talks re being added each day but so far I’ve really liked
  • A Day in the Life of Facebook Operations by Tom Cook
  • Datacenter Infrastructure Innovation by James Hamilton
  • Operations at Twitter – John Adams
  • Mobile Web High Performance – Maximiliano Firtman

  For the videos it’s usually better to view them directly on youtube as they are a bit bigger and easier to read.

• membase has recently been launched by Northscale . It’s a disk-backed distributed storage that talks memcached protocol. Looks quite nice ( I am playing with Tokyo Tyrant for the same thing). The free release on membase.org needs a bit (well any) documentation to be useful but it looks like something to watch.
• Yahoo! have just released boomerang which looks interesting. It’s gives several options for putting javascript web-bugs on your page that communicate back to you with how long the user takes to load the page, their download speed etc. Have a look at the online version of the documentation.

Notes from the Load testing Session on Sunday.

Lonely Planet

• Need tests developed as features are added
• Moved tests to jmeter
• Anyone can run and develop tests ( previous just one test guy who had to write and run tests for 3 agile teams)
• Load test are integrated and tested as part of continious integration

Atlassian

• test important paths and functions daily
• Emails test person and author of recent commit if things (usually load times) get worse
• Graphs of speed of pages

Demo of Cloudtest by one of the participants. Looks amzing (although quite expensive I guess)

Suggested vendors / Products

• neotest
• cloudtest
• siege
• browsermob ( uses Selenium)
• newrelic

Tim Moore – Atlassian

What do we do wrong now?

• Continuous tests? , Against production environment?

Writing Flexable tests

• Current test make assumptions about enviroment (each to recreate DB etc)
• Sample app ( twitter clone (tooter) ), in java, maven to google apps)
• http://github.net/TimMoore/tooter , tooterapp.appspot.com
• Tests in “htmlunit” to perform tests again the actual website
• Logs in checks elements on page. Test from logging in, make sure simple errors on logging in are picked up and right error pages produced.
• First refactor was to generalise the tests to able to reuse things like “login to site as user XXX,  password YYY”
• 2nd refactor puts everything in a seperate class. Low level things like login, host, html library all have their own class. Each to change
• Keep data in production database. Have test users or organisations to test against. Perhaps don’t confirm user signups so new users not created each test. Create “test groups” that test users are members of so changes only affect those test accounts, leverage A/B testing infrastructure
• Other tools include cucumber

Joel Courtney from Energy Australia (cover parts of NSW)

• Fundamental business of power not changed in 100 years. Compared to technology and telecommunications
• Very high reliability requirements
• 60-80 year old infrastructure in places
• instrumentation limited as to home close it goes to customers. Only to distribution stations (30k of these) , sometimes substations
• Rolling out 4G and other wireless networks – priority is coverage not speed
• IP over power not viable solution
• Need to do something with all this data

System

• Need to integrate GIS and other information about state of network to make sense of raw numbers and how they relate to each other
• decided to go with web based system since clients had at used web. Browsers on desktops etc
• Web based systems more mature these days
• Lack of expertise in IT space
• Using GIS info and maps based apps to plan rollouts ( reduce tralvel distances)
• Some inspiration from flickr ( url structure, deep linksing,  simple relationship between assets)
• Everyblock ( Integrating and agregating information on geographical basis)
• Twitter (asset reporting of event on timestamp, everything else is on top that)
• tumblr ( similar to twitter, move through streams of events)
• Google maps ( simple intergration, visualisation, assets in real world

ION

• Application for monitoring
• 750 sub-stations currently
• Previous systems used polling (didn’t scale)
• Push messaging
• Operators at this point able to get better locations of faults than in current systems
• Huge increase in information being collected (every minute vs once every few months in some cases)
• able to do overlay on google maps for assets (putting 30k on one map pushes google API). Percentage utilsation
• Some cases substation in remote areas only have a couple of customers. Not worth deploying especially since next generation metering will provide information
• Looking at future integration once smart meters rolled out. But data from this still be decided by power regulator. Strong privacy concerns.
• Mostly event related browsing now
• Sorter term pushing up into transmission space
• Interested in releasing more information to public. Need input on formats , interaction.

John Ferlio – Commit Early, Deploy Often

Flickr claim to deploy multiple times per day, not for everybody

Various ways to deploy, tarballs, versions control, packages

Usaing Package management to deploy

• Treat internal apps same as external apps
• Works with config management
• Use CM and VMs to give devs a sample of production on their laptop
• Devs can then develop directly to what prod looks like

Example for Deploy for Ruby Apps

• Use bundle to setup ruby dependencies for app
• Simple makefile to install install bundle and run it. Bundle installs apps and then compiles them. Then files put in right dirs (prod and stating versions)
• dh_make to create base deb env
• delete unwanted extra debian template files.
• create staging and a production package, basic fill in of debian control files to buuild packages
• Package contains whole website and files. Maybe 200MB. More advanced users might want to split some stuff off.
• He has little bit to put the bzr revision in a file in the package for later reference.

Lauchpad PPAs – lets you build your source for various debian distributions/archetectires. He Blogged about a simplier system he created (you can host locally) this a month or two back.

Use to deploy rubu-on-rails and WordPress apps.

• Don’t have to compile things on box, don’t have to install gems, don’t have to svn update.
• Maybe use pre-init script to stop app server during the upgrade nd restart afterwards
• Some talk about how to sync deployment of updated app and new database schema that it requires. Thoughts seem to be that new app version should be able to handle the old schema during the transition.
• Rails libararies evolve so fast which is why libraries need to be bundled. Other languages can sometimes use OS provided libraries