NZNOG 2010 – Day 2 – Session 4

Metro WDM for the fiscally prudent – Simon Blake

  • CWDM – Split into various bands – uncooled lasers –
  • Single mode fibre – G.652c ideally – coloured optics – components
  • DOM/DDM support (SFF-8472) – query SFP and see what signal level it’s getting (over or under strength )
  • 1-8 Channel MUX/demux – 8 channels 1471-1611 over a pair of fibre
  • Cisco 8 port mux/demux $6k/end ,
  • ebay 8 port mux/demux $800-1000/end
  • Direct import 8 port mux/demux $US 550/end
  • 2 x 10GE on one pair – 2 channel 1310-1550 CWDW splitter (mux not a splitter) – $40 kit on direct import – vs numbers above
  • 1x10GE on single fibre- optical circulators $NZ 1000k , $US14 imported
  • 6 node network, 4 dark fibres – $27K
  • Trying to solve problem with lots of small hops, upstream building losing power (unpowered gear)
  • Pros: Multiprotocol, Perf/Security/reliability
  • Cons: Short Haul (sub 120km) , only 18 channels , Doesn’t do >10GE per channel, You need fibre
  • Direct Import Pro: cheaper , especially in bulk – design flexability
  • Direct Import con: No support except swaps – Freight – Language/Culture chellenges
  • traps – Waterpeak , Wideband receivers , Near end reflection , Avaibility of 10GE optics – DOM (ask for it) and untrusted optics – Measurement equipment/Circuits recording – Link Budgets and insertion loss

Monkeying around on the APE – Michael Jager

  • Plug in new port at APE and found things very open
  • PAcket sniffer + APE – should see boracast and traffic desinted for me
  • What did sniffer see – lots of APE for non-APE address space – DHCP
  • Borrowing transit – see how many networks will accept packets – 46 out of 75 will accept frame from unknown address detinated for their MAC
  • 3 ports provide proxyarp for random address
  • How many networks have an interface in your mngt network?
  • 6 will accept for
  • Customer can try and grab as many packaets as possible across cheap APE link rather than expensive transit link
  • Possible things untried – ARP spoofing – responding to un-answered ARP requests (old BGP session of removed neighbour ) – respond to DHCP requests
  • Speaking OSPF to OSPF-speaker – sending TCP RSTs – sending IPv6 RAs and answering IPv6 RS (like DHCP but for v6)
  • Read IM2tubes slide from Jonny and Philip’s slides from Monday
  • AMS-IX configuration guide
  • Don’t take packet from IXP if you arn’t expecting it
  • Don’t announce IXP network from anywhere

NZNOG 2010 – Day 2 – Session 3

Announcement at start of session that Telecom New Zealand now has an official Interconnect/Paid Peering Policy and Contact. Details to be Published. Ask Greg from Telecom for help.

Internetnz Update – Jordan Carter

  • General updates and new structure, new CEO
  • 4 main areas ( Openness, rights and responsibilities, security)
  • IPv6 Task force , replace steering group
  • Copyright – replacement policy looks better, but sneaky changes might come back
  • ACTA – Key concern , lack of transparency,
  • DIA filtering – voluntary and uses BGP . Give webpage, can report false alarm
  • Filter – only http, erodes end-to-end , privacy concerns , might be later abused (scope creep)
  • Filter – Send signal that “The government has made the Internet safe”
  • Internet opposed – DIA unhappy with that angle
  • Fibre Stuff – “Last day for 1.5 billion lolly scramble”
  • Regional Networks or one big National Network
  • Hard to tell what will happen – Similar exercise in Aus and Govt went back to drawing board
  • What happens to International Bandwidth?
  • Please join, followon twitter

APNIC update and much more – Elly Tawhai

  • Over 2000 members
  • 1400+ monthly helpdesk enquiries ( 55% growth since last year)
  • Allocations around 100 per month
  • Various Policy changes coming up – Prop-050 (xfering address space ) , Prop-073 (sinple IPv6 allocations – 1 click) , Prop-074 (32 bit ASNs treatment same as 16 bit ones pushed back a year) , Prop-075 (recover historical ASNs)
  • Policies under discussion – Prop-78 ( Final /8 , only people deploying ipv6) , Prop-079 (abuse contact info in objects ) , Prop-080 ( Removal of IPv4 prefix exchange policy )
  • Several more allocation policies in pipeline
  • Recent Survey leading to priorities
  • Various my.apnic updates (web services even), support of research
  • More DNS root servers (Taiwan , Mongolia)
  • Please Participate

RIPE News – Tools and news – George Michaelson

  • RIPE used to be a research place and then became a RIR. RIPE labs is a return to the past
  • Platform to test and evaluate new tools, feedback cycle
  • INRDB – big cloud of assignments, table dumps, dumps
  • Resource explainer
  • Various measurements , visualisation and links to tools. DNS reply size tester
  • Why – fast turnaround, engagement, no service g’tees

IPv6 flow chart – Nathan Ward

  • Make decission which IPv6 or IPv4/Ipv4 translation technology you should use
  • Tunnel Broker, 6to4, 6RD, Teredo, Dual stack lite, Double NAT, Dual stack
  • Other stuff that I wasn’t paying attention two
  • IPv6 addressing schemes
  • Sparse allocations
  • gives a sample which I won’t copy, look at his slides
  • Customer assignmesnt. Nathan likes /56s or RFC recomended /48. Take your pick

Andy is Curious – Andy Linton

  • Are Universities turning out the right people?
  • Good at turning out applications programmers not systems programmers

NZNOG 2010 – Day 2 – Session 2

DNSSEC at the root zone – Joe Abley

  • ICANN – Manges the Ket-signing-key (KSK) – accepts DS records from zone operators – sends update to DoCfor auth and to veriSign for implimentation
  • DoC auth changes and Verisign impliments the change
  • New process has Verisign signs the keys. V gets a few weeks of of KSKs that Doc signs in batches beforehand
  • DNSSEC Practice Statement – describes procedures, currently drafts
  • Around 20 Community Trusted Representative ( TCR ) have an active roll in the mangement of the KSK
  • 2 copies of the Keys, west coats and east coast. Plus distributed backup
  • “ceromony” for each step in procedure, required what you do and how many people and which people are present.
  • Similar to what x.509 CAs do
  • KSK is 2048 RSA key rolled every 2-5 years ( RFC 5011 but not all have that support) –  Signature using SHA-256
  • ZSK is 1024 RSA key – signed with NSEC – rolled 4 times year – Signature is SHA-256
  • Time cycle every 90 days – ZSK overlap of a couple of weeks
  • Root trust Anchor – published in XML document with constant URL – plain DNS record – PKCS#10 cert CSR , as self signed pub key, signable by others if they want
  • DO=1 part of EDNS0 – says client wants DNSSEC – many clients set bit even though most won’t really want them right now – will cause all queries to jump in size
  • Hard to sign root and then rollback
  • Staged deployment – Start servering DNSSEC for 1 root server at a time – L-Root first, then A, then the others with J last
  • DURZ – Unverifiable key published as placeholder
  • Measurement – Packet captures , diologue with operators – wide range of pre-testing with various software – test with clients that drop large packets
  • DS change requests – TLD procedure to be decided – DS requests 1-2 months before zone published
  • Timeline – Test key signing Dec 2009 – Jan 2010 . Jan – July 2010 roll out signed roots . July 2010 Full Production
  • Lots of documentation on website
  • Indication of big jump in tcp queries presumably because udpreplies are too big

ENUM – Jay Daley

  • Why Doesn’t telephony work like email?
  • Email you choose how to published your email record, where to host, what emails to accept, can outsource, totally in control
  • So IP telephony should be easy too?
  • Unfortunately not
  • Non site-local numbers MUST go to telcoto get delivered
  • Missing – single , global directory linking telephone nmbers to voip numbers
  • This is ENUM . Telephone Number -> Domain Name – Simple Algorithm – – 04 931 6970 ->
  • Won’t be typed, Translation done by a device – people still type out over fashon numbers
  • Register your number, create zone. Add NAPTOR records to DNS zone. Special records to specifiy endpoints (usually sip records), receive calls
  • NAPTO records do interesting stuff . eg “dig +short naptr”
  • how? Option 1-  enable on your VOIP PBX that is internet connected
  • Option 2 – on session border controller – “enterprise”
  • Option 3 – ENUM proxy ( if existing SBC doesn’t handle enum)
  • Registration process – not same as for domains since numbers already registered – needs authentication
  • Various methods of authentication in different places
  • No ENUM in NZ . Available in UK, Holland, Ireland, Germany, Austria but not significant takeup
  • Reasons for lack of takeup in those countries – lack of mindshare – hostility from telcos
  • Why not in NZ – TCF 2006 report – Privacy issues (but only publish what you like) – Emergancy services access (no idea where callers are) (but all VOIP has problem ) – Polcy/Goverance – “Carrier Issues”
  • ENUM isabout control – movingit from carrier to you
  • Key users – Call centres , ENUM instead of 0800 – Large supply chains (mandate VOIP ) – Multiple sites , simplyfy provisioning
  • Won’t happen without demand
  • “On the Internet voice is just another application”
  • Significant political and commercial resistence from Telcos

Day in the Life of the Internet – Sabastian Castro

  • 4 years of DNS data
  • DITL motivation – network measurement – collection of data from DNS root servers – yearly since 2006
  • More and more root servers, Alt root servers, gTLDs etc passive traces, 48-72 hours
  • concentrate on root server data
  • Pick best 24 hours out of total window
  • 4-8 billion queries, 3-6 million unique clients – sm5-12% recursive queries
  • Mostly A queries, AAAA increasing due to gluerecords being added (why are IPv4 clients sending AAAA queries when they probably won’t/can’t use)
  • 70% of clients are EDNS are capable ( 90% of these are D0 enabled )
  • However clients sending lots of of queries (probably broken) have good support – But clients that query less have lover level of support
  • 10 invalid TLDs represent 10% of queries ( .local , .localdomain , wpad , invalid , home , belkin , corp , lan )
  • Impossible to track down
  • Most queries from NZgoing to Auckland root and Brisbane root but some going to overseas servers (those might be use simple round-robin picking)
  • Lessons – Data collection is hard – clock skew , dat loss , wrong command line options , bad network taps
  • Data management – moredat , more participants – more formats – big effortto normalize data , fill gaps , fix clock skew .

NZNOG 2010 – Day 2 – Session 1

Lightning Talk

  • Geoff Huston – Stateless TCP and DNS
  • TCp limitations – Rough a high load
  • UDP Limitations – Requires IP fragmentation
  • Problems when response bigger than MTU , Fragments of UDP IPv6 often dropped. Switching to TCP drives up load again
  • Simulate UDP with TCP – do minimal crappy respose to fill headers
  • Ignore options, server doesn’t retransmit, ignore anything else from client, just closes connection
  • No reliability, No Flow Control, bad Idea but seems to work
  • Olof Kasselstrant – IXOR
  • Small IX in Malmo and Copenhagen (2nd site being looked at)
  • DIX only IX in Denmark
  • Sponsors for Fibre and Equipment
  • Exchange in 2 countries. Does it affect “must peer in 4 countries” agreement.
  • Dream to be in 4 sites soon
  • CCIP – Barry Brailey
  • Getting out of rewriting Microsoft patch notices
  • “investigation and analysis” function being dropped
  • Infomation and Alerting – website , newsletter, alerts – alerts targetted and highish threshold –
  • Outreach and partnering – main function – lease with overseas certs – talk to various groups – Education: presentation, newsletters, exercises (CyberStorm III – volenteers )
  • Security Information exchanges – Various groups – traffic light protocol – Looking at some new forums – Maybe ISP SIE
  • Cloud Computing for Service Providers – Richard Wade
  • As a service provider – should I care?
  • Infrastructure Foundation (Cisco, EMC, HP)
  • Infrastructure as a service (Amazon , Sun , Savvis )
  • Platform as a swervice (Amazon, MS Azure )
  • Software as a Service ( Salesforce, Google apps)
  • Integrate mngt ( network, servers, hypervisor, storage ) – unified fabric
  • Why and Why Should I care
  • Customer Ads – Eliminate Capex – Reduce Opex – IT as a utility
  • Customer Probs – No LAN apps (overseas often) – WAN now biz critical – Operational relationship with overseas provider – Legal jurisdiction of data
  • Service Provider ads – Understand managed services – Existing datacentres and infrastructure – OSS , process staff and contacts – SLAs – Domestic provider
  • Sp Probs – Managed cust revenue declining – Race to bottom? – Increase International transit – High expectations of quality and relaibility
  • Lame aternative IX Update technique – Simon Blake
  • New system to update filter lists for IXs
  • Citylink can instead download list of networks from customer URL
  • Pulls list daily
  • If diff email for confirmation or action it immediately
  • ALTO – LLyod
  • Helping p2p users select local/nearby peers
  • GeoIP and anycasting – rough
  • ALTO allows ISP to provide application, localtion, routing information, charging information, performance.
  • ISP puts on network some servers (itrackers) that deliver to p2p client the policy information
  • p2p caches (very close to edge) can be advertised
  • No currently in use in the wild
  • IPv6 taskforce – Dean Pemberton
  • Internetnz+ MED
  • TechSIG – 3 Hui in 2009 – Aimed at CIO/CTO – Went really well
  • Looking at more training (session in 2009 already)
  • Other things Task Force can do?

Building a Datacentre for less than $1 million – Gerald Creamer

  • When it’s your own money you care so much more
  • Had to move datacenter to another building
  • Short is that you can’t do it for less than $1m
  • Significant cost areas – Physical – power – cooling – network – time
  • The right building – 18 m search – 100 sites looked at – 7 sites investigated – 4 site due diligence
  • Engineers – “consultation” vs “converstaion”
  • First culling – all concrete – Not ground , not top floor – Strong 5kPa – high stud – no sprinklers – built between 50 and mid-80s – CBD fringe
  • $400 per m2 to strength building
  • 2nd culling – close to street transformer – shorter power cables runs in building – shorter pipes for colling – outdoor space – generater space – near data networks
  • Useful – friendly landlord – nice bank – recession (kean landlord)
  • Save money – quality pre-owned hardware – “free” stuff – Ask experts – do some stuff yourself – Get experts to do others
  • Cables up abandoned lift shaft
  • 2nd hand generator – not as large as final requirement but bigenough for current build
  • Room to upgrade UPS, generator, cables and space spec’d for more
  • domestic meters to measure power in each rack
  • Process Coolers (cheaper) 28KW each $1500/KW cost – $70k of aircon for $7k – check serial number with manufacter to find product history
  • Seismic Bracing – $30k
  • Helped corps clear out datacenters they were moving out of rooms ( “make good” on leases) and picked up some equipment
  • Citylink and Telstra provisioned fibre. Telecom less helpful.

NZNOG 2010 – Day 1 – Session 4

IPv6 deployment scenarios – Brian Carpenter

  • Assumed v6 deployed by v4 ran out
  • change transition model
  • More internetworking than original ipv6 design originally anticipated. Assume v6 clients will need to access v4 servers forever
  • Tunnels – Dual Stack Lite ( share ipv4 addr amung custs by combining UPv4-in-IPv6 and NAT, Driven by Comcast BB model ) – 6rd ( blend of 6to4 and ISTAP providing atumatic tunning of IPv6-in-IPv4 to ISP subscribers. Deployed by
  • Older mesh and hub+spoke models also documented.
  • NAT64 – old NAT-PT deprecated
  • NAT64 – millions of IPv6-only custs needing access to IPv4-only services
  • NAT64 only solves 1 problem – cannot be met my dual-stack – DNS64 dns server creates AAAA of site only with A record. Packets to NAT64 box and translated
  • Various problems. 7 ietf drafts. Only solving since case
  • V6OPS WG- Emerging Service Provider Scenarios for IPv6 Deployment – ID and survey ISPs then publish draft 03/2010

Rapid IPv6 Deployment in ISp Network – Skeeve Stevens

  • AIM – Get people to use IPv6
  • eintellego runs ISPs
  • What stopping ISps implimenting IPv6
  • Why not? – Too expensive , bigger ISPs yes, smaller ISPs perhaps not, NOT expensive to do enough to be able to play with it
  • Why not? – Too Hard – Lack of internal skills – IPv6 is NOT hard, cisco admin should be basic IPv6 in 2h and IPv6 BGP in under a day – Play now or else you will be overwhelmed later when everybody is yelling
  • Why Not? – Don’t know where to start – Start with a external co-lo box in the US – Allocate small amount of time – Get access to a lab – Start at the border
  • Why Not? – No one asking for it – True enough – Don’t know about Ipv4 exhaustion, but they will
  • Why Not? – Little vendor support – improving – DSL CPE equipment getting better – Carrier Grade NAT ( CGN/LSN)
  • Why Not? – What is IPv6? – From Many IT professionals – Integrators have minimal experience
  • Why Not? – Who can help me? – commerially, very few people – Some training courses – Community helps
  • IPv6 is big, break it down into stages
  • Experiment Externally
  • Get allocation from APNIC
  • Enable your Edge (BGP)
  • Enable Core
  • Enable desktop
  • Enable your hosting
  • Enable Operation Support Systems
  • One hosting company just took 1 week
  • Very rapid training, just a couple of days
  • Simplified addressing – short to medium term – rapid deployment – format – 2406:9800::F: – Use F0 instead of”F” for next pop – Using /128s will increase routing table – “chazwazza” is ipv6 equiv of “octet”
  • We use /64 for all end customer assignments – static routes to make v4-in-v6 work
  • NTP might not work
  • Some security concerns
  • Go through commons OS, Daemons, Hardware ( phones, printers, UPS, gameboys)
  • Might have to tunnel
  • Hassel carrier if not provided
  • Hassel vendors if they don’t work
  • Some parts won’t happen overnight
  • Predictions – Telstra selling IPv6 mid 2010 – Resource rush to grab IPv4 IPs while they can , surge in APNIC membership – exhaustion brought forward – secondary market will come – APNIC will lose control

Simply allocation of ipv6 addr to ipv4 holders – Elly Tawhai

  • Policy 73
  • Encourage greater uptake of IPv6
  • An APNIC member with IPv4 allocation is eligible /32 . Member with assignment gets a /48
  • One-Click IPv6 from

NZ/IPv6 from (offshore) DNS – GGM (no name)

  • Passive tap on DNS servers – spot reverse lookups for
  • Capture all DNS in 1 day look for NZ IPs
  • 1 in 10,000 lookups are doing IPv6
  • 1 in 200 queries for DNS using IPv6
  • 87.5% active delegattions in 24 hour period
  • 45% of V6 networks live in 24 period
  • 52% of v6 is Macs
  • IPv6 not on the phone
  • 6to4 common even with providers that do IPv6 native

Things running late so IPv6 panel skipped.


NZNOG 2010 – Day 1 – Session 3

NZ Internet Task Force – Paul McKitrick

  • Out of Cyberstorm planning session – “what to do about botnets?”
  • Task Force has Steering Committee
  • Trust is essential – New members vetted – slow growth of membership
  • Protocol on how widely specific pieces of information can be shared
  • Information sharing – networking – training courses ( honeynet, shadow server foundation, team cymru )
  • Focus areas – Telecommunications (telecom honeynet, Uni grads seconded to telecom, Walled Gardens)  – Research ( , VUW honeynet , data Brokerage ) – Stretegy ( Phishing site takedowns, Nat Cyber Security day 2010 , NZ Computer crime and Secuity project )
  • NZ Ips sending 110 million spams per day
  • Why – good for “.nz inc” , Opportunities for research, networking, conduit for disclosure

Bits on a Budget – Perry and Jamie

  • chellenging the belief that PCs running linux useful only for slow, small, un-important routing jobs
  • changes in last few years means this may need to be re-evaluated
  • What changed – PC Arch, Intel stopped sucking , Quick Path Interconnect , PCIe , Multicore – Substantial improvement in Linux – Multiqueue RX/TX to take advatage of multicore
  • Intel x520 10 GigE cards – Significant hardwareoffload – TCP segmentation, generic receive offload , checksumming , multiple input/output queues, input flow director
  • Well over 10Gb/s to hardware from CPU to IOwith PCIe
  • Server $9k – Dual intel x5570 – 6 x 4GB DDR3 – SuperMicro X8DTE with 1 io hub – Server grade redundant PSU – NIC $3k , 2x Dual port Intel x520 10GE Nic + optics – Debian Lenny – Linux vanilla
  • created traffic generators as test setup – 45 machines
  • 1 sender 1 receiver ( 11 boxes to 11 boxes ) – 9.8Gb/s – 1.2Mpps
  • 2 senders , 2 receivers – 18Gb/s [ missed getting other stats but saturated links ]
  • 3.5Mpps before collapse , PCIe thrashing, NUMA inefficiencies , Young NIC drivers
  • Bridging instead of routing – L2 filters – performance approx same as IP routing
  • firewalling – Stress box with lots of small TCP connections (hard to create, generator needs to hold up 100s thousands of sessions) – Open, receive 4k data, close  – lots of tweaks to create traffic – Conntrack entrydefaults to 65k, upped to 10mil-
  • firewalling – 150,000 connections/second reached ( 5Gb/s)
  • firewalling – without contrack – saturates 10Gb/s
  • Number of Rules in Fw – 10Gb bi-directional , packetloss at 128-256 rules , no tuning – double that for single-direction – test has each packet going through each rule
  • Do you need to be an expert ? – If very fast, very cheap, then yes
  • Vyatta busy making this very easy – only pay for support, software is free
  • GigE (even lots of ports) is pretty easy
  • What experts do – Results over 90GB/s ( 40 in , 40 out ) on current hardware – People investigating for commercial reasons

Secure BGP – Geoff Huston

  • Anything evil is possible on the Internet
  • If I was evil , Through routing I’d attack DNS and forward to interceptor web server. Attack NZ based banks overseas so appears ok here
  • Through routing attack – route registry system, DNS root, trust anchors for TLS, critcal public servers, overwhelm routing system
  • Large networks advertised ( /8s etc) by various networks with no ovious reasons why. Same with AS numbers – v6 too
  • Nobody notices or cares about bogus routes beingoriginated
  • today’s networking is very insecure
  • Easy to – grab traffic , drop traffic , added false addresses to routing system , isolating or removing router from system . Don’t need to hack router just inject false routing information
  • what to do – protect you routers – standard security ( ssh access, maintain filter lists, user accts mngt, access log maintenance, snmp acls , etc )
  • what to do – bgp filters, md5 , passwords, prefix limits, watch out for errors causing bgp session to reset or come down – look at Rod Thomas’ BGP config templates
  • what to do – Check validity of routes your customers as you to route before adding to access control
  • alternatively – can BGP check each update to make sure it reflects the way things actually
  • RIRs sign who owns IPs , so routing changes for that network are in turned signed, resource certifcates. sign derivtive certs for sub-delegations of that resource
  • “AS 65000 can route” signed by the owner of that network.
  • What about path validation (signed AS above can just be prepended). A bit harder. – some progress and funding and test implimentations
  • Solution must cope with “partial use and deployment” , some good players will not use it any time soon.
  • Partially secured enviroment may be more operationally expensive but no more secure than what we have today.
  • Trust hierarchy is a “concentrating of vulnerability” – single point of attack
  • Only what to achieve useful outcomes?
  • Perhaps just anomaly detection to spot a large percentage of the problems
  • Will need key management systems and processes within companies like with website SSL certs

Trends in Cybercrime – Marcel van der Berg

  • Plenty of bots in NZ
  • Few comand and control servers in NZ
  • Approx 5000 unique IPs in NZ seen each day – trending up slightly long term
  • Increase in http botnets vs IRC botnets more static – around 500 controllers
  • C&C servers – IRC based in US and Eu – http based US , China , Russia
  • 1 million open recursive DNS servers just used in 1 attack
  • Resurgance of “pay per install” business – stable botnet platforms offer lucrative models
  • “dumps” – information on magnetic stripe card – reseller network – from ATMs / POS / Payment processors / personally / In transit / Any datbase holding data
  • “CVV” – personal data (addresses, names, etc )
  • Make credits cards to match info from dump
  • “201” cards with chip on them harder to write/use and numbers are worth less. Perhaps $50 for the blank card
  • It’s all about the people. It’s all about the money

NZNOG 2010 – Day 1 – Session 2

Emergence Video Internet EcoSystem – Bill Norton

  • Tier 1 ISPs , Teir 2 ISPs and Content Providers
  • Recent changes: Big Content companies peering 70%-80% of traffic, agressively pushing out and peering with cable companies. CDNs also disrupting. Big middle
  • Video big growth
  • Perhaps 80% of Internet traffic is video – > Video Internet
  • How hollywood delivers video and how internet delivers video are parallel and clashing
  • Hollywood System: creation/production (IP + money + work= movie )
  • Hollywood Distribution: Staged, theaters, pay-per-view, dvd, premium tv, commercial cable, broadcast TV
  • Hollywood model vs Internet Model clash
  • Lots of room for innovation (eg settop boxes, tive, boxeee, hulu) over commodity internet vs over cable infrastructure.
  • Hollywood system is 100% push
  • Hollywood system adjusting to take account of Internet model
  • Worldwide releases all at once
  • Download buy and rent available
  • Combo packs movie + dvd + soundtrack all in one package
  • Mini revolution achienved Vidoe Internet – Cheap cameras + editing software , Free upload and idstrobution (youtube) , dropping CDN/transit prices , broadband to the eyeballs , Home wifi , setop boxes
  • SkypeTV – killer App – what happens on mothers day?
  • What would purpose built video Internet look like?
  • Portable TV, tablet
  • Video Internet , innovation at lower end of content ( conference, cheap shows ) since cost of movies and primetime shows expensive to make.

Next 3 years – Philip Smith

  • Internet has been grwoing since the start
  • “The Long and Windy ROAD”
  • Work on next generation of IP since mid-1990s
  • Current Situation: Perception IPv6 hasn’t taken hold. Private sector worried about ROI to migrate
  • Stauts: Service providers get prefix automaticly. Much discussion about transition about operators, Deployment experience presentations, Many providers made backbones IPv6 compatable.
  • OS and Apps getting better
  • Content needs to be on IPv4 and IPv6 (not yet)
  • Ongoing debates – IPv6 Multhoming – Rigid IpV6 address allocation model “one size fits all” barrier
  • Ongoing – Not every device is IPv6 cabable (who cares about local lan devices) – We have enough IPv4 – Migration vs Co-existence (both will exist for years, dual-stck OS makes it trivial)
  • What not NAT?  Many serious issues
  • Is IPv4 running out? Yes!
  • IPv4 run-out policiys by RIRs (last /8) – soft landing- keep range for 6/4 NAT
  • Issues today – minimum content on Ipv6 , giving Ipv6 to customers might confuse them
  • Strategies available – Do Nothing  – Extend Ipv4 , push custs to NAT, Buy IPv4 – Deploy Ipv6 , dual stack, Ipv6 and NAT, various others
  • Proposals for prolong IPv4, various NAT options – NAT444/SP NAT – Dual Stack lite – NAT64 and IVI
  • Many require lage NAT box to translate all traffic v4/v6
  • IPv4 address markey – could happen – will addresses need to be registered with RIR to prove buyer has right to advertise them?
  • Spare /24s being grabbed and sold could cause routing table growth
  • Deaggregation various across the globe
  • Large provides marketing dept pointing to high ranking on CIDR report as proof they are “big”. Morons
  • Reports people towards top of list tend to feel flacky when you use them
  • BGP instabilitu report ( >5 updates per minute) – People towards top tend to be rough service.
  • Running low on AS numbers, transition to 32 bit – They are in the wild
  • Reasonable software support for 32-bits ASNs

Do your Fruit hang low – Adam Boileau

  • Adam is a penertration tester, Kiwicon organiser
  • Security guys are Jerks
  • Maybe you need better security guys
  • Secuity is fundimantally asymmetric – defenders do lots more work than attackers – Hackers only have to find one hole
  • completity == insecurity
  • 0day can happen happen to anyone
  • Full disclosure is dead
  • Vulnerabilies are worth money
  • Surity is not a product
  • Security is a property of the system as a whole
  • Why do you care? – Sin’t a network problem any more – Network is getting dumber (passive encryption) – clients arn’t exposed any more
  • Virtual everything – consulation changes everything – VLANs, VRFs, MPLS, Virtul servers, virtual hosting , Virtual firewalls, Virtual network segrigation
  • Lawful Intercept – Harder to hack 1000 people or 1 telcom LI system? – Vodafone Athens , T-mobile – Google vs China
  • The Target is you (again) – You are the management plane- you use crappy IE6 boxes on the corp domain
  • Your Desktop – AD, patch management, AV, outloook, TFTP server, IDS, twitter, facebook, outsourced desktop mangement
  • Security Metrics . Nobody knows how bad it is and who got hacked , media reporting is useless
  • Scanned 6.8 million IPs and put in mongoDB
  • data-mined – lots of A records, self-signed certs , specific apps
  • Presentened stats of various probably vulnerable boxes
  • Tried contacting owners , no luck
  • Crimes Act very vague, no case law, etc
  • what to do? Release? Release the toolchain? Release to some people? Just delete it?
  • Companies: Insomnia or Lateral Security

NZNOG2010 – Day 1 – Session 1

I attended the NZNOF 2010 conference in Hamilton. Notes as below.


  • Overview by Dean and Jonny on developments, especially about the trust

National Library Webharvest

  • 2nd Harvest planned in 2010
  • Harvest planned for April
  • Material from 1st harvest not yet online
  • Feedback requested on “Notification” , “robots Policy” , “Location of Harvester”
  • Would like feedback on the options paper

WAND Group

  • PMTUD (Path MTU discovery) in ipv6
  • Tested how well this is working
  • Sent ICMPv6 PTB message to hosts and see if remote host changes behavour in response to it (drop from >1280 to 1280 byte packets)
  • Tested 1647 websites (working ones from Alexa top 1 Million sites)
  • Used scamper to test
  • 58% PMTU worked, 34% packets too small ( might be working already, unsure)
  • 5% PMTU failed or no response
  • Working on protocols other than port80
  • Multiple vantage points, Other sources of addresses, web interface to toll
  • Conclusion – PMTUD mostly works – read RFC 4890

Anomaly detection in Networks – Andreas Loft

  • Doing this automaticly is good
  • Several existing tools
  • Nothing very concrete

WAND AMP Project

  • Boxes hosted by ISPs and PCs and sit around pinging each other
  • Good coverage of TelstraClear since ISPs use them as upsteeam, less so for Telecom
  • 1 ping / minute , 10 minute average posted
  • Cute interface to graphs
  • -> click on “NZ AMP”
  • Still under development

Shane Hobson – Velocity – Fibre to the home/premises

  • “How to build a Fibre network with a sack full of Government cash”
  • Broadband Challenge Fund $25M
  • Hamilton had 5 companies with some Fibre – Formed Hamilton Fibre Networks Ltd
  • HFN got $3m grant from fund
  • HFN partnered with Velocity Networks
  • 50-60km of Cable around Hamilton
  • Sell layer-2 ethernet services (similar to citylink)
  • Govt Ultra fast Broadband fund of $1500
  • Aim Ultra Fats BB to 75% of NZers
  • 100% of NZers in 25 (or 33) largest towns and cities
  • BB today is 25Mbit on ADSL2 contended to perhaps 250kb/s
  • UltraFats is 100Mb/s+ (50Mb/s upstream) with zero contention on access network
  • Huge amounts of bandwidth potentially ( hundreds of GB/s just for each say Hamilton )
  • ISPs need to decide: Buy Layer 2 or buy dark fibre?
  • ISPs: Different standards/services in different regions
  • ISPs: What content / services ?
  • ISPs: Peer at regional exchanges to reduce haul on Nat links?
  • ISPs: ISPANZ role?
  • ISPs: Caching, CDNs
  • ISPs: Zero rated “on net” traffic , Multicast IPTV, software updates
  • right now Hamilton provider doing:1/3 Dark Fibre, 1/3 L2 within companies , 1/3 to Internet
  • Frustrating to watch City Council digging up ground and not putting down ducts or letting other people do it.
  • Some councils are better

Time to retire some stories

As a sort of New Year’s resolution I’ve decided to retire a few stories that I sometimes tell people. I suspect I repeat some of these a bit too often (and sometimes to the same person) and they are getting a little stale. Feel free to offer other suggestions.

  • Kicking down door at work
  • My day as a court witness
  • Co-Worker electrocuted and comes back for more
  • Co-worker at Gang Party
  • Co-worker mugged on 1st day in Auckland
  • Colour-blind co-worker and windows
  • My Uncle meets Bill Gates
  • Stories about crazy head of the company I used to work for.
  • The day I meet the guy from the Fraud Squad

The above are all retired until Jan 1st 2015 unless specificly requested.


LCA2010 – Day 4

I ended up staying up quite late on Wednesday night so I was a little zonked out on thursday morning.

Keynote – Glyn Moody

  • Interviewed people for “rebel code” , found free software people “very nice” even compared to other people in computer industry
  • setup week before Linux kernel first released (Aug 1991)
  • Overview of public Library of science
  • Human Gnome project – DNA inherently digital
  • Bermuda Principles – finished annotated sequences submitted to public database
  • Jim Kent published and got full human gnome into public domain a short time before Celera finished their work and could have patented everything.
  • open data – data is not published just results – example of recent climate data being released, not a big problem if it had already been in public.
  • open notebook , reqular updates on progress
  • History of sharing art – Project Gutenbery 1971  .10 books 1991 , 1000 in 1997.
  • Various free licenses slightly incompatible , hard to convert between, took several goes to get licences correct
  • wikipedia – easy not programmer example of sharing tht people can understand – “open source is wikipedia for code”
  • Open government is more “Shared Source Government” rather than “Open Source Government”
  • Global economic crisis – tragedy of the commons
  • At least the Financial crisis has some winners
  • Very anti financial system, suggest more  “open source” options and commons
  • “if you share stuff you are destrying property, you are taking jobs away from the poor people” – How the debate is being framed

It was noted by one person that this year’s keynotes are more “Freedom” and “High tech”.

Lindsay Holmwood – Flapjack and Monitoring

  • Check – unit test – good bad ugly
  • Monitoring system – monitors for failing checks
  • 3 questions for monitoring systems – next check? , was check okay?, who do we notify? . Fetch , test , notify
  • fetch – lookup
  • test – execute , verify
  • notify – decide , callout
  • traditionally done in single process
  • but it’s an embarrassingly parallel problem
  • parts can be split. fetch+test fetch+notify – pass id/command between
  • precompile checks – so fetch is less expensive
  • transport between processes is the scheduler
  • no data collection when testing (graph seperately)
  • scheduler – workqueue – filled by populator, assigns stuff to notifier and workers
  • Lots of workers can be created (to do test)
  • flapjack – in ruby , talks to nagios plugin format
  • beanstalk – ansyncrnise workqueue service – ubuntu/debian packages
  • beanstalk – producer  puts jobs on beanstalk , consumer takes jobs off
  • uses named tubes (queues) , multiple tubes per instance
  • flapjack-worker – started up by flapjack-worker-manager starts multiple copies on machine. various control commands
  • worker is simple so linear scaling, spread across multiple machines required
  • flapjck-notifier – has manager to start it.
  • notifier has recipients.conf file with list of people to notify
  • notifier.conf – config for various notifiers (MAIL, SMS)
  • APIs – notifiers, filters, systems
  • notifier API – who , when and how sort of stuff.
  • “how many here use puppet – about a dozen – How many use Chef? – none “thanks a shame” “no it’s not”
  • persistence API – store stuff , mysql, couchdb whatever, standard way to store data.
  • filter API – parent checks hierarchy (so don’t check ports if host down)
  • flapjack-admin – pending – nodes , check templates , checks (check template + node ) , batches (group of checks)
  • 3 types of checks
  • Gaugaes – stuff within range – collectd ( point flapjack at collected output )
  • Behavoural tests – cucumber-nagios
  • Trending – reconoiter – growing area
  • collectd – gets stats from anything – nagios bridge – collectd-nagios queries collectd data
  • collectd client – gathers data from node and sends to collectd server
  • collectd forwarding server – agregates, filters and forwards
  • falapjack – crrently gems, soon to be real packages

Bob Edward – Yubikey authentication in a mid-sized organisation

  • Reusable passwords are dead , hard to remeber, something you know which can be shared and discovered, captured, guessed
  • Alternative – One time Passwords – doesn’t matter if captured.
  • examples – RSA keys, SMS based systems, Yubikey, 2 factor authentication
  • Created by Yubico in sweden, open-source
  • Looks like a USB keyboard to a computer, generates a 44 character OTP each time button is pressed. No batteries, 2st 23 characters fixed for each key
  • $12 each in volumn – $40 as one-off
  • Based on secret AES 128-bit key
  • Yubicoships yubikeys with pre-generated IDs and AES keys. Offer publicauthentication, they know secret 128-bit key, need to trust them
  • secret-id+sess+timestamp+session+rand+CRC  string created by key , then encrypted and public ID prepended.
  • Server decrypts , checks checksums and looks to make sure secret-id matches and session and timestamps are incrimented from previous values.
  • Unless you trust and always want to use Yubicom’s servers you should reprogram you keys with your own keys and IDs. Can’t then be used against Yubicom’s server.
  • weaknesses – requires computer with usb port that accepts usb keyboard – some bugs with 1st generation keys – unused generated keys remian live until the next valid key is used
  • You can run your own server fairly easily – ykaserver – various interfaces, postgress database for storage – can also call out to PAM for two-factor authentication
  • softykey – software Yubikey – can use to generate 1-time pad for stuff without usb keyboard interfaces
  • Tested with ssh, VPNs , web logins – mostly use PAM or LDAP method
  • See Linux Journal and

vimperator – automatic launch prog for netbooks

Jan Schmidt – Towards GStreamer 1.0

  • History of dev, faster bits during hackfests, when switched to git etc
  • Overview of last year, switched to git, slowdown when people busyswitched to binary registry
  • Support for various DVD playback  functions, special subtitles etc.
  • I’m not really in this area so I was just listening to get an idea where things are going. A bit too much detail for me at times.

Adam Jackson – The rebirth of Xinerama

  • Once again this was a bit over my head. It does look like the X guys spend a lot of time fighting assumptions built into the protocol and code 10 years ago however.

Stewart Smith et al – Building a Database kernel with Lego Like parts (Drizzle)

  • What would you change about Mysql – Modular architecture
  • Some crazy legacysuff in the Mysql code – good oppertunity to clean
  • move alot of code out of core, especially option parts – understandable and to reduce load – don’t load if you don’t need
  • more code coverage with tests
  • plugin interfaces – protocols, replication , logging, etc
  • modular replication system
  • general refactoring of storage engines
  • “If part of API sucks then fix API rather than work around it”
  • New this week – rot13() powerful encryption
  • Authentication plugins – auth_pam , auth_http
  • Various Logging plugins – logging_query , logging_syslog
  • Drizzle Community – All contributors equally – All project information public – No contributor license agreeements – Release early and often (~2 weeks ) – 100+ contributors , 500+ on mailing list
  • Milestone releases
  • When production release? – waiting to solidfy compatability – Sounds like a few months. – Reliable but still in flux
  • Pacakages to be pushed out to dists once things stable

Afterwards I had some dinner and went to the Professional Deligates networking session.