NZNOG 2010 – Day 2 – Session 3

Announcement at start of session that Telecom New Zealand now has an official Interconnect/Paid Peering Policy and Contact. Details to be Published. Ask Greg from Telecom for help.

Internetnz Update – Jordan Carter

  • General updates and new structure, new CEO
  • 4 main areas ( Openness, rights and responsibilities, security)
  • IPv6 Task force , replace steering group
  • Copyright – replacement policy looks better, but sneaky changes might come back
  • ACTA – Key concern , lack of transparency, http://www.acta.net.nz
  • DIA filtering – voluntary and uses BGP . Give webpage, can report false alarm
  • Filter – only http, erodes end-to-end , privacy concerns , might be later abused (scope creep)
  • Filter – Send signal that “The government has made the Internet safe”
  • Internet opposed – DIA unhappy with that angle
  • Fibre Stuff – “Last day for 1.5 billion lolly scramble”
  • Regional Networks or one big National Network
  • Hard to tell what will happen – Similar exercise in Aus and Govt went back to drawing board
  • What happens to International Bandwidth?
  • Please join, followon twitter http://twitter.com/internetnz

APNIC update and much more – Elly Tawhai

  • Over 2000 members
  • 1400+ monthly helpdesk enquiries ( 55% growth since last year)
  • Allocations around 100 per month
  • Various Policy changes coming up – Prop-050 (xfering address space ) , Prop-073 (sinple IPv6 allocations – 1 click) , Prop-074 (32 bit ASNs treatment same as 16 bit ones pushed back a year) , Prop-075 (recover historical ASNs)
  • Policies under discussion – Prop-78 ( Final /8 , only people deploying ipv6) , Prop-079 (abuse contact info in objects ) , Prop-080 ( Removal of IPv4 prefix exchange policy )
  • Several more allocation policies in pipeline
  • Recent Survey leading to priorities
  • Various my.apnic updates (web services even), support of research
  • More DNS root servers (Taiwan , Mongolia)
  • Please Participate

RIPE News – Tools and news – George Michaelson

  • RIPE used to be a research place and then became a RIR. RIPE labs is a return to the past
  • http://labs.ripe.net
  • Platform to test and evaluate new tools, feedback cycle
  • INRDB – big cloud of assignments, table dumps, dumps
  • Resource explainer
  • Various measurements , visualisation and links to tools. DNS reply size tester
  • Why – fast turnaround, engagement, no service g’tees

IPv6 flow chart – Nathan Ward

  • Make decission which IPv6 or IPv4/Ipv4 translation technology you should use
  • Tunnel Broker, 6to4, 6RD, Teredo, Dual stack lite, Double NAT, Dual stack
  • Other stuff that I wasn’t paying attention two
  • IPv6 addressing schemes
  • Sparse allocations
  • gives a sample which I won’t copy, look at his slides
  • Customer assignmesnt. Nathan likes /56s or RFC recomended /48. Take your pick

Andy is Curious – Andy Linton

  • Are Universities turning out the right people?
  • Good at turning out applications programmers not systems programmers
Share

NZNOG 2010 – Day 2 – Session 2

DNSSEC at the root zone – Joe Abley

  • ICANN – Manges the Ket-signing-key (KSK) – accepts DS records from zone operators – sends update to DoCfor auth and to veriSign for implimentation
  • DoC auth changes and Verisign impliments the change
  • New process has Verisign signs the keys. V gets a few weeks of of KSKs that Doc signs in batches beforehand
  • DNSSEC Practice Statement – describes procedures, currently drafts
  • Around 20 Community Trusted Representative ( TCR ) have an active roll in the mangement of the KSK
  • 2 copies of the Keys, west coats and east coast. Plus distributed backup
  • “ceromony” for each step in procedure, required what you do and how many people and which people are present.
  • Similar to what x.509 CAs do
  • KSK is 2048 RSA key rolled every 2-5 years ( RFC 5011 but not all have that support) –  Signature using SHA-256
  • ZSK is 1024 RSA key – signed with NSEC – rolled 4 times year – Signature is SHA-256
  • Time cycle every 90 days – ZSK overlap of a couple of weeks
  • Root trust Anchor – published in XML document with constant URL – plain DNS record – PKCS#10 cert CSR , as self signed pub key, signable by others if they want
  • DO=1 part of EDNS0 – says client wants DNSSEC – many clients set bit even though most won’t really want them right now – will cause all queries to jump in size
  • Hard to sign root and then rollback
  • Staged deployment – Start servering DNSSEC for 1 root server at a time – L-Root first, then A, then the others with J last
  • DURZ – Unverifiable key published as placeholder
  • Measurement – Packet captures , diologue with operators – wide range of pre-testing with various software – test with clients that drop large packets
  • DS change requests – TLD procedure to be decided – DS requests 1-2 months before zone published
  • http://www.root-dnssec.org
  • Timeline – Test key signing Dec 2009 – Jan 2010 . Jan – July 2010 roll out signed roots . July 2010 Full Production
  • Lots of documentation on website
  • Indication of big jump in tcp queries presumably because udpreplies are too big

ENUM – Jay Daley

  • Why Doesn’t telephony work like email?
  • Email you choose how to published your email record, where to host, what emails to accept, can outsource, totally in control
  • So IP telephony should be easy too?
  • Unfortunately not
  • Non site-local numbers MUST go to telcoto get delivered
  • Missing – single , global directory linking telephone nmbers to voip numbers
  • This is ENUM . Telephone Number -> Domain Name – Simple Algorithm – e164.arpa – 04 931 6970 -> 0.7.9.6.1.3.9.4.4.6.e164.arpa
  • Won’t be typed, Translation done by a device – people still type out over fashon numbers
  • Register your number, create zone. Add NAPTOR records to DNS zone. Special records to specifiy endpoints (usually sip records), receive calls
  • NAPTO records do interesting stuff . eg “dig +short nsrs.tel naptr”
  • how? Option 1-  enable on your VOIP PBX that is internet connected
  • Option 2 – on session border controller – “enterprise”
  • Option 3 – ENUM proxy ( if existing SBC doesn’t handle enum)
  • Registration process – not same as for domains since numbers already registered – needs authentication
  • Various methods of authentication in different places
  • No ENUM in NZ . Available in UK, Holland, Ireland, Germany, Austria but not significant takeup
  • Reasons for lack of takeup in those countries – lack of mindshare – hostility from telcos
  • Why not in NZ – TCF 2006 report – Privacy issues (but only publish what you like) – Emergancy services access (no idea where callers are) (but all VOIP has problem ) – Polcy/Goverance – “Carrier Issues”
  • ENUM isabout control – movingit from carrier to you
  • Key users – Call centres , ENUM instead of 0800 – Large supply chains (mandate VOIP ) – Multiple sites , simplyfy provisioning
  • Won’t happen without demand
  • “On the Internet voice is just another application”
  • Significant political and commercial resistence from Telcos

Day in the Life of the Internet – Sabastian Castro

  • 4 years of DNS data
  • DITL motivation – network measurement – collection of data from DNS root servers – yearly since 2006
  • More and more root servers, Alt root servers, gTLDs etc passive traces, 48-72 hours
  • concentrate on root server data
  • Pick best 24 hours out of total window
  • 4-8 billion queries, 3-6 million unique clients – sm5-12% recursive queries
  • Mostly A queries, AAAA increasing due to gluerecords being added (why are IPv4 clients sending AAAA queries when they probably won’t/can’t use)
  • 70% of clients are EDNS are capable ( 90% of these are D0 enabled )
  • However clients sending lots of of queries (probably broken) have good support – But clients that query less have lover level of support
  • 10 invalid TLDs represent 10% of queries ( .local , .localdomain , wpad , invalid , home , belkin , corp , lan )
  • Impossible to track down
  • Most queries from NZgoing to Auckland root and Brisbane root but some going to overseas servers (those might be use simple round-robin picking)
  • Lessons – Data collection is hard – clock skew , dat loss , wrong command line options , bad network taps
  • Data management – moredat , more participants – more formats – big effortto normalize data , fill gaps , fix clock skew .
Share

NZNOG 2010 – Day 2 – Session 1

Lightning Talk

  • Geoff Huston – Stateless TCP and DNS
  • TCp limitations – Rough a high load
  • UDP Limitations – Requires IP fragmentation
  • Problems when response bigger than MTU , Fragments of UDP IPv6 often dropped. Switching to TCP drives up load again
  • Simulate UDP with TCP – do minimal crappy respose to fill headers
  • Ignore options, server doesn’t retransmit, ignore anything else from client, just closes connection
  • No reliability, No Flow Control, bad Idea but seems to work
  • Olof Kasselstrant – IXOR
  • Small IX in Malmo and Copenhagen (2nd site being looked at)
  • DIX only IX in Denmark
  • Sponsors for Fibre and Equipment
  • Exchange in 2 countries. Does it affect “must peer in 4 countries” agreement.
  • Dream to be in 4 sites soon
  • CCIP – Barry Brailey
  • Getting out of rewriting Microsoft patch notices
  • “investigation and analysis” function being dropped
  • Infomation and Alerting – website , newsletter, alerts – alerts targetted and highish threshold –
  • Outreach and partnering – main function – lease with overseas certs – talk to various groups – Education: presentation, newsletters, exercises (CyberStorm III – volenteers )
  • Security Information exchanges – Various groups – traffic light protocol – Looking at some new forums – Maybe ISP SIE
  • Cloud Computing for Service Providers – Richard Wade
  • As a service provider – should I care?
  • Infrastructure Foundation (Cisco, EMC, HP)
  • Infrastructure as a service (Amazon , Sun , Savvis )
  • Platform as a swervice (Amazon, MS Azure )
  • Software as a Service ( Salesforce, Google apps)
  • Integrate mngt ( network, servers, hypervisor, storage ) – unified fabric
  • Why and Why Should I care
  • Customer Ads – Eliminate Capex – Reduce Opex – IT as a utility
  • Customer Probs – No LAN apps (overseas often) – WAN now biz critical – Operational relationship with overseas provider – Legal jurisdiction of data
  • Service Provider ads – Understand managed services – Existing datacentres and infrastructure – OSS , process staff and contacts – SLAs – Domestic provider
  • Sp Probs – Managed cust revenue declining – Race to bottom? – Increase International transit – High expectations of quality and relaibility
  • Lame aternative IX Update technique – Simon Blake
  • New system to update filter lists for IXs
  • Citylink can instead download list of networks from customer URL
  • Pulls list daily
  • If diff email for confirmation or action it immediately
  • ALTO – LLyod
  • Helping p2p users select local/nearby peers
  • GeoIP and anycasting – rough
  • ALTO allows ISP to provide application, localtion, routing information, charging information, performance.
  • ISP puts on network some servers (itrackers) that deliver to p2p client the policy information
  • p2p caches (very close to edge) can be advertised
  • No currently in use in the wild
  • IPv6 taskforce – Dean Pemberton
  • Internetnz+ MED
  • TechSIG – 3 Hui in 2009 – Aimed at CIO/CTO – Went really well
  • Looking at more training (session in 2009 already)
  • Other things Task Force can do?

Building a Datacentre for less than $1 million – Gerald Creamer

  • When it’s your own money you care so much more
  • Had to move datacenter to another building
  • Short is that you can’t do it for less than $1m
  • Significant cost areas – Physical – power – cooling – network – time
  • The right building – 18 m search – 100 sites looked at – 7 sites investigated – 4 site due diligence
  • Engineers – “consultation” vs “converstaion”
  • First culling – all concrete – Not ground , not top floor – Strong 5kPa – high stud – no sprinklers – built between 50 and mid-80s – CBD fringe
  • $400 per m2 to strength building
  • 2nd culling – close to street transformer – shorter power cables runs in building – shorter pipes for colling – outdoor space – generater space – near data networks
  • Useful – friendly landlord – nice bank – recession (kean landlord)
  • Save money – quality pre-owned hardware – “free” stuff – Ask experts – do some stuff yourself – Get experts to do others
  • Cables up abandoned lift shaft
  • 2nd hand generator – not as large as final requirement but bigenough for current build
  • Room to upgrade UPS, generator, cables and space spec’d for more
  • domestic meters to measure power in each rack
  • Process Coolers (cheaper) 28KW each $1500/KW cost – $70k of aircon for $7k – check serial number with manufacter to find product history
  • Seismic Bracing – $30k
  • Helped corps clear out datacenters they were moving out of rooms ( “make good” on leases) and picked up some equipment
  • Citylink and Telstra provisioned fibre. Telecom less helpful.
Share

NZNOG 2010 – Day 1 – Session 4

IPv6 deployment scenarios – Brian Carpenter

  • Assumed v6 deployed by v4 ran out
  • change transition model
  • More internetworking than original ipv6 design originally anticipated. Assume v6 clients will need to access v4 servers forever
  • Tunnels – Dual Stack Lite ( share ipv4 addr amung custs by combining UPv4-in-IPv6 and NAT, Driven by Comcast BB model ) – 6rd ( blend of 6to4 and ISTAP providing atumatic tunning of IPv6-in-IPv4 to ISP subscribers. Deployed by Freenet.fr)
  • Older mesh and hub+spoke models also documented.
  • NAT64 – old NAT-PT deprecated
  • NAT64 – millions of IPv6-only custs needing access to IPv4-only services
  • NAT64 only solves 1 problem – cannot be met my dual-stack – DNS64 dns server creates AAAA of site only with A record. Packets to NAT64 box and translated
  • Various problems. 7 ietf drafts. Only solving since case
  • V6OPS WG- Emerging Service Provider Scenarios for IPv6 Deployment – ID and survey ISPs then publish draft 03/2010

Rapid IPv6 Deployment in ISp Network – Skeeve Stevens

  • AIM – Get people to use IPv6
  • eintellego runs ISPs
  • What stopping ISps implimenting IPv6
  • Why not? – Too expensive , bigger ISPs yes, smaller ISPs perhaps not, NOT expensive to do enough to be able to play with it
  • Why not? – Too Hard – Lack of internal skills – IPv6 is NOT hard, cisco admin should be basic IPv6 in 2h and IPv6 BGP in under a day – Play now or else you will be overwhelmed later when everybody is yelling
  • Why Not? – Don’t know where to start – Start with a external co-lo box in the US – Allocate small amount of time – Get access to a lab – Start at the border
  • Why Not? – No one asking for it – True enough – Don’t know about Ipv4 exhaustion, but they will
  • Why Not? – Little vendor support – improving – DSL CPE equipment getting better – Carrier Grade NAT ( CGN/LSN)
  • Why Not? – What is IPv6? – From Many IT professionals – Integrators have minimal experience
  • Why Not? – Who can help me? – commerially, very few people – Some training courses – Community helps
  • IPv6 is big, break it down into stages
  • Experiment Externally
  • Get allocation from APNIC
  • Enable your Edge (BGP)
  • Enable Core
  • Enable desktop
  • Enable your hosting
  • Enable Operation Support Systems
  • One hosting company just took 1 week
  • Very rapid training, just a couple of days
  • Simplified addressing – short to medium term – rapid deployment – format – 2406:9800::F:203.18.102.99 – Use F0 instead of”F” for next pop – Using /128s will increase routing table – “chazwazza” is ipv6 equiv of “octet”
  • We use /64 for all end customer assignments – static routes to make v4-in-v6 work
  • NTP might not work
  • Some security concerns
  • Go through commons OS, Daemons, Hardware ( phones, printers, UPS, gameboys)
  • Might have to tunnel
  • Hassel carrier if not provided
  • Hassel vendors if they don’t work
  • Some parts won’t happen overnight
  • Predictions – Telstra selling IPv6 mid 2010 – Resource rush to grab IPv4 IPs while they can , surge in APNIC membership – exhaustion brought forward – secondary market will come – APNIC will lose control

Simply allocation of ipv6 addr to ipv4 holders – Elly Tawhai

  • Policy 73
  • Encourage greater uptake of IPv6
  • An APNIC member with IPv4 allocation is eligible /32 . Member with assignment gets a /48
  • One-Click IPv6 from my.apnic.net

NZ/IPv6 from (offshore) DNS – GGM (no name)

  • Passive tap on DNS servers – spot reverse lookups for in-addr.arpa
  • Capture all DNS in 1 day look for NZ IPs
  • 1 in 10,000 lookups are doing IPv6
  • 1 in 200 queries for DNS using IPv6
  • 87.5% active delegattions in 24 hour period
  • 45% of V6 networks live in 24 period
  • 52% of v6 is Macs
  • IPv6 not on the phone
  • 6to4 common even with providers that do IPv6 native

Things running late so IPv6 panel skipped.

Share

NZNOG 2010 – Day 1 – Session 3

NZ Internet Task Force – Paul McKitrick

  • Out of Cyberstorm planning session – “what to do about botnets?”
  • Task Force has Steering Committee
  • Trust is essential – New members vetted – slow growth of membership
  • Protocol on how widely specific pieces of information can be shared
  • Information sharing – networking – training courses ( honeynet, shadow server foundation, team cymru )
  • Focus areas – Telecommunications (telecom honeynet, Uni grads seconded to telecom, Walled Gardens)  – Research (Botsearch.py , VUW honeynet , data Brokerage ) – Stretegy ( Phishing site takedowns, Nat Cyber Security day 2010 , NZ Computer crime and Secuity project )
  • NZ Ips sending 110 million spams per day
  • Why – good for “.nz inc” , Opportunities for research, networking, conduit for disclosure

Bits on a Budget – Perry and Jamie

  • chellenging the belief that PCs running linux useful only for slow, small, un-important routing jobs
  • changes in last few years means this may need to be re-evaluated
  • What changed – PC Arch, Intel stopped sucking , Quick Path Interconnect , PCIe , Multicore – Substantial improvement in Linux – Multiqueue RX/TX to take advatage of multicore
  • Intel x520 10 GigE cards – Significant hardwareoffload – TCP segmentation, generic receive offload , checksumming , multiple input/output queues, input flow director
  • Well over 10Gb/s to hardware from CPU to IOwith PCIe
  • Server $9k – Dual intel x5570 – 6 x 4GB DDR3 – SuperMicro X8DTE with 1 io hub – Server grade redundant PSU – NIC $3k , 2x Dual port Intel x520 10GE Nic + optics – Debian Lenny – Linux 3.6.32.5 vanilla
  • created traffic generators as test setup – 45 machines
  • 1 sender 1 receiver ( 11 boxes to 11 boxes ) – 9.8Gb/s – 1.2Mpps
  • 2 senders , 2 receivers – 18Gb/s [ missed getting other stats but saturated links ]
  • 3.5Mpps before collapse , PCIe thrashing, NUMA inefficiencies , Young NIC drivers
  • Bridging instead of routing – L2 filters – performance approx same as IP routing
  • firewalling – Stress box with lots of small TCP connections (hard to create, generator needs to hold up 100s thousands of sessions) – Open, receive 4k data, close  – lots of tweaks to create traffic – Conntrack entrydefaults to 65k, upped to 10mil-
  • firewalling – 150,000 connections/second reached ( 5Gb/s)
  • firewalling – without contrack – saturates 10Gb/s
  • Number of Rules in Fw – 10Gb bi-directional , packetloss at 128-256 rules , no tuning – double that for single-direction – test has each packet going through each rule
  • Do you need to be an expert ? – If very fast, very cheap, then yes
  • Vyatta busy making this very easy – only pay for support, software is free
  • GigE (even lots of ports) is pretty easy
  • What experts do – Results over 90GB/s ( 40 in , 40 out ) on current hardware – People investigating for commercial reasons

Secure BGP – Geoff Huston

  • Anything evil is possible on the Internet
  • If I was evil , Through routing I’d attack DNS and forward to interceptor web server. Attack NZ based banks overseas so appears ok here
  • Through routing attack – route registry system, DNS root, trust anchors for TLS, critcal public servers, overwhelm routing system
  • Large networks advertised ( /8s etc) by various networks with no ovious reasons why. Same with AS numbers – v6 too
  • Nobody notices or cares about bogus routes beingoriginated
  • today’s networking is very insecure
  • Easy to – grab traffic , drop traffic , added false addresses to routing system , isolating or removing router from system . Don’t need to hack router just inject false routing information
  • what to do – protect you routers – standard security ( ssh access, maintain filter lists, user accts mngt, access log maintenance, snmp acls , etc )
  • what to do – bgp filters, md5 , passwords, prefix limits, watch out for errors causing bgp session to reset or come down – look at Rod Thomas’ BGP config templates
  • what to do – Check validity of routes your customers as you to route before adding to access control
  • alternatively – can BGP check each update to make sure it reflects the way things actually
  • RIRs sign who owns IPs , so routing changes for that network are in turned signed, resource certifcates. sign derivtive certs for sub-delegations of that resource
  • “AS 65000 can route 192.2.200.0/24” signed by the owner of that network.
  • What about path validation (signed AS above can just be prepended). A bit harder. – some progress and funding and test implimentations
  • Solution must cope with “partial use and deployment” , some good players will not use it any time soon.
  • Partially secured enviroment may be more operationally expensive but no more secure than what we have today.
  • Trust hierarchy is a “concentrating of vulnerability” – single point of attack
  • Only what to achieve useful outcomes?
  • Perhaps just anomaly detection to spot a large percentage of the problems
  • Will need key management systems and processes within companies like with website SSL certs

Trends in Cybercrime – Marcel van der Berg

  • Plenty of bots in NZ
  • Few comand and control servers in NZ
  • Approx 5000 unique IPs in NZ seen each day – trending up slightly long term
  • Increase in http botnets vs IRC botnets more static – around 500 controllers
  • C&C servers – IRC based in US and Eu – http based US , China , Russia
  • 1 million open recursive DNS servers just used in 1 attack
  • Resurgance of “pay per install” business – stable botnet platforms offer lucrative models
  • “dumps” – information on magnetic stripe card – reseller network – from ATMs / POS / Payment processors / personally / In transit / Any datbase holding data
  • “CVV” – personal data (addresses, names, etc )
  • Make credits cards to match info from dump
  • “201” cards with chip on them harder to write/use and numbers are worth less. Perhaps $50 for the blank card
  • It’s all about the people. It’s all about the money
Share

NZNOG 2010 – Day 1 – Session 2

Emergence Video Internet EcoSystem – Bill Norton

  • Tier 1 ISPs , Teir 2 ISPs and Content Providers
  • Recent changes: Big Content companies peering 70%-80% of traffic, agressively pushing out and peering with cable companies. CDNs also disrupting. Big middle
  • Video big growth
  • Perhaps 80% of Internet traffic is video – > Video Internet
  • How hollywood delivers video and how internet delivers video are parallel and clashing
  • Hollywood System: creation/production (IP + money + work= movie )
  • Hollywood Distribution: Staged, theaters, pay-per-view, dvd, premium tv, commercial cable, broadcast TV
  • Hollywood model vs Internet Model clash
  • Lots of room for innovation (eg settop boxes, tive, boxeee, hulu) over commodity internet vs over cable infrastructure.
  • Hollywood system is 100% push
  • Hollywood system adjusting to take account of Internet model
  • Worldwide releases all at once
  • Download buy and rent available
  • Combo packs movie + dvd + soundtrack all in one package
  • Mini revolution achienved Vidoe Internet – Cheap cameras + editing software , Free upload and idstrobution (youtube) , dropping CDN/transit prices , broadband to the eyeballs , Home wifi , setop boxes
  • SkypeTV – killer App – what happens on mothers day?
  • What would purpose built video Internet look like?
  • Portable TV, tablet
  • Video Internet , innovation at lower end of content ( conference, cheap shows ) since cost of movies and primetime shows expensive to make.

Next 3 years – Philip Smith

  • Internet has been grwoing since the start
  • “The Long and Windy ROAD”
  • Work on next generation of IP since mid-1990s
  • Current Situation: Perception IPv6 hasn’t taken hold. Private sector worried about ROI to migrate
  • Stauts: Service providers get prefix automaticly. Much discussion about transition about operators, Deployment experience presentations, Many providers made backbones IPv6 compatable.
  • OS and Apps getting better
  • Content needs to be on IPv4 and IPv6 (not yet)
  • Ongoing debates – IPv6 Multhoming – Rigid IpV6 address allocation model “one size fits all” barrier
  • Ongoing – Not every device is IPv6 cabable (who cares about local lan devices) – We have enough IPv4 – Migration vs Co-existence (both will exist for years, dual-stck OS makes it trivial)
  • What not NAT?  Many serious issues
  • Is IPv4 running out? Yes!
  • IPv4 run-out policiys by RIRs (last /8) – soft landing- keep range for 6/4 NAT
  • Issues today – minimum content on Ipv6 , giving Ipv6 to customers might confuse them
  • Strategies available – Do Nothing  – Extend Ipv4 , push custs to NAT, Buy IPv4 – Deploy Ipv6 , dual stack, Ipv6 and NAT, various others
  • Proposals for prolong IPv4, various NAT options – NAT444/SP NAT – Dual Stack lite – NAT64 and IVI
  • Many require lage NAT box to translate all traffic v4/v6
  • IPv4 address markey – could happen – will addresses need to be registered with RIR to prove buyer has right to advertise them?
  • Spare /24s being grabbed and sold could cause routing table growth
  • Deaggregation various across the globe
  • Large provides marketing dept pointing to high ranking on CIDR report as proof they are “big”. Morons
  • Reports people towards top of list tend to feel flacky when you use them
  • BGP instabilitu report ( >5 updates per minute) – People towards top tend to be rough service.
  • Running low on AS numbers, transition to 32 bit – They are in the wild
  • Reasonable software support for 32-bits ASNs

Do your Fruit hang low – Adam Boileau

  • Adam is a penertration tester, Kiwicon organiser
  • Security guys are Jerks
  • Maybe you need better security guys
  • Secuity is fundimantally asymmetric – defenders do lots more work than attackers – Hackers only have to find one hole
  • completity == insecurity
  • 0day can happen happen to anyone
  • Full disclosure is dead
  • Vulnerabilies are worth money
  • Surity is not a product
  • Security is a property of the system as a whole
  • Why do you care? – Sin’t a network problem any more – Network is getting dumber (passive encryption) – clients arn’t exposed any more
  • Virtual everything – consulation changes everything – VLANs, VRFs, MPLS, Virtul servers, virtual hosting , Virtual firewalls, Virtual network segrigation
  • Lawful Intercept – Harder to hack 1000 people or 1 telcom LI system? – Vodafone Athens , T-mobile – Google vs China
  • The Target is you (again) – You are the management plane- you use crappy IE6 boxes on the corp domain
  • Your Desktop – AD, patch management, AV, outloook, TFTP server, IDS, twitter, facebook, outsourced desktop mangement
  • Security Metrics . Nobody knows how bad it is and who got hacked , media reporting is useless
  • Scanned 6.8 million IPs and put in mongoDB
  • data-mined – lots of A records, self-signed certs , specific apps
  • Presentened stats of various probably vulnerable boxes
  • http://lowhangingkiwifruit.com
  • Tried contacting owners , no luck
  • Crimes Act very vague, no case law, etc
  • what to do? Release? Release the toolchain? Release to some people? Just delete it?
  • Companies: Insomnia or Lateral Security
Share

NZNOG2010 – Day 1 – Session 1

I attended the NZNOF 2010 conference in Hamilton. Notes as below.

Opening

  • Overview by Dean and Jonny on developments, especially about the trust

National Library Webharvest

  • 2nd Harvest planned in 2010
  • Harvest planned for April
  • Material from 1st harvest not yet online
  • Feedback requested on “Notification” , “robots Policy” , “Location of Harvester”
  • Would like feedback on the options paper

WAND Group

  • PMTUD (Path MTU discovery) in ipv6
  • Tested how well this is working
  • Sent ICMPv6 PTB message to hosts and see if remote host changes behavour in response to it (drop from >1280 to 1280 byte packets)
  • Tested 1647 websites (working ones from Alexa top 1 Million sites)
  • Used scamper to test
  • 58% PMTU worked, 34% packets too small ( might be working already, unsure)
  • 5% PMTU failed or no response
  • Working on protocols other than port80
  • Multiple vantage points, Other sources of addresses, web interface to toll
  • Conclusion – PMTUD mostly works – read RFC 4890

Anomaly detection in Networks – Andreas Loft

  • Doing this automaticly is good
  • Several existing tools
  • Nothing very concrete

WAND AMP Project

  • Boxes hosted by ISPs and PCs and sit around pinging each other
  • Good coverage of TelstraClear since ISPs use them as upsteeam, less so for Telecom
  • 1 ping / minute , 10 minute average posted
  • Cute interface to graphs
  • http://www.wand.net.nz -> click on “NZ AMP”
  • Still under development

Shane Hobson – Velocity – Fibre to the home/premises

  • “How to build a Fibre network with a sack full of Government cash”
  • Broadband Challenge Fund $25M
  • Hamilton had 5 companies with some Fibre – Formed Hamilton Fibre Networks Ltd
  • HFN got $3m grant from fund
  • HFN partnered with Velocity Networks
  • 50-60km of Cable around Hamilton
  • Sell layer-2 ethernet services (similar to citylink)
  • Govt Ultra fast Broadband fund of $1500
  • Aim Ultra Fats BB to 75% of NZers
  • 100% of NZers in 25 (or 33) largest towns and cities
  • BB today is 25Mbit on ADSL2 contended to perhaps 250kb/s
  • UltraFats is 100Mb/s+ (50Mb/s upstream) with zero contention on access network
  • Huge amounts of bandwidth potentially ( hundreds of GB/s just for each say Hamilton )
  • ISPs need to decide: Buy Layer 2 or buy dark fibre?
  • ISPs: Different standards/services in different regions
  • ISPs: What content / services ?
  • ISPs: Peer at regional exchanges to reduce haul on Nat links?
  • ISPs: ISPANZ role?
  • ISPs: Caching, CDNs
  • ISPs: Zero rated “on net” traffic , Multicast IPTV, software updates
  • right now Hamilton provider doing:1/3 Dark Fibre, 1/3 L2 within companies , 1/3 to Internet
  • Frustrating to watch City Council digging up ground and not putting down ducts or letting other people do it.
  • Some councils are better
Share

Time to retire some stories

As a sort of New Year’s resolution I’ve decided to retire a few stories that I sometimes tell people. I suspect I repeat some of these a bit too often (and sometimes to the same person) and they are getting a little stale. Feel free to offer other suggestions.

  • Kicking down door at work
  • My day as a court witness
  • Co-Worker electrocuted and comes back for more
  • Co-worker at Gang Party
  • Co-worker mugged on 1st day in Auckland
  • Colour-blind co-worker and windows
  • My Uncle meets Bill Gates
  • Stories about crazy head of the company I used to work for.
  • The day I meet the guy from the Fraud Squad

The above are all retired until Jan 1st 2015 unless specificly requested.

Share

LCA2010 – Day 4

I ended up staying up quite late on Wednesday night so I was a little zonked out on thursday morning.

Keynote – Glyn Moody

  • Interviewed people for “rebel code” , found free software people “very nice” even compared to other people in computer industry
  • arXiv.org setup week before Linux kernel first released (Aug 1991)
  • Overview of public Library of science
  • Human Gnome project – DNA inherently digital
  • Bermuda Principles – finished annotated sequences submitted to public database
  • Jim Kent published and got full human gnome into public domain a short time before Celera finished their work and could have patented everything.
  • open data – data is not published just results – example of recent climate data being released, not a big problem if it had already been in public.
  • open notebook , reqular updates on progress
  • http://en.wikipedia.org/wiki/Open_Notebook_Science
  • History of sharing art – Project Gutenbery 1971  .10 books 1991 , 1000 in 1997.
  • Various free licenses slightly incompatible , hard to convert between, took several goes to get licences correct
  • wikipedia – easy not programmer example of sharing tht people can understand – “open source is wikipedia for code”
  • Open government is more “Shared Source Government” rather than “Open Source Government”
  • Global economic crisis – tragedy of the commons
  • At least the Financial crisis has some winners
  • Very anti financial system, suggest more  “open source” options and commons
  • “if you share stuff you are destrying property, you are taking jobs away from the poor people” – How the debate is being framed

It was noted by one person that this year’s keynotes are more “Freedom” and “High tech”.

Lindsay Holmwood – Flapjack and Monitoring

  • Check – unit test – good bad ugly
  • Monitoring system – monitors for failing checks
  • 3 questions for monitoring systems – next check? , was check okay?, who do we notify? . Fetch , test , notify
  • fetch – lookup
  • test – execute , verify
  • notify – decide , callout
  • traditionally done in single process
  • but it’s an embarrassingly parallel problem
  • parts can be split. fetch+test fetch+notify – pass id/command between
  • precompile checks – so fetch is less expensive
  • transport between processes is the scheduler
  • no data collection when testing (graph seperately)
  • scheduler – workqueue – filled by populator, assigns stuff to notifier and workers
  • Lots of workers can be created (to do test)
  • flapjack – in ruby , talks to nagios plugin format
  • beanstalk – ansyncrnise workqueue service – ubuntu/debian packages
  • beanstalk – producer  puts jobs on beanstalk , consumer takes jobs off
  • uses named tubes (queues) , multiple tubes per instance
  • flapjack-worker – started up by flapjack-worker-manager starts multiple copies on machine. various control commands
  • worker is simple so linear scaling, spread across multiple machines required
  • flapjck-notifier – has manager to start it.
  • notifier has recipients.conf file with list of people to notify
  • notifier.conf – config for various notifiers (MAIL, SMS)
  • APIs – notifiers, filters, systems
  • notifier API – who , when and how sort of stuff.
  • “how many here use puppet – about a dozen – How many use Chef? – none “thanks a shame” “no it’s not”
  • persistence API – store stuff , mysql, couchdb whatever, standard way to store data.
  • filter API – parent checks hierarchy (so don’t check ports if host down)
  • flapjack-admin – pending – nodes , check templates , checks (check template + node ) , batches (group of checks)
  • 3 types of checks
  • Gaugaes – stuff within range – collectd ( point flapjack at collected output )
  • Behavoural tests – cucumber-nagios
  • Trending – reconoiter – growing area
  • collectd – gets stats from anything – nagios bridge – collectd-nagios queries collectd data
  • collectd client – gathers data from node and sends to collectd server
  • collectd forwarding server – agregates, filters and forwards
  • falapjack – crrently gems, soon to be real packages
  • http://flapjack-project.com

Bob Edward – Yubikey authentication in a mid-sized organisation

  • Reusable passwords are dead , hard to remeber, something you know which can be shared and discovered, captured, guessed
  • Alternative – One time Passwords – doesn’t matter if captured.
  • examples – RSA keys, SMS based systems, Yubikey, 2 factor authentication
  • Created by Yubico in sweden, open-source
  • Looks like a USB keyboard to a computer, generates a 44 character OTP each time button is pressed. No batteries, 2st 23 characters fixed for each key
  • $12 each in volumn – $40 as one-off
  • Based on secret AES 128-bit key
  • Yubicoships yubikeys with pre-generated IDs and AES keys. Offer publicauthentication, they know secret 128-bit key, need to trust them
  • secret-id+sess+timestamp+session+rand+CRC  string created by key , then encrypted and public ID prepended.
  • Server decrypts , checks checksums and looks to make sure secret-id matches and session and timestamps are incrimented from previous values.
  • Unless you trust and always want to use Yubicom’s servers you should reprogram you keys with your own keys and IDs. Can’t then be used against Yubicom’s server.
  • weaknesses – requires computer with usb port that accepts usb keyboard – some bugs with 1st generation keys – unused generated keys remian live until the next valid key is used
  • You can run your own server fairly easily – ykaserver – various interfaces, postgress database for storage – can also call out to PAM for two-factor authentication
  • softykey – software Yubikey – can use to generate 1-time pad for stuff without usb keyboard interfaces
  • Tested with ssh, VPNs , web logins – mostly use PAM or LDAP method
  • See Linux Journal and yubico.com

vimperator – automatic launch prog for netbooks

Jan Schmidt – Towards GStreamer 1.0

  • History of dev, faster bits during hackfests, when switched to git etc
  • Overview of last year, switched to git, slowdown when people busyswitched to binary registry
  • Support for various DVD playback  functions, special subtitles etc.
  • I’m not really in this area so I was just listening to get an idea where things are going. A bit too much detail for me at times.

Adam Jackson – The rebirth of Xinerama

  • Once again this was a bit over my head. It does look like the X guys spend a lot of time fighting assumptions built into the protocol and code 10 years ago however.

Stewart Smith et al – Building a Database kernel with Lego Like parts (Drizzle)

  • What would you change about Mysql – Modular architecture
  • Some crazy legacysuff in the Mysql code – good oppertunity to clean
  • move alot of code out of core, especially option parts – understandable and to reduce load – don’t load if you don’t need
  • more code coverage with tests
  • plugin interfaces – protocols, replication , logging, etc
  • modular replication system
  • general refactoring of storage engines
  • “If part of API sucks then fix API rather than work around it”
  • New this week – rot13() powerful encryption
  • Authentication plugins – auth_pam , auth_http
  • Various Logging plugins – logging_query , logging_syslog
  • Drizzle Community – All contributors equally – All project information public – No contributor license agreeements – Release early and often (~2 weeks ) – 100+ contributors , 500+ on mailing list
  • Milestone releases
  • When production release? – waiting to solidfy compatability – Sounds like a few months. – Reliable but still in flux
  • Pacakages to be pushed out to dists once things stable

Afterwards I had some dinner and went to the Professional Deligates networking session.

Share

LCA2010 – Day 3

Wednesday is the first day of Linux.conf.au proper. I thought that today I’d just keep my notes in a blog post to prevent doubling up.

The keynote was Benjamin Mako Hill talked about various things the most interesting bit was “antifeatures”. Things like DRM, crippling of products etc. The one of these I most hate right now is they way that cheap netbooks have fairly low specs (small resolutions, low RAM, slow CPUs ) partially because they have to keep the spec below a certain value in order to qualify for the really cheap Windows license.

The dreamwidth talk was quiet interesting (although the speakers pre-rehearsed banter between the speakers didn’t really work). Lots of practical examples , war stories and good sound advice.

Selena Deckelmann talked about choosing which open source database your should choose. The quick answer is “what problem are your trying to solve?”. She did a survey of the 50-odd databases out there and got 25 replies. Also did her own research and comparisons. Classified DBs into several categories (which I won’t list) such as

  • General Model – Key-Value, OLTP.
  • Distribution model (replication, partitioning, sharing).
  • Memory vs disk (eg keegin g everything in memory only like memcached).
  • HA options, Node failover.
  • Code dev model – Core +modules , Monolithic , Infrastructure
  • Community dev model – Dictator, Feature driven, Small group, A mix

Results at http://ossdbsurvey.org

  • Databases implement each others protocols
  • Need verification that protocols correctly implimented
  • Need tools/test to check things like replication working
  • More connections between projects/people (eg java seperate)

Ted Ts’o – Production-Ready filesystems

  • Hard to make robust. Many different workloads, lots of state, very parallel
  • Hard to balance getting it out with getting it stable enough to be fairly safe to use
  • 75-100 persons-years for filesystem to be production ready.
  • eg zfs around a dozen people , start 2001, announced 2005, shipped 2006, people confident with it around 2008-2009
  • Ext4 renamed from ext4dev at end 2008
  • Ext4 Shipping is some community distributions, soon in some enterprise distributions, widespread adoption 12+ months later
  • Lots of bugfixes still in ext4, most not real-world and picked up by auto-tools or careful checks in weird conditions.
  • Ted: “my other prefered term for Dbench is ‘random number generator’ “
  • Paths like online resize, online defrag that are not regularly tested by users or testers so source of many bugs.
  • Many bugs were in the recently subsystems and features
  • Making General purpose file system takes longer and a lot more effort than you might expect. Labour of love, hard to justify from business perspective.
  • Solid state drives with “flash translation layer” in place are fairly much the same as spinning disks. Extra optimizations for disks don’t help but they don’t hurt

Matthew Garrett on the Linux community

  • Started by listing things he’s not talked about
  • The Linux community is “Like the Koreas”
  • To be a member of the Linux community “you just have to care, just have to turn”
  • As community we are very hostile, it’s seen okay to flame and it is being rewarded still
  • Should we stop just cause it’s a nice thing to do or because it’ll stop scaring people off?
  • Ubuntu code of conduct has mean’t that users are consider part of the community more than in other distributions
  • Code of Conduct must be enforced or it’s useless
  • “We value code above all else… not a good thing” . We need people to feel that by using software they are part of something
  • Communty entirely based on technical excellence or encompasing everybody who users, cares, contributes to projects
  • Idea for positive examples Wiki with pointers to COPs and best practice examples
  • Not gained behavior standards normally associated with grown communities

Sage Weil – ceph distributed file system

  • How different
  • scaleable to 1000s , grow from a few
  • reliable, HA, replicated data, fast recovery
  • snapshots, quota-like accounting
  • Motivation – avoid bottlenecks and symetrical shared disks
  • avoid manual workload partition, p3p-like protocols, intell storage agents
  • POSIX file system , scaleable metadata server
  • metadata (MDS) servers/clusters and object store boxes seperate
  • CRUSH hash function used to distrubtute objects across devices, works as devices are added. Spread them out explicitly across infrastructure if required
  • fast (no lookups), relieable, stable
  • celp object storage daemon on each node
  • talks to peers on other node: rep data, detect failures, migrate data
  • hashing fuction means nodes don’t have to negotiate with each other, CRUSH says where data is going.
  • monitor storage nodes, moves data around, make sure it’s in the right places, uptodate. fixes if required.
  • raw storage API if you don’t need full filesystem fun (dirs etc)
  • proxy that emulates s3 REST interface
  • metadata cluster , uses object store for all long term storage, needs memory and fast network for performance.
  • metadata streamed to journal. large journal (100s MB) flushed now and then
  • snapshotting on per-directory basisi via simple mkdir
  • snapshot leverages btrfs copy-on-write storage layer
  • file systems client near-posix
  • kernel client, FUSE, Hadoop clients
  • stable but not production ready
  • client should be in mainline kernel soon
  • aim to work in multiple datacentre, across unrelieble links
  • http://ceph.newdream.net/

Paul Fenwick – Worlds Worst Inventions

Not really a technical talk. More a few stories about funny inventions. Quiet amusing but I’m not sure it fits in with the rest of the conference.

Share