Category: Tech

Technical

Linux.conf.au – Day 2 – Session 1 – Sysadmin Miniconf

Configuration Management – A love Story – Javier Turegano

  • June 2008 – Devs want to deploy fast
  • June 2009 – git -> jenkins -> Puppet master
  • But things got pretty complicated and hard to maintain
  • Remove puppet master, puppet noop, but only happens now and then lots of changes but a couple of errors
  • Now doing manual changes
  • June 2010 – Thngs turned into a mess.
  • June 2011 – Devs want prod-like development
  • Cloud! Tooling! Chef! – each dev have their own environment
  • June 2012 – dev environments for all working in ec2
  • dev no longer prod-like. cloud vs datacentre, puppet vs chef , debian vs centos, etc
  • June 2013 – More into cloud, teams re-arranged
  • Build EC2 images and deploy out of jenkins. Eaither as AMI or as rpm
  • Each team fairly separate, doing thing different ways. Had guilds to share skills and procedures and experience
  • June 2014 – Cloudformation, Ansible used by some groups, random

Healthy Operations – Phil Ingram

  • Acquia – Enterprise Drupal as a service. GovCMS Australian Federal Government. 1/4 are remote
  • Went from working in office to working from home
  • Every week had phone call with boss
  • Talk about thing other than with work, ask home people are going, talk to people.
  • Not sleep, waking up at night, not exercising, quick to anger and negative thinking, inability to concentrate
  • Hadn’t taken more than 1 week off work, let exercise work, hobbies was computer stuff
  • In general being in Ops not as much of an option to take time off. Things stay broke until fix
  • Unable to learn via Osmosis, Timing of handing over between shifts
  • People do not understand that computers are run by people not robots
  • Methods: Turn work off at the end of the day, Rubber Ducking, exercise

Developments in PCP (Performance Co-Pilot) : Nathan Scott

  • See my slides from yesterday for intro to PCP
  • Stuff in last 12 months
    • Included in supported in RHEL 6.6 and RHEL 7
    • Regular stable releases
    • Better out of the box experience
    • Tackling some long-standing problems
  • JSON access – pmwebd , interactive web charts ( Graphite, grafana )
  • zero-install look-inside containers
  • Docker support but written to allow use by others
  • Collectors
    • Lots of new kernel metrics additions
    • New applications from web devs (memcached, DNS, web )
    • DB server additions
    • Python PMDA interfaces
  • Monitor work
    • Reporting tools
    • Web tools, GUIs
  • Also improving ease of setup
  • Getting historical data from sar, iostat
  • www.pcp.io

Security options for container implementations – Jay Coles

  • What doesn’t work: rlimits, quotas, blacklisting via ACLs
  • Capabilities: Big list that containers probably shouldn’t have
  • Cgroups – Accounting, Limiting resource usage, tracking of processes, preventing/allowing device access
  • App Armor vs selinux – Use at least one, selinux a little more featured
Share

Linux.conf.au – Day 2 – Keynote by Eben Moglen

Last spoke 10 years ago in Canberra Linux.conf.au

Things have improved in the last ten years

  • $10s of billions of value have been lost in software patent war
  • But things have been so bad that some help was acquired, so worst laws have been pushed back  a little
  • “Fear of God” in industry was enough to push open Patent pools
  • Judges determined that Patent law was getting pathological, 3 wins in Supreme court
  • Likelihood worst patent laws will be applied against free software devs has decreased
  • “The Nature of the problem has altered because the world has altered”

The Next 10 years

  • Most important Patent system will be China’s
  • Lack of rule of law in China will cause problems in environment of patents
  • Too risky for somebody too try and stop a free software project. We have “our own baseball bat” to spring back at them

The last 10 years

  • Changes in Society more important changes in software
  • 21st century vs 20th century social organisations
    • Less need for hierarchy and secrecy
    • Transparency, Participation, non-hierarchical interaction
  • OS invented that organisation structure
  • Technology we made has taken over the creation of software
  • “Where is BitKeeper now?” – Eben Moglen
  • Even Microsoft reorganises that our way of software making won
  • Long term the organisation structure change everywhere will be more important than just it’s application in Software
  • If there has been good news about politics = “we did it”, bad news = “we tried”

Our common Values

  • “Bridge entire environment between vi and emacs”

Snowden

  • Without PGP and free software then things could have been worse
  • The world would be a far more despotic place if PGP was driven underground back in 1993. Imagine today’s Net without HTTPS or SSH!
  • “We now live in the world we are afraid of”
  • “What stands between them and us is our inventions”
  • “Freedom itself depends on how we make use of the technologies we are creating.” – Eben Moglen
  • “You can’t trust what you can’t read”
  • Big power in the wrong is committed against the first law of robotics, they what technology to work for it.
  • From guy in twitter – “You can’t trust what you can’t read.” True, but if OpenSSL teaches us anything you can’t necessarily trust what you can
  • Attitudes in under-18s are a lot more positive towards him than those who are older (not just cause he looks like Harry Potter)
  • GNU Project is 30 years old, almost same age is Snowden

Oppertunity

  • We can’t control the net but opportunity to prevent others from controlling it
  • Opportunity to prevent failure of freedom
  • Society is changing, demographics under control
  • But 1.6 billion people live in China, America is committed to spying, consumer companies are committed to collecting consumer information
  • Collecting everything is not the way we want the net to work
  • We are playing for keeps now.

 

 

Share

Linux.conf.au – Day 1 – Session 3 – Containers

Building a PaaS with Docker, Kubernetes, and Hard Work – Steven Pousty

  • Slides – bit.ly/1AFGACa
  • All about Openshift
  • So why a new Paas?
  • Project Atomic – stripped down RHEL install, everything else as a container. ostree file system, same kernel as RHEL
  • Kubernetes intro
    • Kubernetes Daemon – Routing for services
    • Sceduler etc
  • Openshift
    • Built-in software defined networking – OpenVSwith , HAPRoxy load balancing etc
  • Takeaway
    • PAAS seems to be cool again

 

Galera with Docker: How Synchronous Replication and Linux Containers Mesh Together – Raghavendra Prabhu

  • I got lost in the talk

 

Cloud, Containers, and Orchestration Panel –  Katie Miller

  • Steven Pousty , Bran Philips ,
    Tycho Andersen
    Tycho Andersen
    Tycho Andersen
    Tycho Andersen

    Tycho Andersen

  • Standard is Dockers to lose and they might manage it
  • 3-4 years before we should standardise them. Need to experiment first.
  • The kernel API imposes some limits on diversity
  • Lots of other stuff

 

Share

Linux.conf.au 2015 – Day 1 – Session 2 – Containers

AWS OpsWorks Orchestration War Stories – Andrew Boag

  • Autoscaling too slow since running build-from-scratch every time
  • Communications dependencies
  • Full stack rebuild in 20-40 minutes to use data currently in production
  • A bit longer in a different region
  • Great for load testing
  • If we ere doing again
    • AMI-based better
    • OPSWorks not suitable for all AWS stacks
    • Golden master for flexable
  • Auto-Scaling
    • Not every AMI instance is Good to Go upon provisioning
    • Not a magic bullet, you can’t broadly under-provision
    • needs to be throughly load-tested
  • Tips
    • Dual factor authentication
    • No single person / credentials should be able to delete all cloud-hosted copies of your data
  • Looked at Cloudformation at start, seemed to be more work
  • Fallen out of love with OpsWorks
  • Nice distinction by Andrew Boag: he doesn’t talk about “lock-in” to cloud providers, but about “cost to exit”.   – Quote from Paul

 

Slim Application Containers from Source – Sven Dowideit

  • Choose a base image and make a local version (so all your stuff uses the same one)
  • I’d pick debian (a little smaller) unless you can make do with busybox or scratch
  • Do I need these files? (check though the Dockerfile) eg remove docs files, manpages, timezones
  • Then build, export, import and it comes all clean with just one layer.
  • If all your images use same base, only on the disk once
  • Use related images with all your tools, related to deployment image but with the extra dev, debug, network tools
  • Version the dev images
  • Minimise to 2 layers
    • look at docker-squash
    • Get rid of all the sourc code from your image, just end up with whats need, not junk hidden in layers
  • Static micro-container nginx
    • Build as container
    • export as tar , reimport
    • It crashes 🙁
    • Use inotifywait to find what extra files (like shared libraries) it needs
    • Create new tarball with those extra files and “docker import” again
    • Just 21MB instead of 1.4GB with all the build fragments and random system stuff
    • Use docker build as last stage rather than docker import and you can run nginx from docker command line
    • Make 2 tar files, one for each image, one in libs/etc, second is nginx

 

Containers and PCP (Performance Co-Pilot) –  Nathan Scott

  • Been around for 20+ years, 11 years open source, Not a big mindshare
  • What is PCP?
    • Toolkit, System level analysis, live and historical, Extensible, distributed
    • pmcd daemon on each server, plus for various functions (bit of like collectd model)
    • pmlogger, pmchart, pmie, etc talk (pull or poll) to pmcd to get data
  • With Containers
    • Use –container=  to grab info inside a container/namespace
    • Lots of work still needed. Metrics inside containers limited compared to native OS

 

The Challenges of Containerizing your Datacenter – Daniel Hall

  • Goals at LIFX
    • Apps all stateless, easy to dockerize
    • Using mesos, zookeeper, marathon, chronos
    • Databases and other stuff outside that cloud
  • Mesos slave launches docker containers
  • Docker Security
    • chroot < Docker < KVM
    • Running untrusted Docket containers are a BAD IDEA
    • Don’t run apps as root inside container
    • Use a recent kernel
    • Run as little as possible in container
    • Single static app if possible
    • Run SELinux on the host
  • Finding things
    • Lots of micoroservices, marathon/mesos moves things all over the place
    • Whole machines going up and down
    • Marathon comes with a tool that pushes it’s state into HAProxy, works fairly well, apps talk to localhost on each machines and haproxy forwards
    • Use custom script for this
  • Collecting Logs
    • Not a good solution
    • can mount /dev/log but don’t restart syslog
    • Mesos collects stdout/stderror , hard to work with and no timestamps
    • Centralized logs
    • rsyslog log to 127.0.0.1 -> haproxy -> contral machine
    • Sometimes needs to queue/drop if things take a little while to start
    • rsyslog -> logstash
    • elasticsearch on mesos
    • nginx tasks running kibana
  • Troubleshooting
    • Similar to service discover problem
    • Easier to get into a container than getting out
    • Find a container in marathon
    • Use docker exec to run a shell, doesn’t work so well on really thin containers
    • So debugging tolls can work from outside, pprof or jsonsole can connect to exposed port/pid of container
Share

Linux.conf.au 2015 – Day 1 – Session 1 – Containers

Clouds, Containers, and Orchestration Miniconf

 

Cloud Management and ManageIQ – John Mark Walker

  • Who needs management – Needs something to tie it all together
  • New Technology -> Adoption -> Proliferation -> chaos -> Control -> New Technology
  • Many technologies follow this, flies under the radar, becomes a problem to control, management tools created, management tools follow the same pattern
  • Large number of customers using hybrid cloud environment ( 70% )
  • Huge potential complexity, lots of requirements, multiple vendors/systems to interact with
  • ManageIQ
    • Many vendor managed open source products fail – open core, runt products
    • Better way – give more leeway to upstream developers
    • Article about taking it opensource on opensource.com. Took around a year from when decision was made
    • Lots of work to create a good open source project that will grow
    • Release named after Chess Grandmasters
    • Rails App

 

LXD: The Container-Based Hypervisor That Isn’t –  Tycho Andersen

  • Part of Openstack
  • Based on LXC , container based hypervisor
  • Secure by default: user namespaces, cgroups, Apparmor, etc
  • A EST API
  • A daemon that doesn’t hypervisory things
  • A framework for maintaining container based applications
  • It Isn’t
    • No network configuration
    • No storage management – But storage aware
    • Not an application container tool
    • handwavy difference between it and docker, I’m sure it makes sense to some people. Something about running an init/systemd rather than the app directly.
  • Features
    • Snapshoting – eg something that is slow to start, snapshot after just starts and deploy it in that state
    • Injection – add files into the container for app to work on.
    • Migration – designed to go fairly fast with low downtime
  • Image
    • Public and private images
    • can be published
  • Roadmap
    • MVP 0.1 released late January 2015
    • container management only

 

Rocket and the App Container Spec – Brandon Philips

  • Single binary – rkt – runs everywhere, systemd not required
  • rkt fetch – downloads and discovers images ( can run as non-root user )
  • bash -> rkt -> application
  • upstart -> rkt -> application
  • rkt run coreos.com/etcd-v2.3.1
  • multiple processes in container common. Multiple can be run from command line or specified in json file of spec.
  • Steps in launch
    • stage 0 – downloads images, checks it
    • Stage 1 – Exec as root, setup namespaces and cgroups, run systemd container
    • Stage 2 – runs actual app in container. Things like policy to restart the app
    • rocket-gc garbage collects stuff , runs periodicly. no managmanent daemon
  • App Container spec is work in progress
    • images, files, compressed, meta-data, dependencies on other images
    • runtime , restarts processes, run multiple processes, run extra procs under specified conditions
    • metadata server
    • Intended to be built with test suite to verify
Share

Updating my personal email setup

I’m in the process of moving my personal hosting from one VPS to another ( I host with Linode and am buying a new virtual machine with a similar spec to my current one for half the monthly price ) and I decided to rearrange my home email. My old setup was:

Internet –> Exim on VPS -> Download via fetchmail to home -> Send to spamassassin and dspam at home -> filtering into mboxes on home workstation  -> read via alpine

The main disadvantages of this were:

  • Had to ssh into home to read email (couldn’t read on my phone)
  • Hard to view images in email or HTML emails
  • Sending via my ISP was unreliable and they are  implementing filters
  • No notification of new email

So I decided to make some changes.

Internet -> Postfix on VPS -> procmail to spamassassin on VPS -> procmail to maildirs -> read via imap

This setup is a lot simpler than the previous one and a bit more mainstream.

  • Since the email is online via imap I can read it directly from alpine or my phone (or another client)
  • Online running one anti-spam program (spamasaasin) instead of two (dspam and spamassassin)
  • Email operations on one server (the VPS) insead of 3 (VPS, workstation, home virtual machine)
  • Sending email straight via VPS instead of home VM and my ISP’s mail server

Details of my setup

There are a lot of HOWTOs on getting email to work via postfix and dovecot. I decided that the main feature I needed were virtual aliases for my domains. I also decided that since I only had a few mailboxes (mine own and two others) I could just create accounts on the server rather than maintain virtual users in postfix/dovecot. The server is running Ubuntu 14.04

Roughly speaking I followed the advice on these two pages by Rimuhosting:

I added these lines to my Postfix’s main.cf

virtual_alias_maps = hash:/etc/postfix/virtual
home_mailbox = Maildir/
mailbox_command = /usr/bin/procmail -a "$EXTENSION" DEFAULT=$HOME/Maildir/ MAILDIR=$HOME/Maildir

Theses lines tell postfix to (1) use the virtual file (see below) (2) deliver to Maildirs (3) use procmail for delivery.

and create a /etc/postfix/virtual file like:

darkmere.gen.nz              20140720
simon@darkmere.gen.nz        simon-mail@cyan.usenet.net.nz
root@darkmere.gen.nz         simon@darkmere.gen.nz

The first line indicates the domain should be used (this option is a little hidden in the virtual manpage) and then their are various addresses. simon-mail is a noshell account and cyan is the name of the server to so the email is delivered locally to it.

The simon-mail account just has a simple .procmailrc file with my various filters and a Maildir to store the email. Spam processing is called by procmail via:

:0 fW
* < 280000
| spamc -u simon-mail -d localhost

:0:
* ^X-Spam-Status: Yes, score=([5-9]|1[0-9]|[2-9][0-9])
.junk/

which just puts all email that looks like spam into a junk folder (which I can check now and then until I’m happy with the filters).

Dovecot for imap worked out of the box except I had to tell it the location of my email. I just edited the file /etc/dovecot/conf.d/10-mail.conf and changed the mail_location setting to:

mail_location = maildir:~/Maildir

For sending email I pretty much followed this guide directly.

Overall it wasn’t too hard. The main problem was the fact that there were so many guides (I read over a dozen) each of which differed slightly and which were in many cases designed more much larger sites. I’ve currently got the setup in final testing (it is getting a copy of all my incoming email) and intended to switch over soon. In the short term I’m keeping my old mail folders (all 752 of them adding up to 1.8GB) locally at home but may move them at a later date.

Share

Linux.conf.au 2015 – Getting started

Disclaimer: The below is my personal opinion and does not represent the views of the 2015 LCA organising committee. Some details have been left out, stuff may change, names may be wrong, may contain nuts, etc.

In January 2015 the Linux.conf.au conference will be held in Auckland, New Zealand. Each year the conference brings together 600 ( +-100 ) Linux developers and users for 5 days for talks, chat and social events. LCA 2015 will be the 12th Linux.conf.au I’ve attended (every year since 2004) and the first I’ve helped organise. It will be the 3rd time the conference has been held in New Zealand.

Each year’s LCA is held in a different city by a group who bid for and run it. The Auckland team consists of a “core team” of about 10 under the overall lead of Cherie Ellis, another dozen “supporters” (including me). Others volunteers  will be recruited closer to the time and there are also external groups like the papers committee and people from Auckland University doing various jobs.

The majority of the conference will be held in the Owen G Glenn Building at Auckland University. The is single big building with several large lecture theatres along with big central areas and smaller rooms. The currently plan is for just about the whole conference proper to happen there.

Over half the attendees with probably stay at nearby student accommodation, this is cheap, nearby and lets people mingle with other attendees after-hours. There will also be some planned social events (like the conference dinner) elsewhere in Auckland.

Since January 2014 when Auckland was announced as the winning bid for 2015 the pace has gradually been picking up. Over 30 main positions have been filled (most with both a main and backup person) and the core team is meeting (usually online) weekly and the second supporters meeting is coming up.

The amount of stuff to organise is pretty big. As well as the venues, there is food, travel, accommodation, swag, the programme, the websites, network, dinners, registration, etc etc. A huge amount of stuff which will take up many hours per week for the rest of 2015.

At the end of March there was a “Ghosts visit”, this is where half a dozen previous conference organisers ( “Ghosts of conferences past” ) come over for a weekend to look over the setup and talk to the group. The purpose is twofold, the Ghosts check that everything is on track and look for problems, while the 2015 organisers get to pick the Ghost’s brains

Large Brain possibly belonging to Ghost

Even the Ghosts’ event itself is a small test of the organizers’ ability. They have  to fly, meeting, accommodate, hosts, feed and otherwise look after half a dozen people, a mini rehearsal  for the full conference.

Share

Links: Containers, Performance, Backpack Nukes, New Countries

Share

Linux.conf.au 2014 – Day 5 – Finish

Winner Rusty Wrench Award: Andrew Tridgell

 

Host of LCA2015: Auckland!!

Website:

 

Lightning Talks part 2

  • My toothbrush has a serial number
    • after sales support
    • can they find it for me?
    • In the post-Snowden world this should be investigated
  • DIY Book Scanning for Fun
    • Scanned book useful for good reasons
    • diybookscanner.org
  • Freedom Box project update
    • Almost ready for 0.2 release which will be pretty good
  • OneRNG
    • Open Hardware, Random number generator
    • Trustable, see raw or AES whitened
    • trying various options
    • onerng.info
  • Central Coast LUG
    • cclugtmp@gmail.com
  • Bitcoin Myths
    • Anonymous – Nope, all transactions records
    • Bubble – nope, infrastaructer
    • Giant Ponzi scheme – Not sold as investment, no claims
  • dlect – Lecture recording downloader
    • uqlectures.sf.net
    • Looking for help and the extend to other Universities
  • Debian in Australia
    • Trying to get Debian Australia mailing list started
  • Bitcoin architecture applied to capital markets
  • Learning Opportunities in Rocketry Software
    • Maths makes by head hurt
  • Electronic Frontiers Australia
    • Would like to invite you to volunteer and drink beer
  • LA does other things
    • pycon AU in August
    • Drupal camps
    • Barcamp
    • Join a user group
    • hacker space
    • add your blog to our planet

 

Share

Linux.conf.au 2014 – Day 5 – Session 2

Provisioning Bare Metal with OpenStack by Devananda van der Veen

  • Tried to use the existing NOVA tool (which was used for VM provisioning) but all sorts of limitations
    • Hacked it a lot and then gave up
  • Created a new project “Ironic” for bare-metal provisioning in May 2013
  • Status
    • Being working on but lots of devs
    • Not in main release yet
    • Some push to simplify openstack installation – Triple-O (Openstack on Openstack)
    • “Openstack is not a virtualisation layer”
  • Security not there yet
  • Driver Interface
    • 3 classes of interfaces: core, common, vendor
    • core: power management, deploy,
    • common: console, rescue
    • vendor: fireware? boot-from-volume? something-else?
  • Architecture
    • REST API
    • DB
    • conductor services
    • RPC
    • ( the slide makes more sense )
  • Many conductors, Many drivers
  • If the cluster changes
    • take-over hooks
    • consistent has updates
    • node(s) re-mapped to conductors
  • Feature equiv to bare-metal in a few months
  • Usable by that point, eventually to replace bare-metal, may require re-deployment

Talk only took 20 minutes. That was quick

Share