Author: simon

Linux.conf.au – Day 2 – Session 1 – Sysadmin Miniconf

Configuration Management – A love Story – Javier Turegano

  • June 2008 – Devs want to deploy fast
  • June 2009 – git -> jenkins -> Puppet master
  • But things got pretty complicated and hard to maintain
  • Remove puppet master, puppet noop, but only happens now and then lots of changes but a couple of errors
  • Now doing manual changes
  • June 2010 – Thngs turned into a mess.
  • June 2011 – Devs want prod-like development
  • Cloud! Tooling! Chef! – each dev have their own environment
  • June 2012 – dev environments for all working in ec2
  • dev no longer prod-like. cloud vs datacentre, puppet vs chef , debian vs centos, etc
  • June 2013 – More into cloud, teams re-arranged
  • Build EC2 images and deploy out of jenkins. Eaither as AMI or as rpm
  • Each team fairly separate, doing thing different ways. Had guilds to share skills and procedures and experience
  • June 2014 – Cloudformation, Ansible used by some groups, random

Healthy Operations – Phil Ingram

  • Acquia – Enterprise Drupal as a service. GovCMS Australian Federal Government. 1/4 are remote
  • Went from working in office to working from home
  • Every week had phone call with boss
  • Talk about thing other than with work, ask home people are going, talk to people.
  • Not sleep, waking up at night, not exercising, quick to anger and negative thinking, inability to concentrate
  • Hadn’t taken more than 1 week off work, let exercise work, hobbies was computer stuff
  • In general being in Ops not as much of an option to take time off. Things stay broke until fix
  • Unable to learn via Osmosis, Timing of handing over between shifts
  • People do not understand that computers are run by people not robots
  • Methods: Turn work off at the end of the day, Rubber Ducking, exercise

Developments in PCP (Performance Co-Pilot) : Nathan Scott

  • See my slides from yesterday for intro to PCP
  • Stuff in last 12 months
    • Included in supported in RHEL 6.6 and RHEL 7
    • Regular stable releases
    • Better out of the box experience
    • Tackling some long-standing problems
  • JSON access – pmwebd , interactive web charts ( Graphite, grafana )
  • zero-install look-inside containers
  • Docker support but written to allow use by others
  • Collectors
    • Lots of new kernel metrics additions
    • New applications from web devs (memcached, DNS, web )
    • DB server additions
    • Python PMDA interfaces
  • Monitor work
    • Reporting tools
    • Web tools, GUIs
  • Also improving ease of setup
  • Getting historical data from sar, iostat
  • www.pcp.io

Security options for container implementations – Jay Coles

  • What doesn’t work: rlimits, quotas, blacklisting via ACLs
  • Capabilities: Big list that containers probably shouldn’t have
  • Cgroups – Accounting, Limiting resource usage, tracking of processes, preventing/allowing device access
  • App Armor vs selinux – Use at least one, selinux a little more featured
Share

Linux.conf.au – Day 2 – Keynote by Eben Moglen

Last spoke 10 years ago in Canberra Linux.conf.au

Things have improved in the last ten years

  • $10s of billions of value have been lost in software patent war
  • But things have been so bad that some help was acquired, so worst laws have been pushed back  a little
  • “Fear of God” in industry was enough to push open Patent pools
  • Judges determined that Patent law was getting pathological, 3 wins in Supreme court
  • Likelihood worst patent laws will be applied against free software devs has decreased
  • “The Nature of the problem has altered because the world has altered”

The Next 10 years

  • Most important Patent system will be China’s
  • Lack of rule of law in China will cause problems in environment of patents
  • Too risky for somebody too try and stop a free software project. We have “our own baseball bat” to spring back at them

The last 10 years

  • Changes in Society more important changes in software
  • 21st century vs 20th century social organisations
    • Less need for hierarchy and secrecy
    • Transparency, Participation, non-hierarchical interaction
  • OS invented that organisation structure
  • Technology we made has taken over the creation of software
  • “Where is BitKeeper now?” – Eben Moglen
  • Even Microsoft reorganises that our way of software making won
  • Long term the organisation structure change everywhere will be more important than just it’s application in Software
  • If there has been good news about politics = “we did it”, bad news = “we tried”

Our common Values

  • “Bridge entire environment between vi and emacs”

Snowden

  • Without PGP and free software then things could have been worse
  • The world would be a far more despotic place if PGP was driven underground back in 1993. Imagine today’s Net without HTTPS or SSH!
  • “We now live in the world we are afraid of”
  • “What stands between them and us is our inventions”
  • “Freedom itself depends on how we make use of the technologies we are creating.” – Eben Moglen
  • “You can’t trust what you can’t read”
  • Big power in the wrong is committed against the first law of robotics, they what technology to work for it.
  • From guy in twitter – “You can’t trust what you can’t read.” True, but if OpenSSL teaches us anything you can’t necessarily trust what you can
  • Attitudes in under-18s are a lot more positive towards him than those who are older (not just cause he looks like Harry Potter)
  • GNU Project is 30 years old, almost same age is Snowden

Oppertunity

  • We can’t control the net but opportunity to prevent others from controlling it
  • Opportunity to prevent failure of freedom
  • Society is changing, demographics under control
  • But 1.6 billion people live in China, America is committed to spying, consumer companies are committed to collecting consumer information
  • Collecting everything is not the way we want the net to work
  • We are playing for keeps now.

 

 

Share

Linux.conf.au – Day 1 – Session 3 – Containers

Building a PaaS with Docker, Kubernetes, and Hard Work – Steven Pousty

  • Slides – bit.ly/1AFGACa
  • All about Openshift
  • So why a new Paas?
  • Project Atomic – stripped down RHEL install, everything else as a container. ostree file system, same kernel as RHEL
  • Kubernetes intro
    • Kubernetes Daemon – Routing for services
    • Sceduler etc
  • Openshift
    • Built-in software defined networking – OpenVSwith , HAPRoxy load balancing etc
  • Takeaway
    • PAAS seems to be cool again

 

Galera with Docker: How Synchronous Replication and Linux Containers Mesh Together – Raghavendra Prabhu

  • I got lost in the talk

 

Cloud, Containers, and Orchestration Panel –  Katie Miller

  • Steven Pousty , Bran Philips ,
    Tycho Andersen
    Tycho Andersen
    Tycho Andersen
    Tycho Andersen

    Tycho Andersen

  • Standard is Dockers to lose and they might manage it
  • 3-4 years before we should standardise them. Need to experiment first.
  • The kernel API imposes some limits on diversity
  • Lots of other stuff

 

Share

Linux.conf.au 2015 – Day 1 – Session 2 – Containers

AWS OpsWorks Orchestration War Stories – Andrew Boag

  • Autoscaling too slow since running build-from-scratch every time
  • Communications dependencies
  • Full stack rebuild in 20-40 minutes to use data currently in production
  • A bit longer in a different region
  • Great for load testing
  • If we ere doing again
    • AMI-based better
    • OPSWorks not suitable for all AWS stacks
    • Golden master for flexable
  • Auto-Scaling
    • Not every AMI instance is Good to Go upon provisioning
    • Not a magic bullet, you can’t broadly under-provision
    • needs to be throughly load-tested
  • Tips
    • Dual factor authentication
    • No single person / credentials should be able to delete all cloud-hosted copies of your data
  • Looked at Cloudformation at start, seemed to be more work
  • Fallen out of love with OpsWorks
  • Nice distinction by Andrew Boag: he doesn’t talk about “lock-in” to cloud providers, but about “cost to exit”.   – Quote from Paul

 

Slim Application Containers from Source – Sven Dowideit

  • Choose a base image and make a local version (so all your stuff uses the same one)
  • I’d pick debian (a little smaller) unless you can make do with busybox or scratch
  • Do I need these files? (check though the Dockerfile) eg remove docs files, manpages, timezones
  • Then build, export, import and it comes all clean with just one layer.
  • If all your images use same base, only on the disk once
  • Use related images with all your tools, related to deployment image but with the extra dev, debug, network tools
  • Version the dev images
  • Minimise to 2 layers
    • look at docker-squash
    • Get rid of all the sourc code from your image, just end up with whats need, not junk hidden in layers
  • Static micro-container nginx
    • Build as container
    • export as tar , reimport
    • It crashes 🙁
    • Use inotifywait to find what extra files (like shared libraries) it needs
    • Create new tarball with those extra files and “docker import” again
    • Just 21MB instead of 1.4GB with all the build fragments and random system stuff
    • Use docker build as last stage rather than docker import and you can run nginx from docker command line
    • Make 2 tar files, one for each image, one in libs/etc, second is nginx

 

Containers and PCP (Performance Co-Pilot) –  Nathan Scott

  • Been around for 20+ years, 11 years open source, Not a big mindshare
  • What is PCP?
    • Toolkit, System level analysis, live and historical, Extensible, distributed
    • pmcd daemon on each server, plus for various functions (bit of like collectd model)
    • pmlogger, pmchart, pmie, etc talk (pull or poll) to pmcd to get data
  • With Containers
    • Use –container=  to grab info inside a container/namespace
    • Lots of work still needed. Metrics inside containers limited compared to native OS

 

The Challenges of Containerizing your Datacenter – Daniel Hall

  • Goals at LIFX
    • Apps all stateless, easy to dockerize
    • Using mesos, zookeeper, marathon, chronos
    • Databases and other stuff outside that cloud
  • Mesos slave launches docker containers
  • Docker Security
    • chroot < Docker < KVM
    • Running untrusted Docket containers are a BAD IDEA
    • Don’t run apps as root inside container
    • Use a recent kernel
    • Run as little as possible in container
    • Single static app if possible
    • Run SELinux on the host
  • Finding things
    • Lots of micoroservices, marathon/mesos moves things all over the place
    • Whole machines going up and down
    • Marathon comes with a tool that pushes it’s state into HAProxy, works fairly well, apps talk to localhost on each machines and haproxy forwards
    • Use custom script for this
  • Collecting Logs
    • Not a good solution
    • can mount /dev/log but don’t restart syslog
    • Mesos collects stdout/stderror , hard to work with and no timestamps
    • Centralized logs
    • rsyslog log to 127.0.0.1 -> haproxy -> contral machine
    • Sometimes needs to queue/drop if things take a little while to start
    • rsyslog -> logstash
    • elasticsearch on mesos
    • nginx tasks running kibana
  • Troubleshooting
    • Similar to service discover problem
    • Easier to get into a container than getting out
    • Find a container in marathon
    • Use docker exec to run a shell, doesn’t work so well on really thin containers
    • So debugging tolls can work from outside, pprof or jsonsole can connect to exposed port/pid of container
Share

Linux.conf.au 2015 – Day 1 – Session 1 – Containers

Clouds, Containers, and Orchestration Miniconf

 

Cloud Management and ManageIQ – John Mark Walker

  • Who needs management – Needs something to tie it all together
  • New Technology -> Adoption -> Proliferation -> chaos -> Control -> New Technology
  • Many technologies follow this, flies under the radar, becomes a problem to control, management tools created, management tools follow the same pattern
  • Large number of customers using hybrid cloud environment ( 70% )
  • Huge potential complexity, lots of requirements, multiple vendors/systems to interact with
  • ManageIQ
    • Many vendor managed open source products fail – open core, runt products
    • Better way – give more leeway to upstream developers
    • Article about taking it opensource on opensource.com. Took around a year from when decision was made
    • Lots of work to create a good open source project that will grow
    • Release named after Chess Grandmasters
    • Rails App

 

LXD: The Container-Based Hypervisor That Isn’t –  Tycho Andersen

  • Part of Openstack
  • Based on LXC , container based hypervisor
  • Secure by default: user namespaces, cgroups, Apparmor, etc
  • A EST API
  • A daemon that doesn’t hypervisory things
  • A framework for maintaining container based applications
  • It Isn’t
    • No network configuration
    • No storage management – But storage aware
    • Not an application container tool
    • handwavy difference between it and docker, I’m sure it makes sense to some people. Something about running an init/systemd rather than the app directly.
  • Features
    • Snapshoting – eg something that is slow to start, snapshot after just starts and deploy it in that state
    • Injection – add files into the container for app to work on.
    • Migration – designed to go fairly fast with low downtime
  • Image
    • Public and private images
    • can be published
  • Roadmap
    • MVP 0.1 released late January 2015
    • container management only

 

Rocket and the App Container Spec – Brandon Philips

  • Single binary – rkt – runs everywhere, systemd not required
  • rkt fetch – downloads and discovers images ( can run as non-root user )
  • bash -> rkt -> application
  • upstart -> rkt -> application
  • rkt run coreos.com/etcd-v2.3.1
  • multiple processes in container common. Multiple can be run from command line or specified in json file of spec.
  • Steps in launch
    • stage 0 – downloads images, checks it
    • Stage 1 – Exec as root, setup namespaces and cgroups, run systemd container
    • Stage 2 – runs actual app in container. Things like policy to restart the app
    • rocket-gc garbage collects stuff , runs periodicly. no managmanent daemon
  • App Container spec is work in progress
    • images, files, compressed, meta-data, dependencies on other images
    • runtime , restarts processes, run multiple processes, run extra procs under specified conditions
    • metadata server
    • Intended to be built with test suite to verify
Share

Links: Efficient Software, West Wing history, $20k houses, Damn boomers

Share

Another run in with the Electoral Commission

After already having trouble Electoral Commission banning photography in polling places I now get a threatening email from them.

Yesterday I made this Tweet:

 

and today I get the following email

Subject: Electoral Commission complaint – London exit poll posted on Twitter account

Dear Simon,

The Electoral Commission has received a complaint with regard to an exit poll being taken and then published on the Twitter account of @slyall. We understand that this is your Twitter account.

Under section 197(1)(d) of the Electoral Act 1993, it is an offence to conduct a public opinion poll of persons who have voted (exit polls). Section 197(1)(d) states:

197 Interfering with or influencing voters
(1) Every person commits an offence and shall be liable on conviction to a fine not exceeding $20,000 who at an election—
(d) at any time before the close of the poll, conducts in relation to the election a public opinion poll of persons voting before polling day

In order to assist the Commission in considering this complaint, could you please provide the following information:

1.         Who conducted the exit poll and when was it conducted?
2.         How did you receive this information?
3.         Any other information you believe to be of relevance to the Commission’s consideration.
4.         How you might remedy this matter.

Can you please provide the above information by 5pm, Friday 19 September 2014. In the first instance, to avoid further complaints, you may wish to remove the Twitter post.

Please telephone me if you wish to discuss this further.

Update

I replied with:

I saw this:
http://www.reddit.com/r/newzealand/comments/2gidem/kiwi_did_exit_polling_out
side_london_embassy_note/

and copied it to twitter.

I have no further knowledge of the photo or poll or the people who took it
or even if it actually took place.

and did nothing else. A couple of days later he emailed me with.

Thanks for getting back to us. The Commission understands that the original
tweet in respect of the exit poll has been removed and the Commission is not
taking any further action on the matter.

Thanks again for prompt reply which was much appreciated.

which was a little strange since neither me nor anybody else had removed anything. A little weird and one reason I don’t feel confident with these guys running voting over the Internet.

Share

NZ banning photography from polling places

I just saw on reddit that the New Zealand electoral commission is banning photography from polling places under the grounds that they impeded other voters at the polling and could influence other voters who see the photos. Specifically they say:

Photography in a voting place and sharing photographs on social media

While the Electoral Commission encourages people to take and share photos of themselves with their ‘I’ve voted’ sticker once they’re outside the voting place and unlikely to interrupt or inconvenience other voters, the Commission will be putting up ‘No taking photos’ signs inside all voting places and advance voting places.

The increased interest in voters taking ‘selfies’ inside voting places raises concerns about congestion and disturbance in voting places and can breach other rules in the Electoral Act regarding campaigning on election day and protecting the secrecy of voting.

Voting Place Managers have to ensure that voting proceeds smoothly, that voters are not impeded, and that order is maintained in voting places.  Voting places are for the purpose of voting and people should not remain in the voting place for other purposes.  The increased interest in voters taking ‘selfies’ inside voting places has the potential to create congestion and disturbance and for this reason Managers will be putting up ‘no photography signs’.

Publishing anything on election day that could potentially influence another voter is strictly prohibited, and photos taken earlier in the voting period that are shared, re-shared or reposted on election day could fall foul of the Electoral Act.

If a person posts an image of their completed ballot paper on social media on election day or in the three days prior to election day this is likely to be an offence under section 197 of the Act, which carries a potential penalty of a fine not exceeding $20,000. Section 197 of the Act prohibits a range of activities including:

  • the publication of any statement on election day that is likely to influence voters (section 197(1)(g); and
  • the distribution of an imitation ballot paper on election day or the 3 days before election day indicating the candidate/party for whom any person should vote or having thereon any other matter likely to influence a voter.

It also potentially exposes the voter’s friends to the risk of breaching the rules if they share, re-share or repost the voter’s ‘selfie’ on election day.

As there are risks of congestion and disturbance to other voters and risks with publishing or distributing material that includes a ballot paper, particularly in a medium where material will continue to be published– the Commission will not allow voters to take photos inside voting places.  We will be placing ‘no photos’ signs up in voting places.  Returning Officers will still be able to give permission to candidates for filming in voting places.  Permission for candidates will only be given on the condition that there is no filming behind voting screens, no filming of completed or uncompleted voting papers, and no activities that disrupt voting in the voting place.

I found the reasons they give a little dubious and a complete ban overkill so I’ve written the following to them:

Hello,

I am concerned about the recently published social media policy:

http://www.elections.org.nz/parties-candidates/all-participants/use-social-media

specifically the section banning all photography from polling places.

In the past two elections I have taken photos of the polling place I attended and my unmarked ballot paper and uploaded these to the Wikipedia. These photos (and similar ones) have been used to illustrate photos about elections and even cardboard furniture as well as being used on other sites. Even the official blog of the NZ ambassador to the Philippines used one. http://blogs.mfat.govt.nz/andrew-matheson/elections-theyre-important.

I am thus concerned that there appears to be a new policy that bans all photographs except limited ones by members of the media. This seems to go against the openness of our electoral process and the grounds that are given for the ban are very weak.

The matter of influencing other voters can be dealt with by requesting that photos only be published after voting has closed. Similarly I’m sure there are already rules to handle people who take too long to vote when there are long lines. A specific rule against photographing filled out ballots will also address concerns about voters proving to others they have voted a specific way.

In summary I very much hope you can replace a ban of photography with a more targeted rules against specific problems.

Simon Lyall

 

I receive a reply back from the Electoral Commission:

Dear Mr Lyall,

Photography in the voting place has only ever been allowed with the prior permission of the Returning Officer, but the number of photos being
taken without prior permission has increased hugely this year.  I understand that you feel that people could be allowed to take photos but be
advised not to publish the photos until after 7pm on election day – but unfortunately this is not what voters were doing.

Photos within the voting place, and particularly those taken of marked ballot papers and behind voting screens, have generated a large number of
complaints to the Commission already, and as a result we have re-looked at our rules around photography.

Voting Place Managers have to ensure that voting proceeds smoothly, that voters are not impeded, and that order is maintained in voting places.
Voting places are for the purpose of voting and people should not remain in the voting place for other purposes.  The increased interest in voters
taking ‘selfies’ inside voting places has the potential to create congestion and disturbance and for this reason Managers will be putting up ‘no
photography signs’.

Returning Officers will still be able to give permission to candidates for media or campaign managers to organise filming in voting places.
Permission will be given on the condition that there is no filming behind voting screens, no filming of completed or uncompleted voting papers,
and no activities that disrupt voting in the voting place.

We absolutely encourage people to take and share photos of themselves with their ‘I’ve voted’ sticker once they’re outside the voting place and
unlikely to interrupt or inconvenience other voters, however people taking selfies while behind the voting screen is not a good idea.

 

 

Share

Updating my personal email setup

I’m in the process of moving my personal hosting from one VPS to another ( I host with Linode and am buying a new virtual machine with a similar spec to my current one for half the monthly price ) and I decided to rearrange my home email. My old setup was:

Internet –> Exim on VPS -> Download via fetchmail to home -> Send to spamassassin and dspam at home -> filtering into mboxes on home workstation  -> read via alpine

The main disadvantages of this were:

  • Had to ssh into home to read email (couldn’t read on my phone)
  • Hard to view images in email or HTML emails
  • Sending via my ISP was unreliable and they are  implementing filters
  • No notification of new email

So I decided to make some changes.

Internet -> Postfix on VPS -> procmail to spamassassin on VPS -> procmail to maildirs -> read via imap

This setup is a lot simpler than the previous one and a bit more mainstream.

  • Since the email is online via imap I can read it directly from alpine or my phone (or another client)
  • Online running one anti-spam program (spamasaasin) instead of two (dspam and spamassassin)
  • Email operations on one server (the VPS) insead of 3 (VPS, workstation, home virtual machine)
  • Sending email straight via VPS instead of home VM and my ISP’s mail server

Details of my setup

There are a lot of HOWTOs on getting email to work via postfix and dovecot. I decided that the main feature I needed were virtual aliases for my domains. I also decided that since I only had a few mailboxes (mine own and two others) I could just create accounts on the server rather than maintain virtual users in postfix/dovecot. The server is running Ubuntu 14.04

Roughly speaking I followed the advice on these two pages by Rimuhosting:

I added these lines to my Postfix’s main.cf

virtual_alias_maps = hash:/etc/postfix/virtual
home_mailbox = Maildir/
mailbox_command = /usr/bin/procmail -a "$EXTENSION" DEFAULT=$HOME/Maildir/ MAILDIR=$HOME/Maildir

Theses lines tell postfix to (1) use the virtual file (see below) (2) deliver to Maildirs (3) use procmail for delivery.

and create a /etc/postfix/virtual file like:

darkmere.gen.nz              20140720
simon@darkmere.gen.nz        simon-mail@cyan.usenet.net.nz
root@darkmere.gen.nz         simon@darkmere.gen.nz

The first line indicates the domain should be used (this option is a little hidden in the virtual manpage) and then their are various addresses. simon-mail is a noshell account and cyan is the name of the server to so the email is delivered locally to it.

The simon-mail account just has a simple .procmailrc file with my various filters and a Maildir to store the email. Spam processing is called by procmail via:

:0 fW
* < 280000
| spamc -u simon-mail -d localhost

:0:
* ^X-Spam-Status: Yes, score=([5-9]|1[0-9]|[2-9][0-9])
.junk/

which just puts all email that looks like spam into a junk folder (which I can check now and then until I’m happy with the filters).

Dovecot for imap worked out of the box except I had to tell it the location of my email. I just edited the file /etc/dovecot/conf.d/10-mail.conf and changed the mail_location setting to:

mail_location = maildir:~/Maildir

For sending email I pretty much followed this guide directly.

Overall it wasn’t too hard. The main problem was the fact that there were so many guides (I read over a dozen) each of which differed slightly and which were in many cases designed more much larger sites. I’ve currently got the setup in final testing (it is getting a copy of all my incoming email) and intended to switch over soon. In the short term I’m keeping my old mail folders (all 752 of them adding up to 1.8GB) locally at home but may move them at a later date.

Share