Michael Coté – Not actually a DevOps Talk
Digital Transformation
- Goal: deliver value,
weeklyreliably, with small patches - Management must be the first to fail and transform
- Standardize on a platform: special snow flakes are slow, expensive and error prone (see his slide, good list of stuff that should be standardize)
- Ramping up: “Pilot low-risk apps, and ramp-up”
- Pair programming/working
- Half the advantage is people speed less time on reddit “research”
- Don’t go to meetings
- Automate compliance, have what you do automatic get logged and create compliance docs rather than building manually.
- Crafting Your Cloud-Native Strategy
Sajeewa Dayaratne – DevOps in an Embedded World
- Challenges on Embedded
- Hardware – resource constrinaed
- Debugging – OS bugs, Hardware Bugs, UFO Bugs – Oscilloscopes and JTAG connectors are your friend.
- Environment – Thermal, Moisture, Power consumption
- Deploy to product – Multi-month cycle, hard of impossible to send updates to ships at sea.
- Principles of Devops , equally apply to embedded
- High Frequency
- Reduce overheads
- Improve defect resolution
- Automate
- Reduce response times
- Navico
- Small Sonar, Navigation for medium boats, Displays for sail (eg Americas cup). Navigation displays for large ships
- Dev around world, factory in Mexico
- Codebase
- 5 million lines of code
- 61 Hardware Products supported – Increasing steadily, very long lifetimes for hardware
- Complex network of products – lots of products on boat all connected, different versions of software and hardware on the same boat
- Architecture
- Old codebase
- Backward compatible with old hardware
- Needs to support new hardware
- Desire new features on all products
- What does this mean
- Defects were found too late
- Very high cost of bugs found late
- Software stabilization taking longer
- Manual test couldn’t keep up
- Cost increasing , including opportunity cost
- Does CI/CD provide answer?
- But will it work here?
- Case Study from HP. Large-Scale Agile Development by Gary Gruver
- Our Plan
- Improve tolls and archetecture
- Build Speeds
- Automated testing
- Code quality control
- Previous VCS
- Proprietary tool with limit support and upgrades
- Limited integration
- Lack of CI support
- No code review capacity
- Move to git
- Code reviews
- Integrated CI
- Supported by tools
- Archetecture
- Had a configurable codebase already
- Fairly common hardware platform (only 9 variations)
- Had runtime feature flags
- But
- Cyclic dependancies – 1.5 years to clean these up
- Singletons – cut down
- Promote unit testability – worked on
- Many branches – long lived – mega merges
- Went to a single Branch model, feature flags, smaller batch sizes, testing focused on single branch
- Improve build speed
- Start 8 hours to build Linux platform, 2 hours for each app, 14+ hours to build and package a release
- Options
- Increase speed
- Parallel Builds
- What did
- ccache.clcache
- IncrediBuild
- distcc
- 4-5hs down to 1h
- Test automation
- Existing was mock-ups of the hardware to not typical
- Started with micro-test
- Unit testing (simulator)
- Unit testing (real hardware)
- Build Tools
- Software tools (n2k simulator, remote control)
- Hardware tools ( Mimic real-world data, re purpose existing stuff)
- UI Test Automation
- Build or Buy
- Functional testing vs API testing
- HW Test tools
- Took 6 hours to do full test on hardware.
- PipeLine
- Commit -> pull request
- Automated Build / Unit Tests
- Daily QA Build
- Next?
- Configuration as code
- Code Quality tools
- Simulate more hardware
- Increase analytics and reporting
- Fully simulated test env for dev (so the devs don’t need the hardware)
- Scale – From internal infrastructure to the cloud
- Grow the team
- Lessons Learnt
- Culture!
- Collect Data
- Get Executive Buy in
- Change your tolls and processes if needed
- Test automation is the key
- Invest in HW
- Silulate
- Virtualise
- Focus on good software design for Everything
