My History With Open Source

From CPAN to Design Patterns

Throughout my career I've benefited greatly from being able to utilise open source software that other developers have produced and made freely available.

Some of my earliest commercial project work benefitted from libraries made available for Perl via the Comprehensive Perl Archive Network (CPAN). It sometimes felt like our company had a huge advantage over organisations that used VB Script for developing ASP pages, as they seemed to be tied into the world of closed source and needing to pay to use libraries that other organisations had developed as a licensed product for sale.

In the early two thousands I was continuing my university studies as a part time student while working as a software developer. One of the distributed systems courses gave me some exposure to JBoss and Tomcat, which made me question why we were paying to use commercial application servers for some of our clients' projects in my day job.

Aside from the common day to day helper libraries such as JUnit and log4j, Tomcat was probably the first major open source Java system that I brought into my work, proving out that we didn't need EJBs and all of their standards for getting some data out of a database and onto some web pages. At around the same time we were probably dabbling with Apache JMeter as a mechanism to validate that this new kid on the block (well, our block at least) was going to cope with what we anticipated was going to be thrown at it.

Although we didn't use any particular library for it, I would also consider design patterns as an example of shared knowledge that really helped us to achieve our scalability and performance goals. Safely caching data that was being read frequently, but only being updated only a few times each day. If you went skiing in New Zealand in the early 2000s then I can almost guarantee that you checked snow reports using code that I developed.

Giving Something Back

Open source licenses can be a legal minefield for the owners and the users of products and libraries.

Working in large corporations often involves policies and even clauses in employment contracts - along the lines of "If you develop on company time, or on company hardware then anything produced as a result is the property of the company" and / or "The use of any open source software is expressly forbidden unless it has been formally approved by the XYZ committee".

Even smaller companies need to be aware of the differences between GPL, MIT, Apache and other variations of licenses before building a product up.

So far my main contributions to open source projects have mainly been limited to minor improvements to the documentation, and a couple of minor bug fixes for some smaller projects. Correcting typos and improving grammar can be a small way of helping out - providing that it isn't pedantic or debatable whether the new phrasing is better. So far I have had all of my contributions accepted with gratitude, as the original developers sometimes have English as a second language or just slipped up a little in the rush of getting something out and released.

Personally, I also find that by contributing to explaining how something works I can improve my ability to understand and recall that information later on. So, as well as being a good way to make an initial contribution to an open source community, consider that by improving your understanding you will also be moving some way towards being able to contribute to the code as well.

People Skills In A Software Development Team

Introduction

In my opinion the old stereo type of expert developers being left on their own to solve the big problems isn't the way that robust software should be developed.

As often as not when you hear a story about hero developers who are often arrogant and / or jaded about their experiences and are not open to collaborating. That doesn't make for a nice workplace, or sustainable product development.

If you Google'd "no assholes" you will probably be presented with a long list of articles and blog posts describing how organisations have come to the realisation that it is in their interests to not hire or tolerate individuals who are not team players - even if they are the best in their area of expertise. (NB: In this context I don't mean "team player" in the sense of being pushovers who just do what they are told).

This is yet another post of me trying to remind myself about what I've been doing to work well within a team - a nice refresher before I go into job interviews with their HR situational questions.

Empathy Is Key

In the heat of the moment it can be tempting to assume that a problem is somebody else's fault, or that another team doesn't care as much as we do. Most of the time that isn't really the case. Without knowing the wider context around when, why and how something came to be we really shouldn't assume the worst.

Sometimes the root of the problem will be me - like the time that I made a quick one line change to a block of code that I had been working on and when that change was deployed the service started to randomly stop connecting to a service that it was interacting with. I hadn't looked at the context around when that line of code was being called, so didn't notice that I was preventing connections from being closed. 🤦

Psychological Safety

I used to switch off when I first heard the term, "Psychological Safety" as it seemed like something leading to sitting around a campfire and singing Kumbaya.

Much later in my career I paid a litte bit more attention and realised that it is a very simple concept of how comfortable colleagues feel about working with eachother. I see it as being at ease with speaking your mind, knowing that nobody on the team with belittle your ideas, that you will be listened to and understood, and "there are no stupid questions".

One of the personal highlights of my career was on a spontaneous mini pub crawl that arose from having to be out of the office while movers were coming to relocate our equipment. The lunch and conversations were quite fun, but the thing that has really stuck in my mind was when we were walking between pubs and I mentioned to one of the senior members of another team that I felt like I was fitting in well after only a few weeks in the company, his response was "We like you, you ask intelligent questions".

Care, Respect, and Courtesy

I think this was a school motto, but I can't trace it back as that was over thiry years ago and some schools seem to chop and change their motto - and it's not really that important to know the origin story.

Anyway, lets consider each how term can be applied to the working environment.

I use the term "careful" a lot more than the term "care", mainly because I'm used to talking about solving technical problems and how we need to ensure that we don't lose data or make our systems unavailable for our users. This shows caring about the people that interact with the software that we're developing.

Alongside that, life tends to go much more smoothly when we are considerate to our colleagues, whether that means informing them in advance of upcoming changes, or checking in with them on Slack to understand how they are getting on with situations that arise in or out of the work setting. Not just a, "Get well soon" response if they have to take a sick day.

Respect and courtesy often go together, such as listening to what a person has to say without interrupting, and if there is something to challenge being polite enough to address the idea rather than the person who raised the idea. On my first full-on scrum team some of my German colleagues raised these sorts of manners as part of the team's agreed ways of working, and at the time I wasn't sure whether that needed to be said, however I noticed when that was missing from a team that I worked in later in my career and moved to correct that.

Obligation To Dissent

Obligation to dissent was another unusual term that I came across in a team ways of working discussion. In that context it meant that if you had a strong opinion or objection to something that the team was going to do then it was your responsibility to raise your concern and have it addressed.

This approach could be quite distructive on its own, and should be paired with something like ", but be willing to move forward with the team's agreed approach".

My Operations History - A Journey Towards Appreciating DevOps And Infrastructure As Code

Introduction

These days I consider myself to mainly be a developer, but with a solid background in getting services and apps operational.

This post is intended to focus on some of the work that I have done so far in my career that I would consider as being more focussed on the operations side of technology-driven projects.  It can be read as background to my appreciation of the DevOps approach to developing and deploying applications.

If you came to read about my medical history then that is also covered here: I've never had an operation.

University days

Personal hardware

Back in the 90s PCs were still the dominant home computer, and as my brother was into electronics and electrical engineering I went down the path of assembling my first proper computers from components. The level of component that I'm referring to here is consumer-accessible ready-made modules rather than individual chips and circuit boards. I never took up a soldering iron in anger.

So, when I had enough of my own money (being a poor student, money was not abundant), I selected a case, hard drive, motherboard, CPU, memory, video card, and sound card and went about combining them to form the basis of a working computer.

Back then the Internet wasn't a common thing in every home, so I didn't need a modem or network card to start with.  The display was a CRT monitor that I had received as a hand-me-down when a member of the family had upgraded their system.

Unix, shell scripting and cgi-bin (server side "dynamic" web)

Around the same time I was learning some C programming and various useful utilities on Solaris at university both as part of my coursework, as well as out of general curiosity of how to get things done.

I was very interested in how the web worked, and managed to set up some cgi scripts that would run under my home directory on one of the computer science department's servers, though I think this didn't get much beyond calling a random number generator to determine which image to show from a sub-directory on the server.

Startup Systems Administration

Setting up Workstations

Being one of the first employees at a growing company meant that I was able to get involved in configuring the workstations of the new employees.  Back then that involved installing the OS (Windows NT 4, later Windows 2000) and a few core applications: virus scanner, current version of Java SDK, Perl, IIS, MS Office.  We were quite a small scale operation of fewer than a dozen employees for most of the time, so there was never any great need to automate this work.

Configuring Linux Servers

During the day to day work I would pair up with a colleague when he needed to update some of our critical infrastructure software for the systems that we hosted on our servers.  Sendmail and BIND seemed to need updates relatively often - probably every month or every other month.  Back then we had a few customisations in our installation so an upgrade involved downloading the latest source code and building it from configure scripts and makefiles with specific options and configuration files applied.

Later on I needed to apply the same experience to build up a Postgresql server with the PostGIS extension compiled in for use on a property auction website - enabling the site to present out an up to date visual representation of the sold / available status of each property on the map.  The build of the database server was a bit last minute as we realised that a flat file representation of the data wasn't going to work properly on Linux (and I already had my reservations about that approach being suitable for multiple concurrent updates).

Another significant pieve of work involved upgrading and migrating our internally hosted IMAP email server to a more powerful server, so I went ahead and applied some locking down of access rights to sensitive company folders at around the same time - not because anyone in the company wasn't trustworthy, but just as a general best practice for responsible business data handling.  There were a few gotchas when setting up new email folder permissions after that, but the data migration and version upgrade went seamlessly well.

Configuring Windows Servers

Historically we had hosted several client websites using Microsoft's Internet Information Services - which acted as the runtime environment for Active Server Pages (commonly referred to as "ASP pages").  I took care of setting up IIS and locking down various optional features based on security recommendations.

I used to regularly monitor slashdot, securityfocus.com and various other sites to keep up to date with the latest issues and potential issues to be addressed as proactively as possible.  I also routinely monitored the content being uploaded by our clients, such as ensuring that they weren't slipping into outdated practices for making database credentials available for ASP pages that happened to involve database queries.

Database Access Rights

To minimise the potential risk of having compromised database credentials exploited, all of our databases had table level permissions locked down to the absolute minimum rights required by each client application.  We didn't quite split responsibilities up to have reads handled separately from deletes or writes back then as we were still developing monoliths - which was fine for the scale that we were operating at back then.

Continuous Integration - GoCD plugin

While I was working at Springer (later known as Springer Nature) I gained some hands on experience with two technologies that worked well together: Thoughtwork's GoCD, and Pivotal's Cloud Foundry.

When a competition was announced by Thoughtworks to bolster up the range of plugins available for GoCD, I decided to have a go at producing some open source software that would enable teams that used the combination of GoCD and Cloud Foundry to gain a small additional capability for automating the detection of updates in Cloud Foundry.

My plugin won first prize in the competition, but soon afterwards the somewhat clunky GoCD plugins API was given a significant overhaul meaning that my plugin would not work in later versions of GoCD.

Microservices - Infrastructure As Code

Deploying Microservices

Microservices often need some supporting infrastructure beyond their unit of deployment, such as an object store (e.g. S3) or a database or a persistent shared cache.  All of the major cloud providers provide these types of services and have interfaces to allow us to create them programmatically.  My more recent operations experience has been strongly oriented towards automation of provisioning and configuring such infrastructure components.

The more mature organisations that I have worked in have had systems such as Ansible and Terraform in place to allow developers to specify the expectations of the available supporting services and have any updates be provisioned and applied automatically, sometimes as part of the release process for a new version of a service.

Feature Toggles, Not Feature Branches

When it comes to significant changes to infrastructure as part of a release I've found that it is better to aim for a "roll forward" strategy in the event of any unexpected side-effects rather than expecting to be able to roll back.  This can involve something as simple as toggling the new feature off to give the team enough time to properly diagnose what has gone wrong.  The alternative might involve removal of the newly created infrastructure, which could hide the issue and delay resolution.

Where To From Here?

At this point in time (October 2021) I'm at a potential fork in the road careerwise, I could either continue to be a developer of services for end users and service integrations, or switch over to join a team more focused on enabling developers - sometimes refered to as "Platform engineering" - or I could move even further towards the metal and get involved in developing the tools and services that underpin platforms.