Mark Alan Richards

We need to talk about Agile

computer-science agile

"Agile" is not a software methodology, it is an ideology... it is built on a manifesto, that in practice is often corrupted as meaning something beyond what it states. There's too many articles on what Agile does or doesn't mean, but essentially it does demand design, process, documentation and tooling; it just suggests they should be enriched with greater attention to the functions that lead to results. It was created at at time, when you could still buy software off the shelves with user manuals that were hundreds of pages long and if it didn't work, you couldn't easily update it.

As an ideology it has a desired goal, which is to enable software development, but promoting ways of working that help reach functional goals. It does also tend to be a bad fit for heavily regulated environments or security conscious environments.

It has very obvious missing goals, it does not address non-functional requirements like compliance, performance, reliability, consistency, accessibility, maintainability, backups, ..., the list goes on.

Therefore, it is a an ideology to achieve a function, regardless of how well that function works in an ever changing and complex environment.

So what is missing and what do we need to add to or replace in Agile. I'd like to introduce two very important ideologies that we should add to Agile.

The Right People:  we need expertise, not just devs

Most devs can write great functional code based on business ideas for most logical and presentation requirements but they come unstuck when they need expertise in:

An example: Encryption

I do not know a single software developer who can write encryption libraries, like TLS 1.2 level encryption and I include myself. I believe there are lots of mathematicians and computer scientists who could develop individual encryption functions, but combined into a framework that is secure through the whole layer, I'm not sure many can.

I have not met a single developer qualified to identify what encryption libraries are good or bad.

So why do we let software developers pick encryption libraries and configure their implementation?

AES 256 and RSA 4096 are surely all you need? Well, no, you'll still need to understand at least the following to use them:

But our team is small and only average software developers and QA?

Maintenance: it isn't just bugs

Agile typically does cater for bug fixing, but maintenance is more than that.

With GDPR arriving soon, hopefully everyone is reviewing all systems that hold, transport and ... access PII. However, it's not just GDPR that is  and has been changing, especially in more regulated environments with PCI, MIFID, GCP, Accessibility law, etc often have amendments too. Contracts with third parties typically demand levels of security that must be adhered to as well. Some of these changes are passive (you have to discover a legal change) and some are active (you are told of a contractual change) but both need to cycle into the maintenance of what would otherwise be ignored code running in production.

Access is actually a really worrying problem. Many systems are setup with walls at perimeter, but not inside. So the web frontends and web api gates into the system are typically offered some lifecycle management to check for greater maintenance risks, but the other services can be just as dangerous.

XXE, Remote Code Execution (reflection, SQL, etc) and even internal tooling are all often one step away from attackers and that step might not be designed to worry about the concern.

The last concern is licences and discovering that your licence for proprietary software can leave you in a legal dilemma (do you shutdown to respect the licence or steal some more time, hopefully only to migrate or negotiate renewal) or that the open source project you use has died... do you take a risk and continue with unsupported and increasingly likely vulnerable software or refactor off. To do the right thing means knowing about it.

This requires audit

Where is audit mentioned in the Agile manifesto?

Audit is a function of all business environments have and whether it is a legally qualified audit, like an accountancy audit or regulatory audit for compliance that has a high process demand or whether it is a just a regular check, like are the toilets clean with a tickbox to fill, it happens everywhere... except so much in software development.

How do you check your toilets are clean in your website in production? Sounds a bit strange, but how many of us are testing for vulnerabilities, reviewing log cleanliness, etc. We might be doing the functional parts that we know will break the application: is the server running, does the database have disk space... but beyond that too many places have security problems or even embarrassingly suffer problems like their TLS certificates expiring... when that happens, it is not a failure of the dev/ops who set it up, it is a failure of the business to have a business control around it.

So we might clean the toilets with a quarterly automated penetration test, alarms when the disk levels are high and a Jira scheduled ticket (why doesn't that exist?) for renewing the TLS certs, but what about the other layers of audit.

As a lifecycle event it should be driven by business controls, which means they know what to control and that requires documentation, which is fine in Agile, Agile only really demanded you didn't write thousand page usage manuals like you used to get with software in the 90s (okay, some were only 600 pages).

This is the key reason why Agile Software Development fails as a project management methodology alone: the business behaviours demanded are always changing and yet the project leaves features dead in Jira: with just bug fixing idling along until a new feature is demanded.