Introduction

It’s widely recognized that the bulk of software costs arise after its initial development in maintenance tasks like bug fixes, feature additions, and day-to-day operation. Therefore, it’s crucial to build systems that are easy to modify, extend, and operate, ensuring they remain maintainable over time.

Robust testing, including unit, integration, and end-to-end tests, is essential to enable safe modifications and extensions to a system. Once changes are made, they must be released to production without causing downtime or issues. Operators need tools to monitor system health, diagnose problems, and restore services quickly, often achieved through non-code changes such as toggling feature flags or scaling services.

Chapter 29 – Testing

The cost of fixing a bug increases the longer it goes undetected. Software tests, which verify specific parts of an application’s functionality, are crucial for catching bugs early and ensuring that changes to the codebase don’t introduce new issues.

29.1 Scope

Tests vary in scope, from unit tests that focus on a small part of the codebase to integration tests that verify interactions with external dependencies. Unit tests should be stable over time and should only change when the behavior of the system under test changes. Integration tests can be narrow, focusing on specific interactions, or broad, testing interactions across multiple services.

29.3 Practical Considerations

Testing involves trade-offs. For example, end-to-end testing a specific API endpoint might involve interactions with a data store, an internal service, and a third-party API. Balancing the scope and complexity of tests is essential to maintainable testing practices.

Chapter 30 – Continuous Delivery and Deployment

Manual release processes are time-consuming and error-prone. Automating the release process through continuous delivery and deployment pipelines allows changes to be released to production quickly and safely, reducing the likelihood of issues and freeing up developers’ time.

Chapter 31 – Monitoring

Monitoring is critical for detecting failures in production and providing visibility into the system’s health. It can help identify issues early and trigger alerts to operators for quick resolution.

Chapter 32 – Observability

Observability is essential for understanding the behavior of complex distributed systems. It allows operators to detect and diagnose issues quickly, even in the presence of failures and unpredictable behavior.

Chapter 33 – Manageability

Manageability involves the ability to modify an application’s behavior without changing its code. This can include releasing new versions, changing configurations, or managing secrets. Flexible configuration management is crucial for maintaining a system’s manageability.

Summary

Maintaining a production service involves various activities, including bug fixes, feature additions, and day-to-day operations. Embracing these maintenance activities and focusing on building systems that are easy to modify, extend, and operate is key to becoming a better system designer. By prioritizing maintainability, developers can build systems that are more resilient, scalable, and easier to manage in the long run.