In one of our recent blog posts, we looked at various development models and what makes them different. One of the models we discussed was Waterfall Software Development. In this article we’ll look at the Waterfall model in a bit more detail, exploring its advantages and disadvantages. Show
What is the Waterfall Model?The Waterfall model has been around for more than 40 years. It was first described in a 1970 paper (Dr Winston W Royce), as one of the very first formal software development process models. We often describe Waterfall as a ‘linear-sequential lifecycle model’. This means that it follows a simple structure of phases where the results of each phase cascade down to the next level of development. In other words, we’re not so much looking at one big Niagara Falls, but a series of cascading waterfalls – each with their own little pool of activities. The original model consisted of five stages:
There are several well-recognised waterfall methodologies. One of them is PRINCE2, which was created by the UK Government and is still held in high regard in the public sector. Waterfall is often placed on the opposite end of the software development spectrum to Agile development. One of the key differences between the two methodologies is that Waterfall requires everything to be described in written documentation before any code is created. This approach makes it crucially important that all requirements are meticulously defined from the start, as it might be difficult to revise them later. The pros and cons of Waterfall Development
The conclusionWhen it comes to Waterfall Development, it is critical that the software developers involved are able to guide and advise clients effectively in order to circumvent problems later. This is often the most criticised aspect of waterfall development – that the customer doesn’t really know what they want. In many cases, the genuine two-way interaction between developers and clients doesn’t start to happen until the client can actually see the model in action. (Agile development, as a comparison, lets the client see chunks of working code developed and demonstrated throughout the entire process.) Based on these pros and cons, Waterfall development is generally recommended for projects that are not expected to change or to need new developments during the project lifecycle.
Software Engineer Last Updated on November 23, 2022. We introduced Waterfall, Agile, and DevOps in three previous articles, focusing primarily on their history, advantages, and core principles. To complete the discussion, this article will examine their relative strengths, challenges, limitations, and adoption in the software community. The information we provide is derived from the 15th State of Agile Report and the 2021 State of DevOps Report, supplemented by practical, real-world experience from the field. Finally, you can check these guidelines if you want to migrate your delivery processes but are still unsure which one best fits your team and organization. 2. Table Of Contents3. Approach3.1 Assessment CriteriaA few things to note regarding the approach we intend to take:
4.2 Terms and DefinitionsThe only terms in this article that require additional clarifications are:
4. Assessment of Agile, DevOps, and WaterfallBefore we continue We highly recommend reading the introductory Agile, Waterfall, and DevOps articles for background, context, and an understanding of their main concepts. The below statistics come from the 15th State of Agile Report, and provide us with a snapshot of the current state of Agile in terms of adoption within software communities, organizations, and teams. Agile adoption statistics in 2021The first pie chart with 3% of participants reporting a non-Agile methodology presents a bleak future for Waterfall. No surprises there. What is surprising, however, are the numbers in the second chart indicating that only 18% of participants are fully Agile, whereas the rest still use some form of Waterfall. Why is Waterfall still lingering around two decades after the publication of the Agile manifesto? Is it because Waterfall is easier or even better than models? The answer requires some explanation. We will assess each model based on six criteria in the following sections. This assessment will allow for a fair comparison of all three and perhaps allow us to appreciate more the dynamics of Agile adoption. 4.1 Waterfall4.1.1 AccessibilityThe Waterfall model is the most basic and intuitive software delivery methodology among the three for the below reasons:
4.1.2 Team Structure and MaturityTeams of developers, testers, architects, and business analysts using Waterfall are typically compartmentalized and often work in silos. This lack of cross-team interaction and mutual support between the different groups is one of Waterfall’s significant drawbacks. A typical Waterfall project outcome Frequent misunderstandings, misalignments, and lack of visibility and empathy across boundaries reduce the team’s productivity. On the positive side, Waterfall teams typically have one role per member, which leads to faster maturity, easier integration, and a gentle learning curve. Self-organization is limited, with power relationships leading to complex and unconstructive shadow hierarchies. 4.1.3 Tools and InfrastructureGantt charts, spreadsheets, and a basic source code versioning tool are the necessary and sufficient equipment to implement a solution using Waterfall. This lack of sophistication makes the model ready to deploy and run but also highly inefficient, primarily due to the lack of automation and the dominance of manual and human intervention leading to all sorts of errors. 4.1.4 Vertical and Horizontal ScalabilityThe Waterfall model has no issues with vertical scalability. Managers would align expectations with developers based on effort estimations in person-days provided by the latter. These estimations would then be used to create a schedule and calculate a cost; if these do not align with the stakeholders’ expectations, resource allocation and project constraints are modified. Regarding horizontal scalability, effort across multiple project streams or teams is still required. It is typically done via workshops, project meetings, steering committees, etc. 4.1.5 Cost of ChangeCost of change in Waterfall is effortless to describe; it has been identified as early as 1970 in Winston Royce’s famous paper, “Managing the Development of Large Software Systems”. Cost of Change as a Function of SDLC StageIn this paper, Royce points out that cost of change increases dramatically as you move down the Waterfall stages for two reasons; the farther you are in the implementation:
Two critical enhancements were applied to remedy these problems. First, investment in massive design efforts was made to foresee potential challenges in solution architecture and software design before implementation began. Second, system integration and unit testing were adopted to limit the number of bugs that make it into the software testing phases. While these measures improved the software quality produced by catching problems as early as possible, they did not have much impact on the cost of change of a bug if it managed to proceed undetected. The problem was only half resolved. 4.1.6 Architecture and DesignWaterfall adopted a holistic approach to the solution architecture and software design of software products. A typical project would contain several new features to be implemented. Once the requirements are signed off, architects and software engineers would produce architectural plans and low-level designs for the entire set during the design stage and before any implementation. This method, a holistic top-to-bottom approach, is reliable and easy to manage because architects and engineers would:
This method is convenient and reliable but costly; any revision must go through the full feature set to determine any potential redesign effort. This setup saw a trade-off between control and growth, to the latter’s detriment. 4.2 Agile4.2.1 AccessibilityAgile brought improvements over Waterfall in many dimensions, and at its core is the concept of smaller, frequent releases. This modification alone requires quite some effort to explain to peers or management since, at first sight, its advantages are not evident. On top of that, you have paradigm-shifting concepts such as the backlog and its management (called grooming), the cross-functional self-organizing teams, effort estimation through story points, Kanban boards, and team velocities. To make matters worse, several implementations of Agile (like Scrum and Kanban) exist, making the topic seem overwhelming to new joiners. It’s even more difficult for people to decide which Agile version best fits their teams. Agile is neither intuitive nor cheap, making it difficult to sell, especially to senior management or teams set in old ways. These attributes might explain why the top challenges cited in the 15th State of Agile Report (organizational culture, general resistance to change, insufficient leadership support, and inconsistent processes) remain unchanged year on year. 4.2.2 Team Structure and MaturityTeam structure in Agile displays a substantial deviation from its homogenous, static, and self-contained Waterfall counterpart. Agile teams are supposed to be cross-functional, self-sufficient, self-organizing, disciplined, and educated in the Agile way. They usually consist of several profiles like developers, testers, architects, and tech leads. Agile models have also created new roles, such as Product Owners and Scrum Masters. This considerable variety of skill sets and diversity of profiles and methods make Agile teams look very different from one organization to the next. To work together efficiently, the team must be mature, disciplined, and fluent in the Agile methods. These properties create a complex system of interactions and an atypical hierarchical structure that varies with each group. This ability to self-organize, however, allows teams to find a comfortable setup that best suits their skills, profiles, and expertise while still functioning efficiently. 4.2.3 Tools and InfrastructureTo appreciate the magnitude of change in tools supporting Agile methods, think of the Atlassian suite and how much it has changed in the last decade. The synergy between tools like JIRA and BitBucket and the greater adoption of Agile was perfect; these applications made Agile easier to implement, while more Agile customers allowed Atlassian to invest and grow them beyond recognition. Another equally influential set of tools is those that support build and test automation like Maven, Gradle, Jenkins, and Robot. Without automating many repetitive, time-consuming, and non-value-adding tasks, the overhead that resulted from smaller and more frequent releases would have overtaken any advantages the latter brought. Conversely, fluency in these tools is now a must for Agile practitioners, thus raising the bar for new joiners and steepening the learning curve. 4.2.4 Vertical and Horizontal ScalabilityThe challenge with scaling up Agile is a genuine one. As one project manager once told me, Agile tells you that your feature is in the backlog, but it cannot tell you how much it costs or when it can be delivered. The reader might think that this issue has more to do with poor stakeholder management (or effort estimation, velocity calculation) rather than Agile not being scalable. The truth is that even if better stakeholder management processes were in place, project management could not work in Waterfall while the development uses Agile; they would constantly be out of sync. Does Agile scale up quickly and comfortably? The 15th State of Agile Report dedicated only one page to scaling up, which is not much. The report cites Scaled Agile Framework (SAFe®) as the most used method, with 37% of participants applying it within their organizations. How successful and preponderant is SAFe? That remains an open question. To efficiently collaborate, the customers, suppliers, project managers, senior managers, and developers must all be Agile. This requirement is much harder to achieve than it sounds since the reality is much more complex and messy than we would like it to be. What about horizontal scaling? Agile does not provide specific measures to treat this problem, and the same methods (workshops, meetings) for Waterfall continue to apply here. Coordinating multiple project streams with interdependencies is a bit more challenging. This difficulty arises from release management in Agile (small frequent releases), where teams may run at different speeds. The team will be blocked if interdependencies exist until the dependency is resolved. Reducing this interdependency may lead to siloing, another consequence we would like to avoid. 4.2.5 Cost of ChangeThe cost of change in Agile is not as easily described as in Waterfall. In the latter, we stipulated that the closer the project’s final stages, the costlier the changes. With the introduction of smaller and more frequent releases, less design time is now allowed (more on that in the next section) and significantly more automation, especially around build and quality assurance. Does this mean that cost of change has gone down proportionately? The answer would have been ‘yes’, except the introduction of so much overhead has muddied the picture quite a bit. Think of how much time and how many people are involved in writing and implementing a user story and whether these efforts are proportional to the size of the user story. For example, the time it takes to run a full suite of unit tests does not heavily depend on the size of the change (unless your last user story has doubled the number of these tests!). Small changes, like big ones, have a cost. What about the design and the short sprints? How has that impacted cost of change? The decentralization and downsizing of the design effort may lead, in the long run, to a sharp increase in architectural and technical debt and, therefore, cost of change. 4.2.6 Architecture and DesignTo manage two-week sprints that resulted in software being released into production, radical changes to the delivery process had to be made. In addition to running automation in fifth gear, a sacrifice in software design effort (but not necessarily design quality) had to be made. If software can now be architected and designed within the development teams (and not in s special guild), something must be changed. The first change was that architects and senior engineers were now integrated within Agile teams. The second change was abandoning large-scale solution designs and favouring small-scale, feature-level efforts. Two questions arise:
Cutting down on centralized design efforts has traded control for growth. On the other hand, the architectural and technical debt must be restrained. This relaxed attitude toward design was counterbalanced by stricter rules and a higher skill level in software architecture. Also, technical debt management measures were instituted with the help of specialized software like SonarQube. The emergent architecture was complex, arising from simple but strict rules applied iteratively with new features. This dynamic meant a bottom-up approach (think of complex systems like living organisms) instead of a top-down one. So far, the result seems OK, especially since refactoring exercises (and other maintenance works) could now be budgeted at the management level but managed within an Agile team. 4.3 DevOpsThe assessment of the DevOps model is an extension of Agile’s in that both have the same impediments, albeit by varying degrees. DevOps has added continuous delivery to the original Agile outputs and has brought operations under its umbrella. This supplemental sophistication rendered the complications inherited from Agile more acute. Despite getting the people who built the software to operate it makes much sense, DevOps is no more intuitive and accessible than Agile. The profiles and skill sets are richer and more complex, and a “DevOps engineer” role has emerged. The number of tools and services they render has also increased to cover continuous deployment, cloud support, and operations, thus also raising the bar for new joiners. Every new generation of software project management methodology must accomplish two things. Firstly, it must provide a substantial improvement over its predecessor, and secondly, it must also work with the latest technology, such as cloud deployment, for instance. 5. What Else?External factors almost always play a role in the success or demise of a new idea. These factors can sometimes be entirely unrelated to the merits or limitations of the concept itself. The notions behind Agile and DevOps are no different. In addition to their implicit merits and challenges, external, equally influential factors played a decisive role in speeding up or slowing down their adoption and, therefore, their utility. These factors are organisational culture, process management and governance, and working from home.
5. Final WordsIf this article were trying to deliver a message, it would read like this. Despite the obsolescence of the Waterfall model, Agile and DevOps are not silver bullets, and the question of which methodology is best suited for your situation is far from evident. The recommendation, however, is to use Agile practices to achieve Operational Excellence. We hope this article has shed some light on the potential issues that Agile and DevOps practitioners face and perhaps spurred the discussion on how best to resolve them. 6. Further Readings
12. Featured Articles
|