The systems development life cycle (SDLC) is a conceptual model used to develop information systems with well-defined phases. The phases are user requirements definition, system requirements definition, analysis and system design, system development, testing, implementation and maintenance. Some of the models are waterfall, spiral and rapid prototyping. These models are called plan-driven which lie in one end of the ‘planning emphasis’ continuum. At the other end of the continuum lie the agile methods which are considered as adaptive rather than predictive. This paper presents a systems theory perspective of both the models, the inherent challenges that need to be addressed to transition from traditional to agile methods.
Nature of Systems Development
Software development is a highly complex activity. It is characterized by highly variable requirements, the specialization and diversity of skills required, the changing nature and sophistication of technology used to develop and deploy software and the difficulty in management of people who deal with such complexity every day. It is not uncommon to find many organizations being overwhelmed by the inherent complexity in implementing systems development projects.
Therefore the systems development processes can be described as being complex, unpredictable and not well-defined. In other words, these processes don’t have well-defined inputs and outputs and therefore are considered as unrepeatable.
Traditional Systems Development Model
Almost since its inception, a rational engineering based approach such as waterfall method has been used to develop projects. This seems to be grounded in ‘hard systems thinking’ which assumes that problems can be well-defined and processes can be optimized and results well predicted. Extensive upfront planning is done to measure and control the variations in the development life cycle.
The essential nature of traditional software development life cycle :
- The goals are to thoroughly understand users’ needs, craft a solid design, develop software flawlessly and implement a functional system that satisfies user needs
- Heavily emphasizes thorough planning to deal with risks
- Based on the principles of hard systems thinking – identifying alternate ways of reaching the desired state (S1) from the initial state (S0) and choosing the best way to achieve it (S0 – S1).
- Such an approach assumes that problems are well-defined and an optimum solution can be arrived at by extensive upfront planning.
- It also assumes that the processes are predictable, and can be optimized and made repeatable
- It is also based on the assumption that processes can be adequately measured, sources of variations identified and controlled during the development life cycle.
- In summary, traditional software development life cycle is highly process-centric.
Based on the above understanding of systems development, organizations adopt a management style which is
- command-and-control, with a set hierarchy. Therefore these are predominantly mechanistic organizations geared for high-performance in a stable environment
- characterized by high formalization and standardization. People with different specializations are assigned roles for producing defined outcomes. In addition to this, they also produce significant amount of documentation which explains the software, its technical and design specifications.
- In addition customers play an important role but their participation is maximum only during the specification and the implementation stages
From the customers’ perspective, there are certain pros and cons of traditional approaches
- The definite advantage is its scalability, very large projects and mission-critical projects need a strong plan and close supervision and need provision for stability
- The shortcomings are based on the assumption that customer requirements are well understood at the beginning and don’t change much, which is rarely the case
- Another fundamental problem is processes take too long such that when end users see the system, many things including user requirements have changed drastically
To summarize, there are two broad themes that emerge in traditional systems development, around which the problems are centered.
- development life cycle and its processes – the way processes are understood and designed and managed – it is assumed they are well-defined, predictable, repeatable processes – ‘highly process-centric’. It uses ‘hard systems thinking’ (an engineering-based) in its approach. The focus is towards achieving stability.
- management style – the way people are organized, managed and controlled. Mechanistic organizations with ‘high command-and-control’, formalized and standardized, with limited customer interaction
Agile Methodologies
To address the challenges posed by the traditional methods, a set of light-weight methodologies called as ‘Agile’ were developed nearly 2 decades ago. Some of them are XP, SCRUM, feature-driven development, etc. Agile methodologies try to overcome the limitations of the dynamic nature of the systems development projects with various tactics.
Agile methods address the inherent problems of traditional systems development using two scientific concepts.
- One method is by using ‘empirical process control’. One can also say that this is based on ‘soft systems thinking’
- The other is seeing the systems development as ‘Complex Adaptive Systems’ and designing team structures and methods around that.
How agile addresses the shortcomings of traditional software development life cycle :
- Agile methods adopt the empirical process control model as against the ‘defined process control model’. Empirical process control is meant for processes which are not well-defined and are unpredictable and unrepeatable. It implements control through frequent inspection and adaptation. Since the software development processes are highly complex and variable in nature, empirical process control methods seem to be a best fit to deliver resultsAccording to Complexity Theory, a Complex Adaptive System (CAS) self-organizes and adapts to changes in the environment without any central rules governing their behavior. An agile development system can be likened to a CAS which respond to complex and unpredictable changes in the requirements
- The development model changes from linear life cycle model to an evolutionary-delivery model. This is characterized by short iterative cycles, with periodic reflections and adaptations and continuous integration of code into overall system under development.
- Since the cycles are short and iterative, they provide the necessary flexibility and speed to adapt to changes in requirements through constant feedback from the stakeholders
- Agile methods also need constant collaboration with customers needing their inputs and feedback at various checkpoints during each iterative cycle.
The organization design required is that of an organic management system :
- Organic design provides for the inherent instability in the systems development projects through less formalization, decentralization and less supervisionIn order to get the necessary flexibility and speed, the agile projects self-organize themselves into small teams, with a ‘servant leader’ who facilitates the team’s processes and goals.
- These teams are constituted by people of diverse set of skills who often take different responsibilities as the situation arises
- The teams directly interact with customers to set goals and plan and implement them.
- The entire team goes through repeated cycles of thought-action-reflection, with the primary goal being learning and adapting, enabling them to respond quickly to emergent situations.
- The team produces only the most necessary documentation that is needed to communicate between the different stakeholders
What are the main challenges facing organizations in migrating to an agile environment ?
The transition will call for changes in organizational structure, culture and management practices.
The challenges and difficulties of a shift in organizational and management practices include
- Moving from a mechanistic organization to an organic one, or an effective mix of bothThis shift means relinquishing a ‘command and control’ style to a collaborative style. The project manager becomes a ‘facilitator’ who guides and coaches the team members and coordinates their collaborative efforts. The major challenge here would be to make the manager relinquish the authority he/she enjoyed
- Organization culture affects the way individuals behave and work with each other. Culture has significant impact on organization processes such as decision-making, problem-solving, social relationships, communication, etc. Moving to agile practices would need a good understanding of the existing processes before changing them
- Since agile practices discourage extensive documentation, most of the knowledge remains in the heads of the team members. This creates a high dependency for the management on the individuals. This threat needs to be dealt with by finding ways to retain knowledge and people within the organization
- Reward systems need to change to promote team work and retaining people
The challenges in re-organizing teams would include :
- Agile practices are heavily dependent on collaborative team work. Teams have to be trained and coached to perform as self-organized teams, to learn and adapt with the environment.
- For individuals to work in different functions within the same team would require them to be competent enough. This would mean that the agile projects need to be staffed with above-average people. This can cause serious staffing and morale issues
- Customer collaboration being an important aspect, requires the team to build an atmosphere of mutual trust and open communication. It would also be difficult to get some customers participate actively in many of the processes
The challenges in moving away from ‘process-centric’ organization would include :
- Several organizations have invested heavily in CMM practices to strengthen the processes and ensure stability in their setup. Moving to agile requires them to re-look at these practices and doing away with some rigid processes that hinder teams to work in the new methods.
- Traditional life cycle processes are elaborately defined, measured and controlled to minimize variations. As against this, in agile, project management has to rely on assessment rather than measurement and has to keep the processes highly customizable
- The practice of writing code and then test cases is entrenched in traditional methods. This is totally reversed in agile practices where test cases are written before any code is written. This would redefine the function of quality assurance and control departments in the organization
Conclusion
The successful transition of organizations to agile methodologies depends on careful consideration of organizational, people and process aspects that need to be changed. Organizations should also consider a combination of both the traditional and agile methodologies based on the size, complexity, customer readiness and speed.
References
1. ‘Challenges of Migrating to Agile Methodologies’ by Sridhar Nerur, Radhakanta Mahapatra, George Mangalraj
2. ‘The New New Product Development Game’, Harvard Business Review, Jan-Feb 1986, Hirotaka Takeuchi and Ikujiro Nonaka
3. ‘Agile Project Management : Steering from the Edges’ by Sanjiv Augustine
