Agile Adoption - How to encourage a client or employer to adopt Agile

As we celebrate the 10 year anniversary of the Agile Manifesto, it is clear that Agile has become one of the most important concepts in Software Engineering in these last 10 years. It has been rapidly accepted, adopted and butchered in every possible way. As with anything that rapidly accepted, there have been a number (perhaps a majority) of the "adopters" who claim to be Agile or Agile-minded but are clearly mistaken in their understanding of Agile.

This is a link to an article by Martin Fowler and Jim Highsmith following their signing of the Agile Manifesto: http://www.drdobbs.com/184414755

As I have moved between jobs and also been in consulting at client sites, I have noticed the increased awareness of the Agile buzzword. As much awareness I see, I see the same amount of misinformation.

I have seen two categories:

• The old crusty PMI/mainframe guys who show intense mistrust of Agile. They view our enthusiastic support with the same amused attitude that we show to teenagers who love boy bands - "these kids and their stupid toys" they seem to be saying.
• The new generation developers/project managers/business analysts who are supportive but confused. To them, Agile is periodic delivery of software and a daily status meeting. Managers especially like this. They think - "Boy! I am going to get software delivered every 2 weeks and I can demand what's going on everyday! How cool!". Clearly, they fail to see their responsibility in ensuring that this happens.

I love this quote from a colleague regarding a client - "You can't convince them they are not Agile!".

So, in this blog, I will attempt to articulate the different stages of Agile adoption and the roles/attitudes of personnel I have encountered in my experience. I will use the following roles in my diagrams and explanations.

Agile Champion: The true Agile expert and champion who is attempting to bring about true change.

Analyst/Project Manager: The PMI school adherents who think they know Agile.

Developer: Junior to Mid-level developers who are amenable to adopt Agile but don't know how.

Senior/Experienced Developer: The crusty veteran/mainframe guys.

Senior Management: VP, Director, C-suite officers

Stage I: Pre-Agile

In this stage, the organization is cruising (or not!) along with waterfall and delivering projects with > 6 month timelines for even relatively small efforts. Analyst/Project Manager is aware of Agile having heard it in some training or the other. Developer is also familiar with Agile. Senior Management just wants things done now and without delays.

The diagram shows relative levels of support for Agile in this stage. The Agile Champion is the one with the most enthusiasm and the rest of the company is sort of indifferent to Agile.

Moving to the next stage, the Agile champion is faced with some tough questions. The choices are coerce, enforce, collaborate and to co-opt.

Does he/she approach management to enforce or Project Manager to coerce or Developer to collaborate and to co-opt?

The first instinct is to go to management and in a grand presentation, show-off Agile, bring in an external/recognized export and get them to OK the adoption of Agile. In my experience, this is not only useless but also a sure way to fail. The moment, Senior Management thinks something is going to on to improve delivery, they want it now and are not willing to play their part because they are too busy.

Coercion will not help either because this will quite often incubate a mild to major rebellion among Developers and Senior Developers just because they are coerced and there will also be perceived performance pressures.

So, the best choice is to collaborate and co-opt.

Stage II: Early Adoption

So the Agile champion approaches the Developers first and begins to show them the benefits of Agile thinking. This involves in my opinion Test Driven Development, just enough design and continuous integration. Trying to proselytize without concrete deliverables will only confuse the concept further. So, the collaborate and co-opt phase involves a lot of doing and showing while encouraging learning and practice. Developer ( and usually not the Senior Developer) is willing to try and learn. In most cases, the tools and software are on the Agile champion's personal computer.

As the Agile champion demonstrates TDD and benefits of user scenarios, the Developer slowly begins to voluntarily perform some of the tasks and experience the excitement and joy of having tests to cover their regression testing. Refactoring exercises naturally enhance their just enough design skills and they get more comfortable with talking to the customer or product owner (Analyst).

As you can see, the diagram shows the Early Adoption stage.

What next?

Stage III: Growing Influence

In many cases, if the previous stage was reasonably successful, the Agile adoption has typically gathered mass and moving downhill. This is also the most critical stage in this model. Having established the tools and techniques, it is time to begin driving home some of the concepts behind the actions so far. In most cases, the tools and software are on the team's development server and resources are being shared.

If the Developer is doing his/her job, then he/she is speaking up in requirements meetings asking about developing user scenarios and quick milestones. The Analyst/Project Manager sees that the developers are more into asking the right questions and are forced to find answers and they begin to experience the joy of always talking to the customer and getting feedback and reducing surprises when software is delivered.

So, you can see the growing influence of Agile thinking among Developer and Analyst/Project Manager roles. The customer loves being involved with the developer in refining and validating requirements often. This is the stage when the Agile champion has to formalize some sort of education/training and encourage discussion and self-learning among these personnel. It is important to answer: Why TDD?, Why just enough design? Why refactoring? What is Technical Debt and when to repay it?

This is reflected in the Growing Influence stage diagram. The Agile champion duties are now spread among Developer, Agile Champion and Analyst/Project Manager. It is critical to show positive results in this phase.

More often than not, by the end of this phase the Senior Developer is participating in the Agile adoption out of little choice in the matter. This role has to be managed carefully as they often have the ear of Senior Management and can cause adverse impact to Agile adoption.

One note of caution: At this stage, since Agile is not your official model and many companies are bound by scheduled releases, this may not be true Agile continuous delivery. But, if you have managed to deliver the final product on the scheduled release date but with high quality and high customer confidence of the final product, you have achieved Agile in my opinion. We all cannot be nor indeed need to be Facebook or Twitter or Google and deliver features to production every two weeks. THe goal of Agile is not to achieve this model. Rather, it is to achieve high software quality while delivering as close to customer expectations.

Stage IV: Active Practice

In this stage, Senior Management has become aware of the growing influence of Agile and have seen results. This is the the time to break out the power points and bring in the external expert if necessary to present to Senior Management that Agile is indeed viable and demonstrate your successes so far. If Senior Management sees most of the team willing and active participants, they rarely have incentive to shut it down.

As I said earlier, they usually care about delivering value to the organization's customers as expected.

The Agile Champion's role is reduced to be more of an evangelist in the company and interfacing with Senior Management to promote official adoption.

Stage V: Evangelical

This stage is achieved when the company officially recognizes Agile processes to software development and delivery.Tools and Technologies revolve around augmenting Agile and there is support to buy requisite tools. Potential new hires are evaluated during interviews according to their experience with Agile or willingness to be in that environment. All stakeholders speak and think in terms of Agile processes and are willing to put this at the top of the priority while thinking about delivering value to their customers.

The Agile Champion's role is greatly reduced to be more of an evangelist in the company and externally as the case may be. This role merges into Senior Technical Leadership role either as Senior Developer or Architect depending upon the company's policies.

What's on TV?

I don't watch TV. That's not to say, I don't watch TV. Rather, I don't watch TV in the sense of following a show, following a schedule or remembering show times (unless you count Kids' cartoons on PBS, which I watch with my kids!). I watch my share of temporal sports events like Dallas Cowboys football, College Football and Basketball but beyond that, what ever is on for a few minutes between 6 and 7 in the evenings as we sit for a while after dinner. A big reason also, I must admit, are the kids. They keep me busy and I don't want to watch any crap when they are around the living area anyway.

I don't get TV shows. I am not sure if I am too cynical or the shows are that bad. A lot of people I like and admire watch shows such as "Jersey Shore", "Survivor" and other dramas too. I guess they watch it with a dose of inbuilt reality and detachment that they don't get carried away. They enjoy those shows while watching it and can talk about it later but still be grounded. Me, I watch it for a few minutes and go, "Man! That's a load of crap!" and turn it off.

I guess I should develop a sense of detachment and healthy appreciation for stuff around me and yet be not caught up in it.

What is CQRS? My experience with implementing it. Part 1

A short post about CQRS and my experience with it.

One of the more interesting programming/design patterns as I was learning Java way back was the command pattern. Sure, this pattern has been around a long time and has been well defined and discussed. But the first time I implemented it was through Java.

The pattern itself is conceptually easy. There is a command handler that handles any command submitted to it. The command typically implements an interface that the handler executes. With this flexibility, one can introduce pre- and post- execution methods, execute-around method similar to an Aspect etc.

While this pattern was primarily used as one component of the application's architecture, it was almost never the basis of the architecture of any project I worked on until recently. I had the opportunity to be able to implement a CQRS style framework for a greenfield project.

What is CQRS?

Stands for Command Query Responsibility Separation. The concept as the name (sort of) explains is to separate out concerns where a system is 'comman'ed to do something versus a system is queried for something. Wait! Isn't that pretty much obvious and a given in any system. When I command system to save some data, I use a dufferent syntax and when I query it, I use a different syntax. Well, not so fast. This is not about syntax. Rather it is about the semantics of how one saves and queries, the architecture that supports these operations and how they should be separated for better performance and maintainability.

So, let's look at a typical architecture of a web application. We have the three tiers (perhaps more but let's keep it simple here).

The UI/Client layer interacts with the user and asks the business layer to respond to the user's actions - to save data, to show data, to perform a calculation and show results, to collect data from separate sources and show as a dashboard/portal etc.

The business layer accepts request, acts upon it by either persisting, querying, calculating and returns result back to UI/Client.

A few issues I have with this approach:

1. Traditionally, the business layer providers interface for each of these actions. For instance, there is usually a handler (action/controller etc) for Create, one for Update, one for Query and so on. This makes the architecture open to modifications for any new request in an unnecessary way.

2. This also makes the UI very CRUD based. If one wants to update say a user profile to change address, one opens the entire profile screen with all editable fields and updates the addressline and submits. Lost in this update (unless there is a lot of scripting or code involved) is the fact that the user only wanted to update address as opposed to the entire record.

3. Of course, some architectures combine this into one action that performs the appropriate task based on a parameter in the incoming request. But this is not preferred since the one handler can get large and messy. Needless to say, a simple requirement change makes it susceptible to dangerous side effects.

4. An architecture such as this typically serves one client - a UI, or a web service and leads to code duplication.

5. The ORM becomes the source of persistence as well as query and anyone who has dealt with complex database structures and/or large datasets to be queries will know that the ORM typically can be optimized to persist or read but not both especially if the reads bring back large datasets coupled with complex object maps.

How does CQRS help overcome these issues?

• CQRS separates the read and write concerns while opening a whole new architectural avenue to implement apps. Basically, every operation becomes a command and a command handler will inspect the command to determine the action to take.
• This makes the command and command handler a standard interface of the application to the entire client base. UI, Web Services, JMS Messages etc can all now interact with the business layer in a standard interface.
• The command can implement business logic and hence this will be reused and standardized across the application. Required fields, sizes, formats etc can be validated.
• Command handler can do pre- /post- processing to send notifications, audits, initiate events based on data updates, deletes etc.
• CQRS allows creating task-based UI where a specific task can be targeted. In the earlier example of a user updating address, if the UI allowed a user to change just one field, and this operation becomes a command called update zip code, then the command handler knows exactly what changes, audit trail is granular as well. Any business logic that depends on zip code can be implemented cleanly without a gazillion lines of code to determine whether business logic changed and then take action.
• In this case, the command carries all that is needed to be done.and that command is audited. If partner system need to know that zip code changed, an event/message can be sent to interested parties to listen on it.

An extension of this pattern is to complement this with event sourcing. A command can publish an event out to a queue/topic as it updates the OLTP database. This can be a distributed transaction to write to a queue and database. A queue listener can then push this out to the Datamart and/or other interested applications in the enterprise.

One can even implement the command itself as an event that is consumed by the handler and perhaps updates the OLTP database and the datamart.