Adrian Mowat on Agile Data Integration

Test Driven Development - Building a Magic Funnel

2010-10-28T07:08:00.002+01:00

An automated data integration test suite is like a funnel that scoops up abstract, unstructured and downright weird knowledge from every corner of your problem domain, condenses it onto a concise form, and pipes it out to consumers (i.e. development teams) who turn it into a working system.

It fulfils three important roles that are otherwise missing.

* To define a complete and unambiguous specification held in a form that is directly useful to technical people (n.b. a spec is not directly useful because it is open to interpretation and assumes tacit knowledge about the problem)
* To create single, well understood, vocabulary for discussing requirements and issues in development
* To give ourselves fast, simple way to prove functional correctness

So how do you build the funnel?

Let's assume you a basic technical framework in place that allows you to store test cases and run them through the system under test. The question we need to answer is how to integrate it into your project lifecycle in a cost effective manner. The only way to do that is to use Test Driven Development (TDD). Put very simply, write a failing test that describes a change before you write the code to implement that change.

There is plenty of information out there on how to do TDD in Object Oriented systems. I have worked through a lot of it and built systems of my own to learn the techniques (and do other useful things as well). I found that Object Oriented languages allow much more modular code than ETL systems and give you much more control over the low-level details. Many of the techniques do not map perfectly into the Data Integration world. It's certainly impossible, or at least highly impractical, to start from a single test case and build up a system from there. For a start, a large part of the system already exists in the form of source and target systems and interfaces.

However, I have found that the principles and benefits of doing TDD are relevant and achievable to Data Integration when you tweak the practices a little. Furthermore, data integration projects raise challenges around data quality and opaque interfaces to external systems that are not a concern on most OO systems. I have found that building the funnel using TDD provides an elegant and proactive solution to these problems that integrates well with normal project activities and does not mandate an explicit 'phase','activity' or 'gate'.

Here's how to do it.

Before you write any code...

1. Create an automated test harness.
2. Work with technical team, business users and analysts to define an initial set of test cases based on the current understanding of the business problem. Make sure you can run all your test cases in one pass and you don't need to run batches of test cases through the system under test.
3. Roll technical test cases such as boundary conditions and known data quality issues into the test suite.
- Do your initial data profiling here

At this point, you will find that you have driven out a lot of problems in your understanding of the problem and the solution required by simply thinking through the problem at a detailed level. Some of these will be misunderstandings between the business and technical people, while others will be areas the business users and analysts had not considered in their initial discussions. The test framework has already been valuable and you don't even have any code yet!

You are now ready to start coding and you can move the test suite into "TDD" mode.

For every modification...

4. Check you have test cases for the feature. If you don't, write them.
5. Run the suite to ensure the test fails (this is how you "test the test")
6. Write code to make the test pass - running the tests frequently to make sure you are on the right track
7. Once all the tests pass, refactor your code to make it production ready
8. Go to step (4) and start the next feature

You must expect issues to arise once you start development. Some will be technical, others business-related and the rest due to tests that were missed in the first pass. It is very important to assign collective ownership of the test suite to the development team so everyone has the right and ability to make changes as soon as they find a problem. In practical terms, you should have a version control strategy to ensure changes are propagated around the team in a controlled fashion, but you should avoid assigning ownership or setting up any sort of approval mechanism that will create a bottleneck.

Periodically, you should run a smoke test with a full volume of production or production-like data through the system to drive out issues you have missed. Similarly, if you have a lot of dependencies on other systems you should build your test suite to stub out those dependencies but run regularly in a fully integrated environment to find new cases that should be added to your automated suite.

Footnote: Why you really can't create the tests after the code

Apart from the fact you have already lost the benefits of the funnel during development, it's actually impossible to create a complete test suite after the event.

Ask yourself a few of questions...

* How can you validate the test suite tests what you think it does unless you have made the test fail?
* How can you be sure the test suite tests everything the code base does unless it grew alongside the code base?
* How can you inject test cases into a system that was never built to have test cases inserted into it? Usually you will find a few tricky dependencies that you can't work around.

I'm not saying you should not try to build a test suite if you have to manage a codebase that does not have tests, but you should understand the limitations. You should certainly not plan to build the tests after the code if you have the option.

Fundamentals of data testing: Setup and Teardown Patterns

2010-09-17T12:36:00.001+01:00

A good automated test suite can setup a new environment, run the tests while retaining enough information to debug failures, and then teardown, or reset, the environment ready for next time the test suite runs. Data integration systems, by definition, deal with persistent data stores so the test suite needs to reflect that and deal with the inherent complications.

>> Edit: As Nick pointed out in the comments, it's useful to defer teardown and leave the test data intact after the test runs so you can debug any failures. I usually work on the cycle of teardown -> setup -> test

This article describes 3 patterns I have found useful for setup and teardown.

Truncate and Rebuild
Build from Copy
Test Data Set

They are not mutually exclusive and can be mixed and matched as needed so, for example, you might use a truncate and rebuild strategy in the filesystem and a test keys approach in the database.

The Test Environment

The test environment consists of mixture of code, persistent storage, applications and libraries and tools including, but not limited to...

The system under test and libraries on which it depends
The test framework
Source code repositories
Example data sets and expected results
Source, target and intermediate databases and tables
Source, target and intermediate files and directories
Business applications
ETL environments/tools
Data transport software - e.g. MQSeries and FTP
Scheduling software

The test suite must include enough of these elements to repeatedly verify the data-transformation logic and supporting infrastructure on-demand, at the touch of a button. You should keep it as simple as possible by only including only the elements needed to perform the test. For example, don’t bother running your tests within the scheduler unless the schedule is complex enough to justify the extra effort.

It is important to build a test suite that can run at any time without impacting other teams’ work. Unit test suites must be under the direct control of the developer(s) working on the code so they can build test cases and perform the TDD cycle for every change they make to the code base. “Higher” test environments might run on a scheduled basis (every hour, or after each commit, for example), but they still should be made as accessible as possible.

Real-world constraints

Unfortunately, it is not usually cost-effective to have dedicated access to all the elements needed (especially when first introducing these practices) so you will probably need to be a little creative about how you live alongside other users of the existing development and test systems at your disposal.

The following patterns can be used to create a test environment that coexists happily with other teams.

The “Truncate and Rebuild” Pattern

Overview:

A brute force approach where the test suite assumes it has free reign to delete existing artifacts. It deletes everything it finds, builds everything it needs from scratch and deletes it all afterwards.

Mechanics:

Delete all existing artifacts and re-create them from source as you would when deploying to a new environment.

Advantages:

Very good at driving out deployment problems (missing files, bad checkins etc)

The environment is guaranteed to be clean
You are unlikely to impact anyone else’s work
Simplifies the test framework because it does not need to worry about other users of a given resource.

Disadvantages:

Often not cost effective - especially if applied to elements with expensive licences
Can mask real world problems because real world systems need to coexist with other applications so a totally isolated test suite may miss defects caused by other system’s activities
Must be used with care to avoid accidentally deleting someone else's work or artefacts that cannot be easily recovered

The “Build from Copy” Pattern

Overview:

A close cousin of Truncate and Rebuild. Instead of building an element from scratch, it is created by copying from a master-location.

Mechanics:

Delete all existing artifacts and re-create them by copying from a master location.

Advantages:

As per Truncate and Rebuild
Easy to visualise and manage because the master copy of the artifact is available

Disadvantages:

As per Truncate and Rebuild

The “Test Data Set” Pattern

Overview:

The test framework is built to work alongside other users’ data by managing a data set that is easily distinguishable from, and does not overlap with, any other data in the system. Thus it can be inserted, updated and deleted on demand without any undesirable side-effects.

Mechanics:

Records are ideally identified by primary key, or some compound combination of secondary keys. If this is not possible, a value in a comment field can be used - but this is less than optimal because it means you need to change the system under test to allow the test framework to inject it’s own comment value(s).

Keys must be chosen very carefully to ensure you can always identify them. A useful technique is to pick values that fall way outside the normal range. For instance, if a product code is an 8 digit integer in production, you can use 4 digit product code for testing.

Advantages:

Cost effective because it uses existing resources.
Aids data discovery because the team need to think hard about the data to build the framework

Disadvantages:

Adds complexity to the test framework
Risk that teardown may miss some data and pollute the next test run

A Worked Example

Consider an ETL process that reads a csv file pushed from a source system using FTP into a location on the ETL server and loads it into a data warehouse star schema. The warehouse is an established system that has been live for a number of years. It runs on a relational database and the new import job is being developed as part of the normal release cycle alongside a handful of other, distinct, projects that share the same physical production, development and test environments.

Our new import populates a pre-existing fact table and adds 2 new dimensions. We want to build a test framework for the import process that continues to be useful after the current project has finished so we must assume other users are going to hit the same tables at some point, even if it’s not an immediate concern.

Starting with the source data, the test framework holds a set of input cases and corresponding result(s) for each case (how we store and manage them is beyond the scope of this article). Let’s assume out ETL environment provides a mechanism for a tester to have a dedicated area of a filesystem to store test cases. We can use Build from Copy to recreate source_data.csv by taking a copy of the test framework’s input cases each time the test suite runs.

Next, the ETL process under test is stored under version control. Assuming we have a dedicated area where we can create working copies, we can simply delete all the code and re-create it from the version control system each time the test runs. This is an example of Truncate and Rebuild.

Notice there is a subtle difference between Truncate and Rebuild and Build from Copy. In the first case, we build the test environment programmatically as we would in a real deployment, while when employing Build from Copy the build process is synthetic and only found in the test framework.

Finally, the database that holds the warehouse is probably the most complicated part of the system because it is shared by all the project teams in the data warehousing group. We need to solve 2 problems: and how to manage the test data, and how to test the DDL that adds columns to the Fact Table and creates the new Dimensions.

First, the test data. We use the Test Data Set pattern to structure our input data set in such a way that it can be identified easily once it has been loaded to the data warehouse. Thus we can can also delete it easily when we teardown the test environment.

The DDL is a bit more tricky. The dimensions can be simply deleted and re-created using Truncate and Rebuild. We can also use Truncate and Rebuild on the the fact table but the teardown script needs to be sophisticated enough to drop only the new columns. In both cases, the DDL tests should be optional because future users of the test framework will be testing additions to your business logic in a future release so they can reasonably assume the star schema already exists.

Fast Database Resets

2010-08-23T10:01:00.001+01:00

The GOOS (Growing Object Oriented Systems guided by Tests) mailing list today included a discussion on a method of quickly resetting a database using stored procs.

It's Java focussed, but the ideas are useful on all projects doing automated database testing.

New name for the blog

2010-08-13T15:56:00.001+01:00

Gradually, I have started thinking about what I do as Data Integration rather than Data Management so I have changed the title of my blog accordingly.

Fundamentals of data testing: Characteristics of a test suite

2010-08-13T15:48:00.004+01:00

An automated test suite should be a living thing that is easy to update and easy to run. The test suite only adds value when it is run to find errors so it should be built to run many times in a day. This is just normal Agile thinking.

People who are not used to Agile sometimes think about creating an automated test suite that is open to a privileged few and used only at the end of the project when testing is traditionally performed. At best, this leads to a situation where the suite cannot not deliver maximum return on investment because it has limited chances to catch defects. At worst, the test suite will simply become an overhead that quickly falls out of use.

The most successful automated test solutions on data integration projects I have been involved with have been those that business users, analysts, developers, testers and other relevant parties all provide input to throughout the project. This means unexpected problems late in the project are easy to solve because everyone trusts the test suite covers all possibilities. Just this reason alone is adequate return on the investment needed to setup the test suite.

The tests are run by the developers after every significant change and it’s not unusual for the test suite to be run 10 to 20 times in a day (as an aside unit test suites in object development are run much more frequently, but data integration test suites are limited in speed because they deal with files and databases and it’s often hard to isolate and run the lowest level code the developer writes).

Here, at a high level, is what works for me.

You need 3 things:

An environment you can teardown and rebuild repeatedly many times a day
A way of storing input data sets and expected results
A way of running the input data sets through the system and verifying the output matches the expected results

I usually just build this in the development environment where we have most control over what we can setup and delete and who is able to use it.

The nature of the test suite I suggest is difficult to categorise. It’s like a unit test because it’s designed to be run by the developers and uses a small volume of data. But it’s also like a system and/or user acceptance test because it often runs a sequence of related jobs and should include all the cases we expect to find in the production environment. I suppose “unit test” is the most useful description for practical purposes.

One thing it definitely is not is an integration suite or a basis for application testing. A common mistake is to try and test the flow of the data through a number of systems. It can work under specific conditions, but usually it’s working at just too high a level for your tests to run quickly and you are at the mercy of changes in the internals, workflows and input/output interfaces of the applications and distracts you from the real goal of building good data transformation code.

There is an obvious dilemma, though; what happens if a change to an application has an impact on the data transformations?

Well, it’s OK to run the application interfaces every so often. It’s just that you should try to avoid automating the testing through them. In fact, I strongly recommend you make sure you run the whole application end to end regularly so you can get real feedback on how the system in behaving. It can be manual and doesn’t need to be heavyweight - just enough to spot problems quickly and get enough detail to update the automated tests.

This is actually a similar problem to making sure your tests cover edge cases in the data, but I’ll cover that in a future post.

Fundamentals of data testing: Deterministic vs Heuristic

2010-08-13T11:39:00.019+01:00

The style of test automation I find most useful when testing data integration systems is largely based on the “deterministic” testing style popularised by Agile methods. That means that for every value input, we need to be able to define an expected output from the system under test. This is appropriate for testing most situations that arise in data integration solutions but we find that there are occasional cases we will need to borrow from techniques in systems based on heuristics.

For example, consider the following mapping rules:

Source Column	Rule	Target Column
	System date formatted as “YYYYMMDD HH24:MI:SS”	creation_date
flight_number	==>	flight_number
destination	Convert to Sentence Case	dest_name

The rules for populating flight_number and destination are deterministic. Flight number is straightforward because the value from the source is simply copied to the target. The rules for destination are slightly more involved but we easily construct test cases that define an expected result based on a given input - EDINBURGH -> Edinburgh, gLasgow -> Glasgow... and so on.

The "creation_date" rule is quite different. Timestamps taken from the system clock are non-deterministic because it’s impossible to define a value that will definitely match the system time at the moment the job is run. In some cases, you can get round this by injecting a known value into the system under test using a parameter (or some similar method) but it’s not always possible so we need to take a leaf out of heuristic systems thinking.

Heuristic systems use a combination of rules and data gathered at runtime to make decisions. A scheduling system used by an airline plots a schedule for the airline each day that takes into account the many factors that affect airline scheduling - plane location, crew availability, turnaround time, weather conditions, volcanic activity in iceland, and so forth. There is no “right” answer to the question of how to arrange the planes so it’s hard to test deterministically. Instead, the development team need to ensure that the correct decision making flow is executed when deciding the schedule rather than trying to predict an outcome. Emily and Geoff Bache work on such as system and gave a presentation at Agile2008 explaining how their test suite works.

Back to our system datetime. We can borrow from heuristic concepts by plugging in a set of rules that allow us to validate the date generated. A reasonable heuristic rule for the system date might be to check the date is correctly formatted and within a few minutes of the moment the test suite was run.

Take care though. Heuristic tests make your test suite more complicated because you need capture and process more data in a heuristic style test than a deterministic test. Sometimes it may just not be worth the effort. Furthermore, it’s easy to write heuristic tests that are just as complex than the system being tested. Let’s face it, the risk of getting the system date call wrong is pretty remote so maybe it’s just easier to ignore those fields and focus building watertight tests for rules you can test deterministically. That’s usually where the real business benefits are found anyway.

Above all, you should always try to find a deterministic solution and you should expect to do so at least 95% of the time.

Agile Data Testing

2010-08-13T11:22:00.002+01:00

Those of you who avidly follow my blog :-) will have noticed that there have been one or two quiet spells lately. I would love to say I have been living as a hermit living on seaweed and shellfish scraped from the rocks while building the perfect suite of tools for Agile Data Integration but, really, life just got in the way of writing the past couple of years.

However, I have not been idle. I've been working hard to define strategies for creating automated test solutions for ETL/Data Integration projects for the past 4 years and I have a lot of knowledge I want to share. Some of this has been captured in tools, but not as much as I would like so I am also looking for people to help me develop the tool capability in a warm, fuzzy, open-sourcy kind of way.

I'm about to publish some articles explaining what I have learned and where I am going. I know time is valuable, but I really, really want your feedback so we can start to develop more of a community around this topic. I'm going to start with the theory and get into the tooling once I have laid down my thoughts on the core concepts of what we are trying to achieve.

So please, please, please read the articles, be forthright in your comments and encourage me to write more.

My thanks.

Adrian

Big Data/IT blog

2009-10-12T12:38:00.001+01:00

Check out this new blog I found written by a member of the Big Data community...

http://toastingit.blogspot.com/

Agile Datawarehousing Book

2008-12-20T11:06:00.003Z

I just received my copy of Agile Data Warehousing by Ralph Hughes from Amazon.

At last, someone has written this book! I hope it is good.

One of the problems we face moving the world of agile beyond objects is that and managers in conservative companies need confidence that a different approach is tried and tested. This book has the potential to help do just that.

Since it's Christmas holiday season, I hope to have some time to read the book over the next couple of weeks and publish a review on this blog.

Naked Agilists Conference Preview

2008-07-28T08:42:00.002+01:00

The new Naked Agilists podcast is now available at http://www.nakedagilists.com/

It's been a little while since we have released anything new, but with the Agile year's main event approaching fast we thought it would be a good idea to get some of the Agile2008 conference organisers to tell us about what they have planned. First up we have Grigori Melnik who gives us a 15 minute overview of the conference as as a whole as well as an insight into the selection process introduced this year. This year's conference is organised into stages - rather like a music festival - so we then we have 5 individual, shorter, talks from the organisers of some of the stages who tell us in much more detail what their stages have to offer.

It's much more than throwing out the documents

2008-04-27T13:22:00.021+01:00

I was talking to a fellow Ab Initio developer the other day and he asked me a common question...

“Isn't all this Agile stuff just a case of not writing documentation?”

It's true that Agile teams do less documentation, but it is always replaced with better alternatives. I'll put that another way because it is important. Agilists do not throw away documents for the sake of it - that would be a pointless short term fix - but we are passionate about reducing waste in our projects and we have found there are more effective ways of doing some things that every project needs and they happen not to need as much documentation. (Mary and Tom Poppendieck showed us how to use Lean principles to identify and eliminate waste in software projects, this interview is a great place to start.)

This can be a little difficult to imagine if you are used to waterfall projects, so let's look at a couple of typical activities on data projects and the practices we can use to do them more effectively and reduce the documentation overhead.

Test Procedures

Waterfall teams write test procedures to record the manual tests that need to run against each release. There are problems with this approach: the tests take a long time to run manually, they are subject to human error, and because natural language is imprecise and ambiguous technical details gets lost in translation by the tester.

Agile teams use Test Driven Development and executable specifications to create automated test environments that can be run quickly and easily by anyone on the project (including stakeholders). Automated tests are absolutely precise – they either pass or they fail, and if a bug occurs you can simply add a new test and that error can never happen again.

Requirements and design

In Waterfall, business analysts use requirements documents to capture what the customer needs, then the designers create another document that interprets the requirements in technical language. Again, there are two problems with this. It takes a long time, so requirements go out of date and because natural language is subject to human interpretation the customer's original need gets lost in translation.

Agile teams are cross functional including business analysts, designers, developers and testers all of whom engage directly with an empowered representative from the business. They work in short iterations of a month or less that deliver solutions to the most pressing business need at that time. There's no need to maintain detailed requirements definitions or design specs because the customer is able to communicate what is needed face to face and the tests accurately describe what the code does.

Requirements traceability comes for free because Agile teams never let requirements and code diverge. Put another way, the V model, never opens out into a V.

Change Control and Risk Logs

Honestly, does anyone enjoy working through these?

Requirements and designs go out of date on waterfall projects, so they need a whole extra level of documentation and bureaucracy to track changes and uncertainty.

Agile teams simply do not need any of this stuff because they are always working on immediate priorities and the chunks of work are small enough that they can actively address risks early in each iteration.

Look for the Principles

Of course, I have really just scratched the surface, but if we look for the principles behind the practices we find ways to improve our thought processes.

Documentation is a legitimate form of knowledge capture, but it's not always the best option. There are times when documents are necessary and useful, recording business metadata is an example that springs to mind, but next time someone asks you or your team to write a document, recognise it's a default position – the lowest common denominator of project communication – and everyone involved might find a better way of working if they were to kick the problem around for a while to address the root problem more effectively. Sometimes you will find a document is the best solution and other times you will not. In either case, you will reach a well considered decision based on a reality.

Something In Nottigham

2008-04-26T18:37:00.002+01:00

The Nottingham Agile group has a new page at http://somethinginnottingham.blogspot.com. You can use it to find out about upcoming meetings and link to the group's wiki from there.

Agile at Dundee University

2008-02-28T17:16:00.003Z

A few months ago, I was fortunate enough to bump into Janet Hughes who had the dubious pleasure of teaching me Software Engineering back when I was a funny looking undergraduate at The University of Dundee. We exchanged a few email and, on Monday, I had the great pleasure of giving a talk on Agile Data Warehousing to Janet's 3rd year students and then to the local BCS group.

There have been massive changes since I was last there. Not least the fact that the Department of Applied Computing, as it was, has moved to a fantastic new building and become the School of Computing. The range of courses has expanded to include degrees in E-commerce Computing, Computing with Electronics and Computing with Interactive Media Design, along with the Applied Computing degree in which I graduated in 2000.

The software engineering course has changed radically as well. Janet clearly has a great feel for Agile and a passion for teaching it. She told me how she discovered Agile through her own reading, was immediately taken by the way it is tune with the way that we really develop software - as opposed to waterfall; which isn't. After attending a Scrum Master course to learn more, she has added a very strong Agile focus to the Software Engineering course I took all those years ago. While I was there, her students got their marks back for a group project assignment. Working with a real customer to write a scuba dive planning system for a mobile phone in C#, they used Scrum to manage their projects, co-located their teams and developed the code using pair programming and test driven development using nUnit. Part of the final submission was a report from a Continuous Integration system one of the 4th years was writing as a final-year honours project.

I am very impressed! It is great to see a university department teaching Agile so successfully and I think it is encouraging for the development of Agile that students are starting to come out of universities equipped to work effectively on Agile teams and influencing their future colleagues and managers to adopt better practices.

Both talks went well. I was impressed by the students' knowledge of Agile practices when I ran a short discussion at the start of the talk in which I asked them to suggest practices that support each statement on the Agile Manifesto. For example, one group identified working with a product owner as an example of a practice that values individuals and interactions over processes and tools. One of the students asked a great question about how to organise multiple teams working together on the same code base.

The second talk to the BCS was quite a different audience. I was pleased that there was a good turnout for my talk – especially since the talk was at 5.30 on a Monday evening! Many of the attendees professed no Agile experience at the start, but they asked lots of good questions showed a particular interest in Test Driven Development and if/how to integrate Agile with the wider process frameworks like Prince2 you tend to find in large organisations.

I wanted to say thanks to everyone I met for a fantastic day and for looking after me so well. Hopefully I will get the chance to come back again soon!

Book Reviews Kick Off

2008-02-23T15:48:00.005Z

My Agile/Technical/Consulting bookshelf is coming along nicely.

I have always been a bit of a book hoarder, and being introduced to Agile was more than enough encouragement! Christmas 2007 was especially fruitful since I pointed everyone in my family at my Amazon wish list and I got a nice little haul.

There are also a couple more titles that didn't make it in time for Christmas...

Patterns Of Enterprise Application Architecture - Martin Fowler
Domain Driven Design - Eric Evans
The New Turing Omnibus (66 excursions in Computer Science) - A.K. Dewdney

In fact, it was my birthday last month and I took the unprecedented step of asking people not to buy me any more books. I reckon I can make time to read about one a month so I'll be doing well if I finish this lot before next Christmas!

Reviews

As I pointed out in my previous post, most Agile technical literature is focused on software development using Object Oriented languages and I think it's important for the development of Agile that we publish research into how to apply this vast resource of knowledge and experience to Data Management.

Now I have an influx of new books, I think now is a good time to start publishing reviews. I expect the format and structure of the reviews the evolve as I learn what works best but I think it will be helpful to the reader if I were to give each book a headline score that is easy to digest...

Overall Score - how much did I get out of the book. Ranges from 1 to 5
Relevance to Data Management - how much of this book is relevant to people on data management projects. Ranges from 1 to 5.
Technical vs Management - 1 pure management, 2 bit of both, 3 pure technical

Let's see how we get on with that. I'm off to review a book now...

Naked Agilist Podcast

2008-01-22T18:36:00.000Z

The Naked Agilist conference podcast is now available at http://www.nakedagilists.com/jan-08 complete with my presentation on TDD on Data Management projects.

While I am on the subject, I was very pleased that Brian Marick took the time to review my previous post which was in response to his talk and advertise it on his blog. Thanks Brian!

wevouchfor.org - A Community Approach To Certification

2008-01-22T17:37:00.000Z

Certification is a controversial issue in the Agile community at the moment. Some people think its a great idea that is necessary for Agile to grow, while others argue it makes it harder to separate truly talented contributors from the rest of the pack.

Laurent Bossavit presented an alternative (complementary?) approach on January's Naked Agilists mini-conference which he and Brian Marick have cooked up with a little help from their friends.

It's a website called wevouchfor.org which borrows a few ideas from social networking sites to allow software people to publicly "certify" each other based on hard evidence of a job well done. For example, Clarke Ching certified Laurent as follows..

Clarke Ching certifies that Laurent Bossavit is qualified as a master, capable of innovating in the skill agile adoption, based on this evidence:

I spoke with Laurent on the nakeagilists podcast and I was very impressed with his thinking, intentions and attitude: putting his spare time into producing this website wevouchfor.org (along with Brian)which benefits our community. The website speaks for itself and for both Laruent and Brian's skills and motivations. Well done!

I think this is a wonderful approach, and very well executed too. One of Agile's strengths is its strong focus on team decision making as the only way to get reliably solid decisions on matters which affect the team's ability to deliver - your boss can only act on what he or she can see, but you can never pull the wool over your peers' eyes! This site is typical of that ethos and if it is widely supported it has the potential provide a very reliable indicator of an individual's true credentials. This would be as good for the individual as it would for his or her colleagues or potential employers.

The first iteration is now live and Laurent promises more features are in the pipeline. The registration and certification process is extremely quick and easy so I hope you can find a moment to sign up and give some credit to people who have helped you in the past.

Sentient Life Found Beyond Objects

2008-01-20T11:29:00.000Z

Brian Marick made a bold statement in his presentation on last night's Naked Agilists Skype conference. The podcast will be available soon but, put simply, his point was this.

With reference to Moore's technology adoption life cycle, Agile methodologies have crossed the chasm between visionaries and pragmatists only to find that, in his opinion, there are not many true pragmatists out there and the only option is to skip straight to the conservatives. He feels that the original visionaries are now spread very thinly across the world consulting with Conservative companies and the development of Agile would be better served if they were to retreat back to Visionary companies and work together on real projects creating new innovations.

Well done to Brian for sticking his neck out and making such a bold statement. As one of the original signatories on the Agile Manifesto and a consistent contributor and leader in the community, he is well qualified to comment on how Agile is being applied across the industry. Clearly, he has seen some things that have disturbed him greatly and has been moved to comment this strongly.

He is also the first to admit (listen to the podcast) that he would love to discuss this much further so I would like to offer an opinion on a factor he might not have considered...

The Agile community is too heavily skewed towards Object Oriented languages and people believe it only works with Object Oriented languages.

Back in the day, the patterns community, who were the thought leaders in the objects community, met up with the leaders in the methodology community and created what we now know as Agile (OK, so I skipped a few steps along the way, but please run with it). Naturally, the technical practices on OO projects continued to evolve and the people involved wrote the definitive works on refactoring, TDD and Agile design.

Let's be absolutely clear on something. I don't want to, even slightly, cast any doubt on the excellence and relevance of these books and I am certainly not trying to belittle the efforts or success of the people who wrote or have read them. But there is a problem. No one is writing technical books about anything other then OO technologies.

So why is this important?

Because Big Companies Are Conservative

The converse is not true (there are small conservative companies too), but in order for Agile to become truly mainstream it needs to establish itself in IT departments at multinationals like telcos, banks, media companies, major retailers and energy suppliers (the list is endless!)

Big companies like these employ legions of programmers, designers, architects, testers, managers and support staff who don't know about objects, don't do UML and think Ruby is a sort of red diamond. It's not a problem though. They don't need to know about objects because they have built their careers with languages and environments like Ab Initio, COBOL, Siebel, SAP, SAS, PeopleSoft, Business Objects, Brio and Teradata to name but a very few.

These technologies are the heart and lungs of the company and the latest J2EE or Rails website would be pretty useless without them quietly churning away in the background supporting the business's ability to deliver. Of course, we know that nothing really churns away quietly, so we need people to look after them and projects to update them and add new functions when the business environment changes - just like anything else.

It's not glamorous or sexy, but it's still very challenging and often cutting-edge work which is subject to exactly the same project pressures as the latest marketing initiative.

Unfortunately, when the IT director at a multinational looks at Agile, his or her first reaction is not that of one of Moore's Visionaries who would wonder "will this help me achieve 10X productivity gains". It's more likely to be "how does this affect my ability to keep the lights on". And that's the right place to focus - failure to do so would be reckless, career ending and maybe even illegal (governments tend to take a dim view of telecoms operators that cannot reliably provide, for example, 999 emergency services).

So what can we do?

Scrum, Crystal and DSDM have shown us how to work iteratively, Lean has taught managers how to think about software development, but a team cannot be truly Agile without the Extreme Programming technical practices that allow it to truly embrace change. It is essential that we can provide documented proof and guidance on how to apply the technical practices across a wide range of modern and legacy technologies rather than just the new era of OO toys.

The adoption pattern is there to see from the OO world. We need expert visionaries from untried technical domains to stand forward, say "YES! This is possible using Product X", and then go and make it happen. We need blogs, articles, conference presentations, training courses and, ultimately, books to guide Agile virgins through the technical practices using a range of technologies. We need to show business software vendors that they can differentiate themselves by supporting their clients Agile development needs.

That's what I am doing, but I can only work with what I know so I will end with a rallying call to the next generation of visionaries:

If you are breaking ground in non OO environments - especially business software - please write about it! There has never been a better time to make a real difference and maybe a name for yourself too.

Naked Agilists: 19 Jan 2008

2008-01-16T19:50:00.000Z

I am giving a talk at the Naked Agilists skypecast on the 19th to talk about TDD on data management projects. My synopsis is as follows...

The conventional wisdom in the design and testing of Object Oriented systems is to isolate application functionality from the underlying data model. How then, do we adopt Test Driven Development for projects where the goal is to transform the raw data and Object Oriented languages are not used? This presentation introduces the TDD approach I have successfully pioneered on my own data management projects and FIT4Data - the FIT-based testing framework I wrote for the purpose.

We are having a few problems getting Skypecast to work at the moment, so please join the list to find out the new arrangements as they become clearer.

Yahoo Agile Data Management Group

2008-01-16T19:46:00.000Z

I have started an Agile Data Management group on Yahoo. It's at http://tech.groups.yahoo.com/group/agile-datamgmt/

This group is for the discussion of all aspects of applying Agile principles and practices to data management projects. It is intended to compliment the existing Agile Databases group and the key distinction between them is that this group covers situations where working with raw data is the primary focus of the project or application. For example, data warehousing, data migration, data cleansing and operational data transformations.

The focus of the group is wide ranging covering both technical and management topics and is open to everyone from novices to experienced practitioners.

Hello and Welcome

2008-01-04T15:16:00.000Z

Well 2008 is upon us and it's New Year's resolution time. Mine is to start a professional blog to record and make public the thoughts, musings, discoveries and general tribulations of my working life. I might get into a few other bits and bobs too - but let's see how things go.

So what has led me to the (possibly rash) decision to take some time out of my busy schedule to maintain a blog?

A bit of background

I am a professional software developer based in the UK who specialises in data management using Ab Initio. I studied computing at university for the simple reason that I liked it and was reasonably good at it and I joined the workplace as a graduate developer back in 2000 full of bright eyed enthusiasm for writing great solutions. Like many developers progressing through the ranks from graduate to seniority I became increasingly disillusioned with the reality of the politics and methods used in the real world. It was especially disheartening to find that developers are almost at the bottom of the food chain in most companies (apart from the testers who, as bearers of bad news, are often treated even worse). We were the people who were creating the system so surely everyone involved would be judged on the success of that system from initial release to eventual decommissioning. Why, therefore, was everyone not not lining up to support us?

There had to be a better way! I just couldn't find it that's all.

It's not that everything was bad, of course. I have worked on many projects, some were great, others were not and most fell somewhere in the middle. On every one of them, I worked beside some fantastic, heroic people who made real difference. Sometimes they were even recognised for it! However, my enthusiasm for the job was seriously waning. I had retained enough of a spark for technology to resist the gravitational pull towards management but I was seriously considering leaving the IT industry to do something completely different.

Thankfully, everything changed when I was introduced to Lean Agile by the inspirational Nancy Van Schooenderwoert. She showed me there is a better way and it's even better than I could have ever have imagined. The Agile principles and practices made immediate sense to me especially the way Lean thinking and Agile methods put the team at the centre of everything with spectacular improvements on quality, productivity and team satisfaction - my suspicions were confirmed!

That's all very interesting Adrian, but why the blog?

The problem we face as data management professionals is that the vast majority of the Agile thinking and literature is focused on developing UI based applications like websites and gadgets using Object Oriented technologies so it's not immediately apparent how to apply agile to data management projects like data warehouses. The purpose of this blog is to publish the findings of my work with Nancy and my company M2Consortium on how to bridge this knowledge gap and explore new ideas as they come up.

Over the past year or so I have been greatly inspired by Jeff Attwood's blog Coding Horror. It has nothing to do with data management or Ab Initio and only makes occasional references to Agile but many of his posts are stimulating I like the way his enthusiasm for what he does comes across very clearly. If I can be even a quarter as good as Jeff I will be very pleased but I am up for a challenge and on his advice I have decided to pick a schedule I can realistically expect to meet. So, drum roll please...

I will post at least once every three weeks.

... cue muted applause...

OK, so it's not very often and I will try and do better, but I have no idea how this is going to work out so I would rather commit to something I can definitely manage than set the bar too high. Please watch this space and Happy New Year!