Fundamentals of data testing: Deterministic vs Heuristic

The style of test automation I find most useful when testing data integration systems is largely based on the “deterministic” testing style popularised by Agile methods. That means that for every value input, we need to be able to define an expected output from the system under test. This is appropriate for testing most situations that arise in data integration solutions but we find that there are occasional cases we will need to borrow from techniques in systems based on heuristics.

For example, consider the following mapping rules:

Source Column	Rule	Target Column
	System date formatted as “YYYYMMDD HH24:MI:SS”	creation_date
flight_number	==>	flight_number
destination	Convert to Sentence Case	dest_name

The rules for populating flight_number and destination are deterministic. Flight number is straightforward because the value from the source is simply copied to the target. The rules for destination are slightly more involved but we easily construct test cases that define an expected result based on a given input - EDINBURGH -> Edinburgh, gLasgow -> Glasgow... and so on.

The "creation_date" rule is quite different. Timestamps taken from the system clock are non-deterministic because it’s impossible to define a value that will definitely match the system time at the moment the job is run. In some cases, you can get round this by injecting a known value into the system under test using a parameter (or some similar method) but it’s not always possible so we need to take a leaf out of heuristic systems thinking.

Heuristic systems use a combination of rules and data gathered at runtime to make decisions. A scheduling system used by an airline plots a schedule for the airline each day that takes into account the many factors that affect airline scheduling - plane location, crew availability, turnaround time, weather conditions, volcanic activity in iceland, and so forth. There is no “right” answer to the question of how to arrange the planes so it’s hard to test deterministically. Instead, the development team need to ensure that the correct decision making flow is executed when deciding the schedule rather than trying to predict an outcome. Emily and Geoff Bache work on such as system and gave a presentation at Agile2008 explaining how their test suite works.

Back to our system datetime. We can borrow from heuristic concepts by plugging in a set of rules that allow us to validate the date generated. A reasonable heuristic rule for the system date might be to check the date is correctly formatted and within a few minutes of the moment the test suite was run.

Take care though. Heuristic tests make your test suite more complicated because you need capture and process more data in a heuristic style test than a deterministic test. Sometimes it may just not be worth the effort. Furthermore, it’s easy to write heuristic tests that are just as complex than the system being tested. Let’s face it, the risk of getting the system date call wrong is pretty remote so maybe it’s just easier to ignore those fields and focus building watertight tests for rules you can test deterministically. That’s usually where the real business benefits are found anyway.

Above all, you should always try to find a deterministic solution and you should expect to do so at least 95% of the time.

Agile Data Testing

Those of you who avidly follow my blog :-) will have noticed that there have been one or two quiet spells lately. I would love to say I have been living as a hermit living on seaweed and shellfish scraped from the rocks while building the perfect suite of tools for Agile Data Integration but, really, life just got in the way of writing the past couple of years.

However, I have not been idle. I've been working hard to define strategies for creating automated test solutions for ETL/Data Integration projects for the past 4 years and I have a lot of knowledge I want to share. Some of this has been captured in tools, but not as much as I would like so I am also looking for people to help me develop the tool capability in a warm, fuzzy, open-sourcy kind of way.

I'm about to publish some articles explaining what I have learned and where I am going. I know time is valuable, but I really, really want your feedback so we can start to develop more of a community around this topic. I'm going to start with the theory and get into the tooling once I have laid down my thoughts on the core concepts of what we are trying to achieve.

So please, please, please read the articles, be forthright in your comments and encourage me to write more.

My thanks.

Adrian

Big Data/IT blog

Check out this new blog I found written by a member of the Big Data community...

http://toastingit.blogspot.com/

Agile Datawarehousing Book

I just received my copy of Agile Data Warehousing by Ralph Hughes from Amazon.

At last, someone has written this book! I hope it is good.

One of the problems we face moving the world of agile beyond objects is that and managers in conservative companies need confidence that a different approach is tried and tested. This book has the potential to help do just that.

Since it's Christmas holiday season, I hope to have some time to read the book over the next couple of weeks and publish a review on this blog.

Naked Agilists Conference Preview

The new Naked Agilists podcast is now available at http://www.nakedagilists.com/

It's been a little while since we have released anything new, but with the Agile year's main event approaching fast we thought it would be a good idea to get some of the Agile2008 conference organisers to tell us about what they have planned. First up we have Grigori Melnik who gives us a 15 minute overview of the conference as as a whole as well as an insight into the selection process introduced this year. This year's conference is organised into stages - rather like a music festival - so we then we have 5 individual, shorter, talks from the organisers of some of the stages who tell us in much more detail what their stages have to offer.

It's much more than throwing out the documents

I was talking to a fellow Ab Initio developer the other day and he asked me a common question...

“Isn't all this Agile stuff just a case of not writing documentation?”

It's true that Agile teams do less documentation, but it is always replaced with better alternatives. I'll put that another way because it is important. Agilists do not throw away documents for the sake of it - that would be a pointless short term fix - but we are passionate about reducing waste in our projects and we have found there are more effective ways of doing some things that every project needs and they happen not to need as much documentation. (Mary and Tom Poppendieck showed us how to use Lean principles to identify and eliminate waste in software projects, this interview is a great place to start.)

This can be a little difficult to imagine if you are used to waterfall projects, so let's look at a couple of typical activities on data projects and the practices we can use to do them more effectively and reduce the documentation overhead.

Test Procedures

Waterfall teams write test procedures to record the manual tests that need to run against each release. There are problems with this approach: the tests take a long time to run manually, they are subject to human error, and because natural language is imprecise and ambiguous technical details gets lost in translation by the tester.

Agile teams use Test Driven Development and executable specifications to create automated test environments that can be run quickly and easily by anyone on the project (including stakeholders). Automated tests are absolutely precise – they either pass or they fail, and if a bug occurs you can simply add a new test and that error can never happen again.

Requirements and design

In Waterfall, business analysts use requirements documents to capture what the customer needs, then the designers create another document that interprets the requirements in technical language. Again, there are two problems with this. It takes a long time, so requirements go out of date and because natural language is subject to human interpretation the customer's original need gets lost in translation.

Agile teams are cross functional including business analysts, designers, developers and testers all of whom engage directly with an empowered representative from the business. They work in short iterations of a month or less that deliver solutions to the most pressing business need at that time. There's no need to maintain detailed requirements definitions or design specs because the customer is able to communicate what is needed face to face and the tests accurately describe what the code does.

Requirements traceability comes for free because Agile teams never let requirements and code diverge. Put another way, the V model, never opens out into a V.

Change Control and Risk Logs

Honestly, does anyone enjoy working through these?

Requirements and designs go out of date on waterfall projects, so they need a whole extra level of documentation and bureaucracy to track changes and uncertainty.

Agile teams simply do not need any of this stuff because they are always working on immediate priorities and the chunks of work are small enough that they can actively address risks early in each iteration.

Look for the Principles

Of course, I have really just scratched the surface, but if we look for the principles behind the practices we find ways to improve our thought processes.

Documentation is a legitimate form of knowledge capture, but it's not always the best option. There are times when documents are necessary and useful, recording business metadata is an example that springs to mind, but next time someone asks you or your team to write a document, recognise it's a default position – the lowest common denominator of project communication – and everyone involved might find a better way of working if they were to kick the problem around for a while to address the root problem more effectively. Sometimes you will find a document is the best solution and other times you will not. In either case, you will reach a well considered decision based on a reality.

Something In Nottigham

The Nottingham Agile group has a new page at http://somethinginnottingham.blogspot.com. You can use it to find out about upcoming meetings and link to the group's wiki from there.