For the past month and a half, I've been working on performing automated 
testing of our product. Now the product *already* has over one thousand 
unit tests, which test individual components of the application. But as 
any good tester will tell you, you also need integration tests to check 
that the pieces actually fit together correctly.

We do have a guy who's job is to test the software. But one human poking 
buttons at random to see if anything breaks is far slower and less 
systematic than an automated test system. (On the other hand, some tests 
cannot be automated - e.g., you can't write a test that checks that the 
text is legible, hasn't been cut off the edge of the page, etc.)

In short, I've build a system which allows me to remote-control the 
product over the network, and observe its responses. This allows me to 
control the software more or less the way a user would - click this 
button, select that option, check that the correct data is displayed.

I say "build a system" because, after many, many months of fruitless 
Google searching, we discovered that no such system already exists for 
our software platform. If it were a web application, there would be 
trillions of options for testing tools. If it were written in Qt, we 
would have several strong contenders to look at. But it's GTK+, so 
there's essentially nothing. We found a couple of OSS tools which were 
horribly broken, and that is all.

So I looked at the inner working of one of the tools, and ended up 
reimplementing it in C#. And I've spent a month putting all the 
functionality we need into it. Essentially I run my server code in the 
laptop I want to remote control, and run the client from inside Visual 
Studio on my development box. Then I run my test suite, and the tests 
talk to the client, which commands the server to do stuff.



Now, originally I wrote all the tests by hand. But more recently I've 
started using a curious creation known as SpecFlow. It's a tool for VS 
that lets you write user scenarios in "Gherkin", which is "a member of 
the Cucumber family of languages". (??!) For those of you who haven't 
heard, I will explain further...

Essentially, the idea is that you write a usage scenario in that looks 
like free-form English. For example, you might say

   Scenario: Adding a new user
     Given that I am on the User Admin page
     And there is currently no user named 'Tim'
     When I press the 'Add User' button
     And I type the username 'Tim'
     And I type the real name 'Timothy Robinson'
     And I press the 'Save' button
     Then there should now be a user named 'Tim'

You save this text in a file. You then right-click the file and say "run 
tests". The SpecFlow *generates code*, producing an NUnit test fixture - 
which is then run. Let me say that again: merely by writing the above 
text, a test has been created.

When you run the test, absolutely nothing happens. The test is merely 
marked as "inconclusive". Because, hey, the machine has no idea WTF 
you're talking about. What you have to do is go write a set of 
"bindings", which describe how to transform English into executable code.

If you've been paying attention, you'll have noticed that the above text 
isn't *really* free-form English at all. In fact, it's actually pretty 
rigidly structured. In fact, the format is line-based. Every line of 
text is a "test step". Each such line must begin with "Given", "When" or 
"Then". (Or "And", which duplicates the prefix from the previous line.)

The intention is that "Given" lines are test setup, "When" lines are 
test actions, and "Then" lines are test assertions. (Although you don't 
actually have to follow this, if you're feeling subversive.) What you do 
is you write C# code in methods, and you tag the methods with C# method 
attributes. For example,

    [When("I press the 'Save' button")]
    public void PressSave()
    {
      ...actual executable C# code...
    }

All SpecFlow actually does is look at each line of English text and then 
search the entire VS project (!!) for a method with a matching 
annotation. [It appears SpecFlow combines the English text into C# code, 
but the actual binding lookups happen at run-time.]

So in this way, all you're really doing is writing the test code 
manually, but writing a human-readable summary of what order the code is 
run in.



We began by having the testing guy write miles of SpecFlow scenarios, 
and then I go through them, press the "generate test steps" button, 
which builds a class full of empty step bindings. I then go through each 
method and fill it out with real code that really does something. In 
this way, the tester guy is like a script writer, and I'm the developer 
building the code that implements the tester's vision.

That worked for a while, but then I did what any self-respecting 
developer would do - I started to get lazy, and then I started to work 
smarter.

Firstly, I figured out that if you edit the test scenarios slightly so 
that every test is phrased *exactly* the same way, you can reuse any 
test steps you've already implemented. That can save you some coding. So 
by making minor grammatical changes to the written English, I could 
reduce the amount of code I need to write.

(Lots of scenarios are very similar, to the point that lots of them have 
clearly been copy-pasted multiple times. Either that or the guy make the 
exact same typo exactly the same way fifteen times in a row...)

Second, I noticed that as I wrote all the code, certain common patterns 
of commands popped up again and again. So I wrote a helper class which 
allows me to turn these into a single method call. (E.g., I might want 
to search for a button with a specific name, check that it's visible, 
check that it's enabled, and then click it. That can be made into one 
method call.)

That worked for a while. But then I started to get *really* sneaky. 
SpecFlow allows you to use regular expressions to actually PARSE STUFF 
OUT of test steps. For example,

    [When("I press the '(.*)' button")]
    public void PressButton(string name)
    {
      ...code...
    }

Once I implement the code, I now never, ever have to manually implement 
clicking a button again. Suddenly 60% of the test steps I'd been coding 
can now share this single step binding. In order words, my work just got 
60% easier (and my codebase 60% smaller).

After a while I managed to put together a library of bindings so 
comprehensive that for many of the scenarios that had been written, I 
could merely reword them very slightly and actually generate an entire, 
runnable, valid test WITHOUT WRITING ANY CODE! I could actually build by 
test suite just by rephrasing some English into broken robot-speak.

To see how far you can take this, consider the following example:

   Scenario: Adding a new user
     Given that I am on the User Admin page
     And the table does not contain a row with 'Tim' in the 'Username' 
column
     When I press the 'New User' button
     Then I should be on the Create User Page
     When I type 'Tim' in the Username field
     And I type 'Timothy Robinson' in the Real Name field
     And I select the 'Normal User' radio-button
     And I check the 'Accounting' check-box
     And I press the 'Save' button
     Then I should be on the User Admin page
     And the table should include the following rows:
       | Username | Real Name        | Account Type | Group Membership |
       | Tim      | Timothy Robinson | Normal       | Accounting       |

That final line has a binding where SpecFlow executes a method with a 
"table" as argument. The method actually queries the display, checks 
that the specified columns exist in the table, searches for a row where 
the Username column contains "Tim", and then checks that the other cells 
in that row contain the specified values. This method is *not* specific 
to the user admin page; it works for *any* page that has a single table 
on it. And it works for any number of rows, and it allows those rows to 
appear in any order. (For that matter, the columns may appear in any 
order, not necessarily the one shown. And there may be additional 
columns that we don't care about.)

In the one hand, this is a fairly crazy level of testing. On the other 
hand, this has slowly devolved away from being a high-level user-centric 
description of what the application should do, into a very low-level 
"check the exact status of every widget on the entire page according to 
that I, the application developer, what to test for". Look at the second 
line: we've gone from "there isn't a user named Tim" to "the table does 
not contain a row with Tim in the Username column".

Maybe I'm doing it wrong.



Eventually, I came across a system of features where the test scenarios 
become incredibly repetitive. But SpecFlow has an answer for this also. 
Observe:

   Scenario: Changing user type from Normal User to Super User
     Given that I am on the User Admin page
     And the table includes the following rows:
       | Username | Account Type |
       | Tim      | Normal       |
     When I select the row with 'Tim' in the 'Username' column
     And I press the 'Edit' button
     Then I should be on the Edit User page
     And the 'Username' field should contain 'Tim'
     And the 'Normal User' radio-button should be selected
     When I select the 'Super User' radio-button
     And I press the 'Save' button
     Then I should be on the User Admin page
     And the table should include the following rows:
       | Username | Account Type |
       | Tim      | SUPER USER   |
     When I select the row with 'Tim' in the 'Username' column
     And I press the 'Edit' button
     Then I should be on the Edit User page
     And the 'Super User' radio-button should be selected

This checks that when I change the user's type, the data actually saves, 
the table updates correctly, and when I recall the data it's still 
there. (You'd be surprised how often this fails for obscure properties 
that nobody really cares about...)

Now go write a test checking for all the other user types you can 
possible have. Yeah, that won't talk long; a few copy-paste operations 
later, and this gigantic monstrosity has been duplicated enough that if 
you never need to change it, you'll have to change it five times! o_O

So what do I do? I change it to this:

   Scenario Outline: Changing user type
     Given that I am on the User Admin page
     And the table has 'Tim' in the 'Username' column
     When I select the row with 'Tim' in the 'Username' column
     And I press the 'Edit' button
     Then I should be on the Edit User page
     And the 'Username' field should contain 'Tim'
     When I select the '<type button>' radio-button
     And I press the 'Save' button
     Then I should be on the User Admin page
     And the table should include the following rows:
       | Username | Account Type   |
       | Tim      | <type display> |
     When I select the row with 'Tim' in the 'Username' column
     And I press the 'Edit' button
     Then I should be on the Edit User page
     And the '<type button>' radio-button should be selected

     Examples:
       | type button | type name  |
       | Normal User | Normal     |
       | Supervisor  | Supervisor |
       | Manager     | Manager    |
       | Super User  | SUPER USER |

We now have a *parametrised* test script. SpecFlow takes my string 
template, instantiates it with the data from the table to generate 
psuedo-English, with the test bindings then dismantle back into data 
which they pass to the server to actually execute the tests. We have now 
come completely full circuit and disappeared up our own wotsits!

Look at that final line; SpecFlow replaces by variable with a value, 
which the step binding then uses a regex to parse back out so it can 
make the server call. (In fairness, you *can* write a template such that 
a different method gets called for each instance of the template...) 
Look at the vagueness of that parenthetical statement; you can't even 
tell which level of templating I'm talking about any more!

Now I am become death, the destroyer of worlds. All we need now is for 
SpecFlow to implement conditional branching and iteration, and it'll be 
Turing-complete. AND THEN THE WORLD IS NOT SAFE!!

The madness. The madness!!

Post a reply to this message