Is there a resource that tells how to write good unit tests? Not "given the 
tests, how do you put them in a framework", but "how do you know what tests 
to write, given the framework"?

My problem is, in part, that I'm not sure I can think of any non-trivial 
unit tests that I'd run with any sort of frequency.  Let me illustrate with 
some of the sorts of routines I've recently written tests for, or have been 
unable to write tests for.

1) Given a string, canonicalize it. Take out punctuation, stop words, and 
make it lower case, to increase the likelihood that two song titles or 
artists from different sources will match. Of course I wrote tests for this. 
And I got it wrong the first time without knowing it, because I forgot to 
account for multiple consecutive stop words. But it's a leaf routine, so the 
likelihood I change it once I think it works is almost nil, the likelihood 
that I could possibly change something else and break this routine is almost 
nil (since I'm using a safe language), so I can't imagine why I'd run any 
tests I wrote multiple times.

2) I have a table with 6 million songs from iTunes. I have a table with 
800,000 ringtones. Both get updated approximately weekly. The metadata for 
each is of questionable accuracy. (For example, iTunes has about 4 different 
ways of saying "artist", and it's clearly a manual process adding it, 
because you can see the same mistakes in different clumps of data. The 
ringtone database is in various character sets, but the people building the 
database didn't bother to normalize the characters to utf nor record what 
character encoding was used. Just as examples.) I also have a continuous 
stream of song/artist information coming in from a third company (MG) as 
well as from various individual low-speed streams (like individual radio 
stations, web screen scrapes, etc). I want to be able to rapidly match the 
data from these low-speed streams against the data in the iTunes and 
ringtone tables with the best accuracy possible. How do I write a test to 
ensure I'm serving the right answers for given songs, considering the data 
changes faster than I can make an exhaustive manual test, and that making up 
my own data really doesn't tell me anything of interest? Mind, too, that not 
only does one company call it "the alan parsons project" and the other call 
it "alan parsons", but we also want to find the rendition of Beatle's 
"Yesterday" covered by the Philadelphia Philharmonic if and only if the 
actual Beatles song isn't out there, for example.

So this is basically the question of "how do you write unit tests for data 
sets in the millions where the unobvious errors might be 1% or 0.1% of the 
data?"

3) We have a feed of data coming in from MG. Reading the stream is a 
destructive operation. The data is vital to our ongoing service, in the 
sense that if I consume data while the system is live, people will get the 
wrong answer. The only way to read the stream is to use a binary blob of 
executable provided to us by MG. How do I test responding to outages? (I.e., 
this is basically "how do you mock something whose specs are unspecified and 
which you can't futz with directly?")

4) We output the results to HTML pages. The pages are rather complex, and 
include or exclude lots of stuff depending on what kind of result we found, 
etc. You could unit-test this by generating known data sets (like "it was a 
song from station X, with name and artist Y, with one ringtone perfect 
match, another ringtone with the same artist but different song, and two 
songs on iTunes with the same name but different artists, one of which sells 
better than the other so we want to list it first", and then look at the 
HTML that comes out, but every change to the templates is going to be 
different HTML coming out. I suppose having the data sets and templates will 
let you look at what you think is all the possible HTML coming out 
conveniently, but that's also not the sort of thing you need to run if you 
haven't changed the templates or tests, and whether the answer is "right" is 
not something you can algorithmically check.

It just seems to me that in most everything I do, either I have something 
impossible to mock, something unreasonable to check extensively by hand (and 
which will yield wrong results in a small percentage of cases), or there's 
output from the routines that gets interpreted by something else, so there's 
a huge number of equivalent results that are all "correct". (How do you 
unit-test automatically that your UI javascript does the right thing in all 
the browsers you care about?)

The sophisticated data structures I use all come from someone else. I don't 
need to test the on-disk B-Tree implementation the SQL database engine uses, 
nor do I need unit tests to check that qsort() is actually returning sorted 
lists.

My code tends to work with big piles of messy data, including that where the 
result you get isn't always obviously correct. The actual individual methods 
themselves are 90% of the time straightforward (as in, call these three 
searches and concat the resulting lists), and in the 10% of the time they're 
not, I don't change them but once in a blue moon and only to change things 
that would break unit tests anyway, such that writing unit tests would be 
counter-productive.


-- 
   Darren New, San Diego CA, USA (PST)
   Why is there a chainsaw in DOOM?
   There aren't any trees on Mars.

Post a reply to this message

Darren New wrote:

> It just seems to me that in most everything I do, either I have 
> something impossible to mock, something unreasonable to check 
> extensively by hand (and which will yield wrong results in a small 
> percentage of cases), or there's output from the routines that gets 
> interpreted by something else, so there's a huge number of equivalent 
> results that are all "correct". (How do you unit-test automatically that 
> your UI javascript does the right thing in all the browsers you care 
> about?)

I'm currently trying to implement a PostScript interpretter. Do you have 
*any idea* how non-trivial it is to come up with good tests which cover 
all possible corner-cases? Do you have any idea how you verify what the 
"correct" result is even supposed to be? (Currently I just check it 
against GhostScript, but that just means my program will have the same 
bus that GS has! And even that testing isn't trivial...)

So far, I'm still testing the damned parser! I haven't even got to the 
part where I execute anything... :-(

(PostScript also turns out to be a highly non-trivial language. There 
are so many minor details which screw things up. Like the special 
handling for literal procedures encountered on stdin...)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> I'm currently trying to implement a PostScript interpretter. Do you have 
> *any idea* how non-trivial it is to come up with good tests which cover 
> all possible corner-cases? Do you have any idea how you verify what the 
> "correct" result is even supposed to be? 

I don't think that's "unit tests", tho.  Normally, a "unit test" covers what 
would normally be one class, and not a whole operation. If you're building 
lots of data structures, unit tests can give you confidence that the data 
structure meets its postconditions and invariants. Stuff like inserting into 
a hash table keeps the same size if the key is a repeat, or increases the 
size by one if the key isn't a repeat.  I can imagine lots of tests for a 
postscript parser: The string "one two three" parses into three words, the 
word "three" winds up on top of the stack, that "1.0" parses as a number but 
"1X0" doesn't, etc.

I think my problem is that most of the data structures I use are either 
complex SQL tables, or hash tables and arrays provided by the language, so I 
rarely have any kinds of code where there's an obvious pre- and 
post-condition to it. If there were, someone else would have already written 
it to death. :-?

-- 
   Darren New, San Diego CA, USA (PST)
   Why is there a chainsaw in DOOM?
   There aren't any trees on Mars.

Post a reply to this message

Darren New wrote:
> Orchid XP v8 wrote:
>> I'm currently trying to implement a PostScript interpretter. Do you 
>> have *any idea* how non-trivial it is to come up with good tests which 
>> cover all possible corner-cases? Do you have any idea how you verify 
>> what the "correct" result is even supposed to be? 
> 
> I don't think that's "unit tests", tho.

Depends what you test.

> Normally, a "unit test" covers 
> what would normally be one class, and not a whole operation.

Again, depends what you test.

> I can 
> imagine lots of tests for a postscript parser: The string "one two 
> three" parses into three words, the word "three" winds up on top of the 
> stack, that "1.0" parses as a number but "1X0" doesn't, etc.

That's the kind of thing I'm trying to do. Except that figuring out 
every possible case is surprisingly hard.

In particular, there is a user-level function available to take a string 
and parse the first token off of it, and the PostScript language spec 
stipulates exactly how much of the input should be gobbled up. A small 
number of PS programs might actually rely on this precise behaviour, so 
you have to get it exactly right.

I'm having difficulty figuring out a way to really thoroughly check that 
this works properly. Seemingly you'd need a really big stack of test 
cases, but how do you generate those? (And how do you figure out what 
the "correct" answer is? This is sometimes non-obvious.)

When/if I get to the point of actually executing stuff, that'll be a 
whole *other* picnic!

Post a reply to this message

Invisible wrote:
>> Normally, a "unit test" covers what would normally be one class, and 
>> not a whole operation.
> 
> Again, depends what you test.

Well, I'm kind of taking the definition of "unit tests" as I understand it. 
There are four or five other kinds of tests too.

>> I can imagine lots of tests for a postscript parser: The string "one 
>> two three" parses into three words, the word "three" winds up on top 
>> of the stack, that "1.0" parses as a number but "1X0" doesn't, etc.
> 
> That's the kind of thing I'm trying to do. Except that figuring out 
> every possible case is surprisingly hard.

Yes it is. That's why you need to do it. :-)  Run it thru a 3rd-party 
postscript interpreter if you don't know what the function is supposed to 
be. You don't have to test exhaustively. You just have to figure it out.

Look, you have to figure out every possible case to *write* the code, yes?

> I'm having difficulty figuring out a way to really thoroughly check that 
> this works properly. Seemingly you'd need a really big stack of test 
> cases, but how do you generate those? 

While it's not what most people recommend, you can write the test cases as 
you write the code. When you add a check that looks to see if the first 
character is a digit and if so takes a different branch, you put in a test 
where the first character is a digit and where the first character isn't. 
For example.

Better would be to start with a dozen cases you know the answer to, then 
parse them until they work, and every time you see something funny or think 
of something new, add it to the list of test cases.

It's difficult to exhaustively figure out what you're going to match if you 
don't have the right formal tools. (This is why people like regular 
expressions and BNF, you see.)

 > (And how do you figure out what
> the "correct" answer is? This is sometimes non-obvious.)

If you can't tell from the spec, you have to run it against someone else's 
implementation. If you're attempting to build a compatible implementation, 
it has to match Adobe's(?) regardless of what the spec says.

-- 
   Darren New, San Diego CA, USA (PST)
   Why is there a chainsaw in DOOM?
   There aren't any trees on Mars.

Post a reply to this message

>> That's the kind of thing I'm trying to do. Except that figuring out 
>> every possible case is surprisingly hard.
> 
> Yes it is. That's why you need to do it. :-)  Run it thru a 3rd-party 
> postscript interpreter if you don't know what the function is supposed 
> to be. You don't have to test exhaustively. You just have to figure it out.
> 
> Look, you have to figure out every possible case to *write* the code, yes?

Yeah. But you sit down and write some code which you think should work, 
and then you have to write lots of test cases for

- Does it fail on a zero-length literal name. [The spec explicitly 
*allows* such names.]
- Does it correctly handle [what would otherwise be] comments inside a 
literal string?
- Does it choke if the input contains a comment [and so is non-empty] 
but no further actual tokens?
- Does it parse all possible reals, but not invalid ones such as "." and 
"e1"?
- Does it handle balanced brackets and escaped brackets in strings?

It's easy to *think* you got all these working, only to suddenly 
discover that in some sufficiently obscure case it falls over. :-/

>> (And how do you figure out what
>> the "correct" answer is? This is sometimes non-obvious.)
> 
> If you can't tell from the spec, you have to run it against someone 
> else's implementation. If you're attempting to build a compatible 
> implementation, it has to match Adobe's(?) regardless of what the spec 
> says.

I'm not aware of any Adobe implementation that's freely available. 
(Except perhaps for the one inside the nearest laser printer. And I 
already tried to get that to parse stuff; it didn't seem to want to do 
it for some reason.)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
>> Look, you have to figure out every possible case to *write* the code, 
>> yes?
> 
> Yeah. But you sit down and write some code which you think should work, 
> and then you have to write lots of test cases for
> 
> - Does it fail on a zero-length literal name. [The spec explicitly 
> *allows* such names.]
> - Does it correctly handle [what would otherwise be] comments inside a 
> literal string?
> - Does it choke if the input contains a comment [and so is non-empty] 
> but no further actual tokens?
> - Does it parse all possible reals, but not invalid ones such as "." and 
> "e1"?
> - Does it handle balanced brackets and escaped brackets in strings?

There's a good start. Why do you think it's hard?

> It's easy to *think* you got all these working, only to suddenly 
> discover that in some sufficiently obscure case it falls over. :-/

Testing doesn't prove the absence of bugs, only the presence of bugs.

> I'm not aware of any Adobe implementation that's freely available. 

Ghostscript would probably do pretty close, then. :-)

> (Except perhaps for the one inside the nearest laser printer. 

That one was far from free. :-)

> already tried to get that to parse stuff; it didn't seem to want to do 
> it for some reason.)

Sometimes you need to wrap it up in actual page description stuff, by which 
I mean the magic comments that mark the start and end of pages and such.

-- 
   Darren New, San Diego CA, USA (PST)
   Why is there a chainsaw in DOOM?
   There aren't any trees on Mars.

Post a reply to this message

Darren New wrote:
> Orchid XP v8 wrote:
>>> Look, you have to figure out every possible case to *write* the code, 
>>> yes?
>>
>> Yeah. But you sit down and write some code which you think should 
>> work, and then you have to write lots of test cases for
>>
>> - Does it fail on a zero-length literal name. [The spec explicitly 
>> *allows* such names.]
>> - Does it correctly handle [what would otherwise be] comments inside a 
>> literal string?
>> - Does it choke if the input contains a comment [and so is non-empty] 
>> but no further actual tokens?
>> - Does it parse all possible reals, but not invalid ones such as "." 
>> and "e1"?
>> - Does it handle balanced brackets and escaped brackets in strings?
> 
> There's a good start. Why do you think it's hard?

Because the list is wildly incomplete. Figuring out how to make a 
*complete* list is absurdly hard.

> Testing doesn't prove the absence of bugs, only the presence of bugs.

Sure. But I'd like to reveal as many bugs as possible, so...

>> I'm not aware of any Adobe implementation that's freely available. 
> 
> Ghostscript would probably do pretty close, then. :-)

Yeah, that's what I'm ending up using. (Ghostscript seems to correctly 
process just about every *real* page description I've ever come across, 
so I guess it's not too bad a comparison.)

>> (Except perhaps for the one inside the nearest laser printer. 
> 
> That one was far from free. :-)

Ah, but for me? ;-) I didn't pay...

>> already tried to get that to parse stuff; it didn't seem to want to do 
>> it for some reason.)
> 
> Sometimes you need to wrap it up in actual page description stuff, by 
> which I mean the magic comments that mark the start and end of pages and 
> such.

Nope. If I send it a PostScript program that draws some coloured lines 
and stuff, it spits out a piece of paper with lines drawn on it. If I 
send it another PostScript program [which also works fine in 
Ghostscript] that parses a string and writes the results onto the 
page... the printer appears to not receive the data. Really weird.

(And if I send a PostScript program containing deliberate errors, the 
printer prints out a page with an error message on it.)

Post a reply to this message

Invisible wrote:
> Ghostscript] that parses a string and writes the results onto the 
> page... the printer appears to not receive the data. Really weird.

You invoked showpage and all that stuff, I assume?  (Along the "is it 
plugged in, is it turned on" kind of debugging?)

Take a page that just prints one bit of text, and edit it with notepad or 
whatever you prefer, and see if you can get a base for printing?

-- 
   Darren New, San Diego CA, USA (PST)
   Why is there a chainsaw in DOOM?
   There aren't any trees on Mars.

Post a reply to this message

Darren New wrote:
> Invisible wrote:
>> Ghostscript] that parses a string and writes the results onto the 
>> page... the printer appears to not receive the data. Really weird.
> 
> You invoked showpage and all that stuff, I assume?  (Along the "is it 
> plugged in, is it turned on" kind of debugging?)
> 
> Take a page that just prints one bit of text, and edit it with notepad 
> or whatever you prefer, and see if you can get a base for printing?

As I say, when I run it in Ghostscript, it works just fine. But it 
doesn't seem to want to work on the laser printer I tried it with. (I 
guess I could try another...)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message