![](/i/fill.gif) |
![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Darren New
Subject: Unit testing - simple question with long explanation for discussion...
Date: 5 Jan 2009 17:13:54
Message: <49628622@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Is there a resource that tells how to write good unit tests? Not "given the
tests, how do you put them in a framework", but "how do you know what tests
to write, given the framework"?
My problem is, in part, that I'm not sure I can think of any non-trivial
unit tests that I'd run with any sort of frequency. Let me illustrate with
some of the sorts of routines I've recently written tests for, or have been
unable to write tests for.
1) Given a string, canonicalize it. Take out punctuation, stop words, and
make it lower case, to increase the likelihood that two song titles or
artists from different sources will match. Of course I wrote tests for this.
And I got it wrong the first time without knowing it, because I forgot to
account for multiple consecutive stop words. But it's a leaf routine, so the
likelihood I change it once I think it works is almost nil, the likelihood
that I could possibly change something else and break this routine is almost
nil (since I'm using a safe language), so I can't imagine why I'd run any
tests I wrote multiple times.
2) I have a table with 6 million songs from iTunes. I have a table with
800,000 ringtones. Both get updated approximately weekly. The metadata for
each is of questionable accuracy. (For example, iTunes has about 4 different
ways of saying "artist", and it's clearly a manual process adding it,
because you can see the same mistakes in different clumps of data. The
ringtone database is in various character sets, but the people building the
database didn't bother to normalize the characters to utf nor record what
character encoding was used. Just as examples.) I also have a continuous
stream of song/artist information coming in from a third company (MG) as
well as from various individual low-speed streams (like individual radio
stations, web screen scrapes, etc). I want to be able to rapidly match the
data from these low-speed streams against the data in the iTunes and
ringtone tables with the best accuracy possible. How do I write a test to
ensure I'm serving the right answers for given songs, considering the data
changes faster than I can make an exhaustive manual test, and that making up
my own data really doesn't tell me anything of interest? Mind, too, that not
only does one company call it "the alan parsons project" and the other call
it "alan parsons", but we also want to find the rendition of Beatle's
"Yesterday" covered by the Philadelphia Philharmonic if and only if the
actual Beatles song isn't out there, for example.
So this is basically the question of "how do you write unit tests for data
sets in the millions where the unobvious errors might be 1% or 0.1% of the
data?"
3) We have a feed of data coming in from MG. Reading the stream is a
destructive operation. The data is vital to our ongoing service, in the
sense that if I consume data while the system is live, people will get the
wrong answer. The only way to read the stream is to use a binary blob of
executable provided to us by MG. How do I test responding to outages? (I.e.,
this is basically "how do you mock something whose specs are unspecified and
which you can't futz with directly?")
4) We output the results to HTML pages. The pages are rather complex, and
include or exclude lots of stuff depending on what kind of result we found,
etc. You could unit-test this by generating known data sets (like "it was a
song from station X, with name and artist Y, with one ringtone perfect
match, another ringtone with the same artist but different song, and two
songs on iTunes with the same name but different artists, one of which sells
better than the other so we want to list it first", and then look at the
HTML that comes out, but every change to the templates is going to be
different HTML coming out. I suppose having the data sets and templates will
let you look at what you think is all the possible HTML coming out
conveniently, but that's also not the sort of thing you need to run if you
haven't changed the templates or tests, and whether the answer is "right" is
not something you can algorithmically check.
It just seems to me that in most everything I do, either I have something
impossible to mock, something unreasonable to check extensively by hand (and
which will yield wrong results in a small percentage of cases), or there's
output from the routines that gets interpreted by something else, so there's
a huge number of equivalent results that are all "correct". (How do you
unit-test automatically that your UI javascript does the right thing in all
the browsers you care about?)
The sophisticated data structures I use all come from someone else. I don't
need to test the on-disk B-Tree implementation the SQL database engine uses,
nor do I need unit tests to check that qsort() is actually returning sorted
lists.
My code tends to work with big piles of messy data, including that where the
result you get isn't always obviously correct. The actual individual methods
themselves are 90% of the time straightforward (as in, call these three
searches and concat the resulting lists), and in the 10% of the time they're
not, I don't change them but once in a blue moon and only to change things
that would break unit tests anyway, such that writing unit tests would be
counter-productive.
--
Darren New, San Diego CA, USA (PST)
Why is there a chainsaw in DOOM?
There aren't any trees on Mars.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Orchid XP v8
Subject: Re: Unit testing - simple question with long explanation for discussion...
Date: 5 Jan 2009 17:19:54
Message: <4962878a@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Darren New wrote:
> It just seems to me that in most everything I do, either I have
> something impossible to mock, something unreasonable to check
> extensively by hand (and which will yield wrong results in a small
> percentage of cases), or there's output from the routines that gets
> interpreted by something else, so there's a huge number of equivalent
> results that are all "correct". (How do you unit-test automatically that
> your UI javascript does the right thing in all the browsers you care
> about?)
I'm currently trying to implement a PostScript interpretter. Do you have
*any idea* how non-trivial it is to come up with good tests which cover
all possible corner-cases? Do you have any idea how you verify what the
"correct" result is even supposed to be? (Currently I just check it
against GhostScript, but that just means my program will have the same
bus that GS has! And even that testing isn't trivial...)
So far, I'm still testing the damned parser! I haven't even got to the
part where I execute anything... :-(
(PostScript also turns out to be a highly non-trivial language. There
are so many minor details which screw things up. Like the special
handling for literal procedures encountered on stdin...)
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Darren New
Subject: Re: Unit testing - simple question with long explanation for discussion...
Date: 5 Jan 2009 17:37:34
Message: <49628bae$1@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Orchid XP v8 wrote:
> I'm currently trying to implement a PostScript interpretter. Do you have
> *any idea* how non-trivial it is to come up with good tests which cover
> all possible corner-cases? Do you have any idea how you verify what the
> "correct" result is even supposed to be?
I don't think that's "unit tests", tho. Normally, a "unit test" covers what
would normally be one class, and not a whole operation. If you're building
lots of data structures, unit tests can give you confidence that the data
structure meets its postconditions and invariants. Stuff like inserting into
a hash table keeps the same size if the key is a repeat, or increases the
size by one if the key isn't a repeat. I can imagine lots of tests for a
postscript parser: The string "one two three" parses into three words, the
word "three" winds up on top of the stack, that "1.0" parses as a number but
"1X0" doesn't, etc.
I think my problem is that most of the data structures I use are either
complex SQL tables, or hash tables and arrays provided by the language, so I
rarely have any kinds of code where there's an obvious pre- and
post-condition to it. If there were, someone else would have already written
it to death. :-?
--
Darren New, San Diego CA, USA (PST)
Why is there a chainsaw in DOOM?
There aren't any trees on Mars.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Invisible
Subject: Re: Unit testing - simple question with long explanation for discussion...
Date: 8 Jan 2009 07:22:42
Message: <4965f012$1@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Darren New wrote:
> Orchid XP v8 wrote:
>> I'm currently trying to implement a PostScript interpretter. Do you
>> have *any idea* how non-trivial it is to come up with good tests which
>> cover all possible corner-cases? Do you have any idea how you verify
>> what the "correct" result is even supposed to be?
>
> I don't think that's "unit tests", tho.
Depends what you test.
> Normally, a "unit test" covers
> what would normally be one class, and not a whole operation.
Again, depends what you test.
> I can
> imagine lots of tests for a postscript parser: The string "one two
> three" parses into three words, the word "three" winds up on top of the
> stack, that "1.0" parses as a number but "1X0" doesn't, etc.
That's the kind of thing I'm trying to do. Except that figuring out
every possible case is surprisingly hard.
In particular, there is a user-level function available to take a string
and parse the first token off of it, and the PostScript language spec
stipulates exactly how much of the input should be gobbled up. A small
number of PS programs might actually rely on this precise behaviour, so
you have to get it exactly right.
I'm having difficulty figuring out a way to really thoroughly check that
this works properly. Seemingly you'd need a really big stack of test
cases, but how do you generate those? (And how do you figure out what
the "correct" answer is? This is sometimes non-obvious.)
When/if I get to the point of actually executing stuff, that'll be a
whole *other* picnic!
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Darren New
Subject: Re: Unit testing - simple question with long explanation for discussion...
Date: 8 Jan 2009 12:39:03
Message: <49663a37@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Invisible wrote:
>> Normally, a "unit test" covers what would normally be one class, and
>> not a whole operation.
>
> Again, depends what you test.
Well, I'm kind of taking the definition of "unit tests" as I understand it.
There are four or five other kinds of tests too.
>> I can imagine lots of tests for a postscript parser: The string "one
>> two three" parses into three words, the word "three" winds up on top
>> of the stack, that "1.0" parses as a number but "1X0" doesn't, etc.
>
> That's the kind of thing I'm trying to do. Except that figuring out
> every possible case is surprisingly hard.
Yes it is. That's why you need to do it. :-) Run it thru a 3rd-party
postscript interpreter if you don't know what the function is supposed to
be. You don't have to test exhaustively. You just have to figure it out.
Look, you have to figure out every possible case to *write* the code, yes?
> I'm having difficulty figuring out a way to really thoroughly check that
> this works properly. Seemingly you'd need a really big stack of test
> cases, but how do you generate those?
While it's not what most people recommend, you can write the test cases as
you write the code. When you add a check that looks to see if the first
character is a digit and if so takes a different branch, you put in a test
where the first character is a digit and where the first character isn't.
For example.
Better would be to start with a dozen cases you know the answer to, then
parse them until they work, and every time you see something funny or think
of something new, add it to the list of test cases.
It's difficult to exhaustively figure out what you're going to match if you
don't have the right formal tools. (This is why people like regular
expressions and BNF, you see.)
> (And how do you figure out what
> the "correct" answer is? This is sometimes non-obvious.)
If you can't tell from the spec, you have to run it against someone else's
implementation. If you're attempting to build a compatible implementation,
it has to match Adobe's(?) regardless of what the spec says.
--
Darren New, San Diego CA, USA (PST)
Why is there a chainsaw in DOOM?
There aren't any trees on Mars.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Orchid XP v8
Subject: Re: Unit testing - simple question with long explanation for discussion...
Date: 8 Jan 2009 13:51:12
Message: <49664b20@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
>> That's the kind of thing I'm trying to do. Except that figuring out
>> every possible case is surprisingly hard.
>
> Yes it is. That's why you need to do it. :-) Run it thru a 3rd-party
> postscript interpreter if you don't know what the function is supposed
> to be. You don't have to test exhaustively. You just have to figure it out.
>
> Look, you have to figure out every possible case to *write* the code, yes?
Yeah. But you sit down and write some code which you think should work,
and then you have to write lots of test cases for
- Does it fail on a zero-length literal name. [The spec explicitly
*allows* such names.]
- Does it correctly handle [what would otherwise be] comments inside a
literal string?
- Does it choke if the input contains a comment [and so is non-empty]
but no further actual tokens?
- Does it parse all possible reals, but not invalid ones such as "." and
"e1"?
- Does it handle balanced brackets and escaped brackets in strings?
It's easy to *think* you got all these working, only to suddenly
discover that in some sufficiently obscure case it falls over. :-/
>> (And how do you figure out what
>> the "correct" answer is? This is sometimes non-obvious.)
>
> If you can't tell from the spec, you have to run it against someone
> else's implementation. If you're attempting to build a compatible
> implementation, it has to match Adobe's(?) regardless of what the spec
> says.
I'm not aware of any Adobe implementation that's freely available.
(Except perhaps for the one inside the nearest laser printer. And I
already tried to get that to parse stuff; it didn't seem to want to do
it for some reason.)
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Darren New
Subject: Re: Unit testing - simple question with long explanation for discussion...
Date: 8 Jan 2009 18:58:11
Message: <49669313$1@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Orchid XP v8 wrote:
>> Look, you have to figure out every possible case to *write* the code,
>> yes?
>
> Yeah. But you sit down and write some code which you think should work,
> and then you have to write lots of test cases for
>
> - Does it fail on a zero-length literal name. [The spec explicitly
> *allows* such names.]
> - Does it correctly handle [what would otherwise be] comments inside a
> literal string?
> - Does it choke if the input contains a comment [and so is non-empty]
> but no further actual tokens?
> - Does it parse all possible reals, but not invalid ones such as "." and
> "e1"?
> - Does it handle balanced brackets and escaped brackets in strings?
There's a good start. Why do you think it's hard?
> It's easy to *think* you got all these working, only to suddenly
> discover that in some sufficiently obscure case it falls over. :-/
Testing doesn't prove the absence of bugs, only the presence of bugs.
> I'm not aware of any Adobe implementation that's freely available.
Ghostscript would probably do pretty close, then. :-)
> (Except perhaps for the one inside the nearest laser printer.
That one was far from free. :-)
> already tried to get that to parse stuff; it didn't seem to want to do
> it for some reason.)
Sometimes you need to wrap it up in actual page description stuff, by which
I mean the magic comments that mark the start and end of pages and such.
--
Darren New, San Diego CA, USA (PST)
Why is there a chainsaw in DOOM?
There aren't any trees on Mars.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Invisible
Subject: Re: Unit testing - simple question with long explanation for discussion...
Date: 9 Jan 2009 04:25:09
Message: <496717f5$1@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Darren New wrote:
> Orchid XP v8 wrote:
>>> Look, you have to figure out every possible case to *write* the code,
>>> yes?
>>
>> Yeah. But you sit down and write some code which you think should
>> work, and then you have to write lots of test cases for
>>
>> - Does it fail on a zero-length literal name. [The spec explicitly
>> *allows* such names.]
>> - Does it correctly handle [what would otherwise be] comments inside a
>> literal string?
>> - Does it choke if the input contains a comment [and so is non-empty]
>> but no further actual tokens?
>> - Does it parse all possible reals, but not invalid ones such as "."
>> and "e1"?
>> - Does it handle balanced brackets and escaped brackets in strings?
>
> There's a good start. Why do you think it's hard?
Because the list is wildly incomplete. Figuring out how to make a
*complete* list is absurdly hard.
> Testing doesn't prove the absence of bugs, only the presence of bugs.
Sure. But I'd like to reveal as many bugs as possible, so...
>> I'm not aware of any Adobe implementation that's freely available.
>
> Ghostscript would probably do pretty close, then. :-)
Yeah, that's what I'm ending up using. (Ghostscript seems to correctly
process just about every *real* page description I've ever come across,
so I guess it's not too bad a comparison.)
>> (Except perhaps for the one inside the nearest laser printer.
>
> That one was far from free. :-)
Ah, but for me? ;-) I didn't pay...
>> already tried to get that to parse stuff; it didn't seem to want to do
>> it for some reason.)
>
> Sometimes you need to wrap it up in actual page description stuff, by
> which I mean the magic comments that mark the start and end of pages and
> such.
Nope. If I send it a PostScript program that draws some coloured lines
and stuff, it spits out a piece of paper with lines drawn on it. If I
send it another PostScript program [which also works fine in
Ghostscript] that parses a string and writes the results onto the
page... the printer appears to not receive the data. Really weird.
(And if I send a PostScript program containing deliberate errors, the
printer prints out a page with an error message on it.)
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Darren New
Subject: Re: Unit testing - simple question with long explanation for discussion...
Date: 9 Jan 2009 12:37:02
Message: <49678b3e$1@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Invisible wrote:
> Ghostscript] that parses a string and writes the results onto the
> page... the printer appears to not receive the data. Really weird.
You invoked showpage and all that stuff, I assume? (Along the "is it
plugged in, is it turned on" kind of debugging?)
Take a page that just prints one bit of text, and edit it with notepad or
whatever you prefer, and see if you can get a base for printing?
--
Darren New, San Diego CA, USA (PST)
Why is there a chainsaw in DOOM?
There aren't any trees on Mars.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Orchid XP v8
Subject: Re: Unit testing - simple question with long explanation for discussion...
Date: 9 Jan 2009 13:13:22
Message: <496793c2@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Darren New wrote:
> Invisible wrote:
>> Ghostscript] that parses a string and writes the results onto the
>> page... the printer appears to not receive the data. Really weird.
>
> You invoked showpage and all that stuff, I assume? (Along the "is it
> plugged in, is it turned on" kind of debugging?)
>
> Take a page that just prints one bit of text, and edit it with notepad
> or whatever you prefer, and see if you can get a base for printing?
As I say, when I run it in Ghostscript, it works just fine. But it
doesn't seem to want to work on the laser printer I tried it with. (I
guess I could try another...)
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |