Every work on unit testing I've read basically says you should run all the 
unit tests after each change, or certainly at least before each check-in.

However, those same texts say that unit testing should only test individual 
units, and everything else should be mocked.

Why would I want to run the unit tests for the XML parser after making 
changes to the color picker dialog box?  Or, more generally, why would I 
want to run unit tests that are only executing code that hasn't changed 
since last time the unit tests passed?

(Note I'm talking specifically of unit tests, not automated test suites in 
general.)

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

Le 08/06/2011 22:01, Darren New nous fit lire :
> Every work on unit testing I've read basically says you should run all
> the unit tests after each change, or certainly at least before each
> check-in.
> 
> However, those same texts say that unit testing should only test
> individual units, and everything else should be mocked.
> 
> Why would I want to run the unit tests for the XML parser after making
> changes to the color picker dialog box?  Or, more generally, why would I
> want to run unit tests that are only executing code that hasn't changed
> since last time the unit tests passed?
> 
> (Note I'm talking specifically of unit tests, not automated test suites
> in general.)
> 

If you had designed your software in blocks, you would have as many unit
test blocks, on a 1 for 1 ratio.
And only the integration phase would know about the dependencies of
blocks and how to arrange them.
When they talk about running again the unit tests, it is only for the
updated block(s), not the whole set.

*BUT*

The main issue is that to perform correctly, if a block X relies on the
blocks A, B & C, the unit tests for X would need replacement plug-ins
for A, B & C (which would emulate the expected behaviour).
Because it might be cheaper to use directly A, B & C, you end up with a
contaminated system: when updating block B, you now have to also run the
unit test for X.
And because the architecture sucks about documentation and enforced
isolation (you can never be sure... unless you are using a fascist
language in its most fascist mode, like C++ without "using namespace
std" and other restriction (no global, no singleton resources...)... or
Ada... but certainly not Java or php or perl (or VB)), it is simpler
(dumber) to rerun all the unit tests of all blocks.

It's all in the budget: aeroplane industries required the plug-ins for
tests (and made specifications for even testing the plug-ins!)... it has
an expensive initial cost. But it allows to test each element and
subsystems in a simulator... software industries goes often the
house-builder ways: fast, cheap to produce, will fix later if caught.
(and when you only have a hammer, everything looks like a nail). Using
army of monkey coders, the usual way is to require to run the whole set
of unit tests (using brute force instead of undocumented (and unchecked)
dependencies' graph). They even make tools to automate that.
(with the same quality!)

Post a reply to this message

On 08/06/2011 9:01 PM, Darren New wrote:
> Every work on unit testing I've read basically says you should run all
> the unit tests after each change, or certainly at least before each
> check-in.
>
> However, those same texts say that unit testing should only test
> individual units, and everything else should be mocked.
>
> Why would I want to run the unit tests for the XML parser after making
> changes to the color picker dialog box? Or, more generally, why would I
> want to run unit tests that are only executing code that hasn't changed
> since last time the unit tests passed?
>
> (Note I'm talking specifically of unit tests, not automated test suites
> in general.)
>

It doesn't make sense to me. If you're going to run all unit tests after 
one change then you might as well call it a system test.
It sounds as if someone is trying to cover their back. IMO It is not 
feasible to run all UT after a change. You would never get the 
implementation finished.

But then I've never read anything on the theory although I've 15 full 
implementations. As long as you do unit, system, integration and 
possibly regression testing before UAT then you will be OK. IMO.

-- 
Regards
     Stephen

Post a reply to this message

On 6/8/2011 13:27, Le_Forgeron wrote:
> When they talk about running again the unit tests, it is only for the
> updated block(s), not the whole set.

I have never seen that recommendation. It's begrudingly accepted as perhaps 
necessary if your unit tests take too long to run.

> *BUT*
>
> The main issue is that to perform correctly, if a block X relies on the
> blocks A, B&  C, the unit tests for X would need replacement plug-ins
> for A, B&  C (which would emulate the expected behaviour).

Yep.  The real problem with *this* is that you might update A and its tests, 
then run its unit tests and all goes well, and not realize you had broken X. 
  It never really seemed a good idea to me to be mocking stuff that's 
closely integrated without proper API documentation.

(I.e., mocking the config system that tells you where to load the XSD from 
when testing your XML parser would seem to be reasonable, given that the 
config system is probably well-isolated. Mocking the compiler's lexer while 
testing the parser or mocking the parser while testing the code generator 
never seemed to make much sense to me.)

> Because it might be cheaper to use directly A, B&  C, you end up with a
> contaminated system: when updating block B, you now have to also run the
> unit test for X.

Then that's integration testing.

> And because the architecture sucks about documentation and enforced
> isolation (you can never be sure...

OK. So basically it's "run all the unit tests because there might be 
integration tests in with the unit tests."

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

Am 08.06.2011 22:01, schrieb Darren New:
> Every work on unit testing I've read basically says you should run all
> the unit tests after each change, or certainly at least before each
> check-in.
>
> However, those same texts say that unit testing should only test
> individual units, and everything else should be mocked.
>
> Why would I want to run the unit tests for the XML parser after making
> changes to the color picker dialog box? Or, more generally, why would I
> want to run unit tests that are only executing code that hasn't changed
> since last time the unit tests passed?

(1) You only run the unit tests for those units you actually changed. 
But I guess you knew that.

(2a) If you /know/ beyond doubt that your code change in part X of a 
unit can't affect part Y of the same unit, then X and Y should probably 
be separate units.

(2b) If on the other hand you do /not/ know beyond doubt whether your 
code change in part X of a unit might affect part Y of the same unit, 
then obviously you should test part Y as well to be sure. But I guess 
that's no news to you either.

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> Why would I want to run the unit tests for the XML parser after making 
> changes to the color picker dialog box?  Or, more generally, why would I 
> want to run unit tests that are only executing code that hasn't changed 
> since last time the unit tests passed?

  No amount of unit tests can catch all possible bugs. You may have a
thousand unit tests for a small module, yet there may still be bugs there.

  In some cases these bugs *might* be triggered by seemingly unrelated
changes somewhere else in the program, even if there is no connection
between the two modules (might be less likely in "safe" languages, but
probably not impossible).

  Unless running the unit tests takes hours, does it hurt to run them?

-- 
                                                          - Warp

Post a reply to this message

On 6/9/2011 4:03, clipka wrote:
> (1) You only run the unit tests for those units you actually changed. But I
> guess you knew that.

Well, that's what I figured out. But everything I *read* says stuff like 
"run all unit tests before you check anything in."  I've never heard a 
recommendation that said it's *good* to run partial unit tests.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 6/9/2011 7:56, Warp wrote:
> Darren New<dne### [at] sanrrcom>  wrote:
>> Why would I want to run the unit tests for the XML parser after making
>> changes to the color picker dialog box?  Or, more generally, why would I
>> want to run unit tests that are only executing code that hasn't changed
>> since last time the unit tests passed?
>
>    No amount of unit tests can catch all possible bugs. You may have a
> thousand unit tests for a small module, yet there may still be bugs there.
>
>    In some cases these bugs *might* be triggered by seemingly unrelated
> changes somewhere else in the program, even if there is no connection
> between the two modules (might be less likely in "safe" languages, but
> probably not impossible).

I'd agree with this, except that unit tests are supposed to be independent. 
(Indeed, that's my main complaint about relying on unit tests to tell you if 
something is broken.) So if there's a bug in the XML parser, say, it 
shouldn't even be running in the same process as the unit tests for the 
color picker.

Now, unsafe languages, that's interesting. But I don't think that's a unit 
test thing as much as it's a test-the-hell-out-of-everything thing. Sure, 
the more you test code in different circumstances, the more you're likely to 
find uncaught violations of the language specification. I guess that makes 
sense, tho. That's the first good reason I've heard. :-)

And yes, something like not cleaning up a global might get caught in unit 
tests even in a safe language. Indeed, there are several places in XNA where 
static variables are used that should be instance variables. Indeed, the 
fact that calling "Exit()" during a game not only exits that game at the end 
of the current frame, but also all other future games instantiated in the 
same process, makes it very difficult to automate an entire set of tests in 
one process. :-)

>    Unless running the unit tests takes hours, does it hurt to run them?

No. But I've seen recommendations that one mocks everything *because* that 
makes the unit testing fast enough to test everything every time, as if this 
gave a clear benefit over just testing what you changed.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

Le 09/06/2011 17:11, Darren New a écrit :
> 
> No. But I've seen recommendations that one mocks everything *because*
> that makes the unit testing fast enough to test everything every time,
> as if this gave a clear benefit over just testing what you changed.

Well, the real issue with unit tests is: does the specification know
what to test ? really ?

Most unit tests are just "I checked that 3+3 still gives 6". Which is
only a basic test of nominal functionality. They usually fail on the
coverage of real unit test, due to lack of specification of all possible
issues.

For instance, in Java, you can redefine the value of 3 and 6. Which
means that if your patch keep the 3+3 -> 6, even if now 3 has a value of
14 and 6 has the value of 28, the unit tests would still be ok.

Would have the unit tests checked for some other intrinsics properties
of the module, it might have failed.

And to make matter worst: module often ends up at the top of a huge
pyramid of other modules (at best, when there is at least an ordering),
so emulating the other lower modules is only practical for the very few
lower modules at the base. The higher you go, the more expansive it
became to emulate at the specification time (which is also the time
where you should specify the tests) the interface needed to performed
isolated unit tests.
So they ends with a brute force approach.

-- 
A good Manager will take you
through the forest, no mater what.
A Leader will take time to climb on a
Tree and say 'This is the wrong forest'.

Post a reply to this message

On 6/10/2011 1:18, Le_Forgeron wrote:
> For instance, in Java, you can redefine the value of 3 and 6.

Of all the languages out there, I'm pretty sure Java isn't one of those. :-) 
  How do you redefine 3 to be 14?

> Would have the unit tests checked for some other intrinsics properties
> of the module, it might have failed.

Yeah. I much prefer pre- and post-conditions to be documented, etc. At least 
you can test them.

And you still wind up only testing what you think to test. Clearly, if MS 
had thought to test "what happens if we try to run two consecutive games in 
the same process", they would have noticed half a dozen things fail to work 
right the second time around. But hey, all the unit tests passed. :-)

> And to make matter worst: module often ends up at the top of a huge
> pyramid of other modules (at best, when there is at least an ordering),
> so emulating the other lower modules is only practical for the very few
> lower modules at the base.

Well, the mock frameworks seem to do an OK job, as long as you're willing to 
just specify what calls you expect the unit test to make. If you try to unit 
test something actually complex, it winds up just being equivalent to "run 
the program and compare the output with the previous run."

Unit tests really only work well to test your invariants, your pre- and 
post-conditions, on units that are fairly stand-alone. And it only works if 
you have an objective answer as to what the right result should be. You 
can't unit test for "this indeed looks like water coming out of a fountain."

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

Le 10/06/2011 16:45, Darren New a écrit :
> On 6/10/2011 1:18, Le_Forgeron wrote:
>> For instance, in Java, you can redefine the value of 3 and 6.
> 
> Of all the languages out there, I'm pretty sure Java isn't one of those.
> :-)  How do you redefine 3 to be 14?

three years late slogan (for the USA): Yes We Can...

http://thedailywtf.com/Articles/Disgruntled-Bomb-Java-Edition.aspx

basically:

import java.lang.*;
...
           Field field = Integer.class.getDeclaredField( "value" );
           field.setAccessible( true );
           int i=3;
           field.setInt(i,14);
...


Integer a = 2;
Integer b = 3;

System.out.println( a + b );



-- 
Software is like dirt - it costs time and money to change it and move it
around.

Just because you can't see it, it doesn't weigh anything,
and you can't drill a hole in it and stick a rivet into it doesn't mean
it's free.

Post a reply to this message

On 6/10/2011 8:05, Le_Forgeron wrote:
> Le 10/06/2011 16:45, Darren New a écrit :
>> On 6/10/2011 1:18, Le_Forgeron wrote:
>>> For instance, in Java, you can redefine the value of 3 and 6.
>>
>> Of all the languages out there, I'm pretty sure Java isn't one of thos
e.
>> :-)  How do you redefine 3 to be 14?
>
> three years late slogan (for the USA): Yes We Can...
>
> http://thedailywtf.com/Articles/Disgruntled-Bomb-Java-Edition.aspx
>
> basically:
>
> import java.lang.*;
> ...
>             Field field = Integer.class.getDeclaredField( "value" );
>             field.setAccessible( true );
>             int i=3;
>             field.setInt(i,14);
> ...
>
>
> Integer a = 2;
> Integer b = 3;
>
> System.out.println( a + b );

That's not redefining the value of 3 and 6. That's redefining the value o
f 
Integer(3) and Integer(6).  The expression "2 + 3" will still return 5.

Note that in Fortran and FORTH at least, it's pretty trivial to make the 

literal 3 refer to the value 15.  (By "pretty trivial" I mean "you might 
do 
it by accident.")

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 10/06/2011 04:12 PM, Darren New wrote:

> That's not redefining the value of 3 and 6. That's redefining the value
> of Integer(3) and Integer(6). The expression "2 + 3" will still return 5.
>
> Note that in Fortran and FORTH at least, it's pretty trivial to make the
> literal 3 refer to the value 15. (By "pretty trivial" I mean "you might
> do it by accident.")

I think Smalltalk will also let you accidentally redefine 3 to mean 
something else. (Certainly if you operate on array literals, it 
permanently changes the literal to refer to something other than what it 
says!)

Haskell will definitely let you define, say, 2+2=5. You'd be hard 
pressed to do this "by accident" though...

Post a reply to this message

On 6/10/2011 8:18, Invisible wrote:
> I think Smalltalk will also let you accidentally redefine 3 to mean
> something else.

I think anything the parser sees is hard-coded, but I could be wrong there.

> Haskell will definitely let you define, say, 2+2=5. You'd be hard pressed to
> do this "by accident" though...

And what does this result in?  Is it just that the expression "2+2" matches 
before "<integer> + <integer>" or some such, so 2+2=5 but 2+3=5 also?

"By accident" is pretty easy in fortran. All values, including literals, are 
passed by reference to a subroutine, so if the subroutine assigns to 
something that's a literal, you probably just changed the global value of 
that literal. (I.e., in most fortran implementations, "3" refers to an 
anonymous global variable that holds the value "3".)

In FORTH, you just write
     : 2 3 ;
So, "the subroutine named 2 now returns 3".  The way the parser works is 
that if it can't find the name of a routine, it then tries to parse it as a 
literal. But if it finds the name of a function, it uses it. Not something 
you're likely to do by accident unless you mess up some metaprogramming.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

Invisible <voi### [at] devnull> wrote:
> Haskell will definitely let you define, say, 2+2=5. You'd be hard 
> pressed to do this "by accident" though...

  And some people criticize C++ because it supports operator overloading,
which could potentially be used to make operators do something else than
what they seem...

-- 
                                                          - Warp

Post a reply to this message

On 10/06/2011 05:27 PM, Warp wrote:
> Invisible<voi### [at] devnull>  wrote:
>> Haskell will definitely let you define, say, 2+2=5. You'd be hard
>> pressed to do this "by accident" though...
>
>    And some people criticize C++ because it supports operator overloading,
> which could potentially be used to make operators do something else than
> what they seem...

Yeah, I always thought that was a fairly stupid critisism...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

On 10/06/2011 04:49 PM, Darren New wrote:
> On 6/10/2011 8:18, Invisible wrote:
>> I think Smalltalk will also let you accidentally redefine 3 to mean
>> something else.
>
> I think anything the parser sees is hard-coded, but I could be wrong there.

I did something like define a method that returns #[1 2 3], and then I 
took the result and manipulated it. To my surprise, the method now 
returns the new, manipulated value, instead of what it claims to return. (!)

Apparently you shouldn't do that.

>> Haskell will definitely let you define, say, 2+2=5. You'd be hard
>> pressed to do this "by accident" though...
>
> And what does this result in? Is it just that the expression "2+2"
> matches before "<integer> + <integer>" or some such, so 2+2=5 but 2+3=5
> also?

You can define a new number type, and have it implement arithmetic any 
way you desire. This works because "2" is not an integer. It's a number. 
It's *any* number type.

As insane as that sounds, I have actually /used/ this myself. I 
implemented a type called HalfInteger, which supports numbers rounded to 
the nearest 1/2. So if you say "7", it really stores it as 14. And if 
you say "half", that's really one. And if you do "7 + half", you get 15. 
Ah, but of course, it doesn't /look/ like 15. It looks like 7.5 (via the 
various interfaces that let you inspect it).

The fun part is deciding what the result of half * half should be...

> (I.e., in most fortran implementations, "3" refers to
> an anonymous global variable that holds the value "3".)

Isn't that almost exactly what Smalltalk does with [at least] class names?

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

On 6/10/2011 10:23, Orchid XP v8 wrote:
> On 10/06/2011 04:49 PM, Darren New wrote:
>> On 6/10/2011 8:18, Invisible wrote:
>>> I think Smalltalk will also let you accidentally redefine 3 to mean
>>> something else.
>>
>> I think anything the parser sees is hard-coded, but I could be wrong there.
>
> I did something like define a method that returns #[1 2 3], and then I took
> the result and manipulated it. To my surprise, the method now returns the
> new, manipulated value, instead of what it claims to return. (!)
>
> Apparently you shouldn't do that.

Especially not in a functional language? That's exactly the sort of thing a 
pure functional language is guaranteed not to do. :-)

> You can define a new number type, and have it implement arithmetic any way
> you desire. This works because "2" is not an integer. It's a number. It's
> *any* number type.

I see. So it's not really the integer 2, but the lexical symbol 2 that 
you're talking about.

>> (I.e., in most fortran implementations, "3" refers to
>> an anonymous global variable that holds the value "3".)
>
> Isn't that almost exactly what Smalltalk does with [at least] class names?

I'm not sure what you're talking about. Are you talking about interred 
strings? They're not really *quite* the same thing. FORTRAN only does that 
because most machines existing when FORTRAN was created didn't have 
immediate-mode machine instructions.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 6/10/2011 9:27, Warp wrote:
>    And some people criticize C++ because it supports operator overloading,

I admit, I lolled.

However, I think it's better to go way over the top than (say) to support 
operator overloading without supporting literal overloading. The ability to 
define "*" on complex numbers without the ability to write complex numbers 
inline is understandable but feels kludgey to me.

(Yes, I know for a great many types you can essentially write them inline in 
C++. I'm not necessarily talking about C++.)

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

>> I did something like define a method that returns #[1 2 3], and then I
>> took
>> the result and manipulated it. To my surprise, the method now returns the
>> new, manipulated value, instead of what it claims to return. (!)
>>
>> Apparently you shouldn't do that.
>
> Especially not in a functional language? That's exactly the sort of
> thing a pure functional language is guaranteed not to do. :-)

Yeah, well, Smalltalk is no functional language. ;-)

>> You can define a new number type, and have it implement arithmetic any
>> way
>> you desire. This works because "2" is not an integer. It's a number. It's
>> *any* number type.
>
> I see. So it's not really the integer 2, but the lexical symbol 2 that
> you're talking about.

Basically, yes. I can make it so that when you type an integer or a 
decimal, it resolves to something utterly unexpected.

Of course, monads let you do strange things to what look like simple 
imperative statements. (E.g., time travel, quantum superposition, etc.) 
Strangely, the Haskell list syntax is so hard-wired that you can't touch 
it, however.

>>> (I.e., in most fortran implementations, "3" refers to
>>> an anonymous global variable that holds the value "3".)
>>
>> Isn't that almost exactly what Smalltalk does with [at least] class
>> names?
>
> I'm not sure what you're talking about.

When you define a class named Foo, it defines a global variable named 
Foo which contains a Class object describing the class.

The fun thing is, since True and False are classes, they are also global 
variables, and you can do something like True := 42, which utterly 
confuses everything.

Then again, Smalltalk := Nil is always fun...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

On 6/10/2011 11:26, Orchid XP v8 wrote:
> Yeah, well, Smalltalk is no functional language. ;-)

Oh, sorry, I thought we were still talking about Haskell.

Yeah, the thing is when you compile the expression #[1, 2, 3] it creates a 
list and points the executable code to that, or some such.

The same thing happens with Python default arguments.

> When you define a class named Foo, it defines a global variable named Foo
> which contains a Class object describing the class.

Yes. But "global variable" is a bit of a misnomer in a language where global 
variables are simply names in a hashtable that the compiler looks up at runtime.

> The fun thing is, since True and False are classes, they are also global
> variables, and you can do something like True := 42, which utterly confuses
> everything.

I think true, false, and nil are usually special-cased in the compiler 
(along with ifTrue:ifFalse:, etc) for performance reasons. But yes, in 
theory, you could do that.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 10/06/2011 09:11 PM, Darren New wrote:
> On 6/10/2011 11:26, Orchid XP v8 wrote:
>> Yeah, well, Smalltalk is no functional language. ;-)
>
> Oh, sorry, I thought we were still talking about Haskell.

Uh, yeah. You can't do anything crazy like that in Haskell.

Worst thing you can do is unsafePerformIO. Also known as "tell the 
compiler that its assumptions hold for this expression, because I say 
so". Naturally, if you are mistaken... bad things happen.

> Yeah, the thing is when you compile the expression #[1, 2, 3] it creates
> a list and points the executable code to that, or some such.

That's the one.

>> When you define a class named Foo, it defines a global variable named Foo
>> which contains a Class object describing the class.
>
> Yes. But "global variable" is a bit of a misnomer in a language where
> global variables are simply names in a hashtable that the compiler looks
> up at runtime.

Not just the compiler. The entire runtime system. And since the IDE *is* 
the runtime system...

> I think true, false, and nil are usually special-cased in the compiler
> (along with ifTrue:ifFalse:, etc) for performance reasons. But yes, in
> theory, you could do that.

It probably varies by implementation. I suppose I could dig out my copy 
of VisualWorks if I /really/ cared...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message