Orchid XP v8 wrote:
> Man, that's one hell of a complicated system!

How long does it take to run both compilation stages followed by all the 
unit tests?

...Chambers

Post a reply to this message

Orchid XP v8 <voi### [at] devnull> wrote:
> The guys working on the GHC compiler do this. The compiler's source 
> repository contains an entire directory full of code snippets.

  AFAIK the gcc project also has a gigantic set of automated tests, so
that every new release is first run against it. Every time a legit bug
is reported, a test triggering that bug is added to the set.

-- 
                                                          - Warp

Post a reply to this message

>> The guys working on the GHC compiler do this. The compiler's source 
>> repository contains an entire directory full of code snippets.
> 
>   AFAIK the gcc project also has a gigantic set of automated tests, so
> that every new release is first run against it. Every time a legit bug
> is reported, a test triggering that bug is added to the set.

It would seem the only sensible way to work on a project as huge as GHC 
or GCC, yes.

Given how much more popular GCC is than GHC, I imagine the test set 
is... kinda huge by now.

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

>> Man, that's one hell of a complicated system!
> 
> How long does it take to run both compilation stages followed by all the 
> unit tests?

How fast is your PC?

It's kind of like asking how long it takes to render SKYVASE.POV; it 
depends. ;-)

On my ancient laptop, with it's single-core 32-bit processor clocked at 
less than 2 GHz, it took about 2 hours:

30 minutes to compile stage1 using stage0 + stage0-libs.
30 minutes to compile stage1-libs using stage1.
30 minutes to compile stage2 using state1 + stage1-libs.
15 minutes building the documentation files. (They're embedded in the 
source code as comments.)
45 minutes to run all tests.

You can check the output of the actual buildbots yourself if you like:

http://darcs.haskell.org/buildbot/

It's not very intuitive, but... looks like the test suit contains about 
2,000 files, when produce 9,000 tests (taking into account that some 
tests can be run multiple ways - through the compiler, through the 
interpretter, with debugging on, with debugging off, with profiling on, 
etc.)

Looks like the whole process takes several hours on the buildbots too.

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Warp wrote:
>   I myself am quite fond of putting assert()s everywhere where it makes
> sense. That has helped me catch tons of bugs which would have otherwise
> been quite laborious to find later.

Me too. I think what I understood the articles to be saying is that this is 
very helpful, but only if you are experienced enough to use the right assert()s.

>   Another thing which I cannot stress enough is the importance of developing,
> if possible (ie. if the program/library type is suitable for it), an
> automatic regression testing program.

Sadly, the vast majority of my work these last few years is completely 
antithetic to this sort of testing.

I did this even for programs running off floppies in 64K, where I'd save 
inputs and outputs, and if the input no longer generated the same output, 
I'd manually eyeball it to see if the change I just made should have created 
that output, and if so, save the new output.  That was helpful.

>   One project where I have done this is with my function parser library.

Some programs lend themselves to that, some don't.

Did you do this with Hopido?  What parts could you test?

-- 
   Darren New, San Diego CA, USA (PST)
   I ordered stamps from Zazzle that read "Place Stamp Here".

Post a reply to this message

Orchid XP v8 wrote:
> Man, that's one hell of a complicated system! Much bigger than anything 
> I'll ever work on. 

I saw a better one. When someone made a change, it runs thru the smoke 
tests, and then gets pushed up to about 5% to 10% of the web servers. If 
after 15 minutes the web servers haven't reported any statistically unusual 
number of script failures, it gets pushed out to more and more servers. So 
they do a deploy on their production system every 30 to 60 minutes during 
the workday.

-- 
   Darren New, San Diego CA, USA (PST)
   I ordered stamps from Zazzle that read "Place Stamp Here".

Post a reply to this message

Warp wrote:
> Orchid XP v8 <voi### [at] devnull> wrote:
>> The guys working on the GHC compiler do this. The compiler's source 
>> repository contains an entire directory full of code snippets.
> 
>   AFAIK the gcc project also has a gigantic set of automated tests, so
> that every new release is first run against it. Every time a legit bug
> is reported, a test triggering that bug is added to the set.

It's not hard to organize for big batch process type things with 
well-defined inputs and outputs. To the extent that you can turn difficult 
problems into collections of functional input->output streams, you can apply 
this to messier problems.

-- 
   Darren New, San Diego CA, USA (PST)
   I ordered stamps from Zazzle that read "Place Stamp Here".

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> Did you do this with Hopido?  What parts could you test?

  Interactive graphical programs are quite a lot harder to test automatically
than libraries which take some input and produce some well-defined output as
a result.

  With a library (like a function parser) automatic testing is relatively
easy: The testing program simply uses the library, feeds it some input and
checks what the library returns to see if it's correct (or, in the case of
the function parser, within acceptable limits, as we are talking about
floating point numbers after all, so bit-by-bit equivalence between what
the compiler produces and what the library calculates cannot be expected).

  With an interactive graphical program such automatic testing is not that
easy. You can't simply write a program which calls some functions and checks
what those functions return. You would have to programmatically emulate user
interaction, and somehow check that the program does the right thing.
There's a million things which could theoretically go wrong, and most of
them are things which are very hard if not impossible to check
programmatically (eg. visual glitches, wrong timings, etc.)

  There's also the question of deadlines. Writing a fully automated test
for an interactive graphical program could at worst take even more time
than implementing the program itself, and that's often just not an option,
especially in a commercial project. The cost-benefit of such testing scheme
would be dubious.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:

>   With an interactive graphical program such automatic testing is not that
> easy. You can't simply write a program which calls some functions and checks
> what those functions return. You would have to programmatically emulate user
> interaction, and somehow check that the program does the right thing.
> There's a million things which could theoretically go wrong, and most of
> them are things which are very hard if not impossible to check
> programmatically (eg. visual glitches, wrong timings, etc.)

I suppose you could throw a bunch of user events at it and check that it 
doesn't fail any asserts. Doesn't prove it's working right, but it's a 
start.

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> I suppose you could throw a bunch of user events at it and check that it 
> doesn't fail any asserts. Doesn't prove it's working right, but it's a 
> start.

Yeah. Lots of the "bugs" I find are things just not following the specs. 
It's hard to know in advance (for example) if some randomly mal-formed web 
page will "look wrong" when you try to render it, or even what to do about it.

Testing doesn't tell you if your requirements are incomplete or incorrect, 
which is I think the primary place my bugs come from.

-- 
   Darren New, San Diego CA, USA (PST)
   I ordered stamps from Zazzle that read "Place Stamp Here".

Post a reply to this message