![](/i/fill.gif) |
![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx
--
Darren New, San Diego CA, USA (PST)
I ordered stamps from Zazzle that read "Place Stamp Here".
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Warp
Subject: Re: Pretty cool stuff for professional programmers
Date: 10 Oct 2009 08:22:33
Message: <4ad07c89@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Darren New <dne### [at] san rr com> wrote:
> http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx
I myself am quite fond of putting assert()s everywhere where it makes
sense. That has helped me catch tons of bugs which would have otherwise
been quite laborious to find later.
Another thing which I cannot stress enough is the importance of developing,
if possible (ie. if the program/library type is suitable for it), an
automatic regression testing program.
This would simply be a program which automatically tests the validity of
the program or library being developed. In other words, it runs it against
a set of test data and checks that the results are correct (or within
acceptable margin if the result is expected to be fuzzy). What makes it
regression testing is that every time a new feature is added to the
program/library, a test or tests are added to the testing program and,
most importantly, every time a bug is found in the program/library, a
test which triggers the bug is added to the testing program.
Thus every time a change is made to the program/library, the automatic
testing program can be run against it, and a huge amount of tests will
be performed automatically and easily, catching most bugs right there.
One project where I have done this is with my function parser library.
I have been developing it for something like 10 years, and during those
years I have also developed an automatic regression testing program
alongside it. It has grown to be quite large and extensive during the
years, and it works phenomenally. I'd say at least 90% of all bugs are
caught immediately by just running the testing program. That's a HUGE
timesaver.
--
- Warp
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Orchid XP v8
Subject: Re: Pretty cool stuff for professional programmers
Date: 10 Oct 2009 08:52:45
Message: <4ad0839d@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Warp wrote:
> I myself am quite fond of putting assert()s everywhere where it makes
> sense. That has helped me catch tons of bugs which would have otherwise
> been quite laborious to find later.
This is probably something I should do more of...
Haskell makes it possible to prevent many things at compile-time (e.g.,
no null pointers, no empty lists, etc.), but there will always be things
you can only check at runtime.
> Another thing which I cannot stress enough is the importance of developing,
> if possible (ie. if the program/library type is suitable for it), an
> automatic regression testing program.
The Haskell guys have a tool they call QuickCheck, which (with a little
help from you) will randomly generate X tests and check the results.
Obviously this isn't the same as having a set of hand-written tests. If
you're trying to write a parser of some kind, you probably want
hand-written test strings to check out the edge-cases of the parser.
But, for example, a while back I was extending Haskell's binary parser
library to support bit-level alignment. Having QuickCheck autogenerate
half a trillion random bitstrings, write then and then read them back
and check they haven't changed is a pretty rapid way to find out that
your program is broken.
> What makes it
> regression testing is that every time a new feature is added to the
> program/library, a test or tests are added to the testing program and,
> most importantly, every time a bug is found in the program/library, a
> test which triggers the bug is added to the testing program.
>
> Thus every time a change is made to the program/library, the automatic
> testing program can be run against it, and a huge amount of tests will
> be performed automatically and easily, catching most bugs right there.
The guys working on the GHC compiler do this. The compiler's source
repository contains an entire directory full of code snippets. Some of
them are supposed to be rejected with a compile-time error, some of them
are meant to crash at runtime, and some of them are supposed to run
perfectly and produce a particular output. GHC includes a compiler and
an interpretter, and the interpretter can run source code and object
code, so there's at least 3 seperate ways to run a given program. Using
the test framework, you can check that the program produces the same
result when run all three ways, etc.
Every night, a set of "build bots" sync their repos to the central tree
and try to recompile the entire GHC system from scratch. (By that I mean
build the compiler, the interpretter, the standard libraries, the
runtime system...) This is a two-stage process; first, the source code
for the compiler is compiled using an old, old version of GHC, with half
the features turned off. Then the source code is compiled again with the
new compiler just created (and all features turned on). Assuming all
that works (i.e., it doesn't crash somewhere with Haskell syntax errors
or undefined linker symbols or C preprocessor glitches), the build bot
runs the "validate" script. This runs all XXX-thousand test programs and
checks that invalid programs get rejected, valid ones get accepted, each
program produces the output it should, running it in all possible ways,
etc. There's build bots for a dozen different flavours of Linux,
Windows, Mac OS... you name it.
In short, if somebody commits a patch that breaks something, you'll know
about it in 24 hours when all the build bots start reporting errors.
Also, like Warp says, every time somebody submits a bug to the bug
tracker [which turns out to be a valid bug, not something already known
about, etc.], they try to add a test case for it to the test suite.
On top of all that, there's a series of performance benchmarks - for the
compiler itself and for the programs it compiles...
Man, that's one hell of a complicated system! Much bigger than anything
I'll ever work on. Still, you can see why so much test infrastructure is
necessary. A compiler is a complex thing. I imagine similar projects do
something like this too.
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Chambers
Subject: Re: Pretty cool stuff for professional programmers
Date: 10 Oct 2009 12:16:40
Message: <4ad0b368@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Orchid XP v8 wrote:
> Man, that's one hell of a complicated system!
How long does it take to run both compilation stages followed by all the
unit tests?
...Chambers
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Warp
Subject: Re: Pretty cool stuff for professional programmers
Date: 10 Oct 2009 12:30:52
Message: <4ad0b6bc@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Orchid XP v8 <voi### [at] dev null> wrote:
> The guys working on the GHC compiler do this. The compiler's source
> repository contains an entire directory full of code snippets.
AFAIK the gcc project also has a gigantic set of automated tests, so
that every new release is first run against it. Every time a legit bug
is reported, a test triggering that bug is added to the set.
--
- Warp
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Orchid XP v8
Subject: Re: Pretty cool stuff for professional programmers
Date: 10 Oct 2009 12:50:04
Message: <4ad0bb3c$1@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
>> The guys working on the GHC compiler do this. The compiler's source
>> repository contains an entire directory full of code snippets.
>
> AFAIK the gcc project also has a gigantic set of automated tests, so
> that every new release is first run against it. Every time a legit bug
> is reported, a test triggering that bug is added to the set.
It would seem the only sensible way to work on a project as huge as GHC
or GCC, yes.
Given how much more popular GCC is than GHC, I imagine the test set
is... kinda huge by now.
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Orchid XP v8
Subject: Re: Pretty cool stuff for professional programmers
Date: 10 Oct 2009 13:03:12
Message: <4ad0be50$1@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
>> Man, that's one hell of a complicated system!
>
> How long does it take to run both compilation stages followed by all the
> unit tests?
How fast is your PC?
It's kind of like asking how long it takes to render SKYVASE.POV; it
depends. ;-)
On my ancient laptop, with it's single-core 32-bit processor clocked at
less than 2 GHz, it took about 2 hours:
30 minutes to compile stage1 using stage0 + stage0-libs.
30 minutes to compile stage1-libs using stage1.
30 minutes to compile stage2 using state1 + stage1-libs.
15 minutes building the documentation files. (They're embedded in the
source code as comments.)
45 minutes to run all tests.
You can check the output of the actual buildbots yourself if you like:
http://darcs.haskell.org/buildbot/
It's not very intuitive, but... looks like the test suit contains about
2,000 files, when produce 9,000 tests (taking into account that some
tests can be run multiple ways - through the compiler, through the
interpretter, with debugging on, with debugging off, with profiling on,
etc.)
Looks like the whole process takes several hours on the buildbots too.
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Darren New
Subject: Re: Pretty cool stuff for professional programmers
Date: 10 Oct 2009 13:15:57
Message: <4ad0c14d$1@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Warp wrote:
> I myself am quite fond of putting assert()s everywhere where it makes
> sense. That has helped me catch tons of bugs which would have otherwise
> been quite laborious to find later.
Me too. I think what I understood the articles to be saying is that this is
very helpful, but only if you are experienced enough to use the right assert()s.
> Another thing which I cannot stress enough is the importance of developing,
> if possible (ie. if the program/library type is suitable for it), an
> automatic regression testing program.
Sadly, the vast majority of my work these last few years is completely
antithetic to this sort of testing.
I did this even for programs running off floppies in 64K, where I'd save
inputs and outputs, and if the input no longer generated the same output,
I'd manually eyeball it to see if the change I just made should have created
that output, and if so, save the new output. That was helpful.
> One project where I have done this is with my function parser library.
Some programs lend themselves to that, some don't.
Did you do this with Hopido? What parts could you test?
--
Darren New, San Diego CA, USA (PST)
I ordered stamps from Zazzle that read "Place Stamp Here".
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Darren New
Subject: Re: Pretty cool stuff for professional programmers
Date: 10 Oct 2009 13:18:38
Message: <4ad0c1ee@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Orchid XP v8 wrote:
> Man, that's one hell of a complicated system! Much bigger than anything
> I'll ever work on.
I saw a better one. When someone made a change, it runs thru the smoke
tests, and then gets pushed up to about 5% to 10% of the web servers. If
after 15 minutes the web servers haven't reported any statistically unusual
number of script failures, it gets pushed out to more and more servers. So
they do a deploy on their production system every 30 to 60 minutes during
the workday.
--
Darren New, San Diego CA, USA (PST)
I ordered stamps from Zazzle that read "Place Stamp Here".
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
From: Darren New
Subject: Re: Pretty cool stuff for professional programmers
Date: 10 Oct 2009 13:20:53
Message: <4ad0c275$1@news.povray.org>
|
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Warp wrote:
> Orchid XP v8 <voi### [at] dev null> wrote:
>> The guys working on the GHC compiler do this. The compiler's source
>> repository contains an entire directory full of code snippets.
>
> AFAIK the gcc project also has a gigantic set of automated tests, so
> that every new release is first run against it. Every time a legit bug
> is reported, a test triggering that bug is added to the set.
It's not hard to organize for big batch process type things with
well-defined inputs and outputs. To the extent that you can turn difficult
problems into collections of functional input->output streams, you can apply
this to messier problems.
--
Darren New, San Diego CA, USA (PST)
I ordered stamps from Zazzle that read "Place Stamp Here".
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |