Imagine the unimaginable: Try to picture the worst possible software 
codebase. The kind of thing that only exists in your most darkest 
nightmares, the ones where you wake up screaming. The sort of code that 
sends people stark raving mad.

You know the kind of thing I'm talking about. Code written by people who 
have no God-damned idea what the hell they're doing. The kind of people 
who just type stuff and then hit it until it works (for sufficiently low 
value of "works").

There are no comments. There is only a deeply-nested tangle of folders 
with randomly-generated names, containing randomly-named files that 
range in size from bytes to megabytes. Each file contains a selection of 
randomly-named constants, global variables and functions.

There is no separation of concerns. No modularity. The horrors include 
the function with 26 lines for converting lower-case letters to 
upper-case, plus another 26 lines to redundantly map the upper-case 
letters to themselves. The 25-line "negate double" function. The 
function that generates "unique" IDs by trivially manipulating existing 
ones, and retrying if a clash occurs.

Now imagine that instead of just /one/ broken upper-case function, there 
are /sixteen copies/ of this thing, all doing upper-case conversion on a 
different subset of the Latin and perhaps Greek alphabets. Twenty five 
different functions which all incorrectly percent-encode URLs in a 
different incorrect manner. Four copies of that weird function that you 
don't know what it does yet, all of them identical.

Best of all, during the development process, the code has changes so 
many times that large parts of the codebase are actually DEAD CODE! Huge 
chunks of it are EVER ACTUALLY RUN. They used to be, but no longer are. 
Good luck figuring out which bits those are.

The programmer didn't read the documentation for the tools and libraries 
he's using. (He probably wouldn't understand it anyway!) As such, the 
software actually /depends on/ all the undocumented, undefined or just 
plain erroneous behaviour of the compiler, the runtime and the libraries.

This thing is a recursive nightmare. Makefiles that run scripts which 
write makefiles that build other scripts that run compilers to compile 
programs that write source code. CPP macros everywhere. Hell, some files 
don't even /compile/ - and the [auto-generated] makefiles actually 
/depend on/ this behaviour in order to function correctly.

The code uses pointer arithmetic to access stuff, and as such depends on 
the exact ordering of stack frames, heap placement, malloc behaviour and 
God only knows what else. Compiled code is modified at runtime. Some of 
the machine opcodes are also data constants; if the compiler ever 
decided to move the code around, it would stop working.



When you've finished screaming in abject terror, let me tell you the 
worst part: This application wasn't just /written/. It was in active 
development for many decades. But apparently Mr Incompetent didn't know 
about version control either. So each time a customer purchases the 
software, they get a copy of the source tree as it exists on the 
development system on that particular day. And every time they have a 
problem, Mr Incompetent drives out to the customer site and fixes /their 
copy of/ the software.

In case you haven't come across this scenario, let me spell out the dire 
consequences of this development pattern: EVERY SINGLE CUSTOMER has a 
completely unique, one-of-a-kind variant of the software. Each one has 
its own completely unique set of bugs. Each one has different functions 
with different names in different files to do the same job in slightly 
different ways.

At customer 5, file 1678981 runs payroll. But at customer 6, that file 
doesn't exist, at customer 7 that file's contents are completely 
different and do something unrelated, and customer 9 has that file but 
it's dead code; on /that/ system, files 57781 and 8756494 contain a 
completely different implementation of the same thing. (Which one gets 
used depends on which screen you access the feature from. No, they 
aren't implemented the same way, nor do they give the same results. 
There's a workaround for that in file 778456 which tries to make the 
results look similar.)



Are you trembling in catatonic hysteria yet? Then let me tell you the 
worst part: The software is /really really popular/. Sure, it doesn't 
actually /work/ very well, but it's a completely mission-critical part 
of every customer's infrastructure, and /must/ be supported. And the 
range of functionality it provides and the complexity of the workarounds 
the customers have invented to deal with it make replacing or rewriting 
it unthinkable.



OK, so that's pretty much the work Daily WTF nightmare imaginable. Of 
course, nobody /actually/ writes software like that, right?

Right??

Well, you know what? There *is* software out there that's like this. 
Almost everything I've just written accurately describes THE HUMAN 
GENOME. Or, indeed, ANY GENOME. "Bashing it until it works" is almost 
/literally/ how evolution by natural selection works.

Now do you understand why even though we've got the entire human genome, 
nobody has figured out how it works yet? :-P

Of course, for a genome the problem is far, far worse. A computer 
program directs a computer to do some computation. A genome directs a 
cell to actually BUILD MORE CELLS. Change the program, and you can 
ACTUALLY CHANGE THE HARDWARE!

In case you think I'm talking theoretically, consider that in most 
organisms, the DNA sequence "UAG" (uracil, adenine, guanine) usually 
means "stop", but in certain bacteria it means "pyrrolysine" instead. 
Pyrrolysine is an amino acid similar to lysine, but not found in most 
organisms. Only a few species create it, and only in these species does 
the UAG code have this special, altered meaning.

http://en.wikipedia.org/wiki/Pyrrolysine

It's almost like writing a computer program that adds a new opcode to 
the processor. (If you know what "microcode" is, this shouldn't sound so 
far-fetched...)

This isn't even the only instance. Humans use an amino acid involving 
selenium via post-transcriptional modification, for example. There's no 
special DNA sequence for it; rather, there's a marker that makes the RNA 
strand tangle up in a specific way, which causes an enzyme to modify the 
existing amino acid by adding the selenium. That's like modifying your 
processor so that the effect of one opcode depends on another opcode 
somewhere else in the executable!

Consider also the thing I said about there being multiple copies of 
every function, some of them no longer in use. Humans have (IIRC) 14 
copies of the haemoglobin gene. 6 of these copies are broken. (Some of 
them have a letter or two that's wrong, while others have huge chunks of 
code completely missing or duplicated.) Of the remainder, several are 
completely identical, some are slightly different but make the same 
chemical (e.g., CUC and CUA both mean leucine), some make different 
chemicals which still function the same way, and there's a couple that 
make different chemicals that actually work differently.

[I don't remember the numbers off hand. It's /something like/ 14 copies, 
but I don't have the book in front of my right now.]

For example, the blood of a fetus contains a slightly different type of 
haemoglobin, using one of the gene copies which is only activated in a 
fetus. In adults, this gene is never switched on. This version of 
haemoglobin is almost exactly the same as the [several] normal 
version[s] that adults have, except it binds to oxygen slightly more 
strongly.

As a direct result of this, when the fetal blood supply comes close to 
the maternal blood supply in the placenta, oxygen migrates from the 
mother to the fetus, AND NOT THE OTHER WAY AROUND. If they both had the 
same affinities, there wouldn't be much of a transfer. That would be fatal.

(So how the hell did it evolve in the first place, if not having it is 
fatal? Well, for a smaller animal, or one with a lower metabolic rate, 
it wouldn't be fatal. Just a disadvantage. It only becomes critical for 
larger animals with higher metabolism.)

As if that wasn't enough, haemoglobin isn't just used for oxygen 
transport! It's also used as an antioxidant, and for temporary iron 
storage within cells. (Remember what I said about separation of 
concerns? Natural selection doesn't do that.)

As another example, consider that the red-sensitive pigment of the eye 
is completely different from the blue one, but the green-sensitive 
pigment is almost identical to the red one, with just a few 
modifications. Clearly the gene got accidentally duplicated, and mutated 
over time. Apparently some animals see 6 colours or more. (Note also 
that the wavelengths of "red" and "green" are way closer to each other 
than for "blue". Three guesses why...)

In a sense, what scientists are trying to do is even /harder/ than 
figuring out what a really buggy program does. They're trying to figure 
out which customer got which copy of what software from whom at what 
point in time, and what modifications were done after that. In other 
words, track Mr Incompetent's movements and figure out the linage of all 
the different copies of the software. Jesus that's hard!

Post a reply to this message

It's even worse than that: each customer not only get a full different
program, but installed it (with variations) on billions of computers
(most cells get a full copy, initially).

As cells divide, each copy mutates and get truncated or duplicated code.
Statically, it's the same program... in theory. In practice, the
survival is only based on the ability of the new program to still
duplicate the cell.

-- 
Software is like dirt - it costs time and money to change it and move it
around.

Just because you can't see it, it doesn't weigh anything,
and you can't drill a hole in it and stick a rivet into it doesn't mean
it's free.

Post a reply to this message

On 04/08/2011 01:53 PM, Le_Forgeron wrote:
> It's even worse than that: each customer not only get a full different
> program, but installed it (with variations) on billions of computers
> (most cells get a full copy, initially).
>
> As cells divide, each copy mutates and get truncated or duplicated code.
> Statically, it's the same program... in theory. In practice, the
> survival is only based on the ability of the new program to still
> duplicate the cell.

As best as I can tell, mutations are actually very rare events. There 
are specific mechanisms which exist expressly to detect and correct such 
defects. That includes DNA repair mechanisms, automated destruction of 
misfolded proteins, and extracellular signals used to program mutated 
cells to shut down and die.

Cancer happens when these mechanisms fail to correct a mutation /and/ 
that mutation happens to be do to with growth processes. Observe that 
cancer is quite rare. (E.g., it's not like 50% of people get cancer or 
anything like that.)

So it's not like every single cell in your body has a completely 
different copy of your genome. Actually the differences are tiny, if 
existent at all. The major place where these mutations become 
significant is during reproduction, where the entire organism passes 
through a single genome copy. If /that/ gets changed, it's probably 
permanent.

Post a reply to this message

That is sort of the software engineer's view. You can describe DNA as if 
it is a program, but that misses most of the details.
Sometimes I wish I was 18 again today, so I could absorb enough facts 
about how cells and organisms work to help advance the field. I am now 
too old to understand a large enough portion to have any significant 
contribution :(

Yet, I still try to keep up with the field as much as I can, if only to 
be able to understand the guys and gals further up the corridor.

I do know that if I had to create life I would have done it more logical 
and maintainable. But, that would probably not have survived for more 
than a few generations.

Post a reply to this message

On 08/04/2011 09:22 AM, Invisible wrote:
> The programmer didn't read the documentation for the tools and libraries
> he's using. (He probably wouldn't understand it anyway!) As such, the
> software actually /depends on/ all the undocumented, undefined or just
> plain erroneous behaviour of the compiler, the runtime and the libraries.

Sounds a bit like the early days of squashfs (re: my previous post on 
the topic) ... I never could get the tools to compile correctly for what 
can only be some of the very issue's that you've raised. What ended up 
solving the problem was building a VM, installing an older kernel 
(major/minor number change for the fs type), mount the filesystem, then 
copy to a alternate location, insert the fix, and remake the squashfs 
with tools compiled with a more up to date compiler, libraries ... etc. 
I was ready to scream before it was over and done with! LOL ... there's 
a lot to be said for doing things old school. Lot of poorly executed, 
and not very well thought out on how it's going to be supported, 
software out there these days.

Post a reply to this message

On 04/08/2011 03:48 PM, andrel wrote:
> That is sort of the software engineer's view. You can describe DNA as if
> it is a program, but that misses most of the details.

If you read most books about it, they will tell you that "hey, DNA has 
these 4 letters, and they go in groups of 3, and this table here shows 
you for every 3-letter word which animo acid it selects, and that's how 
you make proteins!"

Which is basically true. But this isn't something some engineer 
designed. This is a crazy-ass thing that happened by accident! Suffice 
to say, it's way more complicated than that.

You know the bit I said about compiled code being modified at runtime? 
It turns out that by attaching (or not attaching) various molecules to a 
DNA chain, you can activate or deactivate various genes. So DNA isn't 
just the program. In some sense it's also part of the cell state or 
"memory". Sometimes I wonder whether a "naked" copy of a given animal's 
genome would actually function at all; even at conception, there are 
probably markers turning genes on or off, to set the program in motion 
from the correct start point.

Then there are the knots and tangles in DNA or RNA that affect the way 
it is processed. For example, human mRNA with a certain type of knot in 
it causes certain amino acids to have selenium added to them after 
transcription. So the whole "codons code amino acids" is a vast 
simplification.

Let us not even get into the fact that many proteins are modified after 
being constructed, some fold up as they are transcribed (so trying to 
simulate folding of the complete, finished protein is a doomed 
endeavour), one require "chaperone" proteins to guide their folding in 
the correct direction, many proteins are synthesized inactive and have 
to be "activated" before they will do anything. Recent research even 
suggests that "cellular crowding" may be important for some protein 
activities.

http://en.wikipedia.org/wiki/Cellular_crowding

On top of all that, it turns out that RNA itself can be both a data 
carrier like DNA *and* a catalyst like proteins. Wrap your head around 
that one!

> Sometimes I wish I was 18 again today, so I could absorb enough facts
> about how cells and organisms work to help advance the field. I am now
> too old to understand a large enough portion to have any significant
> contribution :(
>
> Yet, I still try to keep up with the field as much as I can, if only to
> be able to understand the guys and gals further up the corridor.

The more I find out, the more fascinating it all becomes. And the more 
frustrating that you can't see this stuff in action for yourself. 
(Unless your a subject area expert.)

> I do know that if I had to create life I would have done it more logical
> and maintainable. But, that would probably not have survived for more
> than a few generations.

Computer programs generally have the property that any tiny alteration 
to them or any tiny fault in the hardware running them causes 
catastrophic failure. Living organisms are remarkably devoid of such a 
problem...



Did you hear the one about the guy who tried to "evolve" an oscillator 
circuit in his lab? Used some kind of genetic algorithm to take randomly 
generated circuits and gradually modify them and keep the ones that 
acted most like an oscillator.

When the guy analysed the "winning" design, it looked absolutely nothing 
like an oscillator. And yet it worked perfectly. Baffled, the guy took 
it to a friend's lab, whereupon it utterly stopped working. Now both of 
them are *really* puzzled.

Long story short, it turns out it wasn't an oscillator at all. It was a 
radio receiver. It was picking up some kind of radio interference from 
some equipment in the guy's lab, which wasn't present in the other lab. 
So much for adapting to your environment, eh?

Post a reply to this message

On 8/4/2011 5:22, Invisible wrote:
> OK, so that's pretty much the work Daily WTF nightmare imaginable. Of
> course, nobody /actually/ writes software like that, right?

Of course they do. I've fixed code like that.  We didn't actually distribute 
it to the customer, mind, but everything else you described was in there. 
In contrast with distributing it to the customer, we were instead actually 
editing the production copy while the system was live.

-- 
Darren New, San Diego CA, USA (PST)
   How come I never get only one kudo?

Post a reply to this message

On 04/08/2011 04:21 PM, Darren New wrote:
> On 8/4/2011 5:22, Invisible wrote:
>> OK, so that's pretty much the work Daily WTF nightmare imaginable. Of
>> course, nobody /actually/ writes software like that, right?
>
> Of course they do. I've fixed code like that.

It's too horrifying to imagine...

> In contrast with distributing it to the customer, we were
> instead actually editing the production copy while the system was live.

Yes, I was afraid of that.

Post a reply to this message

On 08/04/2011 11:59 AM, Jim Holsenback wrote:
> On 08/04/2011 09:22 AM, Invisible wrote:
>> The programmer didn't read the documentation for the tools and libraries
>> he's using. (He probably wouldn't understand it anyway!) As such, the
>> software actually /depends on/ all the undocumented, undefined or just
>> plain erroneous behaviour of the compiler, the runtime and the libraries.
>
> Sounds a bit like the early days of squashfs (re: my previous post on
> the topic) ... I never could get the tools to compile correctly for what
> can only be some of the very issue's that you've raised.

clarification ... don't want the squashfs folks thinking some povray 
weenie was slamming the project, so I'll add that my comments relate to 
the many lzma variants out there, all aimed at getting just a little 
more blood out of the turnip (1% or 2% better compression) ... I mean 
maybe a good idea when RAM was more expensive. Now a days that's just an 
exercise in futility ;-)

Post a reply to this message

On 4-8-2011 17:18, Invisible wrote:
> On 04/08/2011 03:48 PM, andrel wrote:
>> That is sort of the software engineer's view. You can describe DNA as if
>> it is a program, but that misses most of the details.
>
> If you read most books about it, they will tell you that "hey, DNA has
> these 4 letters, and they go in groups of 3, and this table here shows
> you for every 3-letter word which animo acid it selects, and that's how
> you make proteins!"
>
> Which is basically true. But this isn't something some engineer
> designed. This is a crazy-ass thing that happened by accident! Suffice
> to say, it's way more complicated than that.

You need for instance the proteins of a living cell to do the 
transcription.

>
> You know the bit I said about compiled code being modified at runtime?
> It turns out that by attaching (or not attaching) various molecules to a
> DNA chain, you can activate or deactivate various genes. So DNA isn't
> just the program. In some sense it's also part of the cell state or
> "memory". Sometimes I wonder whether a "naked" copy of a given animal's
> genome would actually function at all; even at conception, there are
> probably markers turning genes on or off, to set the program in motion
> from the correct start point.

You can not boot a cell from scratch. 
http://www.ted.com/talks/lang/eng/craig_venter_unveils_synthetic_life.html

> Then there are the knots and tangles in DNA or RNA that affect the way
> it is processed. For example, human mRNA with a certain type of knot in
> it causes certain amino acids to have selenium added to them after
> transcription. So the whole "codons code amino acids" is a vast
> simplification.
>
> Let us not even get into the fact that many proteins are modified after
> being constructed, some fold up as they are transcribed (so trying to
> simulate folding of the complete, finished protein is a doomed
> endeavour), one require "chaperone" proteins to guide their folding in
> the correct direction, many proteins are synthesized inactive and have
> to be "activated" before they will do anything. Recent research even
> suggests that "cellular crowding" may be important for some protein
> activities.
>
> http://en.wikipedia.org/wiki/Cellular_crowding
>
> On top of all that, it turns out that RNA itself can be both a data
> carrier like DNA *and* a catalyst like proteins. Wrap your head around
> that one!

The mechanism that reads the code is a set of RNA molecules

>> Sometimes I wish I was 18 again today, so I could absorb enough facts
>> about how cells and organisms work to help advance the field. I am now
>> too old to understand a large enough portion to have any significant
>> contribution :(
>>
>> Yet, I still try to keep up with the field as much as I can, if only to
>> be able to understand the guys and gals further up the corridor.
>
> The more I find out, the more fascinating it all becomes. And the more
> frustrating that you can't see this stuff in action for yourself.
> (Unless your a subject area expert.)

I seem to remember you turned down an opportunity to work in my lab.

>> I do know that if I had to create life I would have done it more logical
>> and maintainable. But, that would probably not have survived for more
>> than a few generations.
>
> Computer programs generally have the property that any tiny alteration
> to them or any tiny fault in the hardware running them causes
> catastrophic failure. Living organisms are remarkably devoid of such a
> problem...
>
>
>
> Did you hear the one about the guy who tried to "evolve" an oscillator
> circuit in his lab? Used some kind of genetic algorithm to take randomly
> generated circuits and gradually modify them and keep the ones that
> acted most like an oscillator.
>
> When the guy analysed the "winning" design, it looked absolutely nothing
> like an oscillator. And yet it worked perfectly. Baffled, the guy took
> it to a friend's lab, whereupon it utterly stopped working. Now both of
> them are *really* puzzled.
>
> Long story short, it turns out it wasn't an oscillator at all. It was a
> radio receiver. It was picking up some kind of radio interference from
> some equipment in the guy's lab, which wasn't present in the other lab.
> So much for adapting to your environment, eh?

I know similar stories, yes.

-- 
Apparently you can afford your own dictator for less than 10 cents per 
citizen per day.

Post a reply to this message