Orchid XP v7 <voi### [at] devnull> wrote:
> (My strong suspecion is that it *is* 12 
> *bytes* - since, after all, a single Unicode code point is 24 bits already.)

  That depends on whether it's a "raw" unicode value (ie. in practice an
integer) or eg. an UTF8-encoded unicode character (which would make each
character of variable size).

> The situation is actually worse than it looks. All this lazy evaluation 
> magic is implemented by storing program state around the place, so an 
> "unevaluated" string probably takes up more space still...

  I have always wondered about that. Lazy evaluation can indeed be of
great help optimizing many things. For example, if you read a file into
a string and then just read a small part of it, the lazy evaluation will
automatically skip reading the rest of the file.

  However, in situations where lazy evaluation does not actually help
much (or at all), which isn't a very uncommon case, it feels like a
useless waste of memory. After all, the haskell interpreter/compiler
has to *somehow* store the info about what is still to be done. This must
take some extra memory. If the laziness was of absolutely no help, then
that memory is wasted.

-- 
                                                          - Warp

Post a reply to this message

Le Forgeron <jgr### [at] freefr> wrote:
> It is past 24 bits since a few...  (even past 32 bits!!!)

  That's not true. The current unicode standard defines about 100000
characters. Thus raw unicode values require only 17 bits

  UTF-8 encoding "wastes" some bits (in order to use less bits for the
most used western characters) and requires at most 4 bytes per character
(even though the characters requiring more than 3 bytes are very rarely
used).

> The real thing is how you encode all these.
> UTF-8 is one way (the popular one these days),
> UTF-16 another...  and raw storage the worst idea ever!

  Why would raw storage be the worst idea? There are several advantages.
For instance, each character takes the same amount of space (instead of
taking a variable amount like with the encodings), which means that you
can directly index the nth character in a string (in an utf8-encoded string
if you want the nth character you have to actually traverse the entire
string up to that point and decode it along the way). It's also the most
efficient way (speedwise) of handling the characters because you don't
need to be doing conversions back and forth between the encoding and
the raw values.
  The disadvantage is, of course, an increased memory requirement.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Orchid XP v7 <voi### [at] devnull> wrote:
>> Notice that a string is a *single-linked* list. (I.e., there are "next" 
>> pointers but no "prev" pointers.)
> 
>   You can't random-access a string in haskell? You can just go through it
> once and that's it? That would be quite limiting...

Not unless you transform it into something else first.

(This "ByteString" thing I keep harping on about gives you something 
like O(n/k) random access time, for some implementation-dependent k.)

Typically, if you read a text string, you're probably going to feed it 
into some kind of parser, and then manipulate the resulting parse tree. 
And parsing only requires linear access.

Post a reply to this message

Warp wrote:
> Orchid XP v7 <voi### [at] devnull> wrote:
>> This is why the "ByteString" library was developed.
> 
>   It just shows that the original article which spawned this thread is
> naive. *In theory* you can have all kinds of fancy high-level uber-abstract
> constructs which abstract away all the dirty internal details. *In practice*,
> however, you still need those dirty details if you want any efficiency.
> The fancy theories may be good in a big bunch of programs, but like so many
> abstractions before, it's not the final silver bullet of programming.

The point being, ByteString is implemented as an array, yet still *looks 
like* a normal linked-list. So you get all the nice fancy theory *and* 
the efficient runtime behaviour, all at once. So I'm not sure it is 
"flawed"...

Post a reply to this message

Warp wrote:
> Orchid XP v7 <voi### [at] devnull> wrote:
>> The nice thing about Haskell is that is *allows* you to come up with a 
>> better implementation and slip it in there later.
> 
>   Is that really so? Can you refactor an existing solution (eg. in a library)
> so that it will be more efficient (eg. memorywise) without breaking any
> existing programs?

It depends on the library, and how carefully it was thought out in the 
first place. (I imagine this applies in most languages.)

For example, there is a current project to introduce 
trippy-new-killer-feature "stream fusion". This involves rewriting 
almost *all* the standard Haskell list processing functions (of which 
there are a fairly mind-blowing number) so that they work completely 
differently internally.

Obviously, these are the most-used functions in all of Haskell [three 
guesses why they're being optimised?], and any change to the interface 
would break practically every Haskell program ever written. And breaking 
almost every Haskell program ever written would be a Very Bad Thing(tm). 
Fortunately, they haven't needed to...

Can the same be said for *every* library? Well, that is a harder 
question to answer. My guess would be "many of them", but I don't have 
any really scientific proof of that.

Post a reply to this message

Warp wrote:
> Orchid XP v7 <voi### [at] devnull> wrote:
>> (My strong suspecion is that it *is* 12 
>> *bytes* - since, after all, a single Unicode code point is 24 bits already.)
> 
>   That depends on whether it's a "raw" unicode value (ie. in practice an
> integer) or eg. an UTF8-encoded unicode character (which would make each
> character of variable size).

My understanding is that single characters are stored as 32-bit 
integers. (Because that's simple, and they fit nicely into CPU 
registers.) I could be mistaken though...

>> The situation is actually worse than it looks. All this lazy evaluation 
>> magic is implemented by storing program state around the place, so an 
>> "unevaluated" string probably takes up more space still...
> 
>   I have always wondered about that. Lazy evaluation can indeed be of
> great help optimizing many things. For example, if you read a file into
> a string and then just read a small part of it, the lazy evaluation will
> automatically skip reading the rest of the file.
> 
>   However, in situations where lazy evaluation does not actually help
> much (or at all), which isn't a very uncommon case, it feels like a
> useless waste of memory.

And time - don't forget time. (Takes time to build a closure, takes time 
to execute it. All of which can be avoided if you know you don't need it.)

For this reason, we have the following facts:

1. There is a special compiler pass for detecting places where laziness 
is *not* necessary, and automatically removing it. (E.g., an inner loop 
where you can *see* that the value is obviously used in the next loop.) 
When you compile a library, the compiler even leaves a note to itself 
telling it how strict each function is with its arguments.

2. There are mechanisms for the programmer to explicitly turn off 
laziness where not required. (Unlike the compiler's automatic detection 
stuff, this operation actually changes the behaviour of the program - 
which is why you, the programmer, must say this is OK.)

Post a reply to this message

Le Forgeron wrote:
> Le 01.11.2007 20:00, Orchid XP v7 nous fit lire :
>> No - it's Unicode. 24 bits per character. ;-)
> 
> Unicode is just a bunch of tables of glyphs (lot of tables, lot of
> glyphs).
> It is past 24 bits since a few...  (even past 32 bits!!!)
> The real thing is how you encode all these.
> UTF-8 is one way (the popular one these days),
> UTF-16 another...  and raw storage the worst idea ever!
> 
> UTF-8 is about 8 bits for ascii range, usually go up to 24 bits (3 x
> 8) for classical japanese, 16 for most french variants...

I was under the impression that these encodings apply to *strings*, not 
individual characters by themselves...

>> But yes, the standard Haskell string type is geared to flexibility, not
>> performance. See my ByteString comments...
> 
> Fixed size unicode... if only they stop adding more tables!

Oh, ByteString (as the name somewhat implies) only supports the first 
256 Unicode code-points. ;-)

Somebody should really fix that eventually...

Post a reply to this message

Warp wrote:

>   UTF-8 encoding "wastes" some bits (in order to use less bits for the
> most used western characters) and requires at most 4 bytes per character
> (even though the characters requiring more than 3 bytes are very rarely
> used).

And thus, like any decent variable-length encoding scheme, it tries to 
assign short codes to common symbols. (Although UTF-8 probably fails 
horribly for, say, Japanese text. I don't actually know...)

>> UTF-16 another...  and raw storage the worst idea ever!
> 
>   Why would raw storage be the worst idea? There are several advantages.
>   The disadvantage is, of course, an increased memory requirement.

You win some, you loose some. Programming is all about these kinds of 
compromises. :-)

Post a reply to this message

Orchid XP v7 <voi### [at] devnull> wrote:
> The point being, ByteString is implemented as an array, yet still *looks 
> like* a normal linked-list. So you get all the nice fancy theory *and* 
> the efficient runtime behaviour, all at once. So I'm not sure it is 
> "flawed"...

  Then why do you need any other type of string?

-- 
                                                          - Warp

Post a reply to this message

>> Yeah, it does take a lot, and it uses unicode for characters too,
>> I think I heard it takes about 12 bytes per character that way.
>> It's perty bad on big files.
>
>  But hey, that's the fad in modern programming: Memory usage and speed
> are irrelevant. Processors are getting faster and memory amounts are
> doubling every couple of years, so who cares if something like a string
> takes 12 times as much memory as it could?

Wasn't that part of  Leo Brodie's point, it's more important that a
solution be adequate, and correct than simple?

Post a reply to this message

Warp wrote:
>   Is that really so? Can you refactor an existing solution (eg. in a library)
> so that it will be more efficient (eg. memorywise) without breaking any
> existing programs?

I don't know about Haskell, but IBM had a language called NIL that was 
very high-level. Everything was processes and SQL-style tables and 
unbounded integers and dynamic code instantiation and such.  They wrote 
the routing software for SNA in it.

When a customer wanted to have two SNA routers sharing the load and 
picking up in the event of a fall-over, the guys on the NIL team 
realized they didn't need to rewrite any NIL code. They simply(*) added 
a flag to the compiler to tell it to generate parallel code with soft 
fail-over. Nothing but the compiler got changed.


(*) For some meaning of "simply" obviously. ;-)

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Warp wrote:
> Orchid XP v7 <voi### [at] devnull> wrote:
>> The point being, ByteString is implemented as an array, yet still *looks 
>> like* a normal linked-list. So you get all the nice fancy theory *and* 
>> the efficient runtime behaviour, all at once. So I'm not sure it is 
>> "flawed"...
> 
>   Then why do you need any other type of string?

You probably don't.

It's just that the language standard ("Haskell 98") defines "String" as 
an alias for "[Char]" ("single linked list of characters"). ByteString 
is a 3rd-party library that somebody independently wrote much more 
recently. Adding a new library is quite easy; actually *replacing* 
something existing is much more work.

My hope is that eventually the work that has been done on ByteString 
will be extended to make *all* list processing trippy-fast. There's just 
not much evidence of it happening right now. You know what volunteer 
projects are like...

Post a reply to this message

Warp wrote:

>   Is that really so? Can you refactor an existing solution (eg. in a library)
> so that it will be more efficient (eg. memorywise) without breaking any
> existing programs?

I guess the key to a question like this is to think about what kinds of 
things would possibly *prevent* such a refactor.



I have never written a nontrivial program in C, but in Pascal I 
frequently found that the type system was one such reason. In Pascal 
(and AFAIK in plain C) you *must* expose all implementation details of a 
type in order for anybody else to be able to touch it. This is one of 
the Big Problems that OOP is supposed to fix. (And, IMHO, does fix very 
well indeed.)

Haskell does quite well too in this regard. I can write a module in 
Haskell that exports a type, without telling you a damn thing about what 
the type is. It might just be an alias for some existing type, or it 
could be something really complex. You don't know. You don't care. And I 
can *change* what that type actually is, and you *still* don't need to 
care. All you see is that there exists a type with a specific name, and 
some functions that can do things with it.



We could summarise all that by simply saying that hiding unecessary 
detail is a Good Thing. So perhaps we should look at where details can 
leak out in Haskell...



One place is I/O. Functions that do I/O have to be "marked" as such. 
This is by design; normal functions can be run in any order without 
consequence, whereas I/O operations *usually* need to be run in a very 
precise order. This information is obviously very important, and so is 
exposed at the type level.

What this means is that you can't flip a normal function into an 
I/O-performing function without altering at least the function's type. 
But then, you're turning an "I can run this at any time" function into a 
"it might well matter when I run this" function. Arguably this demands 
that the client code change anyway, in which case the enforced interface 
change is arguably good. Either way, this kind of code change seems 
quite rare in practice. (Certainly rare for performance tuning.)



What other information is leaked? Well, a function's type tells you 
exactly what inputs it depends on. (Unless it does I/O anyway...) So if 
you change the algorithm to take more information into account, the type 
signature must change.

Let us take an example: Suppose you have a function that decodes 
different character sets. You give it a byte, and it translates it to a 
Unicode code-point using some kind of rule.

Now suppose you want to add UTF-8 decoding. Oops! That's a 
variable-length encoding. To decode a byte, you might need to know what 
the previous byte was. But in Haskell, you can't do that.

Now, in C you could just store some status information in a global 
variable somewhere. So in C you can add UTF-8 without having to change 
your public interface. Yay!

Now suppose some client decides to call the function from multiple 
threads processing several input streams. OOPS!! Chaos ensues... The 
*only* way you can fix this is if you alter your public interface. There 
are several possible fixes, but all involve client-visible change.

On the other hand, Haskell would force you to change the public 
interface in the first place. (And again you have several options here.)

To summarise: In both C and Haskell you end up changing the interface if 
you want the thing to work properly. Haskell just forces you to do it 
right at the start, whereas C potentially does strange things until you 
realise what's up. And in both cases, the key to success is to design 
the public interface properly in the first place! (I.e., write a 
function that converts whole strings, not individual octets.)



I said above that leaking unecessary detail is bad. My arguments below 
that could essentially be read as saying "this is leaking detail, but it 
is *necessary* detail, so that's OK". So it's really whether you 
consider these things "necessary" or not.



Another potential problem area is laziness. Being lazy can speed 
programs up no end. It can also slow them down tremendously. 
Deliberately turning off laziness is a fairly common way to improve the 
efficiency of Haskell code. However, this is often an obvservable 
change. Code that works with the lazy version of a library might well 
lock up given a stricter version.

So here is an implementation change that is totally invisible at the 
interface level, yet might cause client code to stop working properly.

(It's actually fairly common to provide libraries with two versions, so 
the client can choose. And because suddenly changing a library's 
implementation can cause problems...)

Post a reply to this message

Warp nous apporta ses lumieres en ce 2007/11/01 14:40:
> Alain <ele### [at] netscapenet> wrote:
>> It's about 12 BITS per characters, on average, not BYTES! That's useing UTF8 
>> encoding. About 16 BITS per characters if using UTF16 encoding.
>> UTF8 is only 7 BITS per characters if you stick to only standard ASCII 
>> characters set, but it gets bigger if you also use extended ASCII or characters 
>> from foreign alphabets.
> 
>   Except that if each single character is indeed garbage-collected, that
> requires quite a lot of memory per character (compared to the size of the
> character).
> 
Garbage collection is BAAAAD!!!
Any implementation that permit you not to use that is GOOD.

I got hit real bad by garbage collection, ONCE! The programm was running realy 
good, no bug at all, then is just stopped there, nothing at all was hapening, 
the systemwas totaly unresponsive, uterly frozen, then it resumed after around 
15 minutes. It was garbage collecting.
I changed the programm logic to prevent any future garbage collection.

-- 
Alain
-------------------------------------------------
You know you've been raytracing too long when you're starting to find these 
quotes more unsettling than funny.
     -- Alex McLeod a.k.a. Giant Robot Messiah

Post a reply to this message

Alain escribió:
> Garbage collection is BAAAAD!!!
> Any implementation that permit you not to use that is GOOD.
> 
> I got hit real bad by garbage collection, ONCE! The programm was running 
> realy good, no bug at all, then is just stopped there, nothing at all 
> was hapening, the systemwas totaly unresponsive, uterly frozen, then it 
> resumed after around 15 minutes. It was garbage collecting.
> I changed the programm logic to prevent any future garbage collection.
> 

http://www.virtualdub.org/blog/pivot/entry.php?id=176

Post a reply to this message

Orchid XP v7 nous apporta ses lumieres en ce 2007/11/01 15:03:
> Warp wrote:
>> Alain <ele### [at] netscapenet> wrote:
>>> It's about 12 BITS per characters, on average, not BYTES! That's 
>>> useing UTF8 encoding. About 16 BITS per characters if using UTF16 
>>> encoding.
>>> UTF8 is only 7 BITS per characters if you stick to only standard 
>>> ASCII characters set, but it gets bigger if you also use extended 
>>> ASCII or characters from foreign alphabets.
>>
>>   Except that if each single character is indeed garbage-collected, that
>> requires quite a lot of memory per character (compared to the size of the
>> character).
> 
> I can't actually find documentation to hand to clarify whether it's 12 
> bits or 12 bytes per character. (My strong suspecion is that it *is* 12 
> *bytes* - since, after all, a single Unicode code point is 24 bits 
> already.)
> 
> The situation is actually worse than it looks. All this lazy evaluation 
> magic is implemented by storing program state around the place, so an 
> "unevaluated" string probably takes up more space still...
Using UTF8 encoding, a single character can be 1 BYTE, 2, 3 or 4 Bytes long. 
Standard ASCII characters are 1 BYTE. extended ASCII and some more use 2 bytes.
BIT 7 is a flag for 2 bytes codes. BITS 6 and 7 set to 1 = 3 bytes codes. BITS 
5, 6 and 7 set to 1 = 4 bytes codes.
Using UTF16 encoding, any character is 2 BYTES long, for a grand total is 65536 
possible characters, not all of them been printable.

-- 
Alain
-------------------------------------------------
Error in operator: add beer

Post a reply to this message

Le 01.11.2007 22:15, Orchid XP v7 nous fit lire :
 >>> UTF-16 another...  and raw storage the worst idea ever!
>>
>>   Why would raw storage be the worst idea? There are several advantages.
>>   The disadvantage is, of course, an increased memory requirement.
> 
> You win some, you loose some. Programming is all about these kinds of
> compromises. :-)

Raw storage is the worst, because they are still adding glyph in
unicode...
When it started, you could have used only 16 bits... now you can't
anymore.
It's not about programming, but "selling" the concept. Raw storage
means no coding, no signaling of actual number of bits used...

On one side, you have billions of people just happy with only 7
bits. May be even with 6 bits it would be enough (you do not need
all these letters in double, do you ? ten number (roman did it with
6 or 7 glyphs) and a few ponctuations signs.)
And other billions which could probably be fine with a fixed 16
bits... maybe not, they are still adding!
and people in the middle, (japanese recommended 1945... and all the
other countries...)
And there is the few who want to be able to mix everything at the
same time in the same document.


UTF-8 is interesting for the 6 bits people, as they do not loose too
much storage and it's simple.
(far more than all the old trick of "active page" and such coding).
But UTF-8 nowadays can extend to already 4 x 8 bits ( or worse ?).
What would have happened if the basic byte was 9 bits long instead
of 8 ? Would they go by 4, or would 3 have been enough ?
Obviously all latin-european coutries would have fit on the same
page of 512, and maybe we would still be playing with paging-code ?
(No thanks!)


-- 
The superior man understands what is right;
the inferior man understands what will sell.
-- Confucius

Post a reply to this message

Orchid XP v7 nous apporta ses lumieres en ce 2007/11/01 17:15:
> Warp wrote:
> 
>>   UTF-8 encoding "wastes" some bits (in order to use less bits for the
>> most used western characters) and requires at most 4 bytes per character
>> (even though the characters requiring more than 3 bytes are very rarely
>> used).
> 
> And thus, like any decent variable-length encoding scheme, it tries to 
> assign short codes to common symbols. (Although UTF-8 probably fails 
> horribly for, say, Japanese text. I don't actually know...)
For Japanese and Chinese, it average around 3 bytes per characters. It's not so 
bad after all, as each characters in those represent a whole word, some even 
represent a whole phrase or some complexe concept.
A 1000 glyphs text in Chinese would be, roughly, a 1000 to 5000 words text in 
English!
> 
>>> UTF-16 another...  and raw storage the worst idea ever!
>>
>>   Why would raw storage be the worst idea? There are several advantages.
>>   The disadvantage is, of course, an increased memory requirement.
> 
> You win some, you loose some. Programming is all about these kinds of 
> compromises. :-)
All possible characters for all European languages fit in 1 or 2 bytes, and I 
think that it also include Arabic and Cyrilic. The Asiatic glyphs use the bulk 
of the 3 and 4 bytes codes.

-- 
Alain
-------------------------------------------------
If you're ever about to be mugged by a couple of clowns, don't hesitate - go for 
the juggler.

Post a reply to this message

Alain wrote:

> Garbage collection is BAAAAD!!!
> Any implementation that permit you not to use that is GOOD.

Gee, that's not much of a wild generalisation or anything like that...

> I got hit real bad by garbage collection, ONCE! The programm was running 
> realy good, no bug at all, then is just stopped there, nothing at all 
> was hapening, the systemwas totaly unresponsive, uterly frozen, then it 
> resumed after around 15 minutes. It was garbage collecting.

Yeah, old GC algorithms used to do this. Research has been done, 
solutions have been found, etc.

Post a reply to this message

Alain <ele### [at] netscapenet> wrote:
> Using UTF16 encoding, any character is 2 BYTES long, for a grand total is 65536 
> possible characters, not all of them been printable.

  Wrong. UTF16-encoding results in either 2-byte or 4-byte characters,
depending on the unicode value.

  Perhaps you are confusing it with UCS2?

-- 
                                                          - Warp

Post a reply to this message

Alain <ele### [at] netscapenet> wrote:
> > And thus, like any decent variable-length encoding scheme, it tries to 
> > assign short codes to common symbols. (Although UTF-8 probably fails 
> > horribly for, say, Japanese text. I don't actually know...)
> For Japanese and Chinese, it average around 3 bytes per characters. It's not so 
> bad after all, as each characters in those represent a whole word, some even 
> represent a whole phrase or some complexe concept.

  UTF16 is better because it uses 2 bytes for the vast majority of the most
commonly used kanjis and other symbols used in Japanese.

-- 
                                                          - Warp

Post a reply to this message

Invisible wrote:
> (and AFAIK in plain C) you *must* expose all implementation details of a 
> type in order for anybody else to be able to touch it. 

Nah. Just use a forward-declared struct and pass pointers to it. You 
know, like fopen/fread/fwrite/fclose.  Note those haven't changed since 
the first edition of K&R.

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Alain wrote:
> Garbage collection is BAAAAD!!!
> Any implementation that permit you not to use that is GOOD.

Uh, not sure why you'd say this. You're looking at one implementation of 
one garbage collector, and deducing from this that the entire concept is 
bad? You *are* aware there are such things as real-time garbage 
collectors, for example?

I can't imagine a GC that takes 15 minutes on anything that isn't paging 
like mad. In which case the problems isn't the GC, but the paging, in 
which case you should say "paging is baaaaad!"  :-)

I mean, really, how can you imagine any decent GC would take more than 
(say) three times the length of time it takes to read all physical memory?

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Orchid XP v7 wrote:
> Yeah, old GC algorithms used to do this. Research has been done, 
> solutions have been found, etc.

Plus, a lot of the complaints about GC slowness are caused in part by 
the GC not interacting with the paging system well. I.e., the OS isn't 
prepared for programs that're going to do GC in particular patterns.

There are read-aheads on disk files. Why would you wait until the GC 
hits the bad page before starting to read it in?

Of course, paging systems nowadays don't even know the physical layouts 
of the disks, so it's generally hard to do things like select sectors on 
the swap partition in a way that paging things back in will be fast.

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Warp nous apporta ses lumieres en ce 2007/11/02 16:14:
> Alain <ele### [at] netscapenet> wrote:
>> Using UTF16 encoding, any character is 2 BYTES long, for a grand total is 65536 
>> possible characters, not all of them been printable.
> 
>   Wrong. UTF16-encoding results in either 2-byte or 4-byte characters,
> depending on the unicode value.
> 
>   Perhaps you are confusing it with UCS2?
> 
I think that I missed the bit about 4 bytes UTF16 characters...

-- 
Alain
-------------------------------------------------
History, in general, only informs us of what bad government is.
Thomas Jefferson

Post a reply to this message

Darren New nous apporta ses lumieres en ce 2007/11/02 20:04:
> Alain wrote:
>> Garbage collection is BAAAAD!!!
>> Any implementation that permit you not to use that is GOOD.
> 
> Uh, not sure why you'd say this. You're looking at one implementation of 
> one garbage collector, and deducing from this that the entire concept is 
> bad? You *are* aware there are such things as real-time garbage 
> collectors, for example?
> 
> I can't imagine a GC that takes 15 minutes on anything that isn't paging 
> like mad. In which case the problems isn't the GC, but the paging, in 
> which case you should say "paging is baaaaad!"  :-)
> 
> I mean, really, how can you imagine any decent GC would take more than 
> (say) three times the length of time it takes to read all physical memory?
> 
OK, it was a LONG time ago!
There was no paging, there could'nt be any paging, there was no HD to receive 
anything. ALL of the data was in RAM. It was before HD where common on PC's.

-- 
Alain
-------------------------------------------------
You know you've been raytracing too long when you know how to render a truly 
photorealistic compact disc, and you're not going to tell anyone
(least of all a POV user  ;)  ).
     -- Alex McLeod a.k.a. Giant Robot Messiah

Post a reply to this message

Warp nous apporta ses lumieres en ce 2007/11/02 17:10:
> Alain <ele### [at] netscapenet> wrote:
>>> And thus, like any decent variable-length encoding scheme, it tries to 
>>> assign short codes to common symbols. (Although UTF-8 probably fails 
>>> horribly for, say, Japanese text. I don't actually know...)
>> For Japanese and Chinese, it average around 3 bytes per characters. It's not so 
>> bad after all, as each characters in those represent a whole word, some even 
>> represent a whole phrase or some complexe concept.
> 
>   UTF16 is better because it uses 2 bytes for the vast majority of the most
> commonly used kanjis and other symbols used in Japanese.
> 
But then, you don't have place for the Chinese ones, then you need Vietnamese, 
Corean, Hindu, Sanskrit, Latin, Cyrilic, Arabic, Inuctituk, Math symbols,...

-- 
Alain
-------------------------------------------------
Make yourself a better person and know who you are before you try and know 
someone else and expect them to know you.

Post a reply to this message

Alain <ele### [at] netscapenet> wrote:
> >   UTF16 is better because it uses 2 bytes for the vast majority of the most
> > commonly used kanjis and other symbols used in Japanese.
> > 
> But then, you don't have place for the Chinese ones, then you need Vietnamese, 
> Corean, Hindu, Sanskrit, Latin, Cyrilic, Arabic, Inuctituk, Math symbols,...

  UTF16 can handle all those too, although some require 4 bytes.

  I suppose Chinese might be more compact with UTF8 than with UTF16,
especially if you use lots of kanjis not in the 2-byte UTF16 range
(because those require 4 bytes with UTF16, while in UTF8, AFAIK, they
require only 3).

-- 
                                                          - Warp

Post a reply to this message

Darren New wrote:
> Invisible wrote:
>> (and AFAIK in plain C) you *must* expose all implementation details of 
>> a type in order for anybody else to be able to touch it. 
> 
> Nah. Just use a forward-declared struct and pass pointers to it. You 
> know, like fopen/fread/fwrite/fclose.  Note those haven't changed since 
> the first edition of K&R.

I have no concept of what you're talking about - but you're probably 
right...

Post a reply to this message

Orchid XP v7 wrote:
> Darren New wrote:
>> Invisible wrote:
>>> (and AFAIK in plain C) you *must* expose all implementation details 
>>> of a type in order for anybody else to be able to touch it. 
>>
>> Nah. Just use a forward-declared struct and pass pointers to it. You 
>> know, like fopen/fread/fwrite/fclose.  Note those haven't changed 
>> since the first edition of K&R.
> 
> I have no concept of what you're talking about - but you're probably 
> right...

In C, you declare a pointer to a structure without defining the layout 
of the structure. C allows this, because it assumes all pointers are the 
same size and have a unified alignment requirement and can all live in 
the same kinds of registers and things like that.

Then in your code file, you define the structure. Creating an object 
involves allocating the structure and returning the opaque pointer to 
it. Using the object requires passing the opaque pointer back in.

No implementation details are visible at all.

http://en.wikipedia.org/wiki/C_file_input/output

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> In C, you declare a pointer to a structure without defining the layout
> of the structure. C allows this, because it assumes all pointers are the
> same size and have a unified alignment requirement and can all live in
> the same kinds of registers and things like that.
>
> Then in your code file, you define the structure. Creating an object
> involves allocating the structure and returning the opaque pointer to
> it. Using the object requires passing the opaque pointer back in.
>
> No implementation details are visible at all.

good ol' abstract data types.  I still enjoy Haskell's powerful algebraic
datatypes, higher-order functions, currying and concise coding a lot more.

Post a reply to this message

Darren New wrote:

> In C, you declare a pointer to a structure without defining the layout 
> of the structure.

Ah, yes. I believe Borland's TurboPascal allows this too. (There's a 
type just called "Pointer", which you then have to cast to something 
else before you can use it.) Good luck with figuring out how big the 
data structure in question is... (But then, presumably the module that 
uses this thing will handle all creation and destruction.) Also fun, you 
how cannot have an array of this thing.

Post a reply to this message

nemesis wrote:
> good ol' abstract data types.

That's not an abstract data type. It's merely a hidden implementation.

>  I still enjoy Haskell's powerful algebraic
> datatypes, higher-order functions, currying and concise coding a lot more.

*That* is way closer to an abstract data type. :-)  And yah, I'm not 
condoning it as better. Merely as possible.

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Orchid XP v7 wrote:
>  (But then, presumably the module that 
> uses this thing will handle all creation and destruction.)

Well, yes, that's the idea. That's one way it's better than C++'s 
mechanism, which requires you to (for example) recompile every client if 
you change the layout of the private bits of the class.

> Also fun, you 
> how cannot have an array of this thing.

You can't have an array of pointers, or an array of the things? That 
seems odd if the former.

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Darren New wrote:
> Orchid XP v7 wrote:
>>  (But then, presumably the module that uses this thing will handle all 
>> creation and destruction.)
> 
> Well, yes, that's the idea. That's one way it's better than C++'s 
> mechanism, which requires you to (for example) recompile every client if 
> you change the layout of the private bits of the class.
> 
>> Also fun, you how cannot have an array of this thing.
> 
> You can't have an array of pointers, or an array of the things? That 
> seems odd if the former.

No, the latter; you can't have an array who's elements are a particular 
type since the compiler won't know how big it is.

(Notice that Haskell has no such limitation; you can expose no 
implementation information to THE PROGRAMMER whilst this information IS 
still exposed to THE COMPILER. Indeed, you can even define rewrite rules 
on the code in your library, so clients of the library actually have 
their code transparently rewritten, and the clients don't need to know 
this...)

Post a reply to this message

Invisible wrote:
> No, the latter; you can't have an array who's elements are a particular 
> type since the compiler won't know how big it is.

Right. Not unusual in compiled languages, that.

> (Notice that Haskell has no such limitation; you can expose no 
> implementation information to THE PROGRAMMER whilst this information IS 
> still exposed to THE COMPILER. 

So, I can have a module that changes the size of the objects it works 
with, and the compiled code of the caller doesn't have to see that?

I.e., to make this happen, does the caller need the source code to my 
module? Or can I compile that module, give you the executable (or 
library), and you can allocate the right size object without knowing it?

Cause, like, doing that when you're distributing library sources is 
pretty easy.

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Darren New wrote:
> Invisible wrote:
>> No, the latter; you can't have an array who's elements are a 
>> particular type since the compiler won't know how big it is.
> 
> Right. Not unusual in compiled languages, that.

Indeed.

>> (Notice that Haskell has no such limitation; you can expose no 
>> implementation information to THE PROGRAMMER whilst this information 
>> IS still exposed to THE COMPILER. 
> 
> So, I can have a module that changes the size of the objects it works 
> with, and the compiled code of the caller doesn't have to see that?

No.

If your code calls my module, and my module changes internally, you will 
have to recompile your code. You don't need to *change* your source code 
in any way, just recompile it.

The optimiser performs moderately extensive cross-module optimisations.

[My code might get inlined into yours. My types (which you, the 
programmer, can't see the representation for) might get unboxed into CPU 
registers during tight inner loops in your code. My module might define 
transformation rules that rewrite parts of your program when you compile 
it. And you don't have to care about any of this...]

> I.e., to make this happen, does the caller need the source code to my 
> module? Or can I compile that module, give you the executable (or 
> library), and you can allocate the right size object without knowing it?

Suppose I write some module named "Banana". In order to compile your 
code against my module, you will need to possess either

* The source code ("Banana.hs")

* The object code ("Banana.o") + the interface file ("Banana.hi")

As long as you have one or the other, you can compile against my code. 
So yes, I can write Banana.hs, compile it (thus generating Banana.o and 
Banana.hi) and give you the latter files (which are both binary). You 
won't have my source code, but you can still compile against it.

Indeed, the compiler ships with compiled base libraries, and doesn't 
*provide* source code unless you download it seperately. (Which is just 
as well, because it's quite large and takes a while to compile...)

(As an aside, in order to use the object code file, you'll need to be 
using the exact same compiler version as me. On the same platform, 
obviously.)

> Cause, like, doing that when you're distributing library sources is 
> pretty easy.

Um... I'm not sure exactly which statement you're trying to make.

Post a reply to this message

Invisible wrote:
>> Right. Not unusual in compiled languages, that.
> Indeed.

OK, all the stuff you described is pretty much SOP for compiled modular 
languages (for everything but C++, that is).

Ada, for example, won't even let you link against the .o file if you 
compiled against a corresponding .hi file that was compiled more 
recently than the .o file.  So you can't even mix-and-match 
representations incorrectly.

> Um... I'm not sure exactly which statement you're trying to make.

I meant that making it such that you can "hide" the representation from 
the programmer and do all that fancy stuff, but then require the source 
files of the declarations (a la C++) is a pretty easy thing to do. Much 
harder to allow (say) inlining of already-compiled code.

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Darren New escribió:
> Invisible wrote:
>>> Right. Not unusual in compiled languages, that.
>> Indeed.
> 
> OK, all the stuff you described is pretty much SOP for compiled modular 
> languages (for everything but C++, that is).
> 
> Ada, for example, won't even let you link against the .o file if you 
> compiled against a corresponding .hi file that was compiled more 
> recently than the .o file.  So you can't even mix-and-match 
> representations incorrectly.
> 
>> Um... I'm not sure exactly which statement you're trying to make.
> 
> I meant that making it such that you can "hide" the representation from 
> the programmer and do all that fancy stuff, but then require the source 
> files of the declarations (a la C++) is a pretty easy thing to do. Much 
> harder to allow (say) inlining of already-compiled code.
> 

I'm probably missing something... can't you do it with .o + .h in C++?

Post a reply to this message

Nicolas Alvarez wrote:
> I'm probably missing something... can't you do it with .o + .h in C++?

The .h file is source code, hence revealing the internal layout of your 
objects. And the functions you want to inline are presented in source. 
And you can't be sure the .h matches the .o.

-- 
   Darren New / San Diego, CA, USA (PST)
     Remember the good old days, when we
     used to complain about cryptography
     being export-restricted?

Post a reply to this message

Darren New wrote:
> Nicolas Alvarez wrote:
>> I'm probably missing something... can't you do it with .o + .h in C++?
> 
> The .h file is source code, hence revealing the internal layout of your 
> objects. And the functions you want to inline are presented in source. 
> And you can't be sure the .h matches the .o.

...whereas Haskell's .hi files are machine-generated (and not 
human-readable).

Post a reply to this message

On Thu, 01 Nov 2007 21:13:15 +0000, Orchid XP v7 wrote:
> I was under the impression that these encodings apply to *strings*, not 
> individual characters by themselves...

The encoding applies to individual characters, and
from those characters is the string composed.

For example, the string "kääpiö" is composed from six characters:

  code point character   utf8 encoding
   U+006B     k            6B
   U+00E4     ä            C3 A4
   U+00E4     ä            C3 A4
   U+0070     p            70
   U+0069     i            69
   U+00F6     ö            C3 B6

So the UTF-8 encoding of the string becomes 9 bytes long in total.

Similarly, the Czech word for "cat" would be encoded like this:

  code point character   utf8 encoding
   U+006B     k            6B
   U+006F     o            6F
   U+010D     ?            C4 8D
   U+006B     k            6B
   U+0061     a            61

(Note: I'm posting in iso-8859-1, which cannot express
the third character in the word: a "c" with a hacek,
hence substituting with "?".)

And the Japanese word for Japan would be:

  code point character   utf8 encoding
   U+65E5     ?            E6 97 A5
   U+672C     ?            E6 9C AC 

The encoding (UTF-8) has a few clever attained design goals:
- Backwards compatibility with ASCII
- Asciibetical sorting still works the same way
- Forward and backward seeking in the string possible without desynchronization
- Minimal space wasted
- Possibility to extend naturally if the unicode set grows

-- 
Joel Yliluoma - http://iki.fi/bisqwit/

Post a reply to this message