On 17/05/2011 16:19, Darren New wrote:
> More erlang fanning:
>
> The protocol between the deamons:
>
> http://www.erlang.org/doc/apps/erts/erl_dist_protocol.html
>
> The protocol between the individual actors:
>
> http://www.erlang.org/doc/apps/erts/erl_ext_dist.html

Ouch. The pain...

> You want to look at 8.21 and 8.22. It seems the code doesn't ship, since
> logically the function can't really be constructed at runtime. Instead,
> it's a reference into the BEAM file (where a BEAM is like a java jar).

Erlang sends an index into the BEAM file, the eventual Haskell 
implementation is going to send linker labels (i.e., indexes into the 
executable image).

I notice also that Erlang sends "Uniq", which is "the hash value of the 
parse for the fun". In other words, it sounds like it's a cryptographic 
hash function of the syntax tree for the function's source code. Which 
is pretty much what the Haskell people are talking about for version 
control.

> Because of course "external term format" is exactly what anyone would
> google for to find that.

Oh, but of course!

> I'm also rather amused that altho Erlang has
> wonderfully powerful expressions for parsing byte streams, they used
> ASCII art with ambiguous textual descriptions to describe the format of
> the stuff on the wire instead of giving you the Erlang expression that
> would parse it unambiguously.

Myself, I was amused by

"The description here is far from complete and will therefore be further 
refined in upcoming releases. The protocols both from Erlang nodes 
towards EPMD (Erlang Port Mapper Daemon) and between Erlang nodes, 
however, are stable since many years. "

So the description is "far from complete"? That's really helpful. Oh, 
but the protocols "are stable since many years". So I'm guessing English 
isn't your first language then? :-P

Post a reply to this message

On 5/17/2011 8:22, Invisible wrote:
> I was under the impression that the registry is *only* for dynamic linking,
> and has nothing to do with static linking [which is what I'm talking about
> here].

Oh, yeah, I don't know how to use it for static linking.

> This is from the people who actually had to ask "does Windows actually
> support letting multiple processes open the same disk file at once?"

Heh. Did you tell them it's better at that then UNIX is?

> Question: Does that mean that one process can produce so much garbage that
> it forces endless GC cycles, slowing other unrelated processes down? Or does
> the VM partition memory on a per-process basis or something?

Each actor (i.e., Erlang process) has its own heap, so you can generate a 
lot of garbage but it'll only make your own thread slow down.

That said, it's possible to generate specific kinds of garbage that go into 
shared heaps or interned string storage or etc, which will be mean to all 
the processes in the same VM. Hermes (which is somewhat similar to Erlang) 
was designed to let the interpreter avoid such. (Altho Erlang *could* avoid 
such problems with more work in the VM, Erlang wasn't specifically designed 
AFAIK with the concept of being safe to run "untrusted" code on a node, 
while that was pretty much Herme's main point.)

> Well yeah, but if you're trying to use a mere textual "name" to decide what
> type something is, you need a version number or something to go with the
> name, to make it unique.

Sure.

> What Erlang is doing is considering any two types to be "identical" if they
> have the same *structure*. That wouldn't do for Haskell; consider, for
> example, how a PortID and a ThreadID are both 32-bit signed integers, but
> are considered utterly different, incompatible types.

Well, more like you have very few types, but they're dynamic types. You have 
list. You have tuple. You don't have list-of-X or tuple-of-three-Y's.

I.e., just like in Haskell a list of 4 integers is the same type as a list 
of seven integers, in Erlang a list is a list, a tuple is a tuple.

> [Of course, TRWTF is that it's *signed*...]

If they're IDs, it doesn't make sense to talk about math or ordering on 
them, so whether they're signed or not is irrelevant.

> You can upgrade the code, but unless you upgrade it all at once, both
> versions need to speak the same wire protocol.

Again, list of three integers vs list of seven integers.

>>> as well. When you compile v2, the compiler checks what you changed, and
>>> warns you if you don't explicitly handle the v1 data (unless it's
>>> identical).
>>
>> That's fair. It would be interesting to see how that's specified.
>
> I guess we sit back and see if they ever release anything...

I mean, I'd be interested in seeing how you tell the compiler "I can take 
either v1 or v2" in a way that v1 and v2 are cleanly separated from the 
code.  Most languages make a distinction of a channel specification where 
you provide the types for specific channels, but I don't know how Haskell 
would handle such a thing. Would it just be a convention that you put that 
stuff in a separate file?

> Yeah, I guess that works too.
>
> Presumably Erlang allows you to serialise cyclic data structures, which is
> notoriously hard for a pure functional language.

I'm not sure how you'd get a cyclic data structure in Erlang, given that 
it's functional. It would imply something created earlier has to point to 
something created later in order to form a cycle.

>>> And yet, that would seem rather heavy compared to just sending a function
>>> name (or rather, a *unique* function identifier),
>>
>> Not all functions have names. For the functions that have names, I
>> believe you just send the fully-qualified name of the function.
>
> Only works if foo/3 on node X is the same thing as foo/3 on node Y. ;-)

Correct. That's why there's an MD5 of the code along with the identifier.

> Oh, right. So atoms are *globally* unique, not unique to each node?

Well, they're ... atoms. Is 27.5 globally unique?  Two atoms with the same 
textual representation are by definition the same atom.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 5/17/2011 8:32, Invisible wrote:
> Erlang sends an index into the BEAM file, the eventual Haskell
> implementation is going to send linker labels (i.e., indexes into the
> executable image).

Well, note that there's two versions, from different releases of the VM.

> I notice also that Erlang sends "Uniq", which is "the hash value of the
> parse for the fun". In other words, it sounds like it's a cryptographic hash
> function of the syntax tree for the function's source code. Which is pretty
> much what the Haskell people are talking about for version control.

I believe that's correct. And the old version sent the MD5 of the range of 
bytes that implement the function, or something. (It's actually the AST, 
which is available also as part of the macro expansion phase of compilation.)

> So the description is "far from complete"? That's really helpful.

Yet typical for open source, where proponents often claim that the code *is* 
the documentation.

> Oh, but
> the protocols "are stable since many years". So I'm guessing English isn't
> your first language then? :-P

Uh, no. Ericcson.com, corporate headquarters Stockholm, Sweden.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

>> This is from the people who actually had to ask "does Windows actually
>> support letting multiple processes open the same disk file at once?"
>
> Heh. Did you tell them it's better at that then UNIX is?

My understanding is that Unix is *great* at letting multiple processes 
read the same file - it's *stopping* them doing this that's hard.

>> Question: Does that mean that one process can produce so much garbage
>> that
>> it forces endless GC cycles, slowing other unrelated processes down?
>> Or does
>> the VM partition memory on a per-process basis or something?
>
> Each actor (i.e., Erlang process) has its own heap, so you can generate
> a lot of garbage but it'll only make your own thread slow down.

Right, OK. So the worst you can do (presumably) is steal lots of CPU 
time. (?)

Come to mention it, what's Erlang's thread scheduler like? Does it just 
do round-robin for all threads ready to run, or can you assign thread 
priorities and stuff?

> That said, it's possible to generate specific kinds of garbage that go
> into shared heaps or interned string storage or etc, which will be mean
> to all the processes in the same VM. Hermes (which is somewhat similar
> to Erlang) was designed to let the interpreter avoid such. (Altho Erlang
> *could* avoid such problems with more work in the VM, Erlang wasn't
> specifically designed AFAIK with the concept of being safe to run
> "untrusted" code on a node, while that was pretty much Herme's main point.)

OK, fair enough.

>> What Erlang is doing is considering any two types to be "identical" if
>> they
>> have the same *structure*. That wouldn't do for Haskell; consider, for
>> example, how a PortID and a ThreadID are both 32-bit signed integers, but
>> are considered utterly different, incompatible types.
>
> Well, more like you have very few types, but they're dynamic types. You
> have list. You have tuple. You don't have list-of-X or tuple-of-three-Y's.
>
> I.e., just like in Haskell a list of 4 integers is the same type as a
> list of seven integers, in Erlang a list is a list, a tuple is a tuple.

Indeed. Erlang cares only about structure, not meaning. (Presumably the 
idiomatic way to distinguish between a ThreadID and a CustomerID would 
be to put an atom in front of it indicating it's "type".)

>> [Of course, TRWTF is that it's *signed*...]
>
> If they're IDs, it doesn't make sense to talk about math or ordering on
> them, so whether they're signed or not is irrelevant.

Presumably it just means that if you create more than 2^31 Haskell 
threads, the ThreadID counter will overflow and half the standard 
library will malfunction...

Soon you'll have FOUR THOUSAND AND NINETY SIX CHILDREN, and they'll all 
be PARALLEL, and Google will be like SLOOOOOW DOWN MAN! IT'S 
MUNCTIONAL!!!! >_<

>> You can upgrade the code, but unless you upgrade it all at once, both
>> versions need to speak the same wire protocol.
>
> Again, list of three integers vs list of seven integers.

There's more to a wire protocol than what types you exchange. (There's 
ordering, for example.)

>> I guess we sit back and see if they ever release anything...
>
> I mean, I'd be interested in seeing how you tell the compiler "I can
> take either v1 or v2" in a way that v1 and v2 are cleanly separated from
> the code. Most languages make a distinction of a channel specification
> where you provide the types for specific channels, but I don't know how
> Haskell would handle such a thing. Would it just be a convention that
> you put that stuff in a separate file?

I don't really know. In principle the information exists to let you do 
this stuff, but how you'd implement it I'm not sure.

GHC already generates "interface files" which are machine-readable files 
telling the compiler what's exported from each source module. Perhaps 
you could do some trick where you compare the old interface files to the 
new ones or something. (But this requires you to remember to not delete 
this files, which are usually considered "temporary", so probably some 
more specific mechanism is necessary.)

>> Yeah, I guess that works too.
>>
>> Presumably Erlang allows you to serialise cyclic data structures,
>> which is
>> notoriously hard for a pure functional language.
>
> I'm not sure how you'd get a cyclic data structure in Erlang, given that
> it's functional. It would imply something created earlier has to point
> to something created later in order to form a cycle.

Does Erlang not permit recursive definitions then?

   let list = [1, 2, 3] ++ list

That'll create a circular list for you in Haskell.

[Actually, it's not defined whether this is a recursive function, or a 
circular list. It's impossible to tell the difference from inside 
Haskell itself...]

>> Only works if foo/3 on node X is the same thing as foo/3 on node Y. ;-)
>
> Correct. That's why there's an MD5 of the code along with the identifier.

Yeah. And why Haskell is talking about doing something similar. 
Currently it just uses a name, but as you say, that breaks as soon as 
sender and receiver have different code.

>> Oh, right. So atoms are *globally* unique, not unique to each node?
>
> Well, they're ... atoms. Is 27.5 globally unique? Two atoms with the
> same textual representation are by definition the same atom.

Right, I see...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

>> I notice also that Erlang sends "Uniq", which is "the hash value of the
>> parse for the fun". In other words, it sounds like it's a
>> cryptographic hash
>> function of the syntax tree for the function's source code. Which is
>> pretty
>> much what the Haskell people are talking about for version control.
>
> I believe that's correct. And the old version sent the MD5 of the range
> of bytes that implement the function, or something. (It's actually the
> AST, which is available also as part of the macro expansion phase of
> compilation.)

The trouble is, if the compiler optimisations change, the BEAM code 
might change, even if what the function actually does is unaffected. 
That [I would hope] is why it hashes the AST, not the compiled code.

OTOH, depending on how it works, there may be more than one AST which is 
trivially equivilent.

OTFH, I believe Rice's theorum states that function equity is 
undecidable. We're not really trying to prove which functions are equal; 
we're trying to prove which ones are *not* equal...

>> So the description is "far from complete"? That's really helpful.
>
> Yet typical for open source, where proponents often claim that the code
> *is* the documentation.

The source code *is* the documentation = EPIC FAILURE.

The usual response I see from OSS is "you should be greatful we bothered 
to write any documentation at all!"

>> Oh, but
>> the protocols "are stable since many years". So I'm guessing English
>> isn't
>> your first language then? :-P
>
> Uh, no. Ericcson.com, corporate headquarters Stockholm, Sweden.

LOLRUS.

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

On 5/17/2011 12:12, Orchid XP v8 wrote:
> My understanding is that Unix is *great* at letting multiple processes read
> the same file - it's *stopping* them doing this that's hard.

Well, there is that, yes.

> Right, OK. So the worst you can do (presumably) is steal lots of CPU time. (?)

Well, you can break the stuff that's shared, like the large binary store or 
the atom intern stuff.

> Come to mention it, what's Erlang's thread scheduler like? Does it just do
> round-robin for all threads ready to run, or can you assign thread
> priorities and stuff?

I don't think there's any priority stuff that I remember.

> Indeed. Erlang cares only about structure, not meaning. (Presumably the
> idiomatic way to distinguish between a ThreadID and a CustomerID would be to
> put an atom in front of it indicating it's "type".)

Pretty much, yes.

> Presumably it just means that if you create more than 2^31 Haskell threads,
> the ThreadID counter will overflow and half the standard library will
> malfunction...

It might not even mean that, if nobody does any comparisons or math with them.

>>> You can upgrade the code, but unless you upgrade it all at once, both
>>> versions need to speak the same wire protocol.
>>
>> Again, list of three integers vs list of seven integers.
>
> There's more to a wire protocol than what types you exchange. (There's
> ordering, for example.)

Well, yes, if you have two versions of Erlang that don't speak the same 
protocol as the other, they won't connect.

> GHC already generates "interface files" which are machine-readable files
> telling the compiler what's exported from each source module. Perhaps you
> could do some trick where you compare the old interface files to the new
> ones or something. (But this requires you to remember to not delete this
> files, which are usually considered "temporary", so probably some more
> specific mechanism is necessary.)

That was the kind of thing I was asking about, yes.

> Does Erlang not permit recursive definitions then?

Not of data. If "list" is already defined, you can't assign to "list" again.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 5/17/2011 12:16, Orchid XP v8 wrote:
> The trouble is, if the compiler optimisations change, the BEAM code might
> change, even if what the function actually does is unaffected. That [I would
> hope] is why it hashes the AST, not the compiled code.

Right. I expect that's why they changed the definition of what goes over the 
whire.

> OTOH, depending on how it works, there may be more than one AST which is
> trivially equivilent.

That wouldn't really matter. There's one place the function is defined in 
the code. If there's a *different* function that does the same thing, then 
it's still a different function.

All they're doing is ensuring the function they have is the same one you 
have. They don't have to prove they're different, only that they're identical.

> We're not really trying to prove which functions are equal; we're trying to
> prove which ones are *not* equal...

Right.

> The usual response I see from OSS is "you should be greatful we bothered to
> write any documentation at all!"

I've even seen "yes, we know the documentation is all wrong and misleading, 
but we didn't bother to put a note on it saying that" (Apache), as well as 
"please, sir, may I write the documentation for you?" "No." (Blender)

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

>> The usual response I see from OSS is "you should be greatful we
>> bothered to write any documentation at all!"
>
> I've even seen "yes, we know the documentation is all wrong and
> misleading, but we didn't bother to put a note on it saying that"
> (Apache), as well as "please, sir, may I write the documentation for
> you?" "No." (Blender)

Use the force - delete the source!

No, wait, that's not quite right...

Anyway, I submitted a patch to GHC last year containing some trivial 
documentation updates. I'm still waiting for it to be merged. (As in, 
they have the new documentation, they literally just need to press a 
button to insert it into the development head... and nobody has done 
this yet. They haven't got around to it.)

In fairness, I suppose they probably need to check it for correctness 
first, but *man*!

...hold the phone! I just went to look at what date I *actually* 
submitted it, and it turns out they applied it yesterday evening. Just 
hours after I was looking at it, in fact. Coincidence, eh?

I still think 5 months to apply one teeny documentation patch is a tad 
slow...

Post a reply to this message

>> My understanding is that Unix is *great* at letting multiple processes
>> read the same file - it's *stopping* them doing this that's hard.
>
> Well, there is that, yes.

It amuses me that there were operating systems 20 years ago that could 
do this better than Linux. It also amuses me that there exists something 
that Windows can actually do better than Linux.

(I gather it's a similar story for figuring out when a file changed...)

>> Come to mention it, what's Erlang's thread scheduler like? Does it
>> just do
>> round-robin for all threads ready to run, or can you assign thread
>> priorities and stuff?
>
> I don't think there's any priority stuff that I remember.

Certainly I've often wished you could assign priorities to Haskell 
threads. But, unfortunately, the thread scheduler doesn't support that.

For that matter, it doesn't support querying how many threads are 
currently running either. Or what their relationships are. Or, indeed, 
anything about them.

(The other day, I was looking at Java, and was amused to note that "each 
thread has a name for identification purposes. Thread names do not need 
to be unique." Well, uh, they do if you want to use names to identify 
threads. :-P Silly Java...)

>> Presumably it just means that if you create more than 2^31 Haskell
>> threads,
>> the ThreadID counter will overflow and half the standard library will
>> malfunction...
>
> It might not even mean that, if nobody does any comparisons or math with
> them.

Testing for equity will still work. I think the only time ordering 
comparisons happen is if you store thread IDs in a BST - and then the 
ordering is arbitrary, it just needs to be total.

Now, if some library somewhere is trying to use thread IDs to infer 
which order threads were created in, *that* will break nicely.

So I guess the libraries don't break until you have 2^32 threads then.

>>>> You can upgrade the code, but unless you upgrade it all at once, both
>>>> versions need to speak the same wire protocol.
>>>
>>> Again, list of three integers vs list of seven integers.
>>
>> There's more to a wire protocol than what types you exchange. (There's
>> ordering, for example.)
>
> Well, yes, if you have two versions of Erlang that don't speak the same
> protocol as the other, they won't connect.

I was thinking more about two versions of the same program. Presumably 
they'll connect, but malfunction.

>> Does Erlang not permit recursive definitions then?
>
> Not of data. If "list" is already defined, you can't assign to "list"
> again.

Neither can you do this with Haskell. But the definition of "list" can 
refer to "list" itself.

Post a reply to this message

On 5/18/2011 1:32, Invisible wrote:
> It amuses me that there were operating systems 20 years ago that could do
> this better than Linux.

Yep. Annoyed me no end, when I had to use UNIX instead of a "real" operating 
system.

> It also amuses me that there exists something that
> Windows can actually do better than Linux.

There's actually quite a bit that Windows now does better than Linux, and 
quite a bit that Linux today does better than Linux five years ago. For 
example, I read that the clock is now *finally* a file handle, like the 
Amiga figured out 20 years ago and which everyone seemed to conveniently 
ignore for the past 40 years when arguing "Everything's a file in UNIX!"

> (I gather it's a similar story for figuring out when a file changed...)

Yes. The IPC stuff in Linux honestly still hasn't caught up to Windows. 
There's no kernel transactions, the file locking is still a mess especially 
over the network, the privilege and authentication system (at the kernel 
level) is still comparatively weak, performance monitoring and error logging 
is not unified, etc.

> be unique." Well, uh, they do if you want to use names to identify threads.

No. You might want to kill all database worker threads and not all web 
server worker threads.  I.e., the same thing you were talking about in 
Haskell code upgrades, where you want to kill all the threads running *this* 
code but not *that* code.

"Identify" doesn't necessarily imply "identity" except in math. :-)

> Testing for equity will still work. I think the only time ordering
> comparisons happen is if you store thread IDs in a BST - and then the
> ordering is arbitrary, it just needs to be total.

Exactly.

> Now, if some library somewhere is trying to use thread IDs to infer which
> order threads were created in, *that* will break nicely.

Exactly.

> I was thinking more about two versions of the same program. Presumably
> they'll connect, but malfunction.

It depends if one is upward compatible with the other. If the new program 
can handle new data and old data both, then they won't malfunction.

> Neither can you do this with Haskell. But the definition of "list" can refer
> to "list" itself.

Oh. No, you can't do that in Erlang, since "list" doesn't exist until you've 
assigned it.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message