 |
|
|
|
|
|
| |
| |
|
|
|
|
| |
| |
|
|
http://tinyurl.com/3q8tkxc
Haskell is great for web development because:
1. Writing
Iterator<Foo> it = list1.iterator();
ArrayList<Foo> list2 = new ArrayList<Foo>();
while (it.hasNext()) list2.add(bar(it.next()));
is way more work than writing
list2 = map bar list1
2. No null pointers. Tightly constrained side-effects. Static typing
powerful enough to statically detect many common bugs at compile-time.
Automated run-time testing tools such as QuickCheck.
3. Thread-safe by default. Concurrency and parallelism are almost
trivial to implement. [Of course, making it actually *go faster* is
never trivial...] Very low cost for using more threads. Technologies
like STM banish race conditions, deadlock, livelock and world poverty.
4. Really easy to link to C for performance-critical sections.
5. Several Haskell web servers / application frameworks with very
competitive performance. (Of course, only the favourable benchmarks are
shown. Obviously.)
http://tinyurl.com/64bfdmv
Simon P. J. enumerating the various Haskell technologies for multi-core
programming: sparks, threads, locks, transactions, parallel arrays...
http://tinyurl.com/678c9xo
Implementing Erlang's distributed processing model in Haskell. (This is
a work in progress. I doubt it's production-ready yet - or any time soon.)
Notable by its absence is hot code update. Some commentators have noted
that if you actually care about high availability, you'll be using
multiple hot-spares anyway, and so long as you don't take down all
systems at once, you can just take one offline, update it, put it back
online, and move on to the next one.
Erlang's basic idea is that you have threads running on multiple
hardware nodes, and they exchange messages. Well, that's not especially
hard. You just need to define a way to convert the data you want to
send/receive into raw binary for transit over the network.
Slightly more tricky is that if a remote process dies, you want to be
notified about it. That just requires some sort of background process
running on each node to generate the notifications. Likewise, Erlang
lets you spawn new threads on remote nodes; that just means you need to
send a message to the corresponding background thread on the target node
asking it to spawn a thread.
Ah, but there's a snag: How do you send functions over the network?
That's really the bit they've solved here. The basic idea is that when
you ask to send a function, what actually gets sent is a representation
of the function's free variables [which need to be sendable too, of
course], followed by the address of the machine code for that function.
All of which obviously fails spectacularly if sender and receiver aren't
running the same version of the same program. ;-)
Actually, sending the machine code address requires modifications to the
compiler (i.e., a way for Haskell code to look up what the hell its
address actually *is*). Currently they've implemented a kludge using
Template Haskell to look up the function name at compile-time and send
that as a text string instead. Also currently you have to do some
legwork yourself which will eventually become automatic (e.g., working
out which functions need to be sendable, making a list of them, and
telling the compiler to generate machine code for them).
Amusingly, using Amazon EC2 you can actually chuck a Haskell program
onto 100 nodes, run it, and see how fast it goes. (Obviously no normal
human actually has access to 100 physical machines.) The graph of
runtime verses number of nodes looks very much like 1/x to me...
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 13/05/2011 12:32, Invisible wrote:
> http://tinyurl.com/678c9xo
>
> Implementing Erlang's distributed processing model in Haskell. (This is
> a work in progress. I doubt it's production-ready yet - or any time soon.)
> Erlang's basic idea is that you have threads running on multiple
> hardware nodes, and they exchange messages.
Of course, Erlang is dynamically typed. By convention, the messages
exchanged are tuples, starting with an atom which identifies the type of
message.
The Haskell implementation does it differently, with two (incompatible)
methods of exchanging messages:
1. Every process implicitly has a "mailbox" that can be used to send or
receive messages of arbitrary [sendable] type. The "send" function sends
a message; the "receive" message waits for a message of the correct type
to arrive, and then returns that. Any messages of the wrong type stay
queued FIFO until an appropriate receive call is executed.
2. You can explicitly create typed "channels". Each such channel can
carry only one type of data, guaranteeing that the receiving end is able
to handle any type of message that you can send. There is also a
function for reading from multiple channels at once, possibly of
different types.
To me, the whole API looks a bit ad hoc and could do with tidying up a
bit. But the fundamental stuff seems to be in place...
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 13/05/2011 12:32, Invisible wrote:
> http://tinyurl.com/64bfdmv
>
> Simon P. J. enumerating the various Haskell technologies for multi-core
> programming: sparks, threads, locks, transactions, parallel arrays...
Darren was asking me about this a while back. Apparently Haskell's
current STM guarantees the following:
- All committed transactions see a consistent image of the system.
- Deadlock is impossible.
- Livelock is impossible.
That seems to cover just about everything, but in fact there *is* one
bad thing that can still happen: starvation.
It is impossible for the system to get stuck in a condition where no
progress is made. (That would be either deadlock or livelock, which are
guaranteed not to happen.) However, it is perfectly possible for one
particular transaction to get stuck. There are two ways this can happen:
1. If a transaction waits for a condition which will never be fulfilled,
it can never complete. (Well, duh!)
2. It is possible for a long-running transaction to be eternally in
conflict with an endless stream of short transactions, such that the
short transactions complete successfully causing the long transaction to
repeatedly restart.
(This is not livelock, since the system is still making progress: All
the short transactions complete just fine. Only the long transaction
gets stuck.)
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/13/2011 4:32, Invisible wrote:
> 4. Really easy to link to C for performance-critical sections.
Do you have an example of what you need to do in Haskell to link to a simple
C function? Is it easy for C to call Haskell functions?
> Implementing Erlang's distributed processing model in Haskell. (This is a
> work in progress. I doubt it's production-ready yet - or any time soon.)
Yah. Python has one of these too.
> Notable by its absence is hot code update. Some commentators have noted that
> if you actually care about high availability, you'll be using multiple
> hot-spares anyway, and so long as you don't take down all systems at once,
> you can just take one offline, update it, put it back online, and move on to
> the next one.
The problem with this is it takes *incredibly* longer. You have to write all
the current data out to disk, shut down cleanly (including all the processes
running there whose code you are *not* updating), then modify the code, then
start everything up again, and read everything back in off disk.
With Erlang's hot update, the data stays in memory. If you're fixing a bug,
you don't have to save out and then reload dozens of gigabytes of in-memory
database just to change the line. Plus, you now have to write all the code
to shut down cleanly and start up cleanly again, and track which tasks are
running in the same process as you're shutting down, shut them down too, etc.
Yeah, you can do it, and that's the answer I give to people who bitch about
Windows needing to reboot to update widely-used code and such, but it really
sucks for an availability system where you have bunches of threads on the
same machine.
> All of which obviously fails spectacularly if sender and receiver aren't
> running the same version of the same program. ;-)
It also fails if the sender and receiver aren't using the same version of
the data structures. As soon as you have half your code updated to be using
floats instead of integers, now you have to have some way in your *old* code
to be able to know you're getting *new* messages and how to handle them.
That can be rough with static typing.
> 1. Every process implicitly has a "mailbox" that can be used to send or
That's the Erlang approach, except of course it's dynamically typed.
> 2. You can explicitly create typed "channels". Each such channel can
That's the NIL / Hermes approach, both of which were statically typed. The
advantage of this is channels can become first class. You can pass them
between processes and have different channels connected to different
processes. You can use it for security, too - if I don't give you a channel
to the screen but only to your window, you can only write to your window,
and it's trivial to verify that.
> (This is not livelock,
I'm not sure about that. I think livelock is defined for individual tasks,
such that no task spends an unbounded amount of time waiting for access to
the critical section. I.e., a system making progress is defined as no part
of the system failing to make progress. But I can't find anything online to
back that up.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 13/05/2011 05:43 PM, Darren New wrote:
> On 5/13/2011 4:32, Invisible wrote:
>> 4. Really easy to link to C for performance-critical sections.
>
> Do you have an example of what you need to do in Haskell to link to a
> simple C function? Is it easy for C to call Haskell functions?
I'm fairly sure I posted one in the other group a while back. Heck, I
even got qsort() from stdlib.h to work arbitrary Haskell data types,
just because Warp said it wasn't possible.
Actually calling a C function is trivial. Passing a Haskell function
pointer to C is trivial. What is very *non*-trivial is getting data
marshalling and memory management to work.
>> Implementing Erlang's distributed processing model in Haskell. (This is a
>> work in progress. I doubt it's production-ready yet - or any time soon.)
>
> Yah. Python has one of these too.
I'm sure lots of people are doing this. After all, the basic
abstractions are not complex.
>> Notable by its absence is hot code update. Some commentators have
>> noted that
>> if you actually care about high availability, you'll be using multiple
>> hot-spares anyway, and so long as you don't take down all systems at
>> once,
>> you can just take one offline, update it, put it back online, and move
>> on to
>> the next one.
>
> The problem with this is it takes *incredibly* longer. You have to write
> all the current data out to disk, shut down cleanly (including all the
> processes running there whose code you are *not* updating), then modify
> the code, then start everything up again, and read everything back in
> off disk.
Perhaps. Perhaps the data is already in some distributed database, so
it's already persistent regardless of whether a node dies or not. (Isn't
the system supposed to recover from node death anyway?)
> With Erlang's hot update, the data stays in memory. If you're fixing a
> bug, you don't have to save out and then reload dozens of gigabytes of
> in-memory database just to change the line. Plus, you now have to write
> all the code to shut down cleanly and start up cleanly again, and track
> which tasks are running in the same process as you're shutting down,
> shut them down too, etc.
With hot update, you *still* have to manually write a bunch of code to
do any data conversions if data structures have changed, or do invariant
checks if the data's invariants have changed. Either way, you gotta
write code.
>> All of which obviously fails spectacularly if sender and receiver aren't
>> running the same version of the same program. ;-)
>
> It also fails if the sender and receiver aren't using the same version
> of the data structures.
Yeah. As I say, the current system is entirely predicated around
everybody having the same code. As soon as that's not true, it's game over.
Obviously that needs fixing for a non-toy implementation.
> As soon as you have half your code updated to be
> using floats instead of integers, now you have to have some way in your
> *old* code to be able to know you're getting *new* messages and how to
> handle them. That can be rough with static typing.
It's already dealing with receiving messages of arbitrary type even in
the presence of static typing. All that's needed is some way to compare
types beyond "do they have the same name?"
>> 1. Every process implicitly has a "mailbox" that can be used to send or
>
> That's the Erlang approach, except of course it's dynamically typed.
>
>> 2. You can explicitly create typed "channels". Each such channel can
>
> That's the NIL / Hermes approach, both of which were statically typed.
> The advantage of this is channels can become first class. You can pass
> them between processes and have different channels connected to
> different processes. You can use it for security, too - if I don't give
> you a channel to the screen but only to your window, you can only write
> to your window, and it's trivial to verify that.
Yeah, passing channels around looks quite useful. It just seems a pitty
to have more or less duplicated functionality in two incompatible
systems - mailboxes and channels. Couldn't they have implemented a
mailbox as a channel of arbitrary type or something?
>> (This is not livelock,
>
> I'm not sure about that.
Well, OK, let me rephrase: There is no situation where an STM program
gets so stuck that nothing can make progress. The worst that can happen
is that one particular transaction gets stuck. (And they're talking
about adding an API to detect / do something about this if it actually
happens.)
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/13/2011 10:11, Orchid XP v8 wrote:
> I'm fairly sure I posted one in the other group a while back.
Yeah, I was just wondering if you had something convenient. I was curious
waht it looked like.
> Actually calling a C function is trivial. Passing a Haskell function pointer
> to C is trivial. What is very *non*-trivial is getting data marshalling and
> memory management to work.
Well, yeah, that's why I was curious. :-)
> I'm sure lots of people are doing this. After all, the basic abstractions
> are not complex.
They actually are pretty complex if you actually want it to be reliable.
Think this: You spawn a process, but before it starts up, it dies. Do you
handle it? Lots of edge cases to get right.
> Perhaps. Perhaps the data is already in some distributed database, so it's
> already persistent regardless of whether a node dies or not. (Isn't the
> system supposed to recover from node death anyway?)
Yes.
To give an example, mnesia (erlang's db) works by storing all the records in
a tree, with indexes being other trees. When you update a tree, it appends a
record to a data file saying what you changed - i.e., it logs the change. If
your node fails and you restart it, it replays the log to recreate the
database. Every once in a while, it takes the whole database and writes it
out to a new file, and starts a new log.
So restarting a node requires first reloading all the records, one at a
time, into a balanced tree. Then replaying all the changes since the last
time the node was saved out to disk. Then playing all the changes made since
the node went down, as copied from some other node.
On a gigabyte CSV file of a half dozen fields per line, on a 3GHz machine
with 100MBps data transfer, this takes about an hour for the first step of
just reloading the saved file. Say you have 50 machines to upgrade. You're
now talking 2 wall days just to restart the new processors, ignoring however
long it might take to dump the stuff out.
> With hot update, you *still* have to manually write a bunch of code to do
> any data conversions if data structures have changed, or do invariant checks
> if the data's invariants have changed. Either way, you gotta write code.
Sure. But you don't have to write code to cleanly shut down *other*
processes when some unrelated process on the same machine gets changed. If I
want to upgrade firefox, I don't have to write any code in the firefox
installer to save out my current Word document, close Word, update firefox,
reopen Word, and reload my Word document.
You can also convert things slowly, using the old data in the new code until
all the old code is gone, then slowly work thru the data (perhaps updating
it each time it's touched) to the new format.
> Yeah. As I say, the current system is entirely predicated around everybody
> having the same code. As soon as that's not true, it's game over.
That, I think, is the fundamental problem with systems like Haskell, and the
fundamental reason why dynamic typing may very well win in this situation.
> It's already dealing with receiving messages of arbitrary type even in the
> presence of static typing. All that's needed is some way to compare types
> beyond "do they have the same name?"
Oh, OK.
> Yeah, passing channels around looks quite useful. It just seems a pitty to
> have more or less duplicated functionality in two incompatible systems -
> mailboxes and channels. Couldn't they have implemented a mailbox as a
> channel of arbitrary type or something?
Two people designing the two systems, maybe?
> Well, OK, let me rephrase: There is no situation where an STM program gets
> so stuck that nothing can make progress. The worst that can happen is that
> one particular transaction gets stuck. (And they're talking about adding an
> API to detect / do something about this if it actually happens.)
Sure. Linux, way back when, would do the same thing with file read/write
locks. If you had four read transactions lasting 3 seconds, with a 1 second
gap, nd you had them all staggered at 1 second intervals, a write lock would
never take. A hold-over from the original UNIX design where kernel calls got
woken up just based on signaling some address they're waiting on and letting
the processes fight it out. That did get fixed a couple years later, tho.
The problem is that when it happens, it kills you. :-)
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Invisible <voi### [at] dev null> wrote:
> 1. Writing
> Iterator<Foo> it = list1.iterator();
> ArrayList<Foo> list2 = new ArrayList<Foo>();
> while (it.hasNext()) list2.add(bar(it.next()));
> is way more work than writing
> list2 = map bar list1
Argument from verbosity is seldom a good argument in programming.
Just because something is shorter doesn't necessarily mean it's better.
Anyways, many languages have constructs to make that shorter. Curiously,
the next C++ standard will introduce constructs that allow you to write:
for(auto& v: list1) list2.push_back(v);
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 14/05/2011 08:53 AM, Warp wrote:
> Invisible<voi### [at] devnull> wrote:
>> 1. Writing
>
>> Iterator<Foo> it = list1.iterator();
>> ArrayList<Foo> list2 = new ArrayList<Foo>();
>> while (it.hasNext()) list2.add(bar(it.next()));
>
>> is way more work than writing
>
>> list2 = map bar list1
>
> Argument from verbosity is seldom a good argument in programming.
> Just because something is shorter doesn't necessarily mean it's better.
You don't think the intension of the second example is clearer at a glance?
> Anyways, many languages have constructs to make that shorter.
Some have a "foreach" construct that can make this kind of thing shorter.
Curiously, Smalltalk had a few library functions to do this kind of
thing. Then again, in Smalltalk, every block of code is an object. That
allowed Smalltalk to offer a #do: method for most container classes.
That means you can then simplement #collect:, #select: and #reject: in
terms of #do:. For example,
list2 := map collect: [ x | x bar ]
That's nearly the same level of directness as Haskell.
> Curiously,
> the next C++ standard will introduce constructs that allow you to write:
>
> for(auto& v: list1) list2.push_back(v);
And that does what? Copy the list?
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Orchid XP v8 <voi### [at] devnull> wrote:
> > Curiously,
> > the next C++ standard will introduce constructs that allow you to write:
> >
> > for(auto& v: list1) list2.push_back(v);
> And that does what? Copy the list?
I actually typoed the line. It misses the call to bar(). It should have
been:
for(auto& v: list1) list2.push_back(bar(v));
If I understand correctly, that's what the original code and the haskell
version are doing.
Naturally if you simply wanted to copy the list you would write a
simple "list2 = list1;" (or if you want to append instead of replace,
"list2.insert(list2.end(), list1.begin(), list1.end());", or if you want
to move the elements, "list2.splice(list2.end(), list1);".)
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 13/05/2011 06:48 PM, Darren New wrote:
> On 5/13/2011 10:11, Orchid XP v8 wrote:
>> I'm fairly sure I posted one in the other group a while back.
>
> Yeah, I was just wondering if you had something convenient. I was
> curious waht it looked like.
Quoting myself from 3 years ago:
module Raw where
import System.Win32.Types
import Graphics.Win32.GDI.Types
import Graphics.Win32.Message
foreign import stdcall "windows.h PostMessageW"
postMessage :: HWND -> WindowMessage -> WPARAM -> LPARAM -> IO LRESULT
This creates a Haskell function named "postMessage". When you call this
function, it really calls PostMessageW() from the windows.h header. Note
carefully that *you* are responsible for getting the type signature right!
How this actually finds the necessary code to execute, I have no idea.
What I do know is that if you want to call your own C function (rather
than something from the Win32 API), you have to tell the compiler to
link the object code in at link time as well.
To call Haskell from C, you write a similar 1-liner to generate a C stub
function which knows the compiler-specific details for calling the
*real* function code.
>> Actually calling a C function is trivial. Passing a Haskell function
>> pointer
>> to C is trivial. What is very *non*-trivial is getting data
>> marshalling and
>> memory management to work.
>
> Well, yeah, that's why I was curious. :-)
You cannot access Haskell data from C at all. You can access C data from
Haskell if you're careful. For anything beyond primitive data (integers,
floats, etc), it's probably easier to write functions in the native
language to get/set the fields you're interested in.
Memory managemet is fun. (Principly because if you get it wrong... well,
you know what happens.) There are Haskell libraries for allocating and
freeing values that the GC doesn't move around (so-called "pinned
memory"). You can attach finalisers to things to have the GC call the
free routine for you when appropriate. (This of course takes only
Haskell into account; it cannot know whether C is still using the object.)
You can also arrange for the main() function to be implemented in C
rather than Haskell. (I believe you have to make sure main() calls some
Haskell startup functions though.) In this way, you can build a Haskell
program that calls C, or a C program that calls Haskell.
You can also build Haskell DLLs. I have no idea how that works...
Speaking of which, GHC now supports compiling each Haskell library as a
DLL, rather than linking them all statically. Unfortunately, this is
broken out-of-the-box. If you compile this way, you have to manually
find all the DLLs your program requires and move them into the search
path, because the GHC installer doesn't do this for you. Oops!
>> I'm sure lots of people are doing this. After all, the basic abstractions
>> are not complex.
>
> They actually are pretty complex if you actually want it to be reliable.
> Think this: You spawn a process, but before it starts up, it dies. Do
> you handle it? Lots of edge cases to get right.
The *abstraction* is simple. The *implementation* is not. The same is
true of many, many things. ;-)
>> Perhaps. Perhaps the data is already in some distributed database, so
>> it's
>> already persistent regardless of whether a node dies or not. (Isn't the
>> system supposed to recover from node death anyway?)
>
> So restarting a node requires first reloading all the records, one at a
> time, into a balanced tree. Then replaying all the changes since the
> last time the node was saved out to disk. Then playing all the changes
> made since the node went down, as copied from some other node.
>
> On a gigabyte CSV file of a half dozen fields per line, on a 3GHz
> machine with 100MBps data transfer, this takes about an hour for the
> first step of just reloading the saved file. Say you have 50 machines to
> upgrade. You're now talking 2 wall days just to restart the new
> processors, ignoring however long it might take to dump the stuff out.
Wouldn't converting one gigabyte of data to a new format online take
just as long as reloading it from disk?
>> With hot update, you *still* have to manually write a bunch of code to do
>> any data conversions if data structures have changed, or do invariant
>> checks
>> if the data's invariants have changed. Either way, you gotta write code.
>
> Sure. But you don't have to write code to cleanly shut down *other*
> processes when some unrelated process on the same machine gets changed.
I'm not seeing why you would need to do this with cold code update
either. You only need to shut down the processes related to the thing
you're actually changing.
> You can also convert things slowly, using the old data in the new code
> until all the old code is gone, then slowly work thru the data (perhaps
> updating it each time it's touched) to the new format.
This is an unavoidable requirement which *must* be met if you want
distributed processing. Otherwise upgrading the system requires stopping
the entire distributed network.
>> Yeah. As I say, the current system is entirely predicated around
>> everybody
>> having the same code. As soon as that's not true, it's game over.
>
> That, I think, is the fundamental problem with systems like Haskell, and
> the fundamental reason why dynamic typing may very well win in this
> situation.
No, this is the fundamental problem with a small proof-of-concept
implementation exploring whether it's even slightly possible to do this.
Obviously the current implementation does not yet provide everything
necessary for production use. Nobody is claiming it does.
>> It's already dealing with receiving messages of arbitrary type even in
>> the
>> presence of static typing. All that's needed is some way to compare types
>> beyond "do they have the same name?"
>
> Oh, OK.
Currently, every time you send a value, the implementation transmits the
name of the value's type, followed by the value itself. The receiving
end uses the type name to figure out which parser to deserialise with.
Similarly, functions are sent by name (along with their free variables).
This works great for testing that your implementation works. It breaks
worribly as soon as you have more than one version of your code. (Not to
mention two unrelated applications both of which happen to define
unrelated types having identical names...)
What is needed - and the documentation makes plain that the implementors
are quite aware of this - is some way of varifying that the identifiers
you send actually refer to the same thing. I don't actually know how
Erlang manages to do this; looks like the Haskell guys are planning to
use hashes of [some canonical representation of] the type definitions or
something...
>> Yeah, passing channels around looks quite useful. It just seems a
>> pitty to
>> have more or less duplicated functionality in two incompatible systems -
>> mailboxes and channels. Couldn't they have implemented a mailbox as a
>> channel of arbitrary type or something?
>
> Two people designing the two systems, maybe?
Looks more like they built the mailbox system, realised that there was a
guarantee it can't provide, and then engineered a second, independent
system which does.
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/14/2011 0:53, Warp wrote:
> Invisible<voi### [at] devnull> wrote:
>> 1. Writing
>
>> Iterator<Foo> it = list1.iterator();
>> ArrayList<Foo> list2 = new ArrayList<Foo>();
>> while (it.hasNext()) list2.add(bar(it.next()));
>
>> is way more work than writing
>
>> list2 = map bar list1
>
> Argument from verbosity is seldom a good argument in programming.
> Just because something is shorter doesn't necessarily mean it's better.
I would say in this case it is, because all the extra stuff is just
verbiage. It obscures the algorithm you're actually interested in, which is
to make a new list where each element is the old list with 'bar' applied to
it. All the stuff with the iterator and the types and all is just junk you
have to repeat wherever you need to do this.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/14/2011 7:53, Orchid XP v8 wrote:
> foreign import stdcall "windows.h PostMessageW"
> postMessage :: HWND -> WindowMessage -> WPARAM -> LPARAM -> IO LRESULT
Oh yea. Very nice, yes. Amazing how many of the pre-C# languages made such
a thing so tedious.
> How this actually finds the necessary code to execute, I have no idea.
That's why they call it "the registry".
> You cannot access Haskell data from C at all. You can access C data from
> Haskell if you're careful.
OK. That's what I figured. I couldn't imagine how C was getting to lazy
data easily.
> Memory managemet is fun. (Principly because if you get it wrong... well, you
> know what happens.) There are Haskell libraries for allocating and freeing
> values that the GC doesn't move around (so-called "pinned memory").
In C#, you pin the memory from C# and then invoke your C code. Similar idea.
> You can also arrange for the main() function to be implemented in C rather
> than Haskell. (I believe you have to make sure main() calls some Haskell
> startup functions though.) In this way, you can build a Haskell program that
> calls C, or a C program that calls Haskell.
Yep. Ada even standardizes the names of the functions for this stuff.
> You can also build Haskell DLLs. I have no idea how that works...
>
> Speaking of which, GHC now supports compiling each Haskell library as a DLL,
> rather than linking them all statically. Unfortunately, this is broken
> out-of-the-box. If you compile this way, you have to manually find all the
> DLLs your program requires and move them into the search path, because the
> GHC installer doesn't do this for you. Oops!
That's why they call it the registry. :-)
> The *abstraction* is simple. The *implementation* is not. The same is true
> of many, many things. ;-)
It's simple after it's pointed out, methinks. $1 - hammer tap. $99 - knowing
where to tap.
> Wouldn't converting one gigabyte of data to a new format online take just as
> long as reloading it from disk?
I don't know what you mean. You have to get it off the disk and into memory,
if you've terminated the process and start it up again.
And yes, it would take just as long, but you don't have to do it all at
once. You can be servicing requests, and reformatting data while you're not
servicing requests.
> I'm not seeing why you would need to do this with cold code update either.
> You only need to shut down the processes related to the thing you're
> actually changing.
Then you have a bunch of different processes on the same machine fighting
over resources. Sending messages between them means you now have a bunch of
context switches, etc.
>> You can also convert things slowly, using the old data in the new code
>> until all the old code is gone, then slowly work thru the data (perhaps
>> updating it each time it's touched) to the new format.
>
> This is an unavoidable requirement which *must* be met if you want
> distributed processing. Otherwise upgrading the system requires stopping the
> entire distributed network.
Yep. And that's why building a big system like this with static type-erased
data is more difficult than dynamic tagged data.
>>> Yeah. As I say, the current system is entirely predicated around
>>> everybody
>>> having the same code. As soon as that's not true, it's game over.
>>
>> That, I think, is the fundamental problem with systems like Haskell, and
>> the fundamental reason why dynamic typing may very well win in this
>> situation.
>
> No, this is the fundamental problem with a small proof-of-concept
> implementation exploring whether it's even slightly possible to do this.
> Obviously the current implementation does not yet provide everything
> necessary for production use. Nobody is claiming it does.
Sure. I'm just not sure they've proved the concept if they can't handle
that. :-) Sort of like the STM proof of concept in .NET. ;-)
> What is needed - and the documentation makes plain that the implementors are
> quite aware of this - is some way of varifying that the identifiers you send
> actually refer to the same thing. I don't actually know how Erlang manages
> to do this;
Dynamic typing. :-)
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Orchid XP v8 <voi### [at] devnull> wrote:
> I'm fairly sure I posted one in the other group a while back. Heck, I
> even got qsort() from stdlib.h to work arbitrary Haskell data types,
> just because Warp said it wasn't possible.
While calling C's qsort() from Haskell is impressive, you still failed
to demonstrate the next challenge, which was to properly handle C bitfields.
So there's still potentially something that Haskell cannot do with C... :P
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/15/2011 3:22, Warp wrote:
> which was to properly handle C bitfields.
Given that C doesn't define what order bit fields go in, that seems like a
tall order. You can't technically do that from assembler, either. :-)
Otherwise, it would seem pretty easy to write Haskell code (or any other
language that handles integer math) to just use masks and multiplies and
divides.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 15/05/2011 05:41 PM, Darren New wrote:
> On 5/15/2011 3:22, Warp wrote:
>> which was to properly handle C bitfields.
>
> Given that C doesn't define what order bit fields go in, that seems like
> a tall order. You can't technically do that from assembler, either. :-)
>
> Otherwise, it would seem pretty easy to write Haskell code (or any other
> language that handles integer math) to just use masks and multiplies and
> divides.
There's a Haskell standard library that offers all the usual bit-level
operators, so it's not like you have to multiply by 2 when what you
really want is a left-shift.
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 15/05/2011 11:22 AM, Warp wrote:
> Orchid XP v8<voi### [at] devnull> wrote:
>> I'm fairly sure I posted one in the other group a while back. Heck, I
>> even got qsort() from stdlib.h to work arbitrary Haskell data types,
>> just because Warp said it wasn't possible.
>
> While calling C's qsort() from Haskell is impressive, you still failed
> to demonstrate the next challenge, which was to properly handle C bitfields.
I thought the next level was setjmp()?
> So there's still potentially something that Haskell cannot do with C... :P
If it's really that hard, ask C to do it, and call C from Haskell.
For that matter, C can't access [non-trivial] Haskell data either. If
your C program wants to, say, access the 7th element of a Haskell list,
it has to call Haskell and ask Haskell to fetch it and return something
simple enough for C to comprehend. (The 7th element might not *exist*
yet, for example...)
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> Argument from verbosity is seldom a good argument in programming.
>> Just because something is shorter doesn't necessarily mean it's better.
>
> I would say in this case it is, because all the extra stuff is just
> verbiage. It obscures the algorithm you're actually interested in, which
> is to make a new list where each element is the old list with 'bar'
> applied to it. All the stuff with the iterator and the types and all is
> just junk you have to repeat wherever you need to do this.
A far more relevant objection would be "how often does Haskell let you
do less work?"
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> foreign import stdcall "windows.h PostMessageW"
>> postMessage :: HWND -> WindowMessage -> WPARAM -> LPARAM -> IO LRESULT
>
> Oh yea. Very nice, yes. Amazing how many of the pre-C# languages made
> such a thing so tedious.
Hey, our language is *great*! Why would you ever want to call anything else?
>> How this actually finds the necessary code to execute, I have no idea.
>
> That's why they call it "the registry".
Well, no. See, if you call a Win32 function, it just works. If you call
a function that you wrote yourself, you have to tell the linker to link
the corresponding object code.
As I understand it, calling Win32 works by calling a C stub function
which does the necessary machine code magic to actually invoke the
Windows kernel, whatever that may be. But I don't see anywhere where
this gets linked in...
>> You cannot access Haskell data from C at all. You can access C data from
>> Haskell if you're careful.
>
> OK. That's what I figured. I couldn't imagine how C was getting to lazy
> data easily.
You *can* do it of course, but it'll probably break with the next
release of the compiler, so nobody does it. Instead, you write Haskell
stub functions to fetch whatever item of data you want, and call that
from C. (Or just design the program the other way around.)
I'm unsure how much runtime overhead is involved in calling between C
and Haskell.
>> You can also build Haskell DLLs. I have no idea how that works...
>>
>> Speaking of which, GHC now supports compiling each Haskell library as
>> a DLL,
>> rather than linking them all statically. Unfortunately, this is broken
>> out-of-the-box. If you compile this way, you have to manually find all
>> the
>> DLLs your program requires and move them into the search path, because
>> the
>> GHC installer doesn't do this for you. Oops!
>
> That's why they call it the registry. :-)
The general opinion among Haskell developers is that Windows is "that
evil platform that we shouldn't have to support". In particular, DLLs
and the registry are both regarded as being impossibly complex to use,
and to be avoided to the maximum extent possible.
Which is a bit like developing software for Linux and insisting that
everybody should avoid using Bash.
>> Wouldn't converting one gigabyte of data to a new format online take
>> just as long as reloading it from disk?
>
> And yes, it would take just as long, but you don't have to do it all at
> once. You can be servicing requests, and reformatting data while you're
> not servicing requests.
OK, fair enough.
>> I'm not seeing why you would need to do this with cold code update
>> either.
>> You only need to shut down the processes related to the thing you're
>> actually changing.
>
> Then you have a bunch of different processes on the same machine
> fighting over resources. Sending messages between them means you now
> have a bunch of context switches, etc.
I don't follow.
>>> You can also convert things slowly, using the old data in the new code
>>> until all the old code is gone, then slowly work thru the data (perhaps
>>> updating it each time it's touched) to the new format.
>>
>> This is an unavoidable requirement which *must* be met if you want
>> distributed processing. Otherwise upgrading the system requires
>> stopping the
>> entire distributed network.
>
> Yep. And that's why building a big system like this with static
> type-erased data is more difficult than dynamic tagged data.
...which is why the Haskell implementation is using dynamic tagged data.
>> No, this is the fundamental problem with a small proof-of-concept
>> implementation exploring whether it's even slightly possible to do this.
>> Obviously the current implementation does not yet provide everything
>> necessary for production use. Nobody is claiming it does.
>
> Sure. I'm just not sure they've proved the concept if they can't handle
> that. :-)
Can't handle it _yet_. They have concrete plans for how they would
implement it, they just haven't yet.
>> What is needed - and the documentation makes plain that the
>> implementors are
>> quite aware of this - is some way of varifying that the identifiers
>> you send
>> actually refer to the same thing. I don't actually know how Erlang
>> manages
>> to do this;
>
> Dynamic typing. :-)
That's not an answer. Haskell is typing values sent over the wire
dynamically too. That doesn't automatically solve the problem.
There are two things to check: data structure type names and function
names. The Erlang answer seems to be:
1. Make no attempt to verify whether data structures match at all. If
the old code excepts data in one format and the new version expects data
in an almost identical yet slightly different format, when the new code
fires up it will just crash due to the data mismatch. Unless you
specifically tested for this and wrote lots of code to make it work.
(By contrast, Haskell might potentially be able to distinguish the old
type and the new type at the type level, if the structure of the type
has actually changed, and issue a compile-time warning.)
2. I have literally no idea how Erlang sends functions over the wire. I
had assumed it just sends the VM executable code over the wire.
Obviously that only works if you're running on a VM. Haskell isn't.
(There *is* a VM, but it's fairly slow, and has various other gotchas.)
Haskell is sending the function name over the wire, but that only works
if the name means the same thing at the other end.
In both cases, it looks like you need a name plus some kind of version
number, or maybe a hash of a description or something, to distinguish
things that the programmer has assigned identical names to...
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/16/2011 1:06, Invisible wrote:
> Well, no. See, if you call a Win32 function, it just works. If you call a
> function that you wrote yourself, you have to tell the linker to link the
> corresponding object code.
Unless you register it, I expect.
> As I understand it, calling Win32 works by calling a C stub function which
> does the necessary machine code magic to actually invoke the Windows kernel,
> whatever that may be. But I don't see anywhere where this gets linked in...
It's entirely possible the declaration itself generates that code, isn't it?
I'm pretty sure that's how it works in C#: you say "C# XYZ(i) calls
C-languge PDQ with a (short) parameter" and the compiler generates an XYZ
function that casts its argument to a short and jumps to the PDQ function,
basically.
> the compiler, so nobody does it. Instead, you write Haskell stub functions
Yep. Kind of like calling C++ from C.
> Which is a bit like developing software for Linux and insisting that
> everybody should avoid using Bash.
Yep. More like developing distributed programs for Linux and deciding that
sockets are too hard to use.
>>> I'm not seeing why you would need to do this with cold code update
>>> either.
>>> You only need to shut down the processes related to the thing you're
>>> actually changing.
>>
>> Then you have a bunch of different processes on the same machine
>> fighting over resources. Sending messages between them means you now
>> have a bunch of context switches, etc.
>
> I don't follow.
It's less efficient to have the kernel scheduler scheduling the different
processes than to have the Erlang-specific scheduler doing it. That's why
people make 100,000 erlang tasks in one process and doing the same in Linux
will bring the kernel to its knees, if not run you out of memory.
>> Yep. And that's why building a big system like this with static
>> type-erased data is more difficult than dynamic tagged data.
>
> ...which is why the Haskell implementation is using dynamic tagged data.
Yep. And that loses a lot of the benefits of the static typing right there.
> Can't handle it _yet_. They have concrete plans for how they would implement
> it, they just haven't yet.
Fair enough. I look forward to seeing what they come up with.
> 1. Make no attempt to verify whether data structures match at all. If the
> old code excepts data in one format and the new version expects data in an
> almost identical yet slightly different format, when the new code fires up
> it will just crash due to the data mismatch. Unless you specifically tested
> for this and wrote lots of code to make it work.
Right.
Actually, even worse, in Erlang, the likelihood is that you just leave the
mismatched message in the buffer, which then grows until it crashes the
whole machine with no obvious reason. One of the poorly thought-out designs
there, methinks.
> (By contrast, Haskell might potentially be able to distinguish the old type
> and the new type at the type level, if the structure of the type has
> actually changed, and issue a compile-time warning.)
This assumes you know at compile time what types the other side is using,
which also isn't always true. :-)
> 2. I have literally no idea how Erlang sends functions over the wire.
It's out in the erldocs somewhere, but I can't find it after five minutes of
looking around. They specify the wire format of everything, including that.
> assumed it just sends the VM executable code over the wire.
Given that I can write a function on an x86 machine and ship it over to a
program running on a Sparc that has been running since before I bought the
x86 machine, yah, I'd say there's some sort of virtual code being
transfered. :-)
> In both cases, it looks like you need a name plus some kind of version
> number, or maybe a hash of a description or something, to distinguish things
> that the programmer has assigned identical names to...
It depends on whether you're sending a function or the name of the function,
really.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 16/05/2011 20:18, Darren New wrote:
> On 5/16/2011 1:06, Invisible wrote:
>> Well, no. See, if you call a Win32 function, it just works. If you call a
>> function that you wrote yourself, you have to tell the linker to link the
>> corresponding object code.
>
> Unless you register it, I expect.
Well, no, what I'm saying is that if you call a function that's defined
in (say) windows.h, the linker somehow "finds" it. But if you wrote some
C source code of your own, you have to tell the linker to link in the
object code or it complains it can't find it.
It makes sense that you have to tell the linker where to find your own
code. What I can't figure out is how it somehow "knows" when a function
is from the Win32 API and magically finds it.
(The answer is probably something like "GHC automatically includes the
Win32 files at link time".)
>> As I understand it, calling Win32 works by calling a C stub function
>> which
>> does the necessary machine code magic to actually invoke the Windows
>> kernel,
>> whatever that may be. But I don't see anywhere where this gets linked
>> in...
>
> It's entirely possible the declaration itself generates that code, isn't
> it? I'm pretty sure that's how it works in C#: you say "C# XYZ(i) calls
> C-languge PDQ with a (short) parameter" and the compiler generates an
> XYZ function that casts its argument to a short and jumps to the PDQ
> function, basically.
If I do a "foreign import", the Haskell compiler generates a Haskell
function which calls the corresponding C function. Which is fine if
you're statically linking the C code into your binary. But you don't
link the Windows kernel into your application, do you? It does some
weird magic with processor rings and code gates to enter kernel mode to
run the code.
I'm no expert, but as I say, I was under the impression that the usual
way to access a DLL is to link in a small C stub which defines the code
necessary to dynamically find and execute the exported functions from
the DLL.
>> Which is a bit like developing software for Linux and insisting that
>> everybody should avoid using Bash.
>
> Yep. More like developing distributed programs for Linux and deciding
> that sockets are too hard to use.
LOL! Yeah...
Seriously. Everybody talks about "hey, it's cool. Oh, except on Windows,
where we have to solve DLL Hell. But hey, who actually uses Windows?"
Personally, I thought "DLL hell" went away about 10 years ago...
>>>> I'm not seeing why you would need to do this with cold code update
>>>> either.
>>>> You only need to shut down the processes related to the thing you're
>>>> actually changing.
>>>
>>> Then you have a bunch of different processes on the same machine
>>> fighting over resources. Sending messages between them means you now
>>> have a bunch of context switches, etc.
>>
>> I don't follow.
>
> It's less efficient to have the kernel scheduler scheduling the
> different processes than to have the Erlang-specific scheduler doing it.
> That's why people make 100,000 erlang tasks in one process and doing the
> same in Linux will bring the kernel to its knees, if not run you out of
> memory.
Right. So all you're saying is that OS processes are heavier than Erlang
processes.
One thing Erlang can do (and Haskell can't, easily) is that because
Erlang is a VM, you can have several unrelated Erlang applications
running on the same VM. But to run multiple Haskell applications, you
would have to have multiple copies of the Haskell runtime in action.
Which, as you say, isn't something you want to do too much of.
>>> Yep. And that's why building a big system like this with static
>>> type-erased data is more difficult than dynamic tagged data.
>>
>> ...which is why the Haskell implementation is using dynamic tagged data.
>
> Yep. And that loses a lot of the benefits of the static typing right there.
*sigh* This old argument again.
Yes, the entire program is statically typed, but this one tiny part
where you have to do a simple runtime type check automatically
COMPLETELY DESTROYS EVERY SINGLE ADVANTAGE OF STATIC TYPING!
>> Can't handle it _yet_. They have concrete plans for how they would
>> implement it, they just haven't yet.
>
> Fair enough. I look forward to seeing what they come up with.
I look forward to seeing whether they ever finish it.
(I lose count of how many projects targeting Haskell at the GPU have
been started. The number of them which produced a production-grade end
product is zero.)
>> 1. Make no attempt to verify whether data structures match at all.
>
> Right.
>
> Actually, even worse, in Erlang, the likelihood is that you just leave
> the mismatched message in the buffer, which then grows until it crashes
> the whole machine with no obvious reason. One of the poorly thought-out
> designs there, methinks.
I'll say!
Basically you need to manually make sure you send a version number at
the start of your message exchange, or something like that, and make
sure that all servers and clients you write can handle all versions of
the protocol.
Or design the protocol so that it never needs to change. (Which is not
infeasible if you can send code as part of the protocol I suppose...)
>> (By contrast, Haskell might potentially be able to distinguish the old
>> type
>> and the new type at the type level, if the structure of the type has
>> actually changed, and issue a compile-time warning.)
>
> This assumes you know at compile time what types the other side is
> using, which also isn't always true. :-)
Not really. When you compile v1, you assume that the other side will be
v1 as well. When you compile v2, the compiler checks what you changed,
and warns you if you don't explicitly handle the v1 data (unless it's
identical).
>> 2. I have literally no idea how Erlang sends functions over the wire.
>
> It's out in the erldocs somewhere, but I can't find it after five
> minutes of looking around. They specify the wire format of everything,
> including that.
Yeah, I figured.
The Haskell implementation has a potential advantage in that you can
specify precisely how to serialise stuff if you want to. (Usually to
send a more compact representation by leaving off data which can be
recomputed at the other side, making use of the special structure of the
data to skip parts of it, etc.) Of course, the bit that tells the other
side who the message is for and what type it contains isn't open to
negotiation. ;-)
>> assumed it just sends the VM executable code over the wire.
>
> Given that I can write a function on an x86 machine and ship it over to
> a program running on a Sparc that has been running since before I bought
> the x86 machine, yah, I'd say there's some sort of virtual code being
> transfered. :-)
And yet, that would seem rather heavy compared to just sending a
function name (or rather, a *unique* function identifier), so perhaps
Erlang actually does it that way like Haskell is doing. Or maybe it only
sends the code if the other side doesn't already have it or something.
Come to think of it, I'm still figuring out how it manages to send atoms...
>> In both cases, it looks like you need a name plus some kind of version
>> number, or maybe a hash of a description or something, to distinguish
>> things
>> that the programmer has assigned identical names to...
>
> It depends on whether you're sending a function or the name of the
> function, really.
Well, we're sending type names too, which give you the same problem.
(And sending a type definition doesn't help, because we still need to
figure out whether they're *meant* to be the same type or not.)
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/17/2011 1:07, Invisible wrote:
>> Unless you register it, I expect.
>
> Well, no, what I'm saying is that if you call a function that's defined in
> (say) windows.h, the linker somehow "finds" it.
Right. It's probably in the C runtime.
> But if you wrote some C
> source code of your own, you have to tell the linker to link in the object
> code or it complains it can't find it.
Right. Unless you register it. Then the linker looks up the code in the
registry and links it against the file that the registry says the code is
in. That's what the registry is for, and why it's called a registry.
You may need to make it a COM DLL first to register it; I don't remember the
exact details any more.
> (The answer is probably something like "GHC automatically includes the Win32
> files at link time".)
That's my guess.
> If I do a "foreign import", the Haskell compiler generates a Haskell
> function which calls the corresponding C function. Which is fine if you're
> statically linking the C code into your binary. But you don't link the
> Windows kernel into your application, do you? It does some weird magic with
> processor rings and code gates to enter kernel mode to run the code.
In every system I've ever used, the "weird magic" is in a linker file that
always gets loaded. For example, msvcrt.dll (Microsoft visual C run time)
for Windows, glib for GCC under Linux, etc.
> I'm no expert, but as I say, I was under the impression that the usual way
> to access a DLL is to link in a small C stub which defines the code
> necessary to dynamically find and execute the exported functions from the DLL.
Or branch to a jump table that the loader fills in with addresses from the
DLL when it loads the DLL, which it loads based on the headers of the program.
Most systems do almost all of this automatically specifically because C
doesn't have any standard syntax for doing this. I suspect if
dynamically-loaded code was the norm in C from the beginning, much less
would be automated by the OS's loader.
> Personally, I thought "DLL hell" went away about 10 years ago...
Yes, pretty much.
> Right. So all you're saying is that OS processes are heavier than Erlang
> processes.
Well, much heavier than Linux or Windows processes. By many, many orders of
magnitude. There are OSes where processes are very light, like Singularity,
Mach, AmigaOS, etc. Generally what you hear referred to as a "microkernel"
is a kernel where processes are so lightweight that you can afford to put
each device driver and file system in its own process.
> One thing Erlang can do (and Haskell can't, easily) is that because Erlang
> is a VM, you can have several unrelated Erlang applications running on the
> same VM.
Yes. Indeed, the whole point of the VM is to be a virtual machine, so the
idea of an "application" is a user-level construct, not something built into
the language. (In much the same way that a "login" is a user-level
construct in UNIX/Linux, whereas in most other OSes, it's an OS-level
construct.)
> Yes, the entire program is statically typed, but this one tiny part where
> you have to do a simple runtime type check automatically COMPLETELY DESTROYS
> EVERY SINGLE ADVANTAGE OF STATIC TYPING!
I didn't say that. I said it loses a lot of the benefits, where "benefits"
are "guarantees" in my mind.
Now, in this particular case, it probably is doing enough work in the
connected set of processes that converting from dynamic to static typing is
worthwhile. In the discussions we were having before, I was talking about
things like web pages, that almost never do any significant processing on
their own at the server, so statically typing a web page where you're
translating dynamically typed SQL results into HTML doesn't really help to
have static typing. I.e., when it's not one tiny part but rather a majority
of the program, it's worse.
> Basically you need to manually make sure you send a version number at the
> start of your message exchange, or something like that, and make sure that
> all servers and clients you write can handle all versions of the protocol.
Or you just check the types and do the right thing with them. Of course,
knowing what the right thing is can be difficult. E.g., if you have been
working with 2D longitude/latitude, and the new version needs altitude also,
you could have all the places that take a 2-element tuple start also working
with a 3-element tuple and asking how many elements it has, then start
generating 3-element tuples. The number of elements in the tuple can serve
as your version number.
> Or design the protocol so that it never needs to change. (Which is not
> infeasible if you can send code as part of the protocol I suppose...)
Well, they handle it by letting you upgrade the code.
> Not really. When you compile v1, you assume that the other side will be v1
> as well. When you compile v2, the compiler checks what you changed, and
> warns you if you don't explicitly handle the v1 data (unless it's identical).
That's fair. It would be interesting to see how that's specified.
> The Haskell implementation has a potential advantage in that you can specify
> precisely how to serialise stuff if you want to.
You just do that in code with Erlang. I.e., you write your own serializer,
or you send a message to a local agent that then sends the message to the
remote agent that has only what you want.
> And yet, that would seem rather heavy compared to just sending a function
> name (or rather, a *unique* function identifier),
Not all functions have names. For the functions that have names, I believe
you just send the fully-qualified name of the function. (And since function
names are atoms, the actual text of the function name is cached after the
first time, so you don't have to send more than "the 37'th function I've
sent you" next time.)
If you send a lambda, I believe it sends the bytecode. There may be caching
involved there too, but I don't remember.
I was too awed by the landmines in the data format to remember the other
details.
> Come to think of it, I'm still figuring out how it manages to send atoms...
It sends them as text, the first time.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
More erlang fanning:
The protocol between the deamons:
http://www.erlang.org/doc/apps/erts/erl_dist_protocol.html
The protocol between the individual actors:
http://www.erlang.org/doc/apps/erts/erl_ext_dist.html
You want to look at 8.21 and 8.22. It seems the code doesn't ship, since
logically the function can't really be constructed at runtime. Instead, it's
a reference into the BEAM file (where a BEAM is like a java jar).
Because of course "external term format" is exactly what anyone would google
for to find that. I'm also rather amused that altho Erlang has wonderfully
powerful expressions for parsing byte streams, they used ASCII art with
ambiguous textual descriptions to describe the format of the stuff on the
wire instead of giving you the Erlang expression that would parse it
unambiguously.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> But if you wrote some C
>> source code of your own, you have to tell the linker to link in the
>> object
>> code or it complains it can't find it.
>
> Right. Unless you register it. Then the linker looks up the code in the
> registry and links it against the file that the registry says the code
> is in. That's what the registry is for, and why it's called a registry.
>
> You may need to make it a COM DLL first to register it; I don't remember
> the exact details any more.
I was under the impression that the registry is *only* for dynamic
linking, and has nothing to do with static linking [which is what I'm
talking about here].
> In every system I've ever used, the "weird magic" is in a linker file
> that always gets loaded. For example, msvcrt.dll (Microsoft visual C run
> time) for Windows, glib for GCC under Linux, etc.
Right. That's probably it...
> Most systems do almost all of this automatically specifically because C
> doesn't have any standard syntax for doing this. I suspect if
> dynamically-loaded code was the norm in C from the beginning, much less
> would be automated by the OS's loader.
On the other hand, the main advantage of C is that almost nothing is
built-in, which means that everything can be changed... ;-)
>> Personally, I thought "DLL hell" went away about 10 years ago...
>
> Yes, pretty much.
This is from the people who actually had to ask "does Windows actually
support letting multiple processes open the same disk file at once?"
>> One thing Erlang can do (and Haskell can't, easily) is that because
>> Erlang
>> is a VM, you can have several unrelated Erlang applications running on
>> the
>> same VM.
>
> Yes. Indeed, the whole point of the VM is to be a virtual machine, so
> the idea of an "application" is a user-level construct, not something
> built into the language.
Fair enough.
Question: Does that mean that one process can produce so much garbage
that it forces endless GC cycles, slowing other unrelated processes
down? Or does the VM partition memory on a per-process basis or something?
>> Yes, the entire program is statically typed, but this one tiny part where
>> you have to do a simple runtime type check automatically COMPLETELY
>> DESTROYS EVERY SINGLE ADVANTAGE OF STATIC TYPING!
>
> I didn't say that. I said it loses a lot of the benefits, where
> "benefits" are "guarantees" in my mind.
>
> Now, in this particular case, it probably is doing enough work in the
> connected set of processes that converting from dynamic to static typing
> is worthwhile. In the discussions we were having before, I was talking
> about things like web pages, that almost never do any significant
> processing on their own at the server, so statically typing a web page
> where you're translating dynamically typed SQL results into HTML doesn't
> really help to have static typing. I.e., when it's not one tiny part but
> rather a majority of the program, it's worse.
>
>> Basically you need to manually make sure you send a version number at the
>> start of your message exchange, or something like that, and make sure
>> that
>> all servers and clients you write can handle all versions of the
>> protocol.
>
> Or you just check the types and do the right thing with them.
Well yeah, but if you're trying to use a mere textual "name" to decide
what type something is, you need a version number or something to go
with the name, to make it unique.
What Erlang is doing is considering any two types to be "identical" if
they have the same *structure*. That wouldn't do for Haskell; consider,
for example, how a PortID and a ThreadID are both 32-bit signed
integers, but are considered utterly different, incompatible types.
[Of course, TRWTF is that it's *signed*...]
>> Or design the protocol so that it never needs to change. (Which is not
>> infeasible if you can send code as part of the protocol I suppose...)
>
> Well, they handle it by letting you upgrade the code.
You can upgrade the code, but unless you upgrade it all at once, both
versions need to speak the same wire protocol.
>> Not really. When you compile v1, you assume that the other side will
>> be v1
>> as well. When you compile v2, the compiler checks what you changed, and
>> warns you if you don't explicitly handle the v1 data (unless it's
>> identical).
>
> That's fair. It would be interesting to see how that's specified.
I guess we sit back and see if they ever release anything...
>> The Haskell implementation has a potential advantage in that you can
>> specify
>> precisely how to serialise stuff if you want to.
>
> You just do that in code with Erlang. I.e., you write your own
> serializer, or you send a message to a local agent that then sends the
> message to the remote agent that has only what you want.
Yeah, I guess that works too.
Presumably Erlang allows you to serialise cyclic data structures, which
is notoriously hard for a pure functional language.
>> And yet, that would seem rather heavy compared to just sending a function
>> name (or rather, a *unique* function identifier),
>
> Not all functions have names. For the functions that have names, I
> believe you just send the fully-qualified name of the function.
Only works if foo/3 on node X is the same thing as foo/3 on node Y. ;-)
> If you send a lambda, I believe it sends the bytecode. There may be
> caching involved there too, but I don't remember.
>
> I was too awed by the landmines in the data format to remember the other
> details.
I'll have to check it out sometime...
>> Come to think of it, I'm still figuring out how it manages to send
>> atoms...
>
> It sends them as text, the first time.
Oh, right. So atoms are *globally* unique, not unique to each node?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 17/05/2011 16:19, Darren New wrote:
> More erlang fanning:
>
> The protocol between the deamons:
>
> http://www.erlang.org/doc/apps/erts/erl_dist_protocol.html
>
> The protocol between the individual actors:
>
> http://www.erlang.org/doc/apps/erts/erl_ext_dist.html
Ouch. The pain...
> You want to look at 8.21 and 8.22. It seems the code doesn't ship, since
> logically the function can't really be constructed at runtime. Instead,
> it's a reference into the BEAM file (where a BEAM is like a java jar).
Erlang sends an index into the BEAM file, the eventual Haskell
implementation is going to send linker labels (i.e., indexes into the
executable image).
I notice also that Erlang sends "Uniq", which is "the hash value of the
parse for the fun". In other words, it sounds like it's a cryptographic
hash function of the syntax tree for the function's source code. Which
is pretty much what the Haskell people are talking about for version
control.
> Because of course "external term format" is exactly what anyone would
> google for to find that.
Oh, but of course!
> I'm also rather amused that altho Erlang has
> wonderfully powerful expressions for parsing byte streams, they used
> ASCII art with ambiguous textual descriptions to describe the format of
> the stuff on the wire instead of giving you the Erlang expression that
> would parse it unambiguously.
Myself, I was amused by
"The description here is far from complete and will therefore be further
refined in upcoming releases. The protocols both from Erlang nodes
towards EPMD (Erlang Port Mapper Daemon) and between Erlang nodes,
however, are stable since many years. "
So the description is "far from complete"? That's really helpful. Oh,
but the protocols "are stable since many years". So I'm guessing English
isn't your first language then? :-P
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/17/2011 8:22, Invisible wrote:
> I was under the impression that the registry is *only* for dynamic linking,
> and has nothing to do with static linking [which is what I'm talking about
> here].
Oh, yeah, I don't know how to use it for static linking.
> This is from the people who actually had to ask "does Windows actually
> support letting multiple processes open the same disk file at once?"
Heh. Did you tell them it's better at that then UNIX is?
> Question: Does that mean that one process can produce so much garbage that
> it forces endless GC cycles, slowing other unrelated processes down? Or does
> the VM partition memory on a per-process basis or something?
Each actor (i.e., Erlang process) has its own heap, so you can generate a
lot of garbage but it'll only make your own thread slow down.
That said, it's possible to generate specific kinds of garbage that go into
shared heaps or interned string storage or etc, which will be mean to all
the processes in the same VM. Hermes (which is somewhat similar to Erlang)
was designed to let the interpreter avoid such. (Altho Erlang *could* avoid
such problems with more work in the VM, Erlang wasn't specifically designed
AFAIK with the concept of being safe to run "untrusted" code on a node,
while that was pretty much Herme's main point.)
> Well yeah, but if you're trying to use a mere textual "name" to decide what
> type something is, you need a version number or something to go with the
> name, to make it unique.
Sure.
> What Erlang is doing is considering any two types to be "identical" if they
> have the same *structure*. That wouldn't do for Haskell; consider, for
> example, how a PortID and a ThreadID are both 32-bit signed integers, but
> are considered utterly different, incompatible types.
Well, more like you have very few types, but they're dynamic types. You have
list. You have tuple. You don't have list-of-X or tuple-of-three-Y's.
I.e., just like in Haskell a list of 4 integers is the same type as a list
of seven integers, in Erlang a list is a list, a tuple is a tuple.
> [Of course, TRWTF is that it's *signed*...]
If they're IDs, it doesn't make sense to talk about math or ordering on
them, so whether they're signed or not is irrelevant.
> You can upgrade the code, but unless you upgrade it all at once, both
> versions need to speak the same wire protocol.
Again, list of three integers vs list of seven integers.
>>> as well. When you compile v2, the compiler checks what you changed, and
>>> warns you if you don't explicitly handle the v1 data (unless it's
>>> identical).
>>
>> That's fair. It would be interesting to see how that's specified.
>
> I guess we sit back and see if they ever release anything...
I mean, I'd be interested in seeing how you tell the compiler "I can take
either v1 or v2" in a way that v1 and v2 are cleanly separated from the
code. Most languages make a distinction of a channel specification where
you provide the types for specific channels, but I don't know how Haskell
would handle such a thing. Would it just be a convention that you put that
stuff in a separate file?
> Yeah, I guess that works too.
>
> Presumably Erlang allows you to serialise cyclic data structures, which is
> notoriously hard for a pure functional language.
I'm not sure how you'd get a cyclic data structure in Erlang, given that
it's functional. It would imply something created earlier has to point to
something created later in order to form a cycle.
>>> And yet, that would seem rather heavy compared to just sending a function
>>> name (or rather, a *unique* function identifier),
>>
>> Not all functions have names. For the functions that have names, I
>> believe you just send the fully-qualified name of the function.
>
> Only works if foo/3 on node X is the same thing as foo/3 on node Y. ;-)
Correct. That's why there's an MD5 of the code along with the identifier.
> Oh, right. So atoms are *globally* unique, not unique to each node?
Well, they're ... atoms. Is 27.5 globally unique? Two atoms with the same
textual representation are by definition the same atom.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/17/2011 8:32, Invisible wrote:
> Erlang sends an index into the BEAM file, the eventual Haskell
> implementation is going to send linker labels (i.e., indexes into the
> executable image).
Well, note that there's two versions, from different releases of the VM.
> I notice also that Erlang sends "Uniq", which is "the hash value of the
> parse for the fun". In other words, it sounds like it's a cryptographic hash
> function of the syntax tree for the function's source code. Which is pretty
> much what the Haskell people are talking about for version control.
I believe that's correct. And the old version sent the MD5 of the range of
bytes that implement the function, or something. (It's actually the AST,
which is available also as part of the macro expansion phase of compilation.)
> So the description is "far from complete"? That's really helpful.
Yet typical for open source, where proponents often claim that the code *is*
the documentation.
> Oh, but
> the protocols "are stable since many years". So I'm guessing English isn't
> your first language then? :-P
Uh, no. Ericcson.com, corporate headquarters Stockholm, Sweden.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> This is from the people who actually had to ask "does Windows actually
>> support letting multiple processes open the same disk file at once?"
>
> Heh. Did you tell them it's better at that then UNIX is?
My understanding is that Unix is *great* at letting multiple processes
read the same file - it's *stopping* them doing this that's hard.
>> Question: Does that mean that one process can produce so much garbage
>> that
>> it forces endless GC cycles, slowing other unrelated processes down?
>> Or does
>> the VM partition memory on a per-process basis or something?
>
> Each actor (i.e., Erlang process) has its own heap, so you can generate
> a lot of garbage but it'll only make your own thread slow down.
Right, OK. So the worst you can do (presumably) is steal lots of CPU
time. (?)
Come to mention it, what's Erlang's thread scheduler like? Does it just
do round-robin for all threads ready to run, or can you assign thread
priorities and stuff?
> That said, it's possible to generate specific kinds of garbage that go
> into shared heaps or interned string storage or etc, which will be mean
> to all the processes in the same VM. Hermes (which is somewhat similar
> to Erlang) was designed to let the interpreter avoid such. (Altho Erlang
> *could* avoid such problems with more work in the VM, Erlang wasn't
> specifically designed AFAIK with the concept of being safe to run
> "untrusted" code on a node, while that was pretty much Herme's main point.)
OK, fair enough.
>> What Erlang is doing is considering any two types to be "identical" if
>> they
>> have the same *structure*. That wouldn't do for Haskell; consider, for
>> example, how a PortID and a ThreadID are both 32-bit signed integers, but
>> are considered utterly different, incompatible types.
>
> Well, more like you have very few types, but they're dynamic types. You
> have list. You have tuple. You don't have list-of-X or tuple-of-three-Y's.
>
> I.e., just like in Haskell a list of 4 integers is the same type as a
> list of seven integers, in Erlang a list is a list, a tuple is a tuple.
Indeed. Erlang cares only about structure, not meaning. (Presumably the
idiomatic way to distinguish between a ThreadID and a CustomerID would
be to put an atom in front of it indicating it's "type".)
>> [Of course, TRWTF is that it's *signed*...]
>
> If they're IDs, it doesn't make sense to talk about math or ordering on
> them, so whether they're signed or not is irrelevant.
Presumably it just means that if you create more than 2^31 Haskell
threads, the ThreadID counter will overflow and half the standard
library will malfunction...
Soon you'll have FOUR THOUSAND AND NINETY SIX CHILDREN, and they'll all
be PARALLEL, and Google will be like SLOOOOOW DOWN MAN! IT'S
MUNCTIONAL!!!! >_<
>> You can upgrade the code, but unless you upgrade it all at once, both
>> versions need to speak the same wire protocol.
>
> Again, list of three integers vs list of seven integers.
There's more to a wire protocol than what types you exchange. (There's
ordering, for example.)
>> I guess we sit back and see if they ever release anything...
>
> I mean, I'd be interested in seeing how you tell the compiler "I can
> take either v1 or v2" in a way that v1 and v2 are cleanly separated from
> the code. Most languages make a distinction of a channel specification
> where you provide the types for specific channels, but I don't know how
> Haskell would handle such a thing. Would it just be a convention that
> you put that stuff in a separate file?
I don't really know. In principle the information exists to let you do
this stuff, but how you'd implement it I'm not sure.
GHC already generates "interface files" which are machine-readable files
telling the compiler what's exported from each source module. Perhaps
you could do some trick where you compare the old interface files to the
new ones or something. (But this requires you to remember to not delete
this files, which are usually considered "temporary", so probably some
more specific mechanism is necessary.)
>> Yeah, I guess that works too.
>>
>> Presumably Erlang allows you to serialise cyclic data structures,
>> which is
>> notoriously hard for a pure functional language.
>
> I'm not sure how you'd get a cyclic data structure in Erlang, given that
> it's functional. It would imply something created earlier has to point
> to something created later in order to form a cycle.
Does Erlang not permit recursive definitions then?
let list = [1, 2, 3] ++ list
That'll create a circular list for you in Haskell.
[Actually, it's not defined whether this is a recursive function, or a
circular list. It's impossible to tell the difference from inside
Haskell itself...]
>> Only works if foo/3 on node X is the same thing as foo/3 on node Y. ;-)
>
> Correct. That's why there's an MD5 of the code along with the identifier.
Yeah. And why Haskell is talking about doing something similar.
Currently it just uses a name, but as you say, that breaks as soon as
sender and receiver have different code.
>> Oh, right. So atoms are *globally* unique, not unique to each node?
>
> Well, they're ... atoms. Is 27.5 globally unique? Two atoms with the
> same textual representation are by definition the same atom.
Right, I see...
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> I notice also that Erlang sends "Uniq", which is "the hash value of the
>> parse for the fun". In other words, it sounds like it's a
>> cryptographic hash
>> function of the syntax tree for the function's source code. Which is
>> pretty
>> much what the Haskell people are talking about for version control.
>
> I believe that's correct. And the old version sent the MD5 of the range
> of bytes that implement the function, or something. (It's actually the
> AST, which is available also as part of the macro expansion phase of
> compilation.)
The trouble is, if the compiler optimisations change, the BEAM code
might change, even if what the function actually does is unaffected.
That [I would hope] is why it hashes the AST, not the compiled code.
OTOH, depending on how it works, there may be more than one AST which is
trivially equivilent.
OTFH, I believe Rice's theorum states that function equity is
undecidable. We're not really trying to prove which functions are equal;
we're trying to prove which ones are *not* equal...
>> So the description is "far from complete"? That's really helpful.
>
> Yet typical for open source, where proponents often claim that the code
> *is* the documentation.
The source code *is* the documentation = EPIC FAILURE.
The usual response I see from OSS is "you should be greatful we bothered
to write any documentation at all!"
>> Oh, but
>> the protocols "are stable since many years". So I'm guessing English
>> isn't
>> your first language then? :-P
>
> Uh, no. Ericcson.com, corporate headquarters Stockholm, Sweden.
LOLRUS.
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/17/2011 12:12, Orchid XP v8 wrote:
> My understanding is that Unix is *great* at letting multiple processes read
> the same file - it's *stopping* them doing this that's hard.
Well, there is that, yes.
> Right, OK. So the worst you can do (presumably) is steal lots of CPU time. (?)
Well, you can break the stuff that's shared, like the large binary store or
the atom intern stuff.
> Come to mention it, what's Erlang's thread scheduler like? Does it just do
> round-robin for all threads ready to run, or can you assign thread
> priorities and stuff?
I don't think there's any priority stuff that I remember.
> Indeed. Erlang cares only about structure, not meaning. (Presumably the
> idiomatic way to distinguish between a ThreadID and a CustomerID would be to
> put an atom in front of it indicating it's "type".)
Pretty much, yes.
> Presumably it just means that if you create more than 2^31 Haskell threads,
> the ThreadID counter will overflow and half the standard library will
> malfunction...
It might not even mean that, if nobody does any comparisons or math with them.
>>> You can upgrade the code, but unless you upgrade it all at once, both
>>> versions need to speak the same wire protocol.
>>
>> Again, list of three integers vs list of seven integers.
>
> There's more to a wire protocol than what types you exchange. (There's
> ordering, for example.)
Well, yes, if you have two versions of Erlang that don't speak the same
protocol as the other, they won't connect.
> GHC already generates "interface files" which are machine-readable files
> telling the compiler what's exported from each source module. Perhaps you
> could do some trick where you compare the old interface files to the new
> ones or something. (But this requires you to remember to not delete this
> files, which are usually considered "temporary", so probably some more
> specific mechanism is necessary.)
That was the kind of thing I was asking about, yes.
> Does Erlang not permit recursive definitions then?
Not of data. If "list" is already defined, you can't assign to "list" again.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/17/2011 12:16, Orchid XP v8 wrote:
> The trouble is, if the compiler optimisations change, the BEAM code might
> change, even if what the function actually does is unaffected. That [I would
> hope] is why it hashes the AST, not the compiled code.
Right. I expect that's why they changed the definition of what goes over the
whire.
> OTOH, depending on how it works, there may be more than one AST which is
> trivially equivilent.
That wouldn't really matter. There's one place the function is defined in
the code. If there's a *different* function that does the same thing, then
it's still a different function.
All they're doing is ensuring the function they have is the same one you
have. They don't have to prove they're different, only that they're identical.
> We're not really trying to prove which functions are equal; we're trying to
> prove which ones are *not* equal...
Right.
> The usual response I see from OSS is "you should be greatful we bothered to
> write any documentation at all!"
I've even seen "yes, we know the documentation is all wrong and misleading,
but we didn't bother to put a note on it saying that" (Apache), as well as
"please, sir, may I write the documentation for you?" "No." (Blender)
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> The usual response I see from OSS is "you should be greatful we
>> bothered to write any documentation at all!"
>
> I've even seen "yes, we know the documentation is all wrong and
> misleading, but we didn't bother to put a note on it saying that"
> (Apache), as well as "please, sir, may I write the documentation for
> you?" "No." (Blender)
Use the force - delete the source!
No, wait, that's not quite right...
Anyway, I submitted a patch to GHC last year containing some trivial
documentation updates. I'm still waiting for it to be merged. (As in,
they have the new documentation, they literally just need to press a
button to insert it into the development head... and nobody has done
this yet. They haven't got around to it.)
In fairness, I suppose they probably need to check it for correctness
first, but *man*!
...hold the phone! I just went to look at what date I *actually*
submitted it, and it turns out they applied it yesterday evening. Just
hours after I was looking at it, in fact. Coincidence, eh?
I still think 5 months to apply one teeny documentation patch is a tad
slow...
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> My understanding is that Unix is *great* at letting multiple processes
>> read the same file - it's *stopping* them doing this that's hard.
>
> Well, there is that, yes.
It amuses me that there were operating systems 20 years ago that could
do this better than Linux. It also amuses me that there exists something
that Windows can actually do better than Linux.
(I gather it's a similar story for figuring out when a file changed...)
>> Come to mention it, what's Erlang's thread scheduler like? Does it
>> just do
>> round-robin for all threads ready to run, or can you assign thread
>> priorities and stuff?
>
> I don't think there's any priority stuff that I remember.
Certainly I've often wished you could assign priorities to Haskell
threads. But, unfortunately, the thread scheduler doesn't support that.
For that matter, it doesn't support querying how many threads are
currently running either. Or what their relationships are. Or, indeed,
anything about them.
(The other day, I was looking at Java, and was amused to note that "each
thread has a name for identification purposes. Thread names do not need
to be unique." Well, uh, they do if you want to use names to identify
threads. :-P Silly Java...)
>> Presumably it just means that if you create more than 2^31 Haskell
>> threads,
>> the ThreadID counter will overflow and half the standard library will
>> malfunction...
>
> It might not even mean that, if nobody does any comparisons or math with
> them.
Testing for equity will still work. I think the only time ordering
comparisons happen is if you store thread IDs in a BST - and then the
ordering is arbitrary, it just needs to be total.
Now, if some library somewhere is trying to use thread IDs to infer
which order threads were created in, *that* will break nicely.
So I guess the libraries don't break until you have 2^32 threads then.
>>>> You can upgrade the code, but unless you upgrade it all at once, both
>>>> versions need to speak the same wire protocol.
>>>
>>> Again, list of three integers vs list of seven integers.
>>
>> There's more to a wire protocol than what types you exchange. (There's
>> ordering, for example.)
>
> Well, yes, if you have two versions of Erlang that don't speak the same
> protocol as the other, they won't connect.
I was thinking more about two versions of the same program. Presumably
they'll connect, but malfunction.
>> Does Erlang not permit recursive definitions then?
>
> Not of data. If "list" is already defined, you can't assign to "list"
> again.
Neither can you do this with Haskell. But the definition of "list" can
refer to "list" itself.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/18/2011 1:32, Invisible wrote:
> It amuses me that there were operating systems 20 years ago that could do
> this better than Linux.
Yep. Annoyed me no end, when I had to use UNIX instead of a "real" operating
system.
> It also amuses me that there exists something that
> Windows can actually do better than Linux.
There's actually quite a bit that Windows now does better than Linux, and
quite a bit that Linux today does better than Linux five years ago. For
example, I read that the clock is now *finally* a file handle, like the
Amiga figured out 20 years ago and which everyone seemed to conveniently
ignore for the past 40 years when arguing "Everything's a file in UNIX!"
> (I gather it's a similar story for figuring out when a file changed...)
Yes. The IPC stuff in Linux honestly still hasn't caught up to Windows.
There's no kernel transactions, the file locking is still a mess especially
over the network, the privilege and authentication system (at the kernel
level) is still comparatively weak, performance monitoring and error logging
is not unified, etc.
> be unique." Well, uh, they do if you want to use names to identify threads.
No. You might want to kill all database worker threads and not all web
server worker threads. I.e., the same thing you were talking about in
Haskell code upgrades, where you want to kill all the threads running *this*
code but not *that* code.
"Identify" doesn't necessarily imply "identity" except in math. :-)
> Testing for equity will still work. I think the only time ordering
> comparisons happen is if you store thread IDs in a BST - and then the
> ordering is arbitrary, it just needs to be total.
Exactly.
> Now, if some library somewhere is trying to use thread IDs to infer which
> order threads were created in, *that* will break nicely.
Exactly.
> I was thinking more about two versions of the same program. Presumably
> they'll connect, but malfunction.
It depends if one is upward compatible with the other. If the new program
can handle new data and old data both, then they won't malfunction.
> Neither can you do this with Haskell. But the definition of "list" can refer
> to "list" itself.
Oh. No, you can't do that in Erlang, since "list" doesn't exist until you've
assigned it.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> It also amuses me that there exists something that
>> Windows can actually do better than Linux.
>
> There's actually quite a bit that Windows now does better than Linux,
> and quite a bit that Linux today does better than Linux five years ago.
> For example, I read that the clock is now *finally* a file handle, like
> the Amiga figured out 20 years ago and which everyone seemed to
> conveniently ignore for the past 40 years when arguing "Everything's a
> file in UNIX!"
>
>> (I gather it's a similar story for figuring out when a file changed...)
>
> Yes. The IPC stuff in Linux honestly still hasn't caught up to Windows.
> There's no kernel transactions, the file locking is still a mess
> especially over the network, the privilege and authentication system (at
> the kernel level) is still comparatively weak, performance monitoring
> and error logging is not unified, etc.
I don't know a great deal about the inner workings of Windows or Unix.
But there are a number of things that Windows seems to do better:
- File permissions. None of this chmod 664 nonsense. You can set
arbitrary combinations of permissions for any number of users and groups.
[But then they tried to retro-fit permission inheritance, which doesn't
actually work how you'd think it works. And don't even get me started on
the whole Active Directory thing with three different sorts of security
groups that nobody understands...]
- The registry provides a unified place to store configuration, whether
per-user or per-system. (Just a shame that there's no real interface for
managing it yourself - e.g., if you want to revert your MS Word
configuration to how it was yesterday.)
- File types [or, more accurately, file extensions] are associated with
programs. (Shame there's no CLI access to this.)
>> be unique." Well, uh, they do if you want to use names to identify
>> threads.
>
> No. You might want to kill all database worker threads and not all web
> server worker threads. I.e., the same thing you were talking about in
> Haskell code upgrades, where you want to kill all the threads running
> *this* code but not *that* code.
>
> "Identify" doesn't necessarily imply "identity" except in math. :-)
...and here I was thinking that's what the ThreadGroup class is for...
>> Neither can you do this with Haskell. But the definition of "list" can
>> refer to "list" itself.
>
> Oh. No, you can't do that in Erlang, since "list" doesn't exist until
> you've assigned it.
Right. So a variable doesn't come into scope until after it's defined?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/18/2011 8:28, Invisible wrote:
> - File permissions. None of this chmod 664 nonsense. You can set arbitrary
> combinations of permissions for any number of users and groups.
Yep. And the same permission framework works on every resource, not just
files. So you can say "this group can read the attributes of processes but
not write them, and that group can append to error logs but not read them."
There's a handful of things you need to have "administrator" privs to do
that you really should have a separate permission for (like reading the UNC
journal), but not much.
> [But then they tried to retro-fit permission inheritance, which doesn't
> actually work how you'd think it works. And don't even get me started on the
> whole Active Directory thing with three different sorts of security groups
> that nobody understands...]
That is the basic problem with complex permissions. :-)
> - The registry provides a unified place to store configuration, whether
> per-user or per-system. (Just a shame that there's no real interface for
> managing it yourself - e.g., if you want to revert your MS Word
> configuration to how it was yesterday.)
Yeah. It's kind of a big bag at this point.
> - File types [or, more accurately, file extensions] are associated with
> programs. (Shame there's no CLI access to this.)
Like the "assoc" command? Primitive, but you can also fronk the registry if
you need something more sophisticated. The nice thing about it is that it
*is* centralized. No more groping around in the web server's configuration
to figure out such stuff.
Even better, of course, would be having it in the actual metadata of the
file, like the Mac's OS tried to do, except they did it before MIME was
around, so it didn't work out as well as it might.
> ...and here I was thinking that's what the ThreadGroup class is for...
That too. I think the real point of that line in the documentation is that
the runtime doesn't *check* that the name is unique.
> Right. So a variable doesn't come into scope until after it's defined?
I believe that's correct.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>
> I don't know a great deal about the inner workings of Windows or Unix.
> But there are a number of things that Windows seems to do better:
>
> - File permissions. None of this chmod 664 nonsense. You can set
> arbitrary combinations of permissions for any number of users and
> groups.
http://en.wikipedia.org/wiki/Access_control_list#Filesystem_ACLs
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 18/05/2011 07:01 PM, Florian Pesth wrote:
>>
>> I don't know a great deal about the inner workings of Windows or Unix.
>> But there are a number of things that Windows seems to do better:
>>
>> - File permissions. None of this chmod 664 nonsense. You can set
>> arbitrary combinations of permissions for any number of users and
>> groups.
>
> http://en.wikipedia.org/wiki/Access_control_list#Filesystem_ACLs
I'm not sure what the relevance of this link is.
I'm aware that a couple of people have hacked propper ACLs into recent
Linux kernels. This doesn't change the fact that the entire Unix
ecosystem still assumes that permissions are set via flat bitmasks. It's
almost infeasible to change this and still call the result "Unix".
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Am Wed, 18 May 2011 21:52:05 +0100 schrieb Orchid XP v8:
> On 18/05/2011 07:01 PM, Florian Pesth wrote:
>>>
>>> I don't know a great deal about the inner workings of Windows or Unix.
>>> But there are a number of things that Windows seems to do better:
>>>
>>> - File permissions. None of this chmod 664 nonsense. You can set
>>> arbitrary combinations of permissions for any number of users and
>>> groups.
>>
>> http://en.wikipedia.org/wiki/Access_control_list#Filesystem_ACLs
>
> I'm not sure what the relevance of this link is.
>
> I'm aware that a couple of people have hacked propper ACLs into recent
> Linux kernels. This doesn't change the fact that the entire Unix
> ecosystem still assumes that permissions are set via flat bitmasks. It's
> almost infeasible to change this and still call the result "Unix".
It's supported and works since ten years now (and according to wikipedia
is supported in Konqueror and Nautilus since quite some time). Just
because you put Linux and Unix in the same bag and Unix doesn't support
some things (which is by the way in this case not true since ACL's come
from Unix) does not mean that Linux can't do it or that it is a "hack".
And yes, Linux is not Unix and it shows in some details like this.
The fact, that a lot of distributions don't enable it by default means
that most people don't need it, but for sure it is there (same with
Extended Attributes, which are also not enabled by default but you just
have to switch it on).
So yes, you can set arbitrary user rights - you just have to enable it. I
even doubt that there is no solution to the file locking problem - but I
don't really know...
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 18/05/2011 06:20 PM, Darren New wrote:
> On 5/18/2011 8:28, Invisible wrote:
>> - File permissions. None of this chmod 664 nonsense. You can set
>> arbitrary
>> combinations of permissions for any number of users and groups.
>
> Yep. And the same permission framework works on every resource, not just
> files.
Yeah. Processes, printers, event logs, the registry. (Shame you can't
use it for MS Access though...)
More importantly, you don't just have permissions for "read", "write"
and "execute". The permissions are more fine-grained than that. For
example, you can make it so a user can append to a file even though they
can't read from it. Or make it so they can access it via the backup API,
but can't read it normally. And so on.
> There's a handful of things you need to have "administrator"
> privs to do that you really should have a separate permission for (like
> reading the UNC journal), but not much.
There's a couple of things which are "privilages" rather than
"permissions". For example, normally (under Windows XP) only members of
Administrators and Power Users have the ability to start and stop
services. It looks like this is hard-wired, but actually it isn't.
Somewhere deep in the murky depths of the Control Panel, you can
actually change this.
You can't change this in Windows NT v4, but you can in Windows XP. I
should know. We had an issue where we wanted certain users to be able to
start and stop services, but *not* be able to log people off the PC.
Administrators can do both, normal users can do neither. In NT4, you
can't fix this. In XP, you can assign both privilages independently.
There might be a tiny few things which really are hard-coded to
Administrator (the user) and/or Administrators (the group), but I
haven't run into that.
>> [But then they tried to retro-fit permission inheritance, which doesn't
>> actually work how you'd think it works. And don't even get me started
>> on the
>> whole Active Directory thing with three different sorts of security
>> groups
>> that nobody understands...]
>
> That is the basic problem with complex permissions. :-)
I assume that comment is directed at AD.
Regarding my first comment: With NT4, the permissions for an object
default to the permissions of its parent at the time of creation. But if
you change the permissions of the parent, the child's permissions stay
the same. Except usually when you change the parent's permissions,
there's an option to recurse over the children and set their permissions
the same. Except that if you do this, any children who had their
permissions deliberately changed get reset. In short, it sucks.
In Windows XP (and possibly Windows 2000), they added the ability for
one object to actually "inherit" the permissions of its parent. That
means that if you change the permissions on the parent, the permissions
of all children change, *except* the ones which you've set to not
inherit. Those stay the same. And any extra, child-specific permissions
that aren't inherited don't get changed either. This is much better...
...except that it's still implemented the old way. In other words, each
object still has the full permissions list on it, and changing the
permissions on a parent object still recurses over all children. It's
just that now permissions are marked to indicate whether they're manual
or automatic, and only the automatic ones get futzed with.
Which means that changing the parent's permissions has the intended
effect, but if the parent has 80,000,000 children, it still takes
forever to set the permissions, because you literally have to touch
every child and update its ACL on disk.
Still, it's much better than what existed before.
Oh yeah. TRWTF is 8.3 names.
>> - The registry provides a unified place to store configuration, whether
>> per-user or per-system. (Just a shame that there's no real interface for
>> managing it yourself - e.g., if you want to revert your MS Word
>> configuration to how it was yesterday.)
>
> Yeah. It's kind of a big bag at this point.
Which is a pity. It's nice that there's one central settings database,
and you can make it follow the user around the network and stuff. It
also lets you do stuff like group policy. In other words, sitting at my
desk, I can activate the screensaver on every PC on the network, set the
timeout, and make it impossible to disable. Try doing that with a
generic Linux distro.
And although the GUI only works for MS products, you *can* use it to
control *anything* that uses the registry [i.e., the vast majority of
Windows software - *if* you can figure out which ****ing key controls it!]
The downside is...
1. Group policy doesn't apply to groups [which can overlap]. It applies
to *folders* [which are strictly hierachical]. Stupid artificial limitation.
2. Group policy works by overwriting parts of each PC's registry. If you
turn off the group policy, the settings to *not* go back to what they
were before. (!!) You have to apply a new policy to set the settings to
something else. (And you cannot set them to "whatever it was before",
because that information no longer exists. Best you can do is set it
back to whatever the installation default value is.)
>> - File types [or, more accurately, file extensions] are associated with
>> programs. (Shame there's no CLI access to this.)
>
> Like the "assoc" command?
I usually go through the GUI. I don't know how to change it from the
CLI. But basically, the part where every file that ends in ".txt" opens
Notepad when you double-click it, while anything ending ".html" opens
Internet Explorer.
Like I say, it's a shame you can't use this feature to launch stuff
programmatically. (At least, not without linking to C.)
> Primitive, but you can also fronk the registry
> if you need something more sophisticated. The nice thing about it is
> that it *is* centralized. No more groping around in the web server's
> configuration to figure out such stuff.
Linux seems to manage this now. Or at least, Linux file managers seem to
open the correct application for most common file types now. I have no
idea how it's implemented. (I have a sinking feeling it's a type
database for each file manager, rather than something that arbitrary
applications can use.)
> Even better, of course, would be having it in the actual metadata of the
> file, like the Mac's OS tried to do, except they did it before MIME was
> around, so it didn't work out as well as it might.
Is that the whole resource fork / data fork thing? I never heard how
that actually works. (Other than "it doesn't".)
I know AmigaDOS distinguished between "projects" and "tools" at the same
level as it distinguishes between "files" and "directories". (A "tool"
is something that's supposed to be executable.)
>> ...and here I was thinking that's what the ThreadGroup class is for...
>
> That too. I think the real point of that line in the documentation is
> that the runtime doesn't *check* that the name is unique.
Right, OK.
>> Right. So a variable doesn't come into scope until after it's defined?
>
> I believe that's correct.
Oh well. I guess this is why Erlang is the most successful "functional"
programming language around.
It's MUNCTIONAL! You're going to be UNCOMFORTABLY PARALLEL! Soon you'll
have so many children, and you've give then ERLANG, and they'll run
ABNORMALLY FAST! You want to be good at databases? Try PETABASES!
Functional, Erlang, functional, database, FUNCTIONAL!!!!
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> I'm aware that a couple of people have hacked propper ACLs into recent
>> Linux kernels. This doesn't change the fact that the entire Unix
>> ecosystem still assumes that permissions are set via flat bitmasks. It's
>> almost infeasible to change this and still call the result "Unix".
>
> It's supported and works since ten years now (and according to wikipedia
> is supported in Konqueror and Nautilus since quite some time). Just
> because you put Linux and Unix in the same bag and Unix doesn't support
> some things (which is by the way in this case not true since ACL's come
> from Unix) does not mean that Linux can't do it or that it is a "hack".
> And yes, Linux is not Unix and it shows in some details like this.
> The fact, that a lot of distributions don't enable it by default means
> that most people don't need it, but for sure it is there (same with
> Extended Attributes, which are also not enabled by default but you just
> have to switch it on).
> So yes, you can set arbitrary user rights - you just have to enable it. I
> even doubt that there is no solution to the file locking problem - but I
> don't really know...
All Unix tools assume that file permissions are set using bitmasks. If
this doesn't hold, most software designed for Unix-like operating
systems simply stops working.
This is the trouble with having a wildly popular system - you can't
change it. The fact that ACLs are turned off by default almost certainly
has nothing to do with "people not needing it" - more likely it's turned
off because it breaks so much stuff.
Similar problems with file locking. As I understand it, there's now an
API to ask "is this file locked?", but if you don't use it, you bypass
all the locking and can access a file even if it's locked. Yeah, that's
really helpful.
I gather that Linux only just got the ability to notify you if a file
changes. (At least modern Linux desktops manage to *use* this
information correctly, which is more than Windows Explorer manages...)
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Am Wed, 18 May 2011 22:30:32 +0100 schrieb Orchid XP v8:
>
> All Unix tools assume that file permissions are set using bitmasks. If
> this doesn't hold, most software designed for Unix-like operating
> systems simply stops working.
Well, binutils and the file managers of the most used desktop environments
KDE and Gnome seem to support it. Now what other programs do with it I
don't know...
>
> This is the trouble with having a wildly popular system - you can't
> change it. The fact that ACLs are turned off by default almost certainly
> has nothing to do with "people not needing it" - more likely it's turned
> off because it breaks so much stuff.
Well, it wouldn't hurt to turn extended attributes on, even if they would
get lost using the wrong programs. I think in Ubuntu really a lot of
stuff is turned off, because people don't need it or they think it is too
dangerous (like this stupid sudo su stuff and only certain users can
change to root, which was taken from BSD...).
>
> Similar problems with file locking. As I understand it, there's now an
> API to ask "is this file locked?", but if you don't use it, you bypass
> all the locking and can access a file even if it's locked. Yeah, that's
> really helpful.
Sure that there is no call for saying "Lock this file definitely" and the
other lock mechanisms are just kept for POSIX compability?
>
> I gather that Linux only just got the ability to notify you if a file
> changes. (At least modern Linux desktops manage to *use* this
> information correctly, which is more than Windows Explorer manages...)
I think this is also older... there is this tailf program, which does the
same as tail -f but does not poll the file, so I guess there was some
mechanism already. But I agree, that a lot of the GUI's could implement
stuff much better.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/18/2011 14:14, Orchid XP v8 wrote:
> There might be a tiny few things which really are hard-coded to
> Administrator (the user) and/or Administrators (the group), but I haven't
> run into that.
The only one I know of for sure is manipulating file system stuff.
Partitioning a disk, reading the UNC, marking a file system dirty, etc. I
don't know what privilege it is - maybe you actually can assign it.
>> That is the basic problem with complex permissions. :-)
> I assume that comment is directed at AD.
Well, all complex permissions systems.
> ...except that it's still implemented the old way.
No it isn't. Well, maybe with XP, but I don't think that's the case any more.
> Which means that changing the parent's permissions has the intended effect,
> but if the parent has 80,000,000 children, it still takes forever to set the
> permissions, because you literally have to touch every child and update its
> ACL on disk.
Maybe if you're doing it thru the explorer?
> Oh yeah. TRWTF is 8.3 names.
What about them?
>> Like the "assoc" command?
>
> I usually go through the GUI. I don't know how to change it from the CLI.
> But basically, the part where every file that ends in ".txt" opens Notepad
> when you double-click it, while anything ending ".html" opens Internet
> Explorer.
That's the "assoc" command.
> Like I say, it's a shame you can't use this feature to launch stuff
> programmatically. (At least, not without linking to C.)
Well, what are you going to launch? You can't launch *anything* without
linking to some sort of C code.
> how it's implemented. (I have a sinking feeling it's a type database for
> each file manager, rather than something that arbitrary applications can use.)
I wouldn't be surprised.
>> Even better, of course, would be having it in the actual metadata of the
>> file, like the Mac's OS tried to do, except they did it before MIME was
>> around, so it didn't work out as well as it might.
>
> Is that the whole resource fork / data fork thing? I never heard how that
> actually works. (Other than "it doesn't".)
No, it was the Creator/Type thing, each a four-character code. So a text
file made by MS Word might have a creator/type or MSWD/TEXT or something.
Mac files had two forks. One was the data fork, which is just like a UNIX
file. The other is the resource fork, which had structured blobs, each
defined by a type and a size, maybe a name, etc. So what you see in a
Windows executable as a "resource" file was actually the resource fork of a
Mac application or data file.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/18/2011 14:50, Florian Pesth wrote:
> Sure that there is no call for saying "Lock this file definitely" and the
> other lock mechanisms are just kept for POSIX compability?
Yes. The basic problem is that none of the lock mechanisms actually work
reliably in all situations.
> I think this is also older...
One problem is that there's at least 3 ways of doing this in Linux, all
three relatively recent.
One would give you a signal when a file in the one single directory you're
watching (maybe just the cwd) changed. Not very handy except for the
refreshing of a window in a file manager.
One would do some sort of something or other with messages sent to you
somehow I forget when anything under a particular file tree changed, IIRC.
It was poor because, IIRC, you had to block to wait for the messages or
something.
The latest one, done right, lets you open a file handle, writing to the
handle says what you're waiting to see change, reading the (selectable)
handle lets you know when something there changed and what it was.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On Wed, 18 May 2011 22:30:32 +0100, Orchid XP v8 wrote:
> I gather that Linux only just got the ability to notify you if a file
> changes. (At least modern Linux desktops manage to *use* this
> information correctly, which is more than Windows Explorer manages...)
The notification API has been around for several years now.
As for ACLs, it depends on the tools. I've seen proper ACLs overlaid on
*nix filesystems for several years now (as well) and it works properly.
Sometimes you have to map the bitmask rights to something in the new ACL
filesystem, but having MAC (ie, mandatory access controls) can certainly
help enforce permissions that would otherwise not be done.
Jim
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Am Wed, 18 May 2011 15:28:44 -0700 schrieb Darren New:
> On 5/18/2011 14:50, Florian Pesth wrote:
>> Sure that there is no call for saying "Lock this file definitely" and
>> the other lock mechanisms are just kept for POSIX compability?
>
> Yes. The basic problem is that none of the lock mechanisms actually work
> reliably in all situations.
>
How about this:
http://www.hackinglinuxexposed.com/articles/20030623.html
Granted, the described problem, that if a program opens before and writes
after the lock (therefore overwriting the file) persists, but I don't
think this is really a locking problem or is it?
>> I think this is also older...
>
> One problem is that there's at least 3 ways of doing this in Linux, all
> three relatively recent.
>
> One would give you a signal when a file in the one single directory
> you're watching (maybe just the cwd) changed. Not very handy except for
> the refreshing of a window in a file manager.
>
> One would do some sort of something or other with messages sent to you
> somehow I forget when anything under a particular file tree changed,
> IIRC. It was poor because, IIRC, you had to block to wait for the
> messages or something.
>
> The latest one, done right, lets you open a file handle, writing to the
> handle says what you're waiting to see change, reading the (selectable)
> handle lets you know when something there changed and what it was.
Ok, thanks. I just was thinking of tailf which advertises with hard disk
spin down when the file does not change - so it definitely does not poll
and is around since at least 2003. I just searched for it and it seems
that there are two mechanisms: dnotify and the more recently inotify. Now
they might not do what you like to do with them (sorry I didn't
understand what you wanted to do exactly) but for being notified when a
file changes (which is what Andrew was talking about) even dnotify seems
to be fine (and is around since kernel 2.4.). Has your problem something
to do with blocking / non-blocking access of this information?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/18/2011 15:53, Florian Pesth wrote:
> Granted, the described problem, that if a program opens before and writes
> after the lock (therefore overwriting the file) persists, but I don't
> think this is really a locking problem or is it?
That's not the only problem. The other problem is, as he said, taking down
an NFS server from user mode, for example.
Here's the article I was thinking of:
http://0pointer.de/blog/projects/locking
http://0pointer.de/blog/projects/locking2
Contrast with (for example) mainframe locking, where advisory locks usually
don't even use files, and mandatory locks are generally properly enforced
because large amounts of peoples' money depends on them working. (Which is
why all the stuff on Linux where peoples' money *does* depend on locking
working, like an RDBMS, you access all the relevant files via a socket to a
single process that does the locking without going through the file system
to do so.)
And in the 1.x kernels (and maybe later), locking could also lead to
starvation in Linux. It's all getting cleaned up over time, of course, but
us old farts remember this stuff and are leery about anything new. Linux has
it's own "wait for version 3 of anything MS releases".
Here's an exercise for you:
Write a backup program under Linux that correctly backs up files, even ones
you might be editing, without ever losing both the new and old versions of a
file. I don't think that's possible. I.e., I want to run a backup program
using normal file system access (rather than reading raw disk data) that
will reliably copy a local directory full of files, even if you're changing
them. There's several ways of doing this on Windows. I don't know any way of
doing it on Linux.
> Ok, thanks. I just was thinking of tailf which advertises with hard disk
> spin down when the file does not change
The hard disk spins down when the disk doesn't generate interrupts for some
length of time. That doesn't mean there's no file activity. It just means
all the file activity is using the buffers.
The three interfaces I remember are one that generates a SIGIO when your
registered directory changes (which apparently nobody even talks about any
more, but it was quite a ways back), dnotify (which required a separate file
handle for each directory and communicates using signals, which means you
can't really use threads to deal with it and you can't really be doing
something else at the same time in the same process because a signal handler
can't do I/O so you wind up polling to see if a signal happened), and
inotify which finally got it right.
> - so it definitely does not poll
That doesn't follow. The fact the disk spins down just means the kernel
doesn't have to look at the disk to know the file hasn't changed.
> Has your problem something
> to do with blocking / non-blocking access of this information?
I don't really have a problem. We were just comparing areas in which Windows
seems to do a superior job to Linux (or at least Linux as of not too long ago).
The dnotify is just an exceptionally kludgey interface, requiring one handle
for each directory, "the problem of hard links to files has been ignored"
(my favorite gripe about all UNIX kernel functionality), and requiring you
to open each nested directory explicitly along with reading the entire
directory every time a file changes to see what changed. That's why they
replaced it. :-)
inotify is about as close as you're likely to get, letting you add the
sockets to a select() list so you can (for example) wait for data from the
keyboard while polling for directory changes.
None of them provide what Windows provides, which includes notifying you
about changes since last you asked, even if you weren't running when the
files changed.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> There might be a tiny few things which really are hard-coded to
>> Administrator (the user) and/or Administrators (the group), but I haven't
>> run into that.
>
> The only one I know of for sure is manipulating file system stuff.
> Partitioning a disk, reading the UNC, marking a file system dirty, etc.
> I don't know what privilege it is - maybe you actually can assign it.
Control Panel > Administrative Tools > Local Security Settings
Left pane: Local Policies > User Rights Assignment
Right pane says:
Perform volume maintenance tasks: Administrators
Presumably this covers things like mounting/unmounting filesystems,
creating partitions, formatting them, and so forth. (The frustrating
thing about Windows is that you have to *guess*; this isn't written down
anywhere.) If you change the setting here, you can allow other users to
do this.
Also in there is "take ownership of files". In other words, bypass ACLs.
Note that everything in this pane refers to "Administrators" (the user
group, not "Administrator" the user).
If you flip the left page to Local Policies > Security Options, there's
more fun stuff to play with, for example:
Devices: Allowed to format and eject removable media: Administrators
>>> That is the basic problem with complex permissions. :-)
>> I assume that comment is directed at AD.
>
> Well, all complex permissions systems.
As far as I can see, the existence of 3 different kinds of groups is an
artefact of the way AD is implemented internally, and how it maintains
backwards compatibility. Which is frustrating, because I don't care how
it works, and I don't need backwards compatibility. I just want
something comprehensible.
>> ...except that it's still implemented the old way.
>
> No it isn't. Well, maybe with XP, but I don't think that's the case any
> more.
And who's using anything newer than XP? I mean, really.
>> Which means that changing the parent's permissions has the intended
>> effect,
>> but if the parent has 80,000,000 children, it still takes forever to
>> set the
>> permissions, because you literally have to touch every child and
>> update its
>> ACL on disk.
>
> Maybe if you're doing it thru the explorer?
How else would you change a file's permissions?
>> Oh yeah. TRWTF is 8.3 names.
>
> What about them?
They still exist. And grown up programs sometimes end up using them.
>> Like I say, it's a shame you can't use this feature to launch stuff
>> programmatically. (At least, not without linking to C.)
>
> Well, what are you going to launch? You can't launch *anything* without
> linking to some sort of C code.
I mean, if you want to launch a program from the CLI, you just type its
name. But there isn't a command like "open Blue.ogg" which will fire up
the default application and load the named file. If you want to do this,
you have to write C code to query the registry and so forth.
>> how it's implemented. (I have a sinking feeling it's a type database for
>> each file manager, rather than something that arbitrary applications
>> can use.)
>
> I wouldn't be surprised.
Actually, you know what? There's probably one type database for KDE,
which only KDE applications can access using the KDE application
libraries, and then a second, completely independent type database for
GNOME, which only GNOME applications can access using the GNOME
libraries. And then half a dozen CLI applications which use their own
built-in mechanism.
And I've seen at least one Debian package which claims to contain a
program for analysing file contents to determine their actual type...
In short, it's an incoherent nightmare, just like almost everything else
in Unix.
>>> Even better, of course, would be having it in the actual metadata of the
>>> file, like the Mac's OS tried to do, except they did it before MIME was
>>> around, so it didn't work out as well as it might.
>>
>> Is that the whole resource fork / data fork thing? I never heard how that
>> actually works. (Other than "it doesn't".)
>
> No, it was the Creator/Type thing, each a four-character code. So a text
> file made by MS Word might have a creator/type or MSWD/TEXT or something.
>
> Mac files had two forks. One was the data fork, which is just like a
> UNIX file. The other is the resource fork, which had structured blobs,
> each defined by a type and a size, maybe a name, etc. So what you see in
> a Windows executable as a "resource" file was actually the resource fork
> of a Mac application or data file.
I see... I think! o_O
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/19/2011 1:35, Invisible wrote:
> Perform volume maintenance tasks: Administrators
OK, thanks!
> Also in there is "take ownership of files". In other words, bypass ACLs.
Take ownership isn't exactly bypass ACLs.
>> Maybe if you're doing it thru the explorer?
> How else would you change a file's permissions?
Via the API that explorer uses.
>>> Oh yeah. TRWTF is 8.3 names.
>> What about them?
> They still exist. And grown up programs sometimes end up using them.
Oh, well, yes. Backwards compatibility and all.
> I mean, if you want to launch a program from the CLI, you just type its
> name. But there isn't a command like "open Blue.ogg" which will fire up the
> default application and load the named file.
start index.html
The command is "start".
> Actually, you know what? There's probably one type database for KDE,
Originally it was Apache that needed it, since Apache had to deliver mime
types based on file extensions or content.
> In short, it's an incoherent nightmare, just like almost everything else in
> Unix.
A strength and a weakness.
>> Mac files had two forks. One was the data fork, which is just like a
>> UNIX file. The other is the resource fork, which had structured blobs,
>> each defined by a type and a size, maybe a name, etc. So what you see in
>> a Windows executable as a "resource" file was actually the resource fork
>> of a Mac application or data file.
>
> I see... I think! o_O
It was a nice idea. You could install fonts into the global space, or into
the executable for the word processor, or into an individual document that
used that font. You could localize a program after it was completely
finished just by adding resources to the executable. Stuff like that.
Oh, and chunks of code were also resources, so by calling code chunks, you'd
wind up loading dynamically the appropriate code resources, so you got
overlays for free, basically.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 5/19/2011 8:14, Darren New wrote:
>>> Maybe if you're doing it thru the explorer?
>> How else would you change a file's permissions?
>
> Via the API that explorer uses.
Or the CLI for that matter.
--
Darren New, San Diego CA, USA (PST)
"Coding without comments is like
driving without turn signals."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 19/05/2011 16:14, Darren New wrote:
> On 5/19/2011 1:35, Invisible wrote:
>> Perform volume maintenance tasks: Administrators
>
> OK, thanks!
>
>> Also in there is "take ownership of files". In other words, bypass ACLs.
>
> Take ownership isn't exactly bypass ACLs.
No. But if you can take ownership of a file, you can give yourself
permission to access it, even if you didn't have that permission to
start with. In other words, it allows you to bypass ACLs.
(Which is just as well, because our project management application
occasionally deletes the ACL list for files, which makes them rather
difficult to work with...)
>>> Maybe if you're doing it thru the explorer?
>> How else would you change a file's permissions?
>
> Via the API that explorer uses.
Well, yeah, if I were a C programmer that might be an option. :-P
>>>> Oh yeah. TRWTF is 8.3 names.
>>> What about them?
>> They still exist. And grown up programs sometimes end up using them.
>
> Oh, well, yes. Backwards compatibility and all.
Backwards compatibility = EVIL!
>> I mean, if you want to launch a program from the CLI, you just type its
>> name. But there isn't a command like "open Blue.ogg" which will fire
>> up the default application and load the named file.
>
> start index.html
>
> The command is "start".
And that works for any file type which Windows has a file association for?
>> Actually, you know what? There's probably one type database for KDE,
>
> Originally it was Apache that needed it, since Apache had to deliver
> mime types based on file extensions or content.
Well, no, Apache doesn't need to know what to open it with at all. It
just needs to know what MIME type to claim it is. (Which might
potentially vary on a file-by-file basis in a way unrelated to the
extension on the name...)
>> In short, it's an incoherent nightmare, just like almost everything
>> else in Unix.
>
> A strength and a weakness.
The Unix philosophy: Every tool should do one job, and do it well.
Unfortunately this leads to 25 different tools doing the same job
equally poorly in incompatible ways. Hence why your typical Linux distro
has (for example) a dozen different packages which implement MD5.
>> I see... I think! o_O
>
> It was a nice idea. You could install fonts into the global space, or
> into the executable for the word processor, or into an individual
> document that used that font. You could localize a program after it was
> completely finished just by adding resources to the executable. Stuff
> like that.
Still not really seeing why you need this implemented at the filesystem
level, but yeah...
> Oh, and chunks of code were also resources, so by calling code chunks,
> you'd wind up loading dynamically the appropriate code resources, so you
> got overlays for free, basically.
Do overlays matter any more? (Clearly they did once...)
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
|
|
