http://research.microsoft.com/os/Singularity/

Cool stuff. Low-overhead message passing even between separate 
processes. Kind of like Hermes done reasonably well, then used to 
implement AmigaDOS-like OS reasonably well. :-)

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Darren New wrote:
> http://research.microsoft.com/os/Singularity/
> 
> Cool stuff. 

Oh, and it also illustrates what I was saying about not needing 
destructors on your GC'ed resources if your OS actually implements 
things correctly.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Darren New wrote:
> Oh, and it also illustrates what I was saying about not needing 
> destructors on your GC'ed resources if your OS actually implements 
> things correctly.

And right now, one apparently writes system configuration information 
(i.e., what you'd normally feed into a program to generate a setup or 
boot script or "package") in Haskell.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Darren New wrote:
> Oh, and it also illustrates what I was saying about not needing 
> destructors on your GC'ed resources if your OS actually implements 
> things correctly.

It also shows a way of doing basically C++-like allocation management 
while nevertheless proving you're not leaking. That is, with minimal 
additional declarations (like, this procedure consumes its argument, 
like "dispose" does), you can prove at compile time that you aren't 
leaking memory and not using memory you already deallocated.

And they do experiments to show that the software checks for array 
bounds are actually significantly (33%) faster than the hardware checks 
you have to do anyway if you don't check it in software.

Overall, a very cool system with lots of cool results.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

I don't know about Singularity and it's particular design goals, but the 
other day I was thinking: What would happen if you set out to design a 
new OS completely from scratch? What would that look like?

It seems to me that most OS designs today are really quite similar. For 
example, in every OS I'm aware of, there is some kind of CLI that allows 
you to invoke executable programs, optionally passing some arguments, 
and the invoked program inherits 3 streams - stdin, stdout and stderr. 
The exact details vary, but this basic story seems to apply to every 
common OS.

What if we wanted to shake that up a litle? What might we decide to do 
differently?

Well, currently a program's arguments are just a giant blob of text. The 
OS does nothing more than hand it over to the program, which may then 
interpret them in any way it pleases. This is LCD; it works for 
everything, but it's not terribly sophisticated.

How about if, say, the program could somehow "tell" the OS what 
arguments it actually accepts? (In the same way a program file usually 
contains metadata to "tell" the OS all kinds of other stuff about it, 
such as linking information.) Then the OS could report invalid argument 
names without even needing to bother actually starting the program 
itself. And just think of the auto-complete possibilities.

Hey, let's go one better. The majority of CLI arguments are either 
on/off switches or filenames, right? Well what if we *tell* the OS what 
things are on/off switches, and that their default state should be? What 
if we *tell* it which things are supposed to be filenames? (And whether 
the name in question *should* or *should not* exist when the program is 
run? Or whether it should be a *file* or a *directory*? Or maybe even 
the name of another program?)

Once you start thinking this way, you start to see that actually, if we 
get the OS to interpret the arguments and pass *structured* data to the 
program [rather than just a blob of textual data], suddenly all sorts of 
interesting ideas become possible.

If nothing else, it means that CLI arguments now have a standardised 
format, enforced by the OS, which makes it easier to learn how to 
operate each new program. Maybe all the program does is somehow list a 
bunch of settings it requires? Maybe then you can specify those either 
by CLI arguments, or a per-user or per-machine set of defaults? Maybe 
the OS has a database of these default settings somewhere? All kinds of 
interesting ideas to throw around.

Similarly, on a "normal" OS, each program sets three character streams. 
(And Unix programs in particular seem to do weird trickery to discover 
whether the output stream "is a TTY" and behave differently if it is.) 
It's also traditional to pipe data between programs.

Maybe we can do something more interesting here? Maybe we can pass 
*structured* data around instead of just plain character streams? 
(Although now you start having potential difficulties with finding a 
data representation that everybody likes.) Maybe not every program has 
to have exactly 3 such streams? Maybe piping data between programs 
running on physically seperate networked machines shouldn't be too 
different from piping locally? Just a thought...

Tradition dictates that when a program exists, it returns a "status 
code", which is simply an integer. Zero indicates success, anything else 
indicates failure or at least some kind of warning. (And every program 
uses its own slightly different set of conventions here.)

Maybe we can do something better here too? Maybe we could have a small 
set of standard categories like "program bug", "resource exhaustion", 
"the network won't answer me", and provide a set of application-specific 
codes for the actual failures that a particular program can have?

How about logging? Windoze does this slightly better than Linux in that 
there are (typically 3) logs that applications can write to if they 
want. But maybe we could do something better than that? Maybe 
per-application logs? (If a given application wants it.) Maybe tell the 
OS how to invoke different levels of logging? Just some ideas.

Of course, when you look at filesystems, most OSes provide a construct 
known as a "file" which is an opaque sequence of bytes. (And a few 
provide a means to specify what those bytes are supposed to represent.) 
I suppose you could go down the route of having files contain structured 
data - but again you're going to get people arguing over the best way of 
structuring things.

I've often thought about what would happen if, say, Smalltalk was the 
entire OS. Then the OS would "know about" the internal workings of each 
program to a large degree, and that opens up some rather interesting 
possibilities. Things like highly structured IPC and so forth. Trouble 
is, now you can only run stuff implemented in Smalltalk...

In short, once you sit down and start to question the way OSes work 
today, you start to see that there are actually many things we could be 
doing differently - ranging from the conservative to the highly radical. 
(To me, really radical ideas are interesting to think about but probably 
wouldn't work too well in practice.)

Heh, if *I* had 3 years to sit and write an OS, maybe I could experiment 
with a few of these ideas? ;-)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Does anybody *else* find it ironic that Micro$oft - the corporation 
internationally renowned for its poor quality, buggy products - is 
interested in methods of producing high-quality software?

Surely bug-free software isn't very profitable? :-P

Regardless, Singularity has a number of interesting ideas.

- Let the compiler enforce program isolation, not the processor 
hardware. (That works great if everybody uses your compiler, but I'm not 
sure what happens if you allow arbitrary 3rd party code to execute...)

- Make IPC fast and use it liberally. Use IPC for plugins instead of 
dynamic loading. Use statically-checked IPC protocols.

- Put almost everything outside the kernel and make all the security 
decisions there.

- Assign security rights to applications as well as users. (I have often 
wondered why no OS does this already...)

Some interesting ideas there...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Invisible <voi### [at] devnull> wrote:
> - Let the compiler enforce program isolation, not the processor 
> hardware. (That works great if everybody uses your compiler, but I'm not 
> sure what happens if you allow arbitrary 3rd party code to execute...)

  The only way to make an OS secure is to have hardware support. Hardware
is the only thing that can stop a program from accessing what it must not
access.

  (Ok, there's another alternative: Run the programs under an emulator.
Of course this is out of question because of speed issues.)

-- 
                                                          - Warp

Post a reply to this message

>> - Let the compiler enforce program isolation, not the processor 
>> hardware. (That works great if everybody uses your compiler, but I'm not 
>> sure what happens if you allow arbitrary 3rd party code to execute...)
> 
>   The only way to make an OS secure is to have hardware support. Hardware
> is the only thing that can stop a program from accessing what it must not
> access.
> 
>   (Ok, there's another alternative: Run the programs under an emulator.
> Of course this is out of question because of speed issues.)

Their approach seems to be to "verify" each program before it runs, 
checking that it doesn't do any "bad" things.

Presumably verifying whether a program does or does not do something 
"bad" is formally equivilent to the halting problem, so I imagine they 
apply some arbitrary set of restrictions to simplify the problem.

Singularity is of course a research experiment, not a production-grade 
OS. It would be interesting to see if they could make it work in the 
face of hostile 3rd party code...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Invisible <voi### [at] devnull> wrote:
> Their approach seems to be to "verify" each program before it runs, 
> checking that it doesn't do any "bad" things.

  That's impossible. It can be proven that it's an unsolvable problem,
exactly for the same reason as the halting problem is unsolvable. There's
no way for any program to check if a piece of code is executed and how.

  It's also impossible for it to know, for example, the addresses of all
pointers by simply examining the program (for example the address of a
pointer could be calculated from user input).

> Presumably verifying whether a program does or does not do something 
> "bad" is formally equivilent to the halting problem, so I imagine they 
> apply some arbitrary set of restrictions to simplify the problem.

  Those restrictions could seriously hinder compiler optimizations.
For example accessing the nth element of an array can usually be done
with a simple CPU opcode. However, if the system restricts this because
it cannot prove what that n might contain, it means that the compiler
cannot generate the single opcode for accessing that array, but must
perform something much more complicated to keep the system happy.

  Ah, but that's the trend nowadays: Computers get faster and the amount
of RAM grows exponentially with time. There's no need for highly optimized
code.

-- 
                                                          - Warp

Post a reply to this message

>> Their approach seems to be to "verify" each program before it runs, 
>> checking that it doesn't do any "bad" things.
> 
>   That's impossible. It can be proven that it's an unsolvable problem,
> exactly for the same reason as the halting problem is unsolvable.

In the general case, it's definitely unsolvable. I'm not sure precisely 
what they're doing to "make" it solvable - but clearly it must involve 
some kind of limitation or other.

>> Presumably verifying whether a program does or does not do something 
>> "bad" is formally equivilent to the halting problem, so I imagine they 
>> apply some arbitrary set of restrictions to simplify the problem.
> 
>   Those restrictions could seriously hinder compiler optimizations.

It's hard to say, but it *also* appears that Singularity runs some kind 
of portable VM code.

 From what I can gather, when you "install" an application, a verifier 
checks that the VM code doesn't do any "bad" things, and then compiles 
it to native code - whatever that might be. Then when you run the 
application, it just runs the native code, trusting that it can't 
possibly do bad things.

Apparently it "works" in their research prototype. Whether it could work 
in a real-world OS is another matter entirely...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

In a nutshell:

http://msdn.microsoft.com/en-gb/magazine/cc163603.aspx

Make of that what you will...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Warp schrieb:
> Invisible <voi### [at] devnull> wrote:
>> Their approach seems to be to "verify" each program before it runs, 
>> checking that it doesn't do any "bad" things.
> 
>   That's impossible. It can be proven that it's an unsolvable problem,
> exactly for the same reason as the halting problem is unsolvable. There's
> no way for any program to check if a piece of code is executed and how.
> 
>   It's also impossible for it to know, for example, the addresses of all
> pointers by simply examining the program (for example the address of a
> pointer could be calculated from user input).

If you only allow safe-mode managed code, pointer arithmethic is not 
possible. I don't see a big problem to validate managed code, ensuring 
it doesn't do anything "bad" for a fixed definition of "bad".

Manuel

Post a reply to this message

Invisible wrote:
> I don't know about Singularity and it's particular design goals, but the 
> other day I was thinking: What would happen if you set out to design a 
> new OS completely from scratch? What would that look like?

Dude? That's Singularity. :-)

Download the release and read the design notes, too.
http://codeplex.com/singularity

> What if we wanted to shake that up a litle? What might we decide to do 
> differently?

Read the Singularity papers. This is exactly the premise they started with.

> How about if, say, the program could somehow "tell" the OS what 
> arguments it actually accepts? 

Yep! And what if, when you installed the program, it said "Sorry, but 
this program requires the ability to connect to the USB printer, and you 
don't have the right USB driver installed"?  Or "I can't let you install 
telnet[1] until you have some sort of TCP/IP stack installed."

[1] Note that "install" means "make available for running." Of course 
the uninstalled code can sit out there.

> Hey, let's go one better. The majority of CLI arguments are either 
> on/off switches or filenames, right? 

Only in a UNIX-like OS. The majority of arguments are instructions on 
how to treat the other arguments, or references to something in the file 
system namespace, there.

Once you can specify as arguments things beyond stuff represented as 
strings, you get a whole nuther ball of wax. You're not stuck with 
stdin, stdout, and stderr, for example. You can say "Hey, this program 
needs an interrupt and two DMA channels to serve as a driver" for example.

> Once you start thinking this way, you start to see that actually, if we 
> get the OS to interpret the arguments and pass *structured* data to the 
> program [rather than just a blob of textual data], suddenly all sorts of 
> interesting ideas become possible.

Read the bits about "compile-time reflection".  You really don't even 
need to "tell" the OS this info, if it's reflected in the types your 
system supports.

> If nothing else, it means that CLI arguments now have a standardised 
> format, enforced by the OS, which makes it easier to learn how to 
> operate each new program. Maybe all the program does is somehow list a 
> bunch of settings it requires? Maybe then you can specify those either 
> by CLI arguments, or a per-user or per-machine set of defaults? Maybe 
> the OS has a database of these default settings somewhere? All kinds of 
> interesting ideas to throw around.

Yep. Singularity calls it the "application manifest".  An application is 
a first-class object, rather than being "a pile of files full of code".

> Maybe we can do something more interesting here? Maybe we can pass 
> *structured* data around instead of just plain character streams? 

Yes. Structured and typed, including a finite state machine to say when 
it's OK to send and what you need to be ready to receive, checked at 
compile time to ensure your code actually obeys the protocol, then 
compiled down to native code and never checked again.

> Maybe we can do something better here too? Maybe we could have a small 
> set of standard categories like "program bug", "resource exhaustion", 
> "the network won't answer me", and provide a set of application-specific 
> codes for the actual failures that a particular program can have?

Nah. You just answer back on the stream that goes to whoever invoked 
you. :-)  Why would only the parent want to know how you exited?

> OS how to invoke different levels of logging? Just some ideas.

SDN 14 Tracing.pdf

> I suppose you could go down the route of having files contain structured 
> data - but again you're going to get people arguing over the best way of 
> structuring things.

Not any more than saying "you'll have people arguing about the best way 
to represent structures".

You're still thinking UNIXy.  Get rid of the mindset that you have to 
agree on data formats and embrace the mindset that you only have to 
agree on APIs. You don't need to stick some Perl script in the middle of 
a pipeline to transform your data. You present the data in a 
semantically-meaningful way.

I.e., your directory isn't a file with 16-byte entries, the first two 
bytes of which is a i-node number, and if non-zero, is followed by up to 
14 bytes nul-terminated file name.

Your directory, instead, is a set of function calls like "read first", 
"read next", "provide details". You don't have to come up with some 
on-disk format to define.

> I've often thought about what would happen if, say, Smalltalk was the 
> entire OS. Then the OS would "know about" the internal workings of each 
> program to a large degree, and that opens up some rather interesting 
> possibilities. Things like highly structured IPC and so forth. Trouble 
> is, now you can only run stuff implemented in Smalltalk...

Yep. That's traditionally been the problem. Singularity does this, but 
makes MSIL the bottom level for applications and such. So anything you 
can compile into structured typed assembler language you can use. This 
includes C#, F#, Iron Python, etc.

> In short, once you sit down and start to question the way OSes work 
> today, you start to see that there are actually many things we could be 
> doing differently - ranging from the conservative to the highly radical. 
> (To me, really radical ideas are interesting to think about but probably 
> wouldn't work too well in practice.)

It seems to be working well in practice. For example, one radical idea 
(which I always thought would be a good idea) is to use safe languages 
for everything. Singularity does this, and in so doing, can run 
everything in Ring 0 and with no hardware memory protection. It actually 
runs faster, because it takes less time to enforce array bound checks 
(for example) than it does to go thru the memory mapping hardware on 
every access to memory. Turning off memory protection more than makes up 
for doing it in software. And then, doing things like scheduling 
threads, adding space to the thread stack, allocating and freeing memory 
blocks ... all that is in user space, because the compiler can inline 
the kernelesque instructions.

> Heh, if *I* had 3 years to sit and write an OS, maybe I could experiment 
> with a few of these ideas? ;-)

Read the papers first. It's exactly what I've been wanting to do myself, 
except they figured out what seems a really good way of doing it.

I like the stuff on permissions, too. Stuff like "setuid" not being a 
privileged operation is kind of funky. :-)

Really, all the stuff you're speculating about, they've written about in 
detail and implemented. It's very cool.  I highly suggest if the idea 
"what if we started over in *this* millenium?" interests you, you read 
the literature they've published. :)

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Manuel Kasten <kas### [at] gmxde> wrote:
> If you only allow safe-mode managed code, pointer arithmethic is not 
> possible.

  So you can't have arrays?

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Invisible <voi### [at] devnull> wrote:
>> Their approach seems to be to "verify" each program before it runs, 
>> checking that it doesn't do any "bad" things.
> 
>   That's impossible. It can be proven that it's an unsolvable problem,
> exactly for the same reason as the halting problem is unsolvable. There's
> no way for any program to check if a piece of code is executed and how.

This isn't quite true. If you restrict the forms of programs you accept 
to something you can verify, then you can do this pretty easily. It is, 
for example, why people are willing to run java applets in their browser.

While it's true the halting problem prevents you from knowing whether 
any arbitrary TM will halt, it doesn't prevent you from knowing whether 
any arbitrary regular expression will halt, for example. If you 
eliminate the instructions that let you write to arbitrary parts of 
memory you don't own, then it's not too hard to check your language 
works fine.

It's also the case that hardware doesn't 100% solve the problem either. 
You have to (a) trust the hardware not to be buggy, and (b) trust the OS 
to correctly set up the hardware.

>   It's also impossible for it to know, for example, the addresses of all
> pointers by simply examining the program (for example the address of a
> pointer could be calculated from user input).

No, because the OS won't install a program that calculates the address 
of a pointer calculated from user input. Basically, you use C# or one of 
the other .NET languages, that compiles down to a strongly-typed 
assembly language. Then, before you run the program, you gather up all 
the strongly typed assembler,

>> Presumably verifying whether a program does or does not do something 
>> "bad" is formally equivilent to the halting problem, so I imagine they 
>> apply some arbitrary set of restrictions to simplify the problem.
> 
>   Those restrictions could seriously hinder compiler optimizations.

Actually, it turns out the compiler can do a *much* better job, because 
it can track the usage of a whole bunch of stuff that's hard to track 
when you allow arbitrary pointers.

> For example accessing the nth element of an array can usually be done
> with a simple CPU opcode. However, if the system restricts this because
> it cannot prove what that n might contain, it means that the compiler
> cannot generate the single opcode for accessing that array, but must
> perform something much more complicated to keep the system happy.

Right. They actually check this, and discover it's about a 4% overhead 
to do the checks in software. And it's about a 6% overhead to do the 
checks in hardware.  Where Is Your God Now?  Mwa ha ha ha!  ;-)

And it's about a 33% overhead to actually put processes in different 
address spaces and enforce that they can't change the VM mapping by 
taking away the ring-0 instructions, compared to checking at compile 
time that you don't go out of bounds and enforcing at runtime where you 
can't check at compile time, once you count up TLB misses, TLB flushes, 
frobbing stacks around during an interrupt, etc.

>   Ah, but that's the trend nowadays: Computers get faster and the amount
> of RAM grows exponentially with time. There's no need for highly optimized
> code.

You should read the papers. *Because* the input is actually structured, 
they can compile the stuff and throw away (for example) fields and 
methods that aren't used, include a GC that's specific to the problem 
being solved (e.g., a higher-overhead real-time GC only for real-time 
programs), and they get a tremendous efficiency boost.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Warp wrote:
> Manuel Kasten <kas### [at] gmxde> wrote:
>> If you only allow safe-mode managed code, pointer arithmethic is not 
>> possible.
> 
>   So you can't have arrays?
> 

You can, but not in the sense of a contiguous block of memory containing 
the data sense.

The question I have is what if I want to develop an application (such as 
a high performance image analysis package) against a platform that uses 
only managed code. Could it be done using the CPU the most efficiently? 
If I were restricted to "safe" code, is there a way to remove that 
restriction for that app?

Post a reply to this message

Mike Raiford <mra### [at] hotmailcom> wrote:
> >   So you can't have arrays?

> You can, but not in the sense of a contiguous block of memory containing 
> the data sense.

  But then the answer is "no".

-- 
                                                          - Warp

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> > For example accessing the nth element of an array can usually be done
> > with a simple CPU opcode. However, if the system restricts this because
> > it cannot prove what that n might contain, it means that the compiler
> > cannot generate the single opcode for accessing that array, but must
> > perform something much more complicated to keep the system happy.

> Right. They actually check this, and discover it's about a 4% overhead 
> to do the checks in software. And it's about a 6% overhead to do the 
> checks in hardware.  Where Is Your God Now?  Mwa ha ha ha!  ;-)

  I have really hard time believing that if you, for example, calculate
the sum of all the integers in an array, adding boundary checks to every
single read operation will add only 4% of overhead.

  Even if the boundary check would take 1 clock cycle, that would mean
that reading the value from the array and adding its value to a register
takes 25 clock cycles.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:

>   I have really hard time believing that if you, for example, calculate
> the sum of all the integers in an array, adding boundary checks to every
> single read operation will add only 4% of overhead.

Accessing "everything in this array" is a pretty common operation - and 
one that an optimising compiler can presumably spot and optimise pretty 
easily.

Now, if you start accessing an array in some really random order... (And 
let's face it, what the hell are arrays especially good at?)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

>> I don't know about Singularity and it's particular design goals, but 
>> the other day I was thinking: What would happen if you set out to 
>> design a new OS completely from scratch? What would that look like?
> 
> Dude? That's Singularity. :-)

Not quite. They didn't do it the way *I* would do it. ;-)

Actually, I can think of *several* ways to do it, and I'd probably spend 
the rest of my life analysing them and never write any real code! :-/

> Download the release and read the design notes, too.
> http://codeplex.com/singularity

Uh... why? It's an experimental research prototype. It probably doesn't 
even run yet.

>> What if we wanted to shake that up a litle? What might we decide to do 
>> differently?
> 
> Read the Singularity papers. This is exactly the premise they started with.

They were focused specifically on "how do we increase security?" I'm 
just thinking "what widely-used but suboptimal abstractions might we 
change?"

> Or "I can't let you install 
> telnet[1] until you have some sort of TCP/IP stack installed."

Isn't this what RPM does?

> Once you can specify as arguments things beyond stuff represented as 
> strings, you get a whole nuther ball of wax. You're not stuck with 
> stdin, stdout, and stderr, for example. You can say "Hey, this program 
> needs an interrupt and two DMA channels to serve as a driver" for example.

A device driver is a rather unusual type of program. I'm thinking more 
about end-user level stuff. You know - the kind of thing you might 
invoke by hand.

> Yep. Singularity calls it the "application manifest".  An application is 
> a first-class object, rather than being "a pile of files full of code".

As an aside... Whenever I compile a Haskell program, it generates a 
*.manifest file that contains some random XML. Any idea WTF that's about?

>> Maybe we can do something better here too? Maybe we could have a small 
>> set of standard categories like "program bug", "resource exhaustion", 
>> "the network won't answer me", and provide a set of 
>> application-specific codes for the actual failures that a particular 
>> program can have?
> 
> Nah. You just answer back on the stream that goes to whoever invoked 
> you. :-)  Why would only the parent want to know how you exited?

Maybe because it's a lights-out system and you want the failed process 
to be started back up again? IDK.

>> I suppose you could go down the route of having files contain 
>> structured data - but again you're going to get people arguing over 
>> the best way of structuring things.
> 
> Not any more than saying "you'll have people arguing about the best way 
> to represent structures".
> 
> You're still thinking UNIXy.  Get rid of the mindset that you have to 
> agree on data formats and embrace the mindset that you only have to 
> agree on APIs.

I guess if you follow all this to its logical conclusion, you end up 
with "the filesystem is a relational database" - and we all know what a 
bad idea *that* was!

>> I've often thought about what would happen if, say, Smalltalk was the 
>> entire OS. Then the OS would "know about" the internal workings of 
>> each program to a large degree, and that opens up some rather 
>> interesting possibilities. Things like highly structured IPC and so 
>> forth. Trouble is, now you can only run stuff implemented in Smalltalk...
> 
> Yep. That's traditionally been the problem. Singularity does this, but 
> makes MSIL the bottom level for applications and such. So anything you 
> can compile into structured typed assembler language you can use. This 
> includes C#, F#, Iron Python, etc.

(Or Haskell, when they fix the bitrot in the MSIL backend.)

One day, I'll have to sit down and find out how the Java VM or the CLR work.

>> (To me, really radical ideas are interesting to think about 
>> but probably wouldn't work too well in practice.)
> 
> It seems to be working well in practice. For example, one radical idea 
> (which I always thought would be a good idea) is to use safe languages 
> for everything. Singularity does this, and in so doing, can run 
> everything in Ring 0 and with no hardware memory protection.

Yah, but this only really works if you're not going to execute arbitrary 
C code - which would be kind of a problem.

>> Heh, if *I* had 3 years to sit and write an OS, maybe I could 
>> experiment with a few of these ideas? ;-)
> 
> Read the papers first. It's exactly what I've been wanting to do myself, 
> except they figured out what seems a really good way of doing it.
> 
> I like the stuff on permissions, too.

Specifying access control by application seems like a perfectly logical 
thing to want to do. That whole Unixy trip with creating a user and 
group named "apache" and making sure the Apache httpd runs under that 
account just seems like a huge kludge to me...

> Really, all the stuff you're speculating about, they've written about in 
> detail and implemented. It's very cool.  I highly suggest if the idea 
> "what if we started over in *this* millenium?" interests you, you read 
> the literature they've published. :)

...and what do you think I just spent my entire afternoon doing? :-P

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
>>> I don't know about Singularity and it's particular design goals, but 
>>> the other day I was thinking: What would happen if you set out to 
>>> design a new OS completely from scratch? What would that look like?
>>
>> Dude? That's Singularity. :-)
> 
> Not quite. They didn't do it the way *I* would do it. ;-)

Then you should have said "What would happen if *I* set out to design a 
new OS completely from scratch?"  ;-)

>> Download the release and read the design notes, too.
>> http://codeplex.com/singularity
> 
> Uh... why? It's an experimental research prototype. It probably doesn't 
> even run yet.

You say this with such confidence. Yet, surprisingly, you didn't 
actually (say) read any of the papers, and noticed they're all dated 
five years ago.

> They were focused specifically on "how do we increase security?" I'm 
> just thinking "what widely-used but suboptimal abstractions might we 
> change?"

No. You didn't read the papers, so you don't know what they're working 
on. It's better to read what the authors wrote than what some blogger 
says about what the authors wrote.

>> Or "I can't let you install telnet[1] until you have some sort of 
>> TCP/IP stack installed."
> 
> Isn't this what RPM does?

No.

>> Once you can specify as arguments things beyond stuff represented as 
>> strings, you get a whole nuther ball of wax. You're not stuck with 
>> stdin, stdout, and stderr, for example. You can say "Hey, this program 
>> needs an interrupt and two DMA channels to serve as a driver" for 
>> example.
> 
> A device driver is a rather unusual type of program. I'm thinking more 
> about end-user level stuff. You know - the kind of thing you might 
> invoke by hand.

Right. Of course, since they're writing the OS, they're worried about 
the sorts of problems drivers cause. But the same result applies to 
things you invoke by hand or from other programs.

> As an aside... Whenever I compile a Haskell program, it generates a 
> *.manifest file that contains some random XML. Any idea WTF that's about?

Well, a manifest is a list of what's included. Other than that, I 
couldn't help you guess without seeing one.

>> Nah. You just answer back on the stream that goes to whoever invoked 
>> you. :-)  Why would only the parent want to know how you exited?
> 
> Maybe because it's a lights-out system and you want the failed process 
> to be started back up again? IDK.

No, I'm saying why would you want the exit status to *ONLY* go to the 
parent process, and not to whoever you want it to go to? Why not list in 
the application manifest all the applications that'll be interested in 
knowing that program X failed?  Wouldn't you want everyone using the TCP 
stack to know that the nic driver failed?

>> You're still thinking UNIXy.  Get rid of the mindset that you have to 
>> agree on data formats and embrace the mindset that you only have to 
>> agree on APIs.
> 
> I guess if you follow all this to its logical conclusion, you end up 
> with "the filesystem is a relational database" - and we all know what a 
> bad idea *that* was!

Uh, no. You wind up with "everything is strongly typed", not necessarily 
"everything is the same type". I am not sure I've discovered exactly 
what they store in files - I'm still going thru the papers - but I'm 
pretty sure it's not relational.

It's not unlike the Amiga OS in that respect, except safe and strongly 
typed.

>> Yep. That's traditionally been the problem. Singularity does this, but 
>> makes MSIL the bottom level for applications and such. So anything you 
>> can compile into structured typed assembler language you can use. This 
>> includes C#, F#, Iron Python, etc.
> 
> (Or Haskell, when they fix the bitrot in the MSIL backend.)

Yep. Or most anything. I'm pretty impressed that they managed to get 
functional languages doing their thing in an OO assembler language.

> One day, I'll have to sit down and find out how the Java VM or the CLR 
> work.

It's ugly. I think the JVM is probably a little easier to understand. 
But think of it merely as strongly typed assembler language with lots of 
metadata about types and layouts.

> Yah, but this only really works if you're not going to execute arbitrary 
> C code - which would be kind of a problem.

Exactly. Why do you need to execute arbitrary C code, tho? Other than 
compatibility?

> Specifying access control by application seems like a perfectly logical 
> thing to want to do. That whole Unixy trip with creating a user and 
> group named "apache" and making sure the Apache httpd runs under that 
> account just seems like a huge kludge to me...

Yes, exactly. Singularity lets you specify it as both, including the 
history.  So "PHP running from user Fred invoked via Apache" can have 
different permissions from "PHP running from user Fred invoked via bash".

> ....and what do you think I just spent my entire afternoon doing? :-P

OK. Well, some of your assertions about how it works were at odds with 
what they wrote, so I assumed you hadn't read all the way through.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Mike Raiford wrote:
> Warp wrote:
>> Manuel Kasten <kas### [at] gmxde> wrote:
>>> If you only allow safe-mode managed code, pointer arithmethic is not 
>>> possible.
>>
>>   So you can't have arrays?
>>
> 
> You can, but not in the sense of a contiguous block of memory containing 
> the data sense.

Actually, yes, you can. You just bounds-check the array.  You can do 
that in a C implementation, even. People just don't for some reason.

Indeed, the language they use for the OS has "representation structures" 
which are specifically designed to (for example) land in certain 
memory-mapped hardware bits.

There's no problem supporting arrays. Arrays are objects. The problem is 
supporting arbitrary untyped pointers assigned non-pointer values - 
i.e., the problem is casting an integer to a pointer.

> The question I have is what if I want to develop an application (such as 
> a high performance image analysis package) against a platform that uses 
> only managed code. Could it be done using the CPU the most efficiently? 

Sure, why not? If the compiler can prove you're not violating the memory 
constraints, why not? Note that their tests show it's actually more 
efficient to check in software than in hardware, and the software checks 
are pretty efficient.

> If I were restricted to "safe" code, is there a way to remove that 
> restriction for that app?

No. That's the point.

I mean, I suppose, sure, you could. But it's not going to be a regular 
app. You'd need to install it differently and prove you're allowed to. 
That's how the kernel, for example, works.  It's like asking "can I 
write code in a Linux app that bypasses the memory mapping hardware?" 
Sure, but it's far from normal.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Warp wrote:
> Mike Raiford <mra### [at] hotmailcom> wrote:
>>>   So you can't have arrays?
> 
>> You can, but not in the sense of a contiguous block of memory containing 
>> the data sense.
> 
>   But then the answer is "no".

I don't think Mike read the papers. Of course you can have an array of 
contiguous memory. You declare it as an array of structs, just like you 
would in C.

They even have a mechanism whereby you can declare a struct with a 
definite memory layout that multiple different languages can reference, 
and an operator that says "treat this as the representation of an 
object", which basically adds the vtable after the fact for your 
particular program. I.e., you can cast an object in memory from a flat 
data structure into a full object-oriented object with inheritance and 
methods and all that, without moving the memory that holds the fields. 
And since you're sharing that memory with different languages, you can 
have the different languages cast it into different objects without 
munging it up for any one particular language.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Warp wrote:
>   I have really hard time believing that if you, for example, calculate
> the sum of all the integers in an array, adding boundary checks to every
> single read operation will add only 4% of overhead.

The compiler can be pretty smart. You can actually optimize out the 
bounds checking most of the time.

int x[50]; int y;
for (i = 0; i < 50; i++) x[i] = i;
for (i = 0; i < 50; i++) y += x[i];

That won't have any bounds-checking code included.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Orchid XP v8 wrote:
>> Download the release and read the design notes, too.
>> http://codeplex.com/singularity
> 
> Uh... why? It's an experimental research prototype. It probably doesn't 
> even run yet.

BTW, the reason I told you to download the release is because there's 
extensive high-level documentation included in the release. There's a 
bunch of PDF files that aren't on the web site that describe how the 
system works, how the verification works, the graph walking that proves 
everything in a system will boot and shows what order to start drivers 
and applications in, and so on.

Not because I expected you to run the code, or even read the source for 
that matter.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> The compiler can be pretty smart.

  If you are doing a simple linear traversal, maybe, but if it's any more
complicated than that...

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> For example accessing the nth element of an array can usually be done
> with a simple CPU opcode. However, if the system restricts this because
> it cannot prove what that n might contain, it means that the compiler
> cannot generate the single opcode for accessing that array, but must
> perform something much more complicated to keep the system happy.

Here's an example of where the 4% comes from. The compiler might 
generate a single op-code, and that single op-code might take hundreds 
of cycles to run, because it hits a page whose virtual address map isn't 
in the cache.  Or, even worse, it hits a page that isn't in memory at 
all. But sure, I suppose if your program consists primarily of random 
access to an array of stuff that you could do in one cycle, and your 
cache coherency sucks, you might take a slight extra hit for bounds 
checking. I guess things like photoshop plug-ins for distorting an image 
might take something of a hit. Something like a SQL server would 
probably run faster than on hardware-protected processes. The 4% was 
from their compiler/verifier/code generator, IIRC.

There's another cool thing they do. Each thread starts with only a 4K 
stack (i.e., one page). The installer (that compiles from MSIL to native 
code, called "bartok" for some reason) will build a call map, figure out 
which function calls *might* pass a page boundary, and insert in-line 
code to allocate another page of memory. Then it copies the appropriate 
number of arguments to the new stack frame, after including a return 
address which will deallocate that new page of memory. So instead of 
allocating a meg of memory for stack space for each thread, or instead 
of trapping out when you run off the end and trying to rearrange things, 
instead you have a bunch of randomly-allocated pages holding your stack, 
linked together with compiler-generated code to allocate and deallocate 
pages as needed. The compiler also makes sure there's enough space at 
the top of any given page to hold the stack of any interrupt routine 
that might run, so you don't even have to deal with switching pages 
around for that.  And when the code *does* call into the kernel, it just 
allocates a new stack page for that and makes the call, and marks that 
stack page as belonging to the kernel, so the GC doesn't start reaping 
things it shouldn't and so the process can get cleaned up if it exits 
during a call-back from the kernel. But if the compiler can look at the 
call graph and figure out that either you *won't* overflow the stack 
frame, or you *will* overflow the stack frame, there's no need to even 
put in the check - you can just put in the code (or not) do do the right 
thing.

And a lot of this gets inlined in the code, because they know what 
kernel you're "linked" against, and they know you can't execute the 
arbitrary code, so you're often not even "trapping" into the kernel to 
allocate memory or send messages between processes or schedule threads 
or whatever.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Warp wrote:
> Darren New <dne### [at] sanrrcom> wrote:
>> The compiler can be pretty smart.
> 
>   If you are doing a simple linear traversal, maybe, but if it's any more
> complicated than that...

True. One advantage the compiler[1] has is that it has the entire source 
code in front of it when it compiles. So it can check that everywhere an 
index gets called, the value is within range, for example.

But yes, obviously if the compiler could prove *everything*, there 
wouldn't be a 4% slow-down with the compiler adding the checks. :-)


[1] Using the term loosely here...
-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

>>> Dude? That's Singularity. :-)
>>
>> Not quite. They didn't do it the way *I* would do it. ;-)
> 
> Then you should have said "What would happen if *I* set out to design a 
> new OS completely from scratch?"  ;-)

Well :-P to you.

>>> Download the release and read the design notes, too.
>>
>> Uh... why? It's an experimental research prototype. It probably 
>> doesn't even run yet.
> 
> You say this with such confidence. Yet, surprisingly, you didn't 
> actually (say) read any of the papers, and noticed they're all dated 
> five years ago.

And this makes a difference? It's not meant to be an end-user OS. It's 
meant for OS hackers to play with.

>> They were focused specifically on "how do we increase security?" I'm 
>> just thinking "what widely-used but suboptimal abstractions might we 
>> change?"
> 
> No. You didn't read the papers, so you don't know what they're working 
> on.

Sure I did. They have software-enforced type and memory safety so they 
can turn off hardware protection and make IPC really cheap. They also 
statically check IPC. I don't see anything anywhere about, say, how end 
users invoke programs.

>>> Or "I can't let you install telnet[1] until you have some sort of 
>>> TCP/IP stack installed."
>>
>> Isn't this what RPM does?
> 
> No.

Really? I thought that was the entire *point* of package managers. (And 
also the reason that as soon as you attempt to upgrade any Linux 
installation, it breaks on 50,000 instances of "wrong version of glibc" 
or something similar.)

>>> Nah. You just answer back on the stream that goes to whoever invoked 
>>> you. :-)  Why would only the parent want to know how you exited?
>>
>> Maybe because it's a lights-out system and you want the failed process 
>> to be started back up again? IDK.
> 
> No, I'm saying why would you want the exit status to *ONLY* go to the 
> parent process, and not to whoever you want it to go to?

Well sure, other programs might want to know as well. I was just 
pointing out that there may not *be* a human sitting at the console, 
that's all. ;-)

>> I guess if you follow all this to its logical conclusion, you end up 
>> with "the filesystem is a relational database" - and we all know what 
>> a bad idea *that* was!
> 
> Uh, no. You wind up with "everything is strongly typed", not necessarily 
> "everything is the same type".

You still have to get everybody to agree on what constitutes a "type". 
Ask a BASIC programmer and they'll tell you a "type" is either 
"integer", "float" or "string". Ask a Pascal programmer and they'll tell 
you a "type" is a unique identifier that identifies an array or a 
record. Ask an OOP expert and they'll tell you a "type" is a class. You 
don't even wanna *know* what a Haskell programmer has to say about the 
matter...

Seriously, do you have *any idea* how many standards have been put 
forward for "store digital audio in a file"? ;-)

> It's not unlike the Amiga OS in that respect, except safe and strongly 
> typed.

Um... AmigaDOS files are streams of octets, just like every other OS.

>>> Yep. That's traditionally been the problem. Singularity does this, 
>>> but makes MSIL the bottom level for applications and such. So 
>>> anything you can compile into structured typed assembler language you 
>>> can use. This includes C#, F#, Iron Python, etc.
>>
>> (Or Haskell, when they fix the bitrot in the MSIL backend.)
> 
> Yep. Or most anything. I'm pretty impressed that they managed to get 
> functional languages doing their thing in an OO assembler language.

Uh, yeah... Haskell really doesn't fit the MSIL very well. I'm told it's 
not very performant there. (I have no idea about F# - but when I 
researched it, it didn't appear to be very functional.)

>> One day, I'll have to sit down and find out how the Java VM or the CLR 
>> work.
> 
> It's ugly. I think the JVM is probably a little easier to understand. 
> But think of it merely as strongly typed assembler language with lots of 
> metadata about types and layouts.

I'm just wondering how you design assembler so that it can be run 
efficiently on multiple targets, that's all. (I hear it's a stack 
machine rather than a register machine, for example. Aren't Wikipaths fun?)

>> Yah, but this only really works if you're not going to execute 
>> arbitrary C code - which would be kind of a problem.
> 
> Exactly. Why do you need to execute arbitrary C code, tho? Other than 
> compatibility?

Oh, well, other than the "minor detail" of compatibility, there's no 
problem at all! ;-)

(You recall that "Linux" is actually a tiny bit of software which 
inherited compatibility with Unix, thus earning an instant library of 
userland tools, right?)

>> Specifying access control by application seems like a perfectly 
>> logical thing to want to do. That whole Unixy trip with creating a 
>> user and group named "apache" and making sure the Apache httpd runs 
>> under that account just seems like a huge kludge to me...
> 
> Yes, exactly. Singularity lets you specify it as both, including the 
> history.  So "PHP running from user Fred invoked via Apache" can have 
> different permissions from "PHP running from user Fred invoked via bash".

...which makes significantly more sense.

(Actually, maybe PHP is a bad example. Perhaps you want to assign 
different permissions to each PHP script? Rather than just to the PHP 
interpretter?)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> And this makes a difference? It's not meant to be an end-user OS. It's 
> meant for OS hackers to play with.

It's designed to support research into building another OS that uses the 
same technologies. They even tell you the name of the "production" OS, 
whose name starts with an "M" but I don't remember offhand. :-)

> Sure I did. They have software-enforced type and memory safety so they 
> can turn off hardware protection and make IPC really cheap. They also 
> statically check IPC. I don't see anything anywhere about, say, how end 
> users invoke programs.

Fair enough. They actually show it in the videos. It looks like a pretty 
straightforward CLI thing.

>>>> Or "I can't let you install telnet[1] until you have some sort of 
>>>> TCP/IP stack installed."
>>>
>>> Isn't this what RPM does?
>>
>> No.
> 
> Really? I thought that was the entire *point* of package managers. 

To some extent. Package managers tell you which dynamic libraries are 
needed for which programs. They don't enforce anything, and you cannot 
(for example) look at an RPM without installing it and know if it'll 
work right once you're done installing it.

It's a small part of what RPMs are supposed to do, and they do it 
relatively poorly. Enough so that I'd have to say "no, they're different 
things."

>> No, I'm saying why would you want the exit status to *ONLY* go to the 
>> parent process, and not to whoever you want it to go to?
> 
> Well sure, other programs might want to know as well. I was just 
> pointing out that there may not *be* a human sitting at the console, 
> that's all. ;-)

Right.  I'm not sure how we wound up talking past each other. :-)

>> Uh, no. You wind up with "everything is strongly typed", not 
>> necessarily "everything is the same type".
> 
> You still have to get everybody to agree on what constitutes a "type". 

Sure. But there's a least-common-denominator that gets passed back and 
forth, called the "rep types". Basically, stuff everyone can represent. 
And that's why you can take the rep type and cast it to a 
language-specific type.

> Seriously, do you have *any idea* how many standards have been put 
> forward for "store digital audio in a file"? ;-)

Sure. But they're all of the same type by the time you talk to the codec 
to get the data out of them.

>> It's not unlike the Amiga OS in that respect, except safe and strongly 
>> typed.
> 
> Um... AmigaDOS files are streams of octets, just like every other OS.

No they're not. Amiga OS devices are things that listen for and respond 
to typed messages.  Certainly the narrator isn't a "stream of octets", 
nor is the clock, nor is the audio device.

Files, yes, to some extent (i.e., discounting the metadata). But you 
don't access files in the Amiga OS. You access drivers. Files on disk 
are one small part of it. And even the directories aren't arrays of bytes.

> Uh, yeah... Haskell really doesn't fit the MSIL very well. I'm told it's 
> not very performant there. (I have no idea about F# - but when I 
> researched it, it didn't appear to be very functional.)

The only thing I'd heard is that F# is apparently a port of Caml to 
.NET.  You now know as much about it as I do. :-)

> I'm just wondering how you design assembler so that it can be run 
> efficiently on multiple targets, that's all. 

One of the things .NET does, for example, is require that every path 
through the MSIL that gets to the same opcode has to have the same types 
on the stack at that point. For example.  Apparently this makes it 
easier to generate good code, because you can statically assign 
addresses or registers to what's on the stack.

>>> Yah, but this only really works if you're not going to execute 
>>> arbitrary C code - which would be kind of a problem.
>>
>> Exactly. Why do you need to execute arbitrary C code, tho? Other than 
>> compatibility?
> 
> Oh, well, other than the "minor detail" of compatibility, there's no 
> problem at all! ;-)

Right. How much C is there that couldn't be ported with relative ease to 
C#?  Of course, if you want to maintain compatibility, you're not going 
to learn much in a research system. And if your company's reason for 
existence is to write software, maintaining compatibility with the other 
peoples' software isn't that big a deal.

Now, if something's written in C++, it might be harder to port, yes. And 
it would be hard to automate porting anything, of course. Or you could 
just make a C compiler that generates safe code - if your program works, 
it probably wouldn't be too difficult.

And if you're targetting a new platform, compatibility isn't too 
important. How much legacy code is there for in-dash car computers, or 
TiVo-like media systems?

> (You recall that "Linux" is actually a tiny bit of software which 
> inherited compatibility with Unix, thus earning an instant library of 
> userland tools, right?)

Sure. Most of which suck. ;-)  Just look at the file system layout you 
wound up with.

> (Actually, maybe PHP is a bad example. Perhaps you want to assign 
> different permissions to each PHP script? Rather than just to the PHP 
> interpretter?)

That too.  Or different permissions to "Fred logged in via SSH with a 
certificate" vs "Fred logged into the console with a password". So you 
can put more trust on certificates, or smart cards, or whatever, at the 
application level, but built into the ACL system. So you can look at the 
static ACLs in the system, and know you can't get to see your Quicken 
data unless you used the smart card to log in.

What's cool is, you can also ignore what program is claiming to be Fred, 
and just say "Anyone that claims to be Fred can see this, regardless of 
whether he logged in or not as Fred."  It's up to the individual 
programs as to which apps they trust to give good authentication.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Darren New wrote:
>> Uh... why? It's an experimental research prototype. It probably 
>> doesn't even run yet.

>> They were focused specifically on "how do we increase security?" I'm 
>> just thinking "what widely-used but suboptimal abstractions might we 
>> change?"

These two statements are what led me to believe you hadn't looked past 
the very surface of what they wrote. Both of these are contradicted by 
(for example) videos demoing the software and the very first design 
notes on the announcement pages.

Sorry if my misinterpretation led me to believe you didn't read up.

> Isn't this what RPM does? 

Specifically, the best the RPM does is to check that the package 
database says that the other packages you need are installed. It doesn't 
check that (for example) you've actually configured Apache to run 
correctly in order to support mod-php, it doesn't check that services 
this program depends on are turned on, it doesn't check that the 
contents of the files are correct, or that an installed device driver 
will be able to run and support the thing you're trying to add.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Invisible wrote:
> How about if, say, the program could somehow "tell" the OS what 
> arguments it actually accepts?

And if you want to see how Singularity does this, and what the interface 
to the disk subsystem looks like (for example), check out
singularity-6709\base\Applications\Benchmarks\diskrw\diskrw.sg
and look at the attributes on the config class.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

And if you want to see how the file system isn't "array of bytes" files, 
look at the different files in
singularity-6709\base\Contracts\Io.Contracts
which specify the types and state machines you can pass around. Note 
that (for example) the Video Device contract isn't anything like an 
array of bytes.  (Of course, the compiler stores the records in arrays 
of bytes in memory, but that's invisible to the programmers.)

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

>> And this makes a difference? It's not meant to be an end-user OS. It's 
>> meant for OS hackers to play with.
> 
> It's designed to support research into building another OS that uses the 
> same technologies.

OK, the documentation I read didn't say that anywhere.

>> Seriously, do you have *any idea* how many standards have been put 
>> forward for "store digital audio in a file"? ;-)
> 
> Sure. But they're all of the same type by the time you talk to the codec 
> to get the data out of them.

Except that (say) GIF supports animation and only 256 colours and 1-bit 
alpha, whereas PNG supports only single images, but with 24-bit colour 
and 8-bit alpha, and TIFF supports something else again...

>>> It's not unlike the Amiga OS in that respect, except safe and 
>>> strongly typed.
>>
>> Um... AmigaDOS files are streams of octets, just like every other OS.
> 
> No they're not. Amiga OS devices are things that listen for and respond 
> to typed messages.  Certainly the narrator isn't a "stream of octets", 
> nor is the clock, nor is the audio device.

The narrator accepts a stream of octets. It just interprets them as 
ASCII text and attempts to synthesize speach for them. But there's 
nothing stopping you from feeding it with arbitrary binary gibberish.

>> Oh, well, other than the "minor detail" of compatibility, there's no 
>> problem at all! ;-)
> 
> Right. How much C is there that couldn't be ported with relative ease to 
> C#?

Um... surely porting C code to *any* other language is intractably 
difficult?

> And if you're targetting a new platform, compatibility isn't too 
> important. How much legacy code is there for in-dash car computers, or 
> TiVo-like media systems?

My dad has a DVD player which appears to be running a modified version 
of Mencoder. (AFAIK, that's a C application.)

>> (You recall that "Linux" is actually a tiny bit of software which 
>> inherited compatibility with Unix, thus earning an instant library of 
>> userland tools, right?)
> 
> Sure. Most of which suck. ;-)  Just look at the file system layout you 
> wound up with.

What, you mean assigning permissions only to the person that owns the 
file is a bad idea? You do surprise me. ;-)

But my point is... it's much faster than writing an entire OS from 
scratch, all by yourself. And it instantly gives you a huge library of 
usable software. Otherwise I suspect Linux would still be nowhere...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Invisible wrote:
> Except that (say) GIF supports animation and only 256 colours and 1-bit 
> alpha, whereas PNG supports only single images, but with 24-bit colour 
> and 8-bit alpha, and TIFF supports something else again...

Um, yes? So? You don't think you can abstract that?  (BTW, GIF supports 
more than that, as you can change the color table after each row. FWIW.)

>> No they're not. Amiga OS devices are things that listen for and 
>> respond to typed messages.  Certainly the narrator isn't a "stream of 
>> octets", nor is the clock, nor is the audio device.
> 
> The narrator accepts a stream of octets. It just interprets them as 
> ASCII text and attempts to synthesize speach for them. But there's 
> nothing stopping you from feeding it with arbitrary binary gibberish.

And the narrator, when read, returns discreet messages with X/Y 
coordinates. Which isn't a stream of octets.

Of course, at some level of detail, everything is a stream of octets. 
It's how the octets are interpreted that's interesting. At some level of 
detail, everything's a voltage too - that's not an interesting level of 
detail for this discussion either. :-)

>>> Oh, well, other than the "minor detail" of compatibility, there's no 
>>> problem at all! ;-)
>>
>> Right. How much C is there that couldn't be ported with relative ease 
>> to C#?
> 
> Um... surely porting C code to *any* other language is intractably 
> difficult?

Uh, no?  Porting C code to C++ is actually fairly easy (almost trivial), 
for example. C# isn't all that different from C, conceptually speaking.

>>> (You recall that "Linux" is actually a tiny bit of software which 
>>> inherited compatibility with Unix, thus earning an instant library of 
>>> userland tools, right?)
>>
>> Sure. Most of which suck. ;-)  Just look at the file system layout you 
>> wound up with.
> 
> What, you mean assigning permissions only to the person that owns the 
> file is a bad idea? You do surprise me. ;-)

No, I mean putting everything that hasn't anything to do with users in a 
directory called "user" is probably a sign of legacy code. :-)

> But my point is... it's much faster than writing an entire OS from 
> scratch, all by yourself. And it instantly gives you a huge library of 
> usable software. Otherwise I suspect Linux would still be nowhere...

Oh, no doubt doing things the same old way makes it easier to port code. 
Not always trivial, mind, but certainly easier. Most operating systems 
that are still around are based on old cruft from 20 years ago for just 
that reason. And obviously Linux didn't do a *sufficiently* good job of 
it, or the amount of UNIX software that people actually want to use and 
is portable is not enough to give Linux significant market share. It's 
really, really difficult to make something that big *actually* 
compatible. (See, for example, all the strangeness in any autoconf.) 
It's probably almost as easy to port most non-GUI Linux utilities to 
Windows as it is to port them to (say) Solaris. The port may be rather 
poor (like, it might not interact with stuff the way you want it to 
under Windows, such as not being startable with "net start" or not 
recording how many unread emails you have for the login screen, say), 
but it'll probably run as well as on Linux without too much hassle.

I'd guess things are way easier to port from UNIX to Windows than from 
either to AmigaOS or Singularity, for example.

I think it's not so much that Linus T said "If I make it look like UNIX, 
I can use all the tools."  I think it was probably at least as much "If 
I make it look like UNIX, I won't have to figure out how an OS *should* 
work."  Hence, it starts out with all the brokenness of UNIX, then 
slowly piles on even more patches to try to make it useful, as long as 
you're not trying to maintain binary compatibility anyway.

-- 
Darren New / San Diego, CA, USA (PST)
  Ever notice how people in a zombie movie never already know how to
  kill zombies? Ask 100 random people in America how to kill someone
  who has reanimated from the dead in a secret viral weapons lab,
  and how many do you think already know you need a head-shot?

Post a reply to this message

Darren New wrote:

> I think it's not so much that Linus T said "If I make it look like UNIX, 
> I can use all the tools."  I think it was probably at least as much "If 
> I make it look like UNIX, I won't have to figure out how an OS *should* 
> work."  Hence, it starts out with all the brokenness of UNIX, then 
> slowly piles on even more patches to try to make it useful, as long as 
> you're not trying to maintain binary compatibility anyway.

The mental image of coding a broken system from scratch and then piling 
more complexity on top really amused me for some reason.

I guess because it's so true. ;-)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> The mental image of coding a broken system from scratch and then piling 
> more complexity on top really amused me for some reason.

Welcome to the wonderful world of backward compatibility! :-)

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

>> The mental image of coding a broken system from scratch and then 
>> piling more complexity on top really amused me for some reason.
> 
> Welcome to the wonderful world of backward compatibility! :-)

"Darren puts the 'backwards' in 'backwards compatibility'." ;-)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> "Darren puts the 'backwards' in 'backwards compatibility'." ;-)

Me? Heaven forbid. It's the bane of my career. :-)

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

Darren New wrote:
> And if you want to see how Singularity does this,

Actually, I kind of like their implementation of "compile time 
reflection", which is sort of like C++ templates. You can pre-compile 
them, ensure they'll work at compile time, and write templates in a 
language different from the one you're using them in. Pretty funky. 
Seems pretty straightforward, as well, and I'm guessing Turing complete 
also, given you have the full power of the language available to 
generate the code, without any weirdness necessary.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Wed, 13 Aug 2008 14:27:48 -0700, Darren New wrote:

>>>>> Or "I can't let you install telnet[1] until you have some sort of
>>>>> TCP/IP stack installed."
>>>>
>>>> Isn't this what RPM does?
>>>
>>> No.
>> 
>> Really? I thought that was the entire *point* of package managers.
> 
> To some extent. Package managers tell you which dynamic libraries are
> needed for which programs. They don't enforce anything, and you cannot
> (for example) look at an RPM without installing it and know if it'll
> work right once you're done installing it.

Well, RPMs aren't package *managers*, they're packages.  RPM is a package 
manager, and it does a reasonably good job of enforcing dependencies - 
you can override with --nodeps, but IME it does a good job for those who 
need them enforced and lets those who know better if a dependency is 
reasonable or not override.

Jim

Post a reply to this message

On Thu, 14 Aug 2008 09:13:53 -0700, Darren New wrote:

> I think it's not so much that Linus T said "If I make it look like UNIX,
> I can use all the tools."  I think it was probably at least as much "If
> I make it look like UNIX, I won't have to figure out how an OS *should*
> work."  Hence, it starts out with all the brokenness of UNIX, then
> slowly piles on even more patches to try to make it useful, as long as
> you're not trying to maintain binary compatibility anyway.

Um, I think you'll find that Linux is a derivative of Minix, not UNIX.  
At best it's Unix-like.

Jim

Post a reply to this message

Jim Henderson wrote:
> it does a reasonably good job of enforcing dependencies - 

There are relatively few kinds of dependencies it can enforce, and 
nothing enforces that the dependecy declarations are correct, for 
example. If the package doesn't allow it, the package manager isn't 
going to be able to do it.

You cannot, for example, look at the list of packages installed on the 
system and tell whether another package will install correctly - there 
may be unwritable files in the way that aren't tracked by the package 
manager, for example. The RPM may install files not listed in the 
manifest, and may not install every file listed in the manifest. If the 
package needs to add a user to the FTP server or something, there's 
nothing in the RPM that lets you look at it and tell automatically that 
adding that user will be necessary and needs to succeed before the 
package is installed. There is nothing in an RPM, as far as I know, that 
says which system services need to be enabled before you can start this 
one. (Sure, it's in the init.d script, but that's not in the package 
manifest, AFAIK.)

The Singularity package manager doesn't have this flaw, because the 
manifest controls what gets installed. There's no shell script in the 
package.

I'm not sure what you were trying to say with
> Well, RPMs aren't package *managers*, they're packages.  RPM is a package 
> manager,

I know that. That's what I was talking about. Nothing I said conflicts 
with this, as far as I can see.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Wed, 13 Aug 2008 14:57:25 +0100, Invisible wrote:

> Does anybody *else* find it ironic that Micro$oft - the corporation
> internationally renowned for its poor quality, buggy products - is
> interested in methods of producing high-quality software?

No.  Well, at least, I don't.

I'm not a fan of Microsoft, *however* it doesn't surprise me that they 
would look for ways to improve code quality without needing to invest 
massive amounts of energy, time, and money to do so.  Better production 
methods are one way of accomplishing this goal.

I've always said that Microsoft is outstanding at producing software 
that's "just good enough" - ie, it is buggy, but it's good *enough* that 
people aren't flocking away.

That doesn't mean that they wouldn't/couldn't/shouldn't go through a 
process of striving for continuous improvement in their development 
processes.  And clearly that's something they do (I've known people who 
have worked in MS Engineering, so this isn't conjecture on my part - it's 
based on conversations with former colleagues who worked at MS in that 
capacity).

Jim

Post a reply to this message

Jim Henderson wrote:
> Um, I think you'll find that Linux is a derivative of Minix, not UNIX.  
> At best it's Unix-like.

I don't know you'd call it a "derivative" of either, really. Clearly the 
whole thing is very UNIX-like, and since I'm only talking about the 
design of the OS (the UI, the API, the file system layout, etc), it 
doesn't really matter either way, since Minix and Unix both share the 
whole *ix bit.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

Warp wrote:
>   If you are doing a simple linear traversal, maybe, but if it's any more
> complicated than that...

Hmmm... Apparently there are mechanisms in place (which I don't really 
understand) that let you interactively prove to the compiler that some 
sequence of operations is safe, and then that gets recorded and the 
compiler can take advantage of it.  I.e., if you can prove that you 
never go out of bounds, even if the proof is "non-obvious" to the 
machine, then the compiler will omit the run-time checks.  Awesome.  :-)

Personally, I can't imagine how you go about doing such a thing, except 
maybe adding stuff to the code describing what/why you think it's true 
and running it thru the compiler again, which doesn't seem like 
"interactive" to me. But I'm not finding anything on line that isn't 
either "it's really cool" or "here's 40 pages of mathematics describing 
how it works."


Also, I imagine you could put in appropriate assertions, such that if 
you say (for example)

void flog(int[] myints, int startinx) {
   assert myints.length > 500;
   assert startinx > 100 && startinx < 400;
   for (int i = startinx - 50; i < startinx + 50; i++)
      myints[i] = myints[i+10];
}

then the compiler could track the possible ranges of values, and you'd 
get runtime checks at the entry to the function but not inside the loop, 
as an example.

But yeah, figuring out which next bit of object to bounce a ray off of 
is obviously going to take some run-time checks.

But honestly, I've never seen code where which element gets accessed 
next is obvious to a programmer but not to the compiler. I've never seen 
code where you could prove to a person's satisfaction that it was 
correctly accessing the array but couldn't prove it in a formal way 
given what's in the code itself, assuming you have all the code in front 
of you, of course.

Do you have any examples of that? I'm sure there must be some out there, 
but I don't do that sort of programming, I think. I think the closest 
I've gotten is knowing that the program that generated the file put 
things in it such that the program reading the file doesn't have to 
check. (E.g., the writer of the file never puts more than 80 chars per 
line, so the reader doesn't have to check, and that's because I wrote 
them both myself.)

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

Jim Henderson wrote:

> I've always said that Microsoft is outstanding at producing software 
> that's "just good enough" - ie, it is buggy, but it's good *enough* that 
> people aren't flocking away.

And I've always said that M$'s greatest achievement is in *redefining* 
what people will consider to be "good enough".

Not so many years ago, software that wasn't 100% crash-free was 
unacceptable. Today this is considered "normal". And it's all due to M$.

> That doesn't mean that they wouldn't/couldn't/shouldn't go through a 
> process of striving for continuous improvement in their development 
> processes.  And clearly that's something they do (I've known people who 
> have worked in MS Engineering, so this isn't conjecture on my part - it's 
> based on conversations with former colleagues who worked at MS in that 
> capacity).

Really? It's actually to their best advantage economically to make their 
software as inefficient as possible. (Although making it work 
*correctly* would be beneficial to them, making it work *efficiently* 
would cause them to lose money.)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

On Thu, 14 Aug 2008 17:25:13 -0700, Darren New wrote:

> Jim Henderson wrote:
>> Um, I think you'll find that Linux is a derivative of Minix, not UNIX.
>> At best it's Unix-like.
> 
> I don't know you'd call it a "derivative" of either, really. Clearly the
> whole thing is very UNIX-like, and since I'm only talking about the
> design of the OS (the UI, the API, the file system layout, etc), it
> doesn't really matter either way, since Minix and Unix both share the
> whole *ix bit.

That's more of a POSIX thing IIRC.  Tannenbaum would say that they're 
different as well, but Linux started as a free MINIX (since MINIX was 
distributed under a restricted license at the time).

Jim

Post a reply to this message

On Fri, 15 Aug 2008 18:46:00 +0100, Orchid XP v8 wrote:

> And I've always said that M$'s greatest achievement is in *redefining*
> what people will consider to be "good enough".
> 
> Not so many years ago, software that wasn't 100% crash-free was
> unacceptable. Today this is considered "normal". And it's all due to M$.

Well, again, fair play to Microsoft - computing has gotten a lot more 
complex over the last 20 years.

>> That doesn't mean that they wouldn't/couldn't/shouldn't go through a
>> process of striving for continuous improvement in their development
>> processes.  And clearly that's something they do (I've known people who
>> have worked in MS Engineering, so this isn't conjecture on my part -
>> it's based on conversations with former colleagues who worked at MS in
>> that capacity).
> 
> Really? It's actually to their best advantage economically to make their
> software as inefficient as possible. (Although making it work
> *correctly* would be beneficial to them, making it work *efficiently*
> would cause them to lose money.)

Are you old enough to be *that* cynical? ;-)

There is something to what you say, though; one of the factors that I've 
seen (and heard discussed) that caused the decline of NetWare was that it 
was *too* stable.  People installed the server and forgot about it.  Look 
at the rather well-known story about the school that actually closed in a 
NetWare 2.x server in a closet because they forgot about it.  Not an 
urban legend, this actually happened (University of North Carolina IIRC).

There were other factors as well that contributed to the decline of 
NetWare, including some really bad missteps on Novell's part, rebranding 
it to "IntraNetWare", which I consider one of the biggest blunders the 
company has made *and* not necessarily learned from as well as it should 
have been).  Having a bit of instability keeps the system in mind, and MS 
does an outstanding job of keeping people on the "upgrade treadmill".

Jim

Post a reply to this message

Orchid XP v8 wrote:
> Not so many years ago, software that wasn't 100% crash-free was 
> unacceptable. 

Nonsense.  I'm guessing it actually crashed at a higher rate, but 
nowadays you have orders of magnitude more people using software.

Or do you forget "sad mac" and "guru meditation" and "kernel panic". Of 
course all these things are common terms in the industry because they 
never, ever happened.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message