Is there a way to call write() without blocking?

In more detail:

Say I'm taking a real-time stream (like live audio) in off a piece of 
hardware. I want to write it to the disk as it comes in. The hardware buffer 
is fairly small and needs to be emptied promptly into memory.

Obviously, I can set up a producer/consumer to write the buffer into memory 
from the device, then write() that out to disk. Assume I have plenty of RAM. 
But if the disk blocks for some reason (like, someone makes gigabytes of 
changes to the disk then calls sync), the write() might block. And I'm 
pretty sure select() or poll() isn't going to tell me this, right?

I'm thinking I allocate more memory in a circular kind of buffer, allocating 
memory if the disk slows, potentially freeing it if it speeds up, but I'd 
need to know when it's safe to reuse the buffer.

So I'm guessing the best way to do it is to have a thread for reading the 
hardware into memory, and a thread for calling write() when each buffer is 
ready, in your classic producer/consumer pattern.

But is there any other way to do it under Linux? Is there any call that (for 
example) will start writing, then give you a signal or some such when the 
buffer is available for refilling?

Now that I think on it, I guess a non-blocking write will write *some* bytes 
of the buffer, and select() will tell you what's writable, but the write 
itself will block while it copies bytes from your buffer into the kernel's 
memory and such, yes? But that shouldn't be long compared to waiting on disk 
revolutions or something, because it's all memory-to-memory, right? And does 
that work for disk-based files? Or does that only apply to sockets, pipes, etc?

I've just never had to deal with real-time processing under Linux as such 
before.  The other real-time systems I've done have had message-passing type 
kernels, so there wasn't blocking of buffers, but you had to be careful not 
to reuse buffers before they were done. Is there anything like message 
passing I/O in Linux these days, or is it just threads vs O_NONBLOCK?

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> Is there a way to call write() without blocking?

Search for fcntl() and O_NONBLOCK.

Post a reply to this message

clipka wrote:
> Darren New <dne### [at] sanrrcom> wrote:
>> Is there a way to call write() without blocking?
> Search for fcntl() and O_NONBLOCK.

I'm aware of that, but traditionally in UNIX that will still block on disk 
I/O. (Like, if the disk isn't spun up or something like that.) Does it do 
that in modern Linux kernels, or has that been changed?

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> > Search for fcntl() and O_NONBLOCK.
>
> I'm aware of that, but traditionally in UNIX that will still block on disk
> I/O. (Like, if the disk isn't spun up or something like that.) Does it do
> that in modern Linux kernels, or has that been changed?

Unfortunately that's all I know about it. Maybe you'll need to do some
multithreading.

Post a reply to this message

clipka wrote:
> Unfortunately that's all I know about it. Maybe you'll need to do some
> multithreading.

Very tl;dr of you. ;-)

It looks like the Linux *I* have at least never fails to block on disk I/O 
even with O_NONBLOCK.  Run time of several minutes, system time of a second, 
user time of even less, and write() never returned less that writing a full 
buffer, so I guess that means you get to block for disk and (grumble) learn 
how to do Linux threading in C. :-)  Damn.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Darren New wrote:
> clipka wrote:
>> Unfortunately that's all I know about it. Maybe you'll need to do some
>> multithreading.
> 
> Very tl;dr of you. ;-)
> 
> It looks like the Linux *I* have at least never fails to block on disk 
> I/O even with O_NONBLOCK.  Run time of several minutes, system time of a 
> second, user time of even less, and write() never returned less that 
> writing a full buffer, so I guess that means you get to block for disk 
> and (grumble) learn how to do Linux threading in C. :-)  Damn.
> 

 From what I recall, 'fork' in Linux is not all that bad to use.
Direct from wikipedia:

http://en.wikipedia.org/wiki/Fork_(operating_system)
http://en.wikipedia.org/wiki/Fork-exec

I've never had to implement fork, but from reading about it, it doesn't 
seem too bad.

While not exactly what you were looking for, I hope this helps some.




Tom

Post a reply to this message

Tom Austin wrote:
>  From what I recall, 'fork' in Linux is not all that bad to use.

I think processes and threads in Linux are both created with the same system 
call, just using different flags. Standard (v7-style) fork wouldn't work in 
this case because the processes wouldn't be sharing memory.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

On Mon, 06 Apr 2009 01:37:49 +0200, Darren New <dne### [at] sanrrcom> wrote:

> Is there a way to call write() without blocking?

http://lmgtfy.com/?q=asynchronous+IO+linux

Apparently there is a POSIX library for this stuff, with kernel support  
 from 2.6 onward. Perhaps that could be of use.



-- 
FE

Post a reply to this message

Fredrik Eriksson wrote:
> http://lmgtfy.com/?q=asynchronous+IO+linux

I take it you didn't actually read past the first line of the question? :-)

"A more subtle problem with non-blocking I/O is that it generally doesn't 
work with regular files."

Given there are about 8 ways to do asynch I/O, none of which seem to work 
with regular files, I was wondering if there was a 9th yet, or whether one 
just does blocking I/O in a thread, which scales poorly but is good enough 
for my application.

> Apparently there is a POSIX library for this stuff, with kernel support 
> from 2.6 onward. Perhaps that could be of use.

Yeah, and the second link on the page you gave says "It sucks and its 
broken." :-)

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message