POV-Ray : Newsgroups : povray.off-topic : Re: Tales from a Renderfarm Server Time
24 Apr 2024 18:10:53 EDT (-0400)
  Re: Tales from a Renderfarm (Message 1 to 8 of 8)  
From: dick balaska
Subject: Re: Tales from a Renderfarm
Date: 20 Sep 2018 23:35:33
Message: <5ba46705$1@news.povray.org>
On 09/08/2018 09:07 AM, dick balaska wrote:

> 
> Now if the vpn dies, the queued frames keep chugging, and if a machine
> reboots he just restarts and goes. yay.

Ah, computers...

The bsacs service (renderfarm server) fails to launch povrayws on reboot.
If I stop/start it by hand, it works.
If I upgrade, the system stops, upgrades, and starts the service correctly.
But on reboot, no.  wtf.
-- 
dik
Rendered 1024 of 921600 pixels (0%)


Post a reply to this message

From: Le Forgeron
Subject: Re: Tales from a Renderfarm
Date: 21 Sep 2018 02:08:53
Message: <5ba48af5$1@news.povray.org>
Le 21/09/2018 à 05:35, dick balaska a écrit :
> On 09/08/2018 09:07 AM, dick balaska wrote:
> 
>>
>> Now if the vpn dies, the queued frames keep chugging, and if a machine
>> reboots he just restarts and goes. yay.
> 
> Ah, computers...
> 
> The bsacs service (renderfarm server) fails to launch povrayws on reboot.
> If I stop/start it by hand, it works.
> If I upgrade, the system stops, upgrades, and starts the service correctly.
> But on reboot, no.  wtf.
> 
violent shutdown for reboot have a trend to leave some files behind them 
which make the system believe the process is already running.

Typical wtf:
1. the start process write the psid in a file
2. on clean shutdown, file with psid is deleted
3. system upgrades cleanly stop all processes before restart (see 2)
4. on power-reboot, file with psid exist, so the system is happy and do 
nothing.

The basic solution is that the psid in the file should also be checked 
to see:
1. Is such psid currently in use (does the process exist)
2. Is such process matching the command that is associated with the file 
(collisions on psid can occurs, there is only 65535 id, and they get reused)

Especially after a power-reboot, in which more processes are forked at 
start to check the disks, so another service can now use the psid logged 
in the file in the previous session.

When you forget point 2, you get problem, randomly.


Post a reply to this message

From: Mike Horvath
Subject: Re: Tales from a Renderfarm
Date: 21 Sep 2018 03:32:06
Message: <5ba49e76$1@news.povray.org>
On 9/20/2018 11:35 PM, dick balaska wrote:
> On 09/08/2018 09:07 AM, dick balaska wrote:
> 
>>
>> Now if the vpn dies, the queued frames keep chugging, and if a machine
>> reboots he just restarts and goes. yay.
> 
> Ah, computers...
> 
> The bsacs service (renderfarm server) fails to launch povrayws on reboot.
> If I stop/start it by hand, it works.
> If I upgrade, the system stops, upgrades, and starts the service correctly.
> But on reboot, no.  wtf.
> 


How big is the farm?


Mike


Post a reply to this message

From: dick balaska
Subject: Re: Tales from a Renderfarm
Date: 21 Sep 2018 11:43:25
Message: <5ba5119d$1@news.povray.org>
On 09/21/2018 03:32 AM, Mike Horvath wrote:

> 
> How big is the farm?

5 nodes. http://www.buckosoft.com/bsac/meta/ click the "Servers" tab.
> 
> 
> Mike


-- 
dik
Rendered 1024 of 921600 pixels (0%)


Post a reply to this message

From: Mike Horvath
Subject: Re: Tales from a Renderfarm
Date: 21 Sep 2018 21:48:22
Message: <5ba59f66$1@news.povray.org>
On 9/21/2018 11:43 AM, dick balaska wrote:
> On 09/21/2018 03:32 AM, Mike Horvath wrote:
> 
>>
>> How big is the farm?
> 
> 5 nodes. http://www.buckosoft.com/bsac/meta/ click the "Servers" tab.
>>
>>
>> Mike
> 
> 

OMG what happened to Elvis and Emma?! Are they okay?


Mike


Post a reply to this message

From: dick balaska
Subject: Re: Tales from a Renderfarm
Date: 22 Sep 2018 02:16:03
Message: <5ba5de23$1@news.povray.org>
On 09/21/2018 09:48 PM, Mike Horvath wrote:
>>
> 
> OMG what happened to Elvis and Emma?! Are they okay?

Elvis died of heat exhaustion. joe is his replacement (my buddy's linux
server).  Emma has been retired.  She was my firewall/tunnel/dns/dhcp
server and has been replaced with a Raspberry Pi. (5 watts vs. 250 watts).


-- 
dik
Rendered 1024 of 921600 pixels (0%)


Post a reply to this message

From: Stephen
Subject: Re: Tales from a Renderfarm
Date: 22 Sep 2018 04:11:09
Message: <5ba5f91d$1@news.povray.org>
On 22/09/2018 07:16, dick balaska wrote:
> On 09/21/2018 09:48 PM, Mike Horvath wrote:
>>>
>>
>> OMG what happened to Elvis and Emma?! Are they okay?
> 
> Elvis died of heat exhaustion. 


On the toilet?  ;)



-- 

Regards
     Stephen


Post a reply to this message

From: dick balaska
Subject: Re: Tales from a Renderfarm
Date: 30 Sep 2018 06:33:24
Message: <5bb0a674$1@news.povray.org>
On 09/21/2018 02:08 AM, Le Forgeron wrote:

>>
> violent shutdown for reboot have a trend to leave some files behind them
> which make the system believe the process is already running.
> 
> Typical wtf:
> 1. the start process write the psid in a file
> 2. on clean shutdown, file with psid is deleted
> 3. system upgrades cleanly stop all processes before restart (see 2)
> 4. on power-reboot, file with psid exist, so the system is happy and do
> nothing.
> 
> The basic solution is that the psid in the file should also be checked
> to see:
> 1. Is such psid currently in use (does the process exist)
> 2. Is such process matching the command that is associated with the file
> (collisions on psid can occurs, there is only 65535 id, and they get
> reused)
> 
> Especially after a power-reboot, in which more processes are forked at
> start to check the disks, so another service can now use the psid logged
> in the file in the previous session.
> 
> When you forget point 2, you get problem, randomly.

This was the problem, but sideways.
For about 6 years, Debian mounts /var/run as a tmpfs so it is always
empty on boot.  This was done specifically to solve the problem you mention.
On install, I created /var/run/bsac (chown bsac:bsac) because I can't
write to root owned /var/run. The solution was to check for
/var/run/bsac at service startup and create as needed.


-- 
dik
Rendered 1024 of 921600 pixels (0%)


Post a reply to this message

Copyright 2003-2023 Persistence of Vision Raytracer Pty. Ltd.