I'm looking for some advise from anyone who has experience rendering long
animations to virtual cloud servers. Any feedback on what services have given
the best price/performance with POV-Ray (AWS, Azure, Google Cloud, etc). Are
there any scripts available to automatically assign portions of the task to
cloud servers or have you found it simpler to just split the task between
servers manually.

Post a reply to this message

On 8/2/19 2:19 PM, Steve Anger wrote:
> I'm looking for some advise from anyone who has experience rendering long
> animations to virtual cloud servers. Any feedback on what services have given
> the best price/performance with POV-Ray (AWS, Azure, Google Cloud, etc). Are
> there any scripts available to automatically assign portions of the task to
> cloud servers or have you found it simpler to just split the task between
> servers manually.
> 

Jakob Flierl (Koppi on newsgroups) posted a pointer to:

https://www.youtube.com/watch?v=PyRjMQ0-Xik

in the thread:

http://news.povray.org/povray.general/thread/%3Cweb.57190929c0fb63c55802fb0%40news.povray.org%3E/?mtop=407886

where he posted additional information such as a link to some set up 
scripts at :

https://github.com/bullet-physics-playground/bpp/issues/8


There have been other posts about cloud rendering - including an example 
of an implementation by AWS folks for some feature using POV-Ray as the 
vehicle, but I didn't quickly find those links in my notes. A POV-Ray 
site search on cloud would likely would turn most of those up.

If you make progress, I'm interested in what you find. Such work is 
still on my, try-it-someday list. My current interest is mostly with 
running the thousands of test cases I've accumulated over the past few 
years.

Years back, cost wise, I looked at AWS and thought their bid for free 
time mode would be cheaper than having a second local machine or two 
onto which to offload work given bid history at the time. I have though, 
complications in sometimes wanting machines not running other things for 
performance comparison work(1). Dedicated machines looked more expensive.

I have, off and on, been working on creating virtual machines local to 
my machine to learn. It looks like the kvm, qemu, libvirt stuff can 
emulate to the cpu type level in addition to OS images. Are there 
virtual hardware counters and such I could use to measure performance in 
a repeatable, dedicated machine way. This would make a cloud solution 
more attractive to me. Are such virtual CPUs affected by other VMs 
running on the same real machine - I don't know. My VM play is still 
very basic.

Bill P.

(1) My image comparison test cases capture and compare time for gross 
ideas of performance change. These times are not reliable unless the 
machine is doing nothing but a chain of single, one threaded render 
tests, at a time.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> I have, off and on, been working on creating virtual machines local to
> my machine to learn. It looks like the kvm, qemu, libvirt stuff can
> emulate to the cpu type level in addition to OS images. Are there
> virtual hardware counters and such I could use to measure performance in
> a repeatable, dedicated machine way. ...

I wonder whether the 'cpuset' pseudo file system stuff (man section 7) could be
useful to you.


regards, jr.

Post a reply to this message

On 8/2/2019 2:19 PM, Steve Anger wrote:
> I'm looking for some advise from anyone who has experience rendering long
> animations to virtual cloud servers. Any feedback on what services have given
> the best price/performance with POV-Ray (AWS, Azure, Google Cloud, etc). Are
> there any scripts available to automatically assign portions of the task to
> cloud servers or have you found it simpler to just split the task between
> servers manually.
> 
> 
> 

I used the $200 trial period for Azure a few years ago. I didn't network 
more than one VM, though.


Michael

Post a reply to this message

On 8/3/19 4:42 PM, jr wrote:
> hi,
> 
> I wonder whether the 'cpuset' pseudo file system stuff (man section 7) could be
> useful to you.
> 

Maybe. Don't know. Unsure to which tool's man page you are referring 
(virsh?). I have been reading about the qemu/libvirt named and host 
model capabilities - which look to be tangled.

I'm not actually working at this complexity at the moment. What I'm 
trying with VMs currently is to create - again, but statically linked 
this time - POV-Ray compiles at various commits in the chain of commits 
back to 3.7-stable. I'm still working through compile issues at various 
positions in the commit chain within the VMs.

I had a large set of dynamically linked 'commit' modules while on Ubuntu 
16.04 - yep, hindsight... I want to have similar capability in 18.04 and 
future environments. Being able to do a binary search of the commit 
chain makes isolating issues introduced over time much easier.

Bill P.

Post a reply to this message

hi,

William F Pokorny <ano### [at] anonymousorg> wrote:
> On 8/3/19 4:42 PM, jr wrote:
> > I wonder whether the 'cpuset' pseudo file system stuff (man section 7) could be
> > useful to you.
>
> Maybe. Don't know.

unsurprising.  I snipped the wrong part(s) of your post, totally misleading.
(sorry)

> Unsure to which tool's man page you are referring (virsh?).

man [7] cpuset


> I have been reading about the qemu/libvirt named and host
> model capabilities - which look to be tangled.

I did install qemu last year but have, as yet, not found time/inclination to
work with it; currently not running any virtualisation s/ware.

> I'm not actually working at this complexity at the moment.

as I said, I inadvertently mislead, no idea what was going on in my head.  :-)
I responded to your writing:

"I have though, complications in sometimes wanting machines not running other
things for performance comparison work. ... My image comparison test cases
capture and compare time for gross ideas of performance change. These times are
not reliable unless the machine is doing nothing but a chain of single, one
threaded render tests, at a time."

as I understand it[*] the cpuset mechanism is designed to facilitate resource
isolation, by process(es).  it occurred to me that it should be possible to
split cpu + memory resources, say 75%:25%, and run the system in the 75% set
while doing your measured runs in the 25% set.


regards, jr.

[*] disclaimer, I am trying to make headway with a s/ware project of mine and
haven't, currently, time or opportunity to play with configuration(s).  so the
above is based on my reading of the documentation.

Post a reply to this message

"Steve Anger" <nomail@nomail> wrote:
> I'm looking for some advise from anyone who has experience rendering long
> animations to virtual cloud servers. Any feedback on what services have given
> the best price/performance with POV-Ray (AWS, Azure, Google Cloud, etc). Are
> there any scripts available to automatically assign portions of the task to
> cloud servers or have you found it simpler to just split the task between
> servers manually.

I was looking at the Azure VM options and they have performance VM's with up to
44 cores. Is there a limit to the number of cores that POV-Ray can efficiently
use? Wondering if I can get the same performance out of one 44 core VM that I
can get out of ten 4 core VMs.

Post a reply to this message

hi,

"Steve Anger" <nomail@nomail> wrote:

the most recent discussion on use of multiple cores:

http://news.povray.org/povray.general/thread/%3Cweb.5c0ac10623560652ee75e2ca0%40news.povray.org%3E/


regards, jr.

Post a reply to this message

On 8/9/2019 10:10 AM, Steve Anger wrote:
> I was looking at the Azure VM options and they have performance VM's with up to
> 44 cores. Is there a limit to the number of cores that POV-Ray can efficiently
> use? Wondering if I can get the same performance out of one 44 core VM that I
> can get out of ten 4 core VMs.
> 
> 
> 

IIRC, the number of cores on Azure go up to 64.


Michael

Post a reply to this message