RenderMan & RISpec, prman 11.5 dying when using multiple processors

Hey everybody,

I'm generating fairly simple RIBs with my homebrew animation system,
and have been making pictures fairly happily for years with it. But,
when I try to render using multiple processors on our 2- and
4-processor boxes, sometimes the rendering dies just as the last few
scanlines are being rendered, and I'm left with an incomplete image.

I'm using multiple processors by invoking

prman -p:2

or

prman -p:4

I have included the rib at the end of this file, it's rediculously
simple.

My hypothesis is that some of the processes are finishing earlier than
others, and somehow are causing the program to crash at that point. No
core files are created, the program just seems to die relatively
happily (albiet without writing the image header/footer.) It occurs
fairly randomly, about 10% to 20% of the time -- rerendering the same
rib again typically works. The problem exists identically when writing
TIFF and SGI RGB files. The problem appears to manifest itself only
when rendering multiple frames -- either on multiple machines using our
RUSH queue software, or sequentially on my own workstation -- when
making single frames during testing it always seems to work (I cannot
reconcile this particular behavior with any reasonable hypothesis of
failure.)

The problem does not seem to manifest itself when rendering RIBs
generated by Maya using Alfred. This, again, seems terribly
mysterious.

I haven't tried (although I'm going to, now that I'm thinking about
it!) turning on the end-of-rendering statistics, to see if that helps
it not die.

I know I'm using an ancient version of RenderMan (version 11.5 is a
year-and-a-half old) but if this tickles anybodies memories I'd love to
hear about it.

Thad Beier
Hammerhead Productions

-------------------------------------------
This rib renders a semi-transparent textured polygon with a deep
shadow.
Typically I'm rendering a few dozen to a few hundred of these to make
clouds.
-------------------------------------------

##RenderMan RIB
version 3.03
Option "searchpath" "shader"
[".:/usr/hammerhead/lib/shaders:/usr/local/prman/lib/shaders"]
Option "searchpath" "texture"
[".:/usr/hammerhead/lib/textures:/usr/local/prman/lib/textures"]
Declare "shadowname" "uniform string"
Declare "volumeinterpretation" "uniform string"
FrameBegin 1
PixelFilter "box" 1 1
Display "/usr/tmp/shadow_1.zbuf" "zfile" "z"
Display "+/usr/tmp/shadow_1.dshad" "deepshad" "deepopacity"
"volumeinterpretation" ["continuous"]
PixelSamples 3 3
Declare "a" "uniform float"
Orientation "lh"
Format 512 512 1
Projection "orthographic"
ScreenWindow -125 125 -125 125
Rotate -35.2644 1 0 0
Rotate 45 0 1 0
Translate -500 -500 500
WorldBegin
Scale 1 1 -1
LightSource "distantlight" 1 "intensity" [0.1]"lightcolor" [0.6 0.8
1]"from" [0 0 0]"to" [0 -1 0]
Opacity [0.15 0.15 0.15]
Surface "flythru" "a" [0.025]
Polygon "P" [-100 0.25 100 100 0.25 100 100 0.25 -100 -100 0.25
-100]"N" [1 1 1 1 1 1 1 1 1 1 1 1]"st" [0 0 1 0 1 1 0 1]
WorldEnd
FrameEnd
PixelFilter "triangle" 2 2
PixelSamples 2 2
Hider "hidden" "jitter" [1]
FrameBegin 2
Orientation "lh"
Display "ri.rgb" "sgif" "rgba"
Format 512 512 1
Shutter 0 0.5
Projection "perspective" "fov" [30]
ScreenWindow -1 1 -1 1
Rotate -0 0 0 1
Rotate -71 1 0 0
Rotate 44 0 1 0
MotionBegin [0 0.5]
Translate -200 -800 200
Translate -199.449 -797.778 199.404
MotionEnd
WorldBegin
Scale 1 1 -1
TransformBegin
Translate 500 500 500
Rotate 45 0 1 0
Rotate -35.2644 1 0 0
LightSource "shadowdistant" 2 "shadowname"
["/usr/tmp/shadow_1.dshad"]"intensity" [1]"lightcolor" [1 1 1]"to" [0 0
-1]
TransformEnd
LightSource "distantlight" 3 "intensity" [0.1]"lightcolor" [0.6 0.8
1]"from" [0 0 0]"to" [0 -1 0]
Opacity [0.15 0.15 0.15]
Surface "flythru" "a" [0.025]
Polygon "P" [-100 0.25 100 100 0.25 100 100 0.25 -100 -100 0.25
-100]"N" [1 1 1 1 1 1 1 1 1 1 1 1]"st" [0 0 1 0 1 1 0 1]
WorldEnd
FrameEnd

Well, i don't use prman, but the one thing that occurs to me as what I
would do while trying to narrow down the problem is to use a standard
shader, and see if the problem still occurs. Likewise, use your shader
on a RIB that otherwise works. Then, do the same thing, but by taking
out the deep shadow. Do you do anything "strange" in the shader?

You may well have already done this. It does sound like just a plain
old bug in prman, so I wouldn't be at all surprised if somebody who is
more familiar with it says that upgrading is your best bet.

So, are you allowed to tell us where we should look for the clouds? ;)

Forkazoo,

Thanks for the comments. I'll do more investigation with respect to
the shader or the deep shadows soon -- but what appears to be true is
that by diverting the input and output of the
render to /dev/null, it appears to not have the problem any more. I'm
saying in my shell script
"prman -p:2 foo.rib < /dev/null > /dev/null 2> /dev/null"

This is not particularly surprising, really -- processes can be killed
if their pipe inputs and outputs are closed. Perhaps RenderMan is
trying to output to the terminal through some thread-safe standard IO
package, and when one of the child processes dies it gets fouled up.

More testing is in order, but today I've rendered 300 frames in a row
with no failures -- I would have expected 20 or 30 failures based on
previous experience.

The clouds will be in "Martian Child". Look for it this Oscar season
:)

Thad