dtr

dtr

Distributed single frame render for Blender Cycles (macOS, Linux, Cygwin/Windows)

d1440247 wording tidy · by Alan Taylor
Name Last Update
man Loading commit data...
bench.blend Loading commit data...
dtr.py Loading commit data...
dtr_analysis.py Loading commit data...
dtr_benchmark.py Loading commit data...
dtr_data_struct.py Loading commit data...
dtr_file_io.py Loading commit data...
dtr_init.py Loading commit data...
dtr_utils.py Loading commit data...
gpl.txt Loading commit data...
readme.md Loading commit data...
render.blend Loading commit data...
user_settings.conf Loading commit data...

Distributed single frame render for Blender Cycles (macOS, Linux, Cygwin/Windows)

This Python script spreads the work of rendering a single frame over multiple computers.

It is free and Open Source software, and is made available under the The GNU General Public License.


It will:

  • distribute your Blender file, textures and library Blender files to all the render nodes you specify
  • calculate the best number of blocks to split your image into
  • manage rendering the blocks on the remote nodes, and collect them as they complete
  • stitch all the blocks together to form the final image
  • restart cleanly after being interrupted, without losing any completed blocks
  • generate a heatmap for your rendered image showing which regions render slowly/quickly

Please note:

  • renders to PNG
  • post processing (compositing and sequencer) will be disabled by the script

A video showing a test run of the script is available here:

There's more information on the wiki: https://gitlab.com/skororu/dtr/wikis/home


What it will run on

It should in principle work on any modern Unix-like environment, and has been tested with:

You can use render nodes running any operating system a Blender version exists for.


Not sure which branch to choose?

Check the wiki for more details.


How to get it working

required for the script required for the render nodes required for the optional backup computer
Blender no yes no
Python yes no no
Pillow yes no no
Rsync yes yes yes

(1) Python requirements

On the machine the script is run on, you'll need Python 3.4 (or newer) and Pillow.

  • Pillow is used to combine rendered blocks/chunks to form the final image

(2) User accounts

You will need to set up a user account for the script to log in to on each machine you wish to use as a render node. Unless you specify otherwise in the configuration file user_settings.conf, the default user account the script will use for all machines, is render.

It is worth noting that if you are creating new users for remote rendering on macOS/OS X, REMEMBER TO LOGIN TO THE ACCOUNT MANUALLY IN THE GUI before using this script, otherwise the accounts are not fully set up by the OS, and Blender instances run over SSH will crash as they start to render (even though remote logins over SSH appear to work just fine).

(3) Configure SSH

You will need to ensure that you can remotely run commands on your chosen remote render nodes using SSH without entering passwords.

(3.1) macOS / OS X

On the local machine (192.168.0.129 is the remote machine in this example):

ssh-keygen -t rsa
cd .ssh
ssh-add id_rsa
cat ~/.ssh/id_rsa.pub | ssh render@192.168.0.129 'cat >> .ssh/authorized_keys'

Then still on the local machine, check with:

ssh render@192.168.0.129 uname

If all has worked well, the check command should run without requesting a password.

(3.1.1) Potential Issues

If ssh-add id_rsa generates the error "Could not open a connection to your authentication agent", then you may need to run exec ssh-agent bash first.

If there are issues running the final command, on the remote machine make sure the permissions are set correctly on authorized_keys using chmod 600 .ssh/authorized_keys.

(3.2) Linux

The process seems a little simpler on Linux, whatever operating systems the render nodes are running.

On the local machine (192.168.0.129 is the remote machine in this example):

ssh-keygen -t rsa
ssh-copy-id render@192.168.0.129

Then still on the local machine, check with:

ssh render@192.168.0.129 uname

If all has worked well, the check command should run without requesting a password.

(4) Install Blender

The script will attempt to find the Blender binary in some common places, which works well for render nodes running Linux distributions where Blender is installed with a package manager, and Blender can be found using which.

If all else fails, the script will try to find the binary in ~/Blender/:

macOS/OS X    ~/Blender/blender.app/Contents/MacOS/blender
Linux         ~/Blender/blender
Cygwin        ~/Blender/blender.exe

If you're using Cygwin (that doesn't offer a packaged Blender installation) or macOS (which has made it increasingly difficult in recent versions to set paths for use over ssh in a consistent manner), it's probably best to install Blender in the appropriate location for your OS as given above. You can either install Blender in the given location, or insert a symbolic link in ~/Blender/ to point to it.

The oldest Blender version that can be used with the script is 2.65, as this was the first version to support manually setting tile sizes.

(5) Edit the file user_settings.conf:

  • specify all the render nodes you wish to use by IP address
  • specify the resolution of the image you want rendered
  • specify the seed value
  • specify the frame to render

(6) Set the file you want to render to be render.blend:

  • it should be placed in the same directory as this script
  • if your file depends on any textures or supporting Blender library files, place them in directories in the same directory as render.blend. Let the script know what the directories are called by setting textures_directory and library_directory in file user_settings.conf, and they will be copied over to the render nodes along with render.blend.
~/gitlab_dtr
|-- bench.blend
|-- dtr.py
|-- dtr_analysis.py
|-- dtr_benchmark.py
|-- dtr_data_struct.py
|-- dtr_file_io.py
|-- dtr_init.py
|-- dtr_utils.py
|-- gpl.txt
|-- library
|   |-- furniture.blend
|   |-- ornaments.blend
|-- man
|   |-- dtr.1
|   |-- user_settings.conf.1
|-- readme.md
|-- render.blend
|-- textures
|   |-- plaster.png
|   |-- painted_timber.png
|-- user_settings.conf

(7) Run the script

A successful run of the script should look something like this:

render@rpi:~/dtr-master-ee05bdf030cfb77dbdabb2fe560ca36591e2040f $ ./dtr.py
>> normal start
>> checking user settings file
>> checking if render nodes are responsive
>> benchmarking
using cached benchmarks for all nodes
cached: node   127.0.0.1 benchmark time 0:01:30 active
cached: node 192.168.0.3 benchmark time 0:03:27 active
cached: node 192.168.0.2 benchmark time 0:04:40
2048 * 2048 image, frame 1 with seed 0
the frame is split into 256 blocks (arranged as 16 * 16) distributed over 2 nodes
giving a block size of 128 * 128 pixels
>> rendering
block 121 issued to node 192.168.0.3
block 120 issued to node 127.0.0.1
block 136 issued to node 127.0.0.1
block 120 retrieved from 127.0.0.1
block 137 issued to node 127.0.0.1
block 136 retrieved from 127.0.0.1
block 104 issued to node 192.168.0.3
block 121 retrieved from 192.168.0.3
block 105 issued to node 127.0.0.1
block 137 retrieved from 127.0.0.1
block 119 issued to node 127.0.0.1
block 105 retrieved from 127.0.0.1
.
.
.
block 242 issued to node 192.168.0.3
block  17 retrieved from 192.168.0.3
block 255 issued to node 127.0.0.1
block 240 retrieved from 127.0.0.1
block   1 issued to node 127.0.0.1
block 255 retrieved from 127.0.0.1
block  16 issued to node 127.0.0.1
block   1 retrieved from 127.0.0.1
block 241 issued to node 192.168.0.3
block 242 retrieved from 192.168.0.3
block 256 issued to node 127.0.0.1
block  16 retrieved from 127.0.0.1
block 241 retrieved from 192.168.0.3
block 256 retrieved from 127.0.0.1
>> compositing
>> result written to: composite_seed_0.png
>> tidying up
>> finished

            node  blocks       mean duration
       127.0.0.1     183             0:04:49
     192.168.0.3      73             0:11:19

                                   real time
                                    14:51:24

                            performance gain
                                       27.7%
render@rpi:~/dtr-master-ee05bdf030cfb77dbdabb2fe560ca36591e2040f $ 

Notes for specific systems

macOS / OS X

If your render nodes have their firewalls enabled, you will need to set Remote Login (SSH) and sshd-keygen-wrapper to allow incoming connections. You are free to leave Stealth Mode enabled.

Raspbian

For this example let's assume that we have the full version of Raspbian (not the -lite version). We want to run the script on a Raspberry Pi which has the IP address 192.168.0.130, and we want to be able to use two other machines for rendering, 192.168.0.128 and 192.168.0.129. We'll also use the Raspberry Pi as a render node.

From a spare computer, log in as default user pi with ssh pi@192.168.0.130 and install Blender.

When I last checked in late 2016, the Blender version available from the package manager was very old: version 2.63a was released in 2012. If you want to work with a current version of Blender, you'll either need to compile it yourself (non-trivial) or use a different Linux distribution.

sudo apt-get install blender

Generate the ssh key

ssh-keygen -t rsa

Then copy the ssh key to all the render nodes you wish to use (this will allow the script to run without constantly asking for passwords):

ssh-copy-id render@192.168.0.128
ssh-copy-id render@192.168.0.129
ssh-copy-id render@192.168.0.130

Try this quick test to make sure you can run a simple command on each render node without a password being requested (the last two query the Raspberry Pi):

ssh render@192.168.0.128 uname
ssh render@192.168.0.129 uname
ssh render@192.168.0.130 uname
ssh render@127.0.0.1 uname

The first time you connect you may see a message similar to the one shown below, which is normal.

pi@rpi:~ $ ssh-copy-id user@192.168.0.128
The authenticity of host '192.168.0.128 (192.168.0.128)' can't be established.
RSA key fingerprint is a7:01:ff:a3:5f:12:aa:4f:9a:22:4a:12:6e:1a:d3:f0.
Are you sure you want to continue connecting (yes/no)? yes

Add user render then add it to group pi

sudo adduser render
sudo adduser render pi

Log out user pi.

Log in as user render with ssh render@192.168.0.130

Create a link to Blender:

render@rpi:~ $ which blender
/usr/bin/blender
render@rpi:~ $ mkdir Blender
render@rpi:~ $ ln -s /usr/bin/blender ./Blender/blender

Log out user render.

Log in as user pi with ssh pi@192.168.0.130

Get the script:

render@rpi:~ $ wget https://gitlab.com/skororu/dtr/repository/archive.zip
render@rpi:~ $ unzip archive.zip 
Archive:  archive.zip
f6cf9b4342275981be0a655126aa6111cce2a266
   creating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/bench.blend
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/dtr.py
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/dtr_benchmark.py
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/dtr_data_struct.py
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/dtr_file_io.py
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/dtr_init.py
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/dtr_utils.py
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/gpl.txt
   creating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/man/
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/man/dtr.1
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/man/user_settings.conf.1
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/readme.md
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/render.blend
  inflating: dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/user_settings.conf
render@rpi:~ $ ls
archive.zip  Blender  dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266
render@rpi:~ $ cd dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266/

The default settings contained in user_settings.conf will use the computer the script is running on (in this case the Raspberry Pi) as a render node (it's listed as 127.0.0.1), so there's no need to edit anything right now, and you can perform a test run of the script:

pi@rpi:~/dtr-master-f6cf9b4342275981be0a655126aa6111cce2a266 $ ./dtr.py 
>> normal start
>> checking if render nodes are responsive
>> benchmarking
running benchmark on 1 node
benchmark cache updated
800 * 450 image, frame 1 with seed 0 will be sent whole to the single available node
>> rendering
tile 0 issued to node 127.0.0.1
tile 0 retrieved from 127.0.0.1
>> result written to: composite_seed_0.png
>> tidying up
>> finished

            node  blocks       mean duration
       127.0.0.1       1             2:03:41

                                  total time
                                     2:03:44

Windows

  • Install Cygwin with the default installation items plus:
    • python3
    • python3-imaging
    • openssh
    • rsync
    • wget (optional, but handy for collecting the latest version of this script)
  • Configure Cygwin SSHD
  • Create user render in Windows
  • Configure ssh so manual password entry for user render is not necessary.
    • Follow the instructions given for Linux, as ssh-copy-id is supported by Cygwin.
  • Install Blender for user render.
    • Unpack the Blender archive with Windows rather than Cygwin, otherwise execute permissions seem to get lost.
  • Get the latest version of this script with wget https://gitlab.com/skororu/dtr/repository/archive.zip?ref=master

Files the script uses

Files:
    benchmark_cache.p    : holds cached benchmark results (so the script doesn't have to benchmark nodes on every run)
    bench.blend          : file rendered when benchmarking remote nodes
    composite_seed_*.png : final rendered image
    dtr.py               : run this file to perform the render
    dtr_benchmark.py     : functions that support benchmarking the remote nodes
    dtr_data_struct.py   : data structures used by the script
    dtr_file_io.py       : functions supporting file based operations
    dtr_init.py          : functions involved in initialising the distributed render
    dtr_utils.py         : general utility functions
    render.blend         : the file you want rendered
    textures             : directory containing any unpacked textures that render.blend requires (optional - user defined)
    library              : directory containing any supporting library blender files that render.blend requires (optional - user defined)
    user_settings.conf   : contains all user configuration options (this is the only user editable file - do not edit any other files)

Temporary files created on the local machine:
    block_*_seed_*.png   : individual block renders
    render_block.py      : file used to configure Blender to render a specific block
    restart_config.p     : holds the key render configuration data, in case we need to interrupt and restart the script
    restart_progress.p   : holds details about the work completed so far, in case we need to interrupt and restart the script

Temporary files created on remote machines:
    0001.png             : output from benchmark render (not used for anything)
    block_*_seed_*.png   : individual block renders
    render_block_*.py    : file used to configure Blender to render a specific block

Notes on usage

(1) Performance

A distributed render works best when the file to be rendered takes a long time to render, and the render nodes are fairly equal in performance. So use the script on complex, slow to render Blender files to get the benefits.

(1.1) Tile size

Currently (Q2 2017), the script will set Blender's internal tile size to a value that's optimal for CPU rendering.

A solution that works more efficiently for GPU rendering is in progress, noting that different GPUs prefer different tile sizes, and that performance can be substantially affected by using non-preferred tile sizes. Also see section 5.2 below.

(2) Benchmarking

bench.blend should be relatively quick to render, but reflective of the general complexity of typical final renders. It is used to get a rough idea of the relative performance of the remote nodes, so we can estimate the number of blocks to split the image into. This allows us to efficiently render the image without nodes sitting around idle at the end for too long. Remember to set the number of threads to automatic.

If in doubt, just use the provided bench.blend.

(3) Avoiding a restarted render

If you don't want to restart after an interrupted render, start up the script with the --clean option. This will delete any temporary files associated with a previously interrupted render. Type: dtr.py --clean

(4) Forcing the script to benchmark nodes again

If you want to force the script to run the benchmarks again, start up the script with the --flush option, this will delete the benchmark cache. Type: dtr.py --flush.

Note that the use of --flush implies --clean; if a decision has been made to flush the cache, it makes sense to discard any previously rendered blocks.

(5) Noise

(5.1) Keep Blender versions the same on render nodes

Try to keep the Blender version on each node the same. This helps to avoid subtle differences in image noise that may occur with different versions.

(5.2) Avoid mixed GPU/CPU clusters

Because of the differences in supported feature sets and noise characteristics between GPU and CPU rendering, it's not currently practical to distribute single frame renders over a cluster of workstations where some use CPU rendering and others use GPU rendering. Keep to all CPU or all GPU clusters.