How to Cache FLIP and Pyro Simulations Overnight in Houdini

Ever stared at the progress bar of a heavy FLIP or Pyro simulation in Houdini, wishing you could step away while it crunches frames?

Waiting hours for particles to bake can stall your entire schedule and leave you staring at a blank screen wondering if you missed a setting.

What if you could hit “cache” at quitting time and come back to fully processed data instead of rerunning overnight?

This guide will show you how to automate an overnight caching workflow for simulations in Houdini, so busy scenes finish while you’re away and you stay on track.

What prerequisites and project setup do I need before caching FLIP and Pyro overnight?

Before launching an overnight FLIP or Pyro sim, establish a consistent project structure. Create a root folder with subfolders for cache, hip files, renders, and logs. Reference these locations using Houdini’s environment variables ($JOB, $HIP, $HIPNAME). This ensures your File Cache SOP or ROP Output Driver writes to the correct path without manual redirects.

Verify your scene uses relative paths. In Edit > Aliases and Variables, set aliases for “cache” and “render” to point inside $JOB. Relative references prevent broken links when moving between machines or transferring to a farm. For flip simulations, prefix cache files with “flip_sim_” and for pyro, “pyro_sim_” plus frame tokens ($F4).

• Disk space: estimate 100–300 MB per frame (HD) for FLIP; adjust for field resolution on Pyro.
• Hardware: SSD or fast RAID for caches; separate OS and simulation drives.
• Permissions: ensure write access if using network storage or render farm shares.

Finally, set up your ROP network. Use a File Cache SOP for low-level control or ROP Geometry Output to leverage frame range batching. Define frame stepping and simulation ranges in the DOP Import. Lock down your sim parameters (particle separation, temperature fields) before caching—any change mid-render invalidates the cache and wastes overnight compute time.

How do I configure FLIP simulations to cache reliably overnight (nodes, memory and file settings)?

Running a FLIP simulation for hours demands that you offload particle data to disk and prevent memory spikes. Houdini’s built-in disk cache inside the Flip Solver node, combined with a dedicated File Cache SOP or Geometry ROP, gives you a robust overnight workflow. The key is to partition your data, compress files, and define clear output paths.

• Enable Disk Cache in Flip Solver
• Configure Partition Size to limit memory
• Use compressed .bgeo.sc file format
• Use File Cache SOP or Geometry ROP for file writes
• Set frame range and overwrite policies

Inside your DOP Network’s Flip Solver, open the Output tab, then under Disk Cache check “Save to Disk.” Point File Name to a dedicated cache folder (e.g. $HIP/cache/flip.$F4.bgeo.sc), and enable compression by appending .bgeo.sc. Adjust Partition Size to 150–300 MB: Houdini will offload each chunk once it reaches this threshold, keeping your RAM usage flat.

At SOP level, append a File Cache SOP to your DOP Import output. Set File Mode to “Write Files Sans Loading,” use the same naming convention, and enable “Load From Disk” to skip in-viewport reload. Alternatively, add a Geometry ROP in /out, target your final SOP, specify the same file pattern, select your frame range (e.g. 1–240), and choose “Overwrite” mode. Trigger this ROP in batch mode for unattended execution.

How do I configure Pyro simulations to cache reliably overnight (voxel resolution, openvdb and compression best practices)?

To cache a Pyro sim overnight without interruptions, begin by choosing a voxel size that balances detail and file size. In the Pyro Solver’s Shape tab, set the Division Size to 0.05–0.1 units for medium-scale explosions. Next, create a ROP Geometry node and point its Output Path to $HIP/cache/pyro/$HIPNAME.$F4.vdb. This ensures consistent frame indexing and easy resume.

• Enable OpenVDB export in the ROP: switch Output Driver Type to “Geometry” and format to “VDB”.
• Activate Asynchronous I/O to decouple disk writes from the sim thread.
• Use tile sizes of 32³ or 64³ to speed up data packing without bloating file headers.
• Arrange your file sequence in numbered subfolders (e.g., frame 0001–0010) for partial reloads.

Compression settings directly impact write performance and disk usage. LZ4 offers the fastest read/write, reducing CPU overhead when caching large volumes. If storage is critical, switch to ZLIB with a level of 2–4 to halve file sizes at a modest cost in speed. Finally, test a short segment overnight to confirm throughput and adjust settings before committing full-length sims.

How can I automate and schedule overnight caches using hython/hbatch and render managers (Deadline, Royal, etc.)?

Offloading simulation caching to non-production hours frees workstations and ensures consistent throughput. Houdini’s headless interpreters (hython and hbatch) can be scripted to load .hip files, trigger ROPs and write out caches without GUI overhead. Integrating these calls into a render manager ensures parallel jobs, retry logic and resource quotas.

Use hbatch for pure caching tasks, as it omits GUI modules and reduces memory footprint. Write a Python script that loads the scene via hou.hipFile.load(), locates your FLIP and Pyro ROP nodes under /out, and calls rop_node.render(). Keep the script modular so you can pass scene paths and ROP names as arguments.

In Deadline, choose the HoudiniBatch plugin to invoke hbatch directly. Configure the “HBatch Script” field to point at your Python file, and set plugin parameters for threads, memory and retries. For shell submissions, use the HoudiniShell plugin and embed your hython command line. Royal Render follows a similar pattern: assign an Houdini shell task, specify your hython call and capture stdout for logs.

When scheduling, leverage dependency chaining. For example, trigger the Pyro cache only after the FLIP job succeeds. Both Deadline and Royal support pre- and post-job scripts for cleanup or notification. Set job limits to avoid saturating overnight clusters, and tag tasks with priorities so caching runs at designated time slots.

Example hython/hbatch script and common command-line flags for overnight caching

A minimal hbatch script (cache_sim.py) to load caches and render:

import hou
hou.hipFile.load(“scenes/flip_pyro.hip”)
hou.node(“/out/flip_cache_rop”).render()
hou.node(“/out/pyro_cache_rop”).render()

Launch via CLI or render manager:

hbatch -n 8 -M 12000 -d cache_sim.py

How do I monitor, validate and recover failed caches, and quantify time/storage savings to roll this into a studio pipeline?

Effective monitoring starts by running your FLIP or Pyro cache through a TOP Network with PDG Monitor. Watch task states—waiting, cooking, complete or failed—and inspect detailed error logs on failure. To validate, compare frame counts, bounding‐box extents and particle volumes in the Geometry Spreadsheet or using SOP Query nodes against baseline values. Automated checks ensure you catch missing frames or unexpected data spikes before downstream artists consume the cache.

For recovery, leverage PDG’s built-in retry logic or manually resubmit only failed tasks. In your TOP Network, set a “Max Retries” parameter on the ROP Fetch node or expose a “Resubmit Failed Tasks” button on the Digital Asset’s interface. A small Python snippet can scan PDG task states, extract the failed frame range, and trigger a re-cook of that segment. This selective re-cooking saves compute time compared to rerunning the full sim.

• Enable “Retry on Failure” and configure backoff intervals in your TOP node.
• Use ROP Fetch to isolate and recook single frames via a frame list expression.
• Automate failure detection with a Python Shell TOP for custom notifications.
• Archive logs and failed-task metadata alongside your .sim or .bgeo files.

To quantify time savings and storage savings, parse PDG’s performance report or export task timings as JSON. A Python script can traverse your output folder, sum file sizes by pattern (e.g., *.bgeo.sc) and compare against an unchached workflow. Generate a CSV or HTML report summarizing per-task cook time and disk footprint. Finally, wrap both caching logic and metrics generation into an HDA scheduled via HQueue or your in-house scheduler. Store results in a central database for ongoing reporting, making full integration into your studio pipeline seamless and transparent.