A Block (Square) engineer shares how they safely upgrade long-running payment processing workflows built on Temporal at scale (15M+ workflows/day). The talk covers three versioning strategies: worker versioning, patching, and their custom 'versioning by dispatch' approach. The dispatch pattern uses a single deployed app with a dispatcher class that routes workflows to the correct versioned implementation based on a LaunchDarkly feature flag stored in workflow input. To upgrade long-running workflows stuck in unbounded waits, they use Temporal's 'continue as new' feature triggered via a batch signal operation, migrating ~900K workflows at ~50/sec. Key lessons include always forking a new version when touching workflow code, using protobuf objects for forward/backward-compatible inputs, the 'fork behind' git pattern to preserve history, and always calling 'every handler finished' before continue-as-new.

1h 0m watch time

Sort: