Executable Workflow Engines Part 2 - Reactive Workflows
Let's explore a different approach to manage long-running workflows in cloud native environments
In previous post, I've covered a few concerns about the popularity of procedural workflow engines and the need to properly assess their benefits and limitations before integrating them in your microservice ecosystem. In this post, I want to propose an alternative approach to manage workflows that might be more aligned to the need of certain organizations. As previously discussed, one of the weakest aspect of procedural workflows is the need to think through the whole process at once, which requires highly skills experts in modeling these processes and in understanding your organization. For simpler process, this might not be a big deal, but you need the power of workflow engines when processes are complex!
I've been working on reactive architectures for the past 15 years or so. I think we design systems are way too orderly, trying to think through all aspects instead of creating chaordic systems where users and developers can innovate and evolve. I found that rigid systems are often very expensive to develop and maintain and are more fragile than loose systems. I'll surely talk about this in another blog, but for now, let's think of a workflow as a series of event that needs to happen in a certain order within a certain timeframe.
Let's use a simple BPMN process like this one:
Basically, this process is performing
step 1 when it's starting then it waits for one event, either
C. Each event trigger a different sequence which converge to the same
step 3 conclusion. I've put a timer event on
step 3 to ensure that this will be completed after a certain duration or at a specific date. Of course, you can design much more complex workflows with BPMN, but at the end of the day, it's all just waiting for some signal/event, checking internal state conditions and triggering actions, either sequentially or in parallel.
This workflow is just a state machine getting mutated through a web app with a number of observers monitoring specific trigger conditions to execute their action. These two diagrams express about the same process, but with very different attributes. Some would rightly argue that the BPMN provides a better understanding of the sequence of things and I would agree. But the second one is not enforcing a sequence and that's the point! If you want to add more steps, you just connect more observers to the state machine, no need to change anything that is already there, you can evolve incrementally.
In their latest engineering blog, the Netflix team hints at the benefits of moving from their procedural workflow engine to their new forward-chaining rule engine. For them, they like the fact that these conditionally executing components are always there listening, no need to start process instances, they watch whatever state they were connected to and execute when they detect a match.
My agile (and little chaotic) mind prefers this approach to modeling workflows. I'm not against the BPMN approach at all, I just think that it is expensive and constraining for the level of benefits you get. I would say, as a rule of thumb, that the larger and stable the organization, the more benefits you get. If you manage a dynamic, evolving business, with a lot of small intertwined processes, you should think about reactive workflows instead.
There's not a big offering in this space right now. A lot of reactive event streams, triggering serverless functions, which are all interesting fundamental components. But the missing part is the scalable state machine components, which is the critical one to get consistent state on which to base workflow rules. State must be abstracted from underlying data source. I'll go deeper into this with concrete example from one of my upcoming startup LiquidScale, which provides this kind of advanced state management components inside Kubernetes. For now, if you need to implement reactive workflows, you can still rely on querying a database or other source to produce the reference state.