Queue and Notify

Given two sequential stages (e.g. microservices, modules, etc.) in a pipeline, store the output of the first stage in a queue, and notify the second stage about it.

The second stage can retrieve and process the data at its leisure, retrying as needed if something goes wrong. Stages can be unit tested independently by hydrating the queue and simulating the notification.

     Stage M            Queue            Stage N
     =======            =====            =======
        |                 |                 |
 input  |                 |                 |
------->|                 |                 |
        |                 |                 |
        | process         |                 |
        |---------.       |                 |
        |         |       |                 |
        |<--------'       |                 |
        |                 |                 |
        | enqueue output  |                 |
        |---------------->|                 |
        |                 |                 |
        | notify          |                 |
        |---------------------------------->|
        |                 |                 |
        |                 |            peek |
        |                 |<----------------|
        |                 |                 |
        |                 |                 | process
        |                 |                 |---------.
        |                 |                 |         |
        |                 |                 |<--------'
        |                 |                 |
        |                 |         dequeue |
        |                 |<----------------|
        |                 |                 |
        |                 |                 | output
        |                 |                 |-------->
        |                 |                 |

Pros:

• Fault tolerance
• Error recovery
• Controlled testing
• Manual intervention

Cons:

• Queue overhead (persistence, (de)serialization, network hops)