Queue and Notify

November 07, 2019

Summary

Given two sequential stages (e.g. microservices, modules, etc.) in a pipeline, store the output of the first stage in a queue, and notify the second stage about it.

The second stage can retrieve and process the data at its leisure, retrying as needed if something goes wrong. Stages can be unit tested independently by hydrating the queue and simulating the notification.

     Stage M            Queue            Stage N
     =======            =====            =======
        |                 |                 |
 input  |                 |                 |
------->|                 |                 |
        |                 |                 |
        | process         |                 |
        |---------.       |                 |
        |         |       |                 |
        |<--------'       |                 |
        |                 |                 |
        | enqueue output  |                 |
        |---------------->|                 |
        |                 |                 |
        | notify          |                 |
        |---------------------------------->|
        |                 |                 |
        |                 |            peek |
        |                 |<----------------|
        |                 |                 |
        |                 |                 | process
        |                 |                 |---------.
        |                 |                 |         |
        |                 |                 |<--------'
        |                 |                 |
        |                 |         dequeue |
        |                 |<----------------|
        |                 |                 |
        |                 |                 | output
        |                 |                 |-------->
        |                 |                 |

Pros

Cons