Beams

Interceptors are responsible for connecting to massive data streams and ensuring stable consumption of the data. They act as a conduit, executing Beams on the data flows. Beams, in this context, represent small, modular pieces of code that are designed to perform specific data processing tasks.

Beams, in this context, represent small, modular pieces of code that are designed to perform specific data processing tasks. They can be compared to serverless functions, as they can be combined and orchestrated to achieve the desired data processing results. Beams are designed to be lightweight, flexible, and easy to integrate into data processing pipelines.

Currently, the majority of Beams are built within the Interceptor framework by the DH3 team, which provides a structured environment for managing data flows and processing. Shortly anyone will be able to provide their Beams through WASM (WebAssembly) modules, enabling custom logic and extending the capabilities of the data processing system. This will allow for more complex and specialized data processing tasks, such as incorporating external data sources, dictionaries, and other custom logic.