Mortar: Wide-Scale Stream Processing

Mortar is a light-weight distributed stream processing engine for federated systems. Such systems, including large-scale datacenters or wireless networks, may have tens of thousands of nodes. This environment presents unique challenges including: scalable query deployment and management, resiliency to node and network failures, and intelligent placement of in-network operators. Mortar provides a robust platform for the scalable collection and processing of data streams across such systems.

Simple In-Network Operator API:
Most stream processors support a standard set of in-network operators, e.g.,  join, min, max, select, union.   Mortar provides a simple API, merge(newValue), remove(oldValue), allowing users to write their own in-network functions.   For example, we have developed a trilateration operator that outputs the location of a Wi-Fi device given a stream of signal strength indicators.

Failure Resilient:
Underneath, Mortar manages the data flow, routing around network obstructions and node failures, providing high availability with failure resilience.  A novel operator placement algorithm, leveraging network coordinates, determines operator placement.  

The lack of clock synchronization, such as the presence of different clock skews (frequencies), can harm result fidelity by changing the relative time reported between nodes (relative clock offset). For distributed stream processing this can increase latency and pollute the final result with values produced at the wrong time. Mortar’s syncless mechanism replaces traditional timestamps with ages, eliminating the effect of clock offset on results and improving result latency by a factor of 8.

Mortar Stream Language:
Mortar allows users to publish, subscribe, and transform any other existing stream in the system.  The Mortar Stream Language allows users to quickly define new streams as a sequence of transformations on existing streams.   Users may attach user-defined triggers to any stream, receiving upcalls as important events occur.  The language draws heavily from the “boxs-and-arrows” approach of the Aurora/Borealis stream processors.  

SSL.UCSD: 2012