ISTC Releases Open Source Code for S-Store Transactional Streaming System

By John Meehan and Stan Zdonik, Brown University & Nesime Tatbul, Intel Labs and MIT

Today, the ISTC for Big Data released the first version of our S-Store transactional stream processing system. S-Store is open-source software and available for download at http://github.com/s-store/s-store/.

S-Store is also an inherent component of the ISTC’s BigDAWG polystore software stack, which itself can be downloaded at http://bigdawg.mit.edu/ and be used in conjunction with S-Store.

S-Store is the world’s first streaming OLTP engine, which seeks to seamlessly combine online transactional processing with push-based stream processing for real-time applications. We provide an API based on Java and SQL, which facilitates the creation of dataflow graphs of computations operating over both streaming and stored datasets. S-Store builds on a scalable in-memory OLTP infrastructure (H-Store) and extends it with efficient mechanisms to execute these dataflow graphs with strong correctness guarantees that include ACID, ordering, and exactly-once processing.

We have applied S-Store to several real-life use cases including real-time alert monitoring and streaming ETL, spanning diverse applications from bicycle sharing to intensive-care patient monitoring to real-time ship navigation.

Bicycle Sharing: S-Store provides real-time position and availability tracking for bikes rented out by a hypothetical “BikeShare” company [6]

Bicycle Sharing: S-Store provides
real-time position and availability
tracking for bikes rented out by a
hypothetical “BikeShare” company [6]

Intensive-Care Unit Patient Monitoring: S-Store provides real-time alert detection based on MIMIC-II datasets [7].

Intensive-Care Unit Patient Monitoring: S-Store provides real-time alert detection based on MIMIC-II datasets [7].

Ocean Metagenomic Analysis: S-Store provides real-time ship-navigation suggestions based on SeaFlow datasets [8].

Ocean Metagenomic Analysis: S-Store provides real-time ship-navigation suggestions based on SeaFlow datasets [8].

 

S-Store version 0.1 runs natively on 64-bit Linux systems with 4+ GB of memory. It also comes with a Dockerized version, which runs on a virtual machine on any system that supports Docker containers. Version 0.1 supports single-node deployment and full connection with BigDAWG through Docker containers. For more information, please visit http://sstore-doc.readthedocs.io/en/latest/index.html.

You are welcome to visit our project web page for more details including copies of our publications. Please contact us if you are interested in contributing to S-Store, or have ideas for applications or research based on S-Store.

Acknowledgements

  • Cansu Aslantas, Bikong Wang, Tommy Yao, Christian Mathiesen (Brown University)
  • Jiang Du (University of Toronto)
  • Andy Pavlo (CMU)
  • Chris Giossi, Hong Quach, Erik Sutherland, Kristen Tufte, Dave Maier (PSU)
  • Adam Dziedzic, Aaron Elmore (University of Chicago)
  • Hao Wang, Sam Madden, Mike Stonebraker (MIT)
  • Tim Mattson, Jeff Parkhurst (Intel)

Additional Reading

Previous Blog Posts:

  1. “S-Store: Real-Time Analytics Meets Transaction Processing,” February 2014.
  2. “S-Store: A Big Velocity Database System,” December 2014.
  3. “Rethinking Streaming: Correct State Matters,”  June 2016.

Core S-Store:

  1. J. Meehan, N. Tatbul, S. Zdonik, C. Aslantas, U. Cetintemel, J. Du, T. Kraska, S. Madden, D. Maier, A. Pavlo, M. Stonebraker, K. Tufte, H. Wang, “S-Store: Streaming Meets Transaction Processing,” PVLDB 8(13), September 2015.
  2. N. Tatbul, S. Zdonik, J. Meehan, C. Aslantas, M. Stonebraker, K. Tufte, C. Giossi, H. Quach, “Handling Shared, Mutable State in Stream Processing with Correctness Guarantees,” IEEE Data Engineering Bulletin 38(4), Special Issue on Next-Generation Stream Processing, December 2015.

Demos & Applications:

  1. U. Cetintemel, J. Du, T. Kraska, S. Madden, D. Maier, J. Meehan, A. Pavlo, M. Stonebraker, E. Sutherland, N. Tatbul, K. Tufte, H. Wang, S. Zdonik, “S-Store: A Streaming NewSQL System for Big Velocity Applications,” PVLDB 7(13), August 2014.
  2. A. Elmore, J. Duggan, M. Stonebraker, M. Balazinska, U. Cetintemel, V. Gadepally, J. Heer, B. Howe, J. Kepner, T. Kraska, S. Madden, D. Maier, T. Mattson, S. Papadopoulos, J. Parkhurst, N. Tatbul, M. Vartak, S. Zdonik, “A Demonstration of the BigDAWG Polystore System,” PVLDB 8(12), August 2015.
  3. T. Mattson, V. Gadepally, Z. She, A. Dziedzic, J. Parkhurst, “Demonstrating the BigDAWG Polystore System for Ocean Metagenomics Analysis,” CIDR’17, January 2017.

Polystore Integration & Data Ingestion/ETL:

  1. J. Meehan, S. Zdonik, S. Tian, Y. Tian, N. Tatbul, A. Dziedzic, A. Elmore, “Integrating Real-Time and Batch Processing in a Polystore,” IEEE HPEC’16, September 2016.
  2. J. Meehan, C. Aslantas, S. Zdonik, N. Tatbul, J. Du, “Data Ingestion for the Connected World,” CIDR’17, January 2017.
  3. J. Du, J. Meehan, N. Tatbul, S. Zdonik, “Dynamic Data Placement for Real-Time Polystore Ingestion,” June 2017.

 

This entry was posted in Big Data Applications, Big Data Architecture, DBMS, ISTC for Big Data Blog, Polystores, Streaming Big Data and tagged , , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *


three + = 4