CSR: Small: Algorithms and Abstractions for Efficient Virtual-Memory Streaming and Big-Data Computing
Grant
Overview
Affiliation
Other
View All
Overview
abstract
As the field stands today, operating systems provide a poor interface for data-intensive computing, requiring programmers to engage in tedious, non-reconfigurable, and error-prone code development. These software-engineering practices often lead to easily exploitable vulnerabilities and devastating security breaches. This project builds various algorithms and implementations for a virtual-stream interface that enables our society to develop big-data software that is more easily managed, simpler to understand, inherently faster, and less buggy.Operating systems have used virtual memory and paging for decades; however, user-level applications are still required to process input/output in blocks of fixed size. This project takes a different approach by creating a zero-copy streaming abstraction that offers sequential access to bulk data with unprecedented simplicity, flexibility, and speed. The outcomes of this research not only improve the internal functionality of operating systems and hardware, but also permit reuse of existing libraries in external-memory operation, lead to significantly faster in-place sorting and inter-thread communication, and pave the way to more scalable database computing.The project delivers novel system-level concepts and prototypes that simplify algorithm design, enable faster processing of large-scale data streams, reduce software cost, and help produce better technology for the 21st century. The project also engages students at Texas A&M University in research-intensive education in cross-disciplinary fields, broadens integration of fundamental research into classroom teaching, mentors students, and permits related research in the industry and institutions around the world through publicly shared outcomes of our work.Project data will be maintained online for as long as it is feasible. The shared products include publications, data, software, and various research artifacts. The project URL is http://irl.cs.tamu.edu/projects/streams/
