A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems Conference Paper uri icon

abstract

  • 2015 IEEE. With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques that can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into sub graphs that fit in RAM and uses a paging-like technique to load sub graphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores.

name of conference

  • 2015 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

published proceedings

  • 2015 IEEE International Parallel and Distributed Processing Symposium

author list (cited authors)

  • Harshvardhan, .., West, B., Fidel, A., Amato, N. M., & Rauchwerger, L.

citation count

  • 3

complete list of authors

  • West, Brandon||Fidel, Adam||Amato, Nancy M||Rauchwerger, Lawrence

publication date

  • May 2015

publisher