BNL Home

Computer Science and Mathematics Projects

Multicore-Aware Data Transfer Middleware

Harnessing Multicore Parallelism to Scale Data Movement 

Harnessing multicore architecture is an important problem that has been worked upon for many years, especially in HPC communities to execute high-end applications in multiprocessor systems. We leverage the large body of multicore research already accomplished within the HPC community and implement NUMA aware schedulers to improve processor, memory, and I/O affinities for individual tasks (file caching, compression, encryption, and network transport) involved in the end to end data transfer. Traditional synchronous processing, storage I/O, and network send/receive, even easy to implement, become bottlenecks in harnessing multi-core/many-core architectures. Asynchronous operations, commonly found in RDMA, advanced storage I/O, and exascale computing, demonstrate their superior performance and great flexibility over their synchronous counterparts.

We will design an asynchronous high throughput data transfer/on-fly processing system for multi-/many- core computer platforms to allow users to plug in comprehensive libraries for data compression, encryption, transformation, and checksum for different processing environments. This research will pave a way for advancing large scale data transfers in excess of 100Gbps, and bridging the gap between the bare metal network performance and effective end-to-end data transfer capability. The expected research outcomes will have preeminent visibility in high-speed networks, data management middleware, cloud computing, and exascale supercomputing.

asynchronous application design

Asynchronous application design


Y. Ren, T. Li, D. Yu, S. Jin, T. Robertazzi, Design, Implementation, and Evaluation of a NUMA-Aware Cache for iSCSI Storage Servers, IEEE Transactions on Parallel and Distributed Systems (TPDS), vol. 26, no. 2, pp. 413-422, Feb. 2015.