Discamus continentiam augere, luxuriam coercere
Home -> Publications
Home
  Publications
    
edited volumes
  Awards
  Research
  Teaching
  Miscellaneous
  Full CV [pdf]
  BLOG






  Events








  Past Events





Publications of Torsten Hoefler
Nikoli Dryden, Roman Böhringer, Tal Ben-Nun, Torsten Hoefler:

 Clairvoyant Prefetching for Distributed Machine Learning I/O

(In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC21), presented in St. Louis, Missouri, ACM, Nov. 2021)

Publisher Reference

Abstract

I/O is emerging as a major bottleneck for machine learning training, especially in distributed environments. Indeed, at large scale, I/O takes as much as 85% of training time. Addressing this I/O bottleneck necessitates careful optimization, as optimal data ingestion pipelines differ between systems, and require a delicate balance between access to local storage, external filesystems, and remote nodes. We introduce NoPFS, a machine learning I/O middleware, which provides a scalable, flexible, and easy-to-use solution to the I/O bottleneck. NoPFS uses clairvoyance: Given the seed generating the random access pattern for training with SGD, it can exactly predict when and where a sample will be accessed. We combine this with an analysis of access patterns and a performance model to provide distributed caching policies that adapt to different datasets and storage hierarchies. NoPFS reduces I/O times and improves end-to-end training by up to 5.4x on the ImageNet-1k, ImageNet-22k, and CosmoFlow datasets.

Documents

download article:     
download slides:


Recorded talk (best effort)

 

BibTeX

@inproceedings{nopfs,
  author={Nikoli Dryden and Roman Böhringer and Tal Ben-Nun and Torsten Hoefler},
  title={{Clairvoyant Prefetching for Distributed Machine Learning I/O}},
  year={2021},
  month={Nov.},
  booktitle={Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC21)},
  location={St. Louis, Missouri},
  publisher={ACM},
  source={http://www.unixer.de/~htor/publications/},
}


serving: 18.232.188.122:50718© Torsten Hoefler