Publications – Darshan

If you use Darshan in a scholarly work, then we recommend that you cite one of the papers highlighted in blue below.

For references related to the Darshan architecture/implementation, cite the “Modular HPC I/O characterization with Darshan” paper.
For references related to Darshan analysis tools, cite the “Enabling agile analysis of I/O performance data with PyDarshan” paper.
For references related to I/O studies based on Darshan data, cite the “Understanding and improving computational science storage access through continuous characterization” paper.

Publications involving the Darshan team:

Jakob Luettgau, Shane Snyder, Tyler Reddy, Nikolaus Awtrey, Kevin Harms, Jean Luca Bez, Rui Wang, Rob Latham, and Philip Carns. “Enabling Agile Analysis of I/O Performance Data with PyDarshan.” In Proceedings of the SC’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, pp. 1380-1391. 2023.
Mihailo Isakov, Mikaela Currier, Eliakin Del Rosario, Sandeep Madireddy, Prasanna Balaprakash, Philip Carns, Robert B. Ross, Glenn K. Lockwood, and Michel A. Kinsy. “A taxonomy of error sources in HPC I/O machine learning models”, in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 205-218. IEEE, 2022.
Bing Xie, Zilong Tan, Philip Carns, Jeff Chase, Kevin Harms, Jay Lofstead, Sarp Oral, Sudharshan S Vazhkudai, and Feiyi Wang. Interpreting write performance of supercomputer I/O systems with regression models. In 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 557–566. IEEE, 2021.
Mihailo Isakov, Eliakin Del Rosario, Sandeep Madireddy, Prasanna Balaprakash, Philip Carns, Robert B. Ross, and Michel A. Kinsy. “HPC I/O throughput bottleneck analysis with explainable local models”, in SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-13. IEEE, 2020.
Eliakin Del Rosario, Mikaela Currier, Mihailo Isakov, Sandeep Madireddy, Prasanna Balaprakash, Philip Carns, Robert B. Ross, Kevin Harms, Shane Snyder, and Michel A. Kinsy. “Gauge: An interactive data-driven visualization tool for HPC application I/O performance analysis”, in 2020 IEEE/ACM Fifth International Parallel Data Systems Workshop (PDSW), pp. 15-21. IEEE, 2020.
Mihailo Isakov, Eliakin del Rosario, Sandeep Madireddy, Prasanna B laprakash, Philip Carns, Robert B Ross, and Michel A Kinsy. Toward generalizable models of I/O throughput. In 2020 IEEE/ACM International Workshop on Runtime and Operating Syst ms for Supercomputers (ROSS), 2020.
Zhengchun Liu, Ryan Lewis, Rajkumar Kettimuthu, Kevin Harms, Philip Carns, Nageswara Rao, Ian Foster, and Michael E Papka. Characterization and identification of HPC applications at leadership computing facility. In Proceedings of the 34th ACM International Conference on Supercomputing, pages 1–12, 2020.
Tirthak Patel, Suren Byna, Glenn K Lockwood, Nicholas J Wright, Philip Carns, Robert Ross, and Devesh Tiwari. Uncovering access, reuse, and sharing characteristics of I/O-intensive files on large-scale production HPC systems. In 18th USENIX Conference on File and Storage Technologies (FAST20), 2020.
Glenn K. Lockwood, Shane Snyder, Suren Byna, Philip Carns, Nicholas J. Wright. “Understanding Data Motion in the Modern HPC Data Center”, in Proceedings of the 4th International Parallel Data Systems Workshop (PDSW’19). PDF/slides
Bing Xie, Zilong Tan, Philip Carns, Jeff Chase, Kevin Harms, Jay Lofstead, Sarp Oral, Sudharshan Vazhkudai, and Feiyi Wang, “Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems”, in Proceedings of the 4th International Parallel Data Systems Workshop (PDSW’19). PDF/slides
Sandeep Madireddy, Prasanna Balaprakash, Philip Carns, Robert Latham, Glenn K Lockwood, Robert Ross, Shane Snyder, and Stefan M Wild. Adaptive learning for concept drift in application performance modeling. In Proceedings of the 48th International Conference on Parallel Processing, pages 1–11, 2019.
Teng Wang, Suren Byna, Glenn K. Lockwood, Shane Snyder, Philip Carns, Sunggon Kim, and Nicholas J. Wright. “A Zoom-in Analysis of I/O Logs to Detect Root Causes of I/O Performance Bottlenecks”, 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), Larnaca, Cyprus, 2019, pp. 102-111.
Glenn K. Lockwood, Shane Snyder, Teng Wang, Suren Byna, Philip Carns, Nicholas J. Wright. “A Year in the Life of a Parallel File System”, in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC’18).
Jakob Luttgau, Shane Snyder, Philip Carns, Justin M. Wozniak, Julian Kunkel, Thomas Ludwig. “Toward Understanding I/O Behavior in HPC Workflows”, in Proceedings of the 3rd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems (PDSW-DISCS ’18). PDF/Slides
Julian Martin Kunkel, Eugen Betke, Matt Bryson, Philip Carns, Rosemary Francis, Wolfgang Frings, Roland Laifer, and Sandra Mendez. Tools for analyzing parallel I/O. In International Conference on High Performance Computing, pages 49–70. Springer, Cham, 2018.
Teng Wang, Shane Snyder, Glenn K. Lockwood, Philip Carns, Nicholas J. Wright, Suren Byna. “IOMiner: Large-scale Analytics Framework for Gaining Knowledge from I/O Logs”, in Proceedings of Cluster 2017. PDF
Glenn K. Lockwood, Wucherl Yoo, Suren Byna, Nicholas J. Wright, Shane Snyder, Kevin Harms, Zachary Nault, and Philip Carns, “UMAMI: a recipe for generating meaningful metrics through holistic I/O performance analysis”, in Proceedings of the 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems 2017 (PDSW-DISCS ’17). ACM, New York, NY, USA, 55-60. PDF
Cong Xu, Shane Snyder, Omkar Kulkarni, Vishwanath Venkatesan, Philip Carns, Suren Byna, Robert Sisneros, and Kalyana Chadalavada, “DXT: Darshan eXtended Tracing”, Cray User Group Conference 2017 (CUG 2017). PDF
Shane Snyder, Philip Carns, Kevin Harms, Robert Ross, Glenn K. Lockwood, Nicholas J. Wright. “Modular HPC I/O characterization with Darshan”, in Proceedings of 5th Workshop on Extreme-scale Programming Tools (ESPT 2016), 2016. PDF
Shane Snyder, Philip Carns, Kevin Harms, Robert Latham, and Robert Ross. “Performance Evaluation of Darshan 3.0.0 on the Cray XC30”, Technical Memorandum ANL/MCS-TM-362, Argonne National Laboratory, April 2016. PDF
Shane Snyder, Philip Carns, Robert Latham, Misbah Mubarak, Robert Ross, Christopher Carothers, Babak Behzad, Huong Vu Thanh Luu, Surendra Byna, and Prabhat. “Techniques for Modeling Large-scale HPC I/O Workloads”, in International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS15). PDF/Slides
Xiaoqing Luo, Frank Mueller, Philip Carns, John Jenkins, Robert Latham, Robert Ross and Shane Snyder, “HPC I/O Trace Extrapolation”, ESPT2015: Workshop on Extreme-Scale Programming Tools.
H. Luu, M. Winslett, W. Gropp R. Ross, P. Carns, K. Harms Prabhat, S. Byna, and Y. Yao. “A Multiplatform Study of I/O Behavior on Petascale Supercomputers”, 24th International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC 2015), 2015.
Janardhan Kodavasal, Kevin Harms, Priyesh Srivastava, Sibendu Som, Shaoping Quan, Keith Richards, and Marta García. “Development of a stiffness-based chemistry load balancing scheme, and optimization of I/O and communication, to enable massively parallel high-fidelity internal combustion engine simulations”, in ASME 2015 Internal Combustion Engine Division Fall Technical Conference. American Society of Mechanical Engineers, 2015.
Dong Dai, Robert B Ross, Philip Carns, Dries Kimpe, and Yong Chen. “Using property graphs for rich metadata management in HPC systems”, in Proceedings of the 9th Parallel Data Storage Workshop (PDSW 2014), 2014.
Argonne Leadership Computing Facility, “ALCF I/O Data Repository”, Technical Memorandum ANL/ALCF/TM-13/1, Argonne National Laboratory, February 2013. PDF
P. Carns, Y. Yao, K. Harms, R. Latham, R. Ross, and K. Antypas, “Production I/O Characterization on the Cray XE6”, in Proceedings of the Cray User Group meeting 2013 (CUG 2013). PDF
P. Carns, K. Harms, R. Latham, and R. Ross, “Performance Analysis of Darshan 2.2.3 on the Cray XE6 Platform,” Technical Memorandum ANL/MCS-TM-331, Argonne National Laboratory, October 2012. PDF
Rob Latham, Chris Daley, Wei-keng Liao, Kui Gao, Rob Ross, Anshu Dubey, and Alok Choudhary. “A case study for scientific I/O: improving the FLASH astrophysics code”, in Computational Science & Discovery, 5(1):015001, 2012. PDF
N Liu, J Cope, P Carns, C Carothers, R Ross, G Grider, A Crume, C Maltzahn. “On the Role of Burst Buffers in Leadership-Class Storage Systems”, in Proceedings of 28th IEEE MSST conference, 2012.
Philip Carns, Kevin Harms, William Allcock, Charles Bacon, Samuel Lang, Robert Latham, and Robert Ross. “Understanding and improving computational science storage access through continuous characterization”, in ACM Transactions on Storage, 7:8:1-8:26, October 2011. PDF
Philip Carns, Kevin Harms, William Allcock, Charles Bacon, Samuel Lang, Robert Latham, and Robert Ross. “Understanding and improving computational science storage access through continuous characterization”, in Proceedings of 27th IEEE Conference on Mass Storage Systems and Technologies (MSST 2011), 2011. (Best Paper Award) PDF
J. Fu, M. S. Min, R. Latham, C. D. Carothers. “Parallel I/O Performance for Application-Level Checkpointing on the Blue Gene/P System”, the Workshop on Interfaces and Architectures for Scientific Data Storage (IASDS), in conjunction with IEEE International Conference on Cluster Computing, September 2011. PDF
Philip Carns, Robert Latham, Robert Ross, Kamil Iskra, Samuel Lang, and Katherine Riley. “24/7 characterization of petascale I/O workloads”, in Proceedings of 2009 Workshop on Interfaces and Architectures for Scientific Data Storage, September 2009. PDF

Presentations:

Shane Snyder. “Understanding and Improving the I/O Behavior of Scientific Computing Applications”, Argonne MCS CS Seminar Series, March 2023. LINK
Philip Carns. “Can we Gamify I/O Performance?”, lightning talk at Dagstuhl Seminar 21332: Understanding I/O Behavior in Scientific and Data-Intensive Computing, August 2021. LINK
Shane Snyder. “Darshan: Enabling Application I/O Understanding in an Evolving HPC Landscape”, NERSC Data Seminar, National Energy Research Scientific Computing Center, March 2021. LINK
Shane Snyder. “Characterizing and understanding the behavior of HDF5 I/O workloads with Darshan”, HDF Users Group (HUG), October 2020. LINK
Philip Carns. “Understanding and Tuning HPC I/O: How hard can it be?”, keynote presentation at the 4th annual HPC I/O in the Data Center Workshop (HPC-IODC) and Workshop on Performance and Scalability of Storage Systems (WOPSSS), June 2018. LINK
Philip Carns. Characterizing data-intensive scientific applications with Darshan. CS/NERSC Data Seminar, National Energy Research Scientific Computing Center. June 2017.
Philip Carns. “Characterizing HPC I/O: from Applications to Systems.” ZIH Colloquium at Technische Universität Dresden, Dresden, DE. April 2017. PDF
Philip Carns. HPC I/O for Computational Scientists: Understanding I/O. Argonne Training Program on Extreme-Scale Computing (ATPESC), August 2017. PDF
Philip Carns. Understanding I/O. Argonne Training Program on Extreme-Scale Computing (ATPESC), August 2016. PDF
Philip Carns and Shane Snyder. Darshan: state of the project and new features. Analyzing Parallel I/O BOF presentation at the International Conference for High Performance Computing, Networking, Storage and Analysis (SC15), November 2015. PDF
Michaela Zimmer, Michael Kluge, and Philip Carns. Analyzing Parallel I/O. BOF meeting at the International Conference for High Performance Computing, Networking, Storage and Analysis (SC14), November 2014. PDF
Philip Carns. I/O characterization of large-scale applications with Darshan. The SciDAC Institute of Scalable Data Management, Analysis and Visualization All Hands Meeting, February 2014. Poster/Slides
Kevin Harms. I/O Profiling and Tuning – Darshan. SUPER SciDAC – Fall 2012 Meeting. PDF
Philip Carns. HEC I/O Measurement and Understanding Panel Presentation. HEC FSIO 2011 Workshop. PDF

Other publications that have utilized Darshan:

E. Costa, T. Patel, B. Schwaller, J. M. Brandt, and D. Tiwari. “Systematically inferring I/O performance variability by examining repetitive job behavior.” In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15. 2021.
H. Devarajan, H. Zheng, A. Kougkas, X. -H. Sun and V. Vishwanath, “DLIO: A Data-Centric Benchmark for Scientific Deep Learning Applications,” 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid), 2021, pp. 81-91, doi: 10.1109/CCGrid51090.2021.00018.
Chien, Steven & Podobas, Artur & Peng, Ivy & Markidis, Stefano. (2020). tf-Darshan: Understanding Fine-grained I/O Performance in Machine Learning Workloads. 359-370. 10.1109/CLUSTER49012.2020.00046.
Bing Xie, Houjun Tang, Suren Byna, Quincey Koziol, and Sarp Oral, “Tuning I/O Performance on Summit – HDF5 Write Use Case Study”, Invited talk at the HPC I/O in the Data Center Workshop (HPC-IODC) 2020, in conjunction with the ISC High Performance 2020.
Sunggon Kim, Alex Sim, Kesheng Wu, Suren Byna, Yongseok Son, and Hyeonsang Eom, “Towards HPC I/O Performance Prediction through Large-scale Log Analysis”, 29th International Symposium on High-Performance Parallel and Distributed Computing (HPDC) 2020.
Jiwoo Bang, Chungyong Kim, Kesheng Wu, Alex Sim, Suren Byna, Sunggon Kim, and Hyeonsang Eom, “HPC Workload Characterization Using Feature Selection and Clustering”, 3rd International Workshop on System and Network Telemetry and Analytics (SNTA’20), 2020
Fahim Chowdhury, Yue Zhu, Todd Heer, Saul Paredes, Adam Moody, Robin Goldstone, Kathryn Mohror, and Weikuan Yu. “I/O Characterization and Performance Evaluation of BeeGFS for Deep Learning”, in Proceedings of the 48th International Conference on Parallel Processing (ICPP 2019). Association for Computing Machinery, New York, NY, USA, Article 80, 1–10. DOI:https://doi.org/10.1145/3337821.3337902
Harsh Khetawat, Christopher J Zimmer, Frank Mueller, Edward S Atchley, Sudharshan S Vazhkudai, and Misbah Mubarak, “Evaluating Burst Buffer Placement In HPC Systems”, in Proceedings of the 2019 IEEE International Conference on Cluster Computing (Cluster2019).
James Dickson, Steven A. Wright, Satheesh Maheswaran, J. A. Herdman, Duncan Harris, Mark C. Miller, and Stephen A. Jarvis,“Enabling portable I/O analysis of commercially sensitive HPC applications through workload replication”, in Cray User Group 2017 Proceedings (CUG2017) pp. 1-14. PDF
J. Han, D. Koo, G. K. Lockwood, J. Lee, H. Eom, and S. Hwang, “Accelerating a Burst Buffer via User-Level I/O Isolation”, in 2017 IEEE International Conference on Cluster Computing (CLUSTER), 2017, pp. 245–255. doi:10.1109/CLUSTER.2017.60
George S. Markomanolis, “ExPBB: A framework to explore the performance of Burst Buffers”, CUG 2017. PDF
James Dickson, Stephen Wright, Satheesh Maheswaran, Andy Herdman, and Mark Miller, “Replicating HPC I/O workloads with proxy applications”, in 2016 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems (PDSW-DISCS 2016). PDF
Ashok Srinivasan, C. D. Sudheer, and Sirish Namilae, “Optimizing Massively Parallel Simulations of Infection Spread Through Air-Travel for Policy Analysis”, IEEE/ACM CCGrid 2016. PDF
Wahid Bhimji, Deborah Bard, Melissa Romanus, David Paul, Andrey Ovsyannikov, Brian Friesen, Matt Bryson, Joaquin Correa, Glenn K. Lockwood, Vakho Tsulaia, Suren Byna, Steve Farrell, Doga Gursoy, Chris Daley, Vince Beckner, Brian Van Straalen, David Trebotich, Craig Tull, Gunther H. Weber, Nicholas J. Wright, Katie Antypas, and Prabhat, “Accelerating Science with the NERSC Burst Buffer Early User Program”, CUG 2016. PDF
Dharshi Devendran, Suren Byna, Bin Dong, Brian van Straalen, Hans Johansen, Noel Keen, and Nagiza Samatova, “Collective I/O Optimizations for Adaptive Mesh Refinement Data Writes on Lustre File System”, CUG 2016. PDF
Huong Luu, Amirhossein Aleyasen, Marianne Winslett, Yasha Mostofi, Kaidong Peng, “The Dashboard: HPC I/O Analysis Made Easy”, International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS15). PDF
Sean McDaniel. “Comparing decoupled I/O kernels versus real traces in the I/O analysis of the HACC scientific application on large-scale systems”, poster at the International Conference for High Performance Computing, Networking, Storage and Analysis (SC14), 2014. Poster Abstract
Babak Behzad, Joseph Huchette, Huong Vu Thanh Luu, Ruth Aydt, Surendra Byna, Yushu Yao, Quincey Koziol, and Prabhat. “A framework for auto-tuning HDF5 applications”, in Proceedings of the 22nd international symposium on High-performance parallel and distributed computing, HPDC ’13, pages 127-128, New York, NY, USA, 2013. PDF
B. Behzad, J. Huchette, H. Luu, R. Aydt, S. Byna, M. Chaarawi, Q. Koziol, Prabhat, and Y. Yao, “Auto-tuning of parallel I/O parameters for HDF5 applications”, Poster at the ACM/IEEE SuperComputing Conference (SC’12), November 2012. PDF
Bjørn Lindi. I/O-profiling with Darshan. Whitepaper published by the Partnership for Advanced Computing in Europe, 2012. PDF
Ning Liu, Jing Fu, Christopher D. Carothers, Onkar Sahni, Kenneth E. Jansen, and Mark S. Shephard. “Massively parallel I/O for partitioned solver systems”, Parallel Processing Letters, 20(4):377–395, 2010. PDF
Steven H. Langer, Charles H. Still, Denise Hinkel, A. Bruce Langdon, and Edward A. Williams. “pF3D simulations of laser-plasma interactions using more than 100 thousand BlueGene processors”, in Proceedings of 2010 Nuclear Explosives Code Development Conference (NECDC), 2010.

Please contact us if you would like to have your paper added to the list.