Uncategorized

Darshan 3.0.0, the new modularized implementation of Darshan, is now officially ready to be released! This is the first stable release of the Darshan 3.0 fork and builds off the experimental pre-releases we have been providing over the last few months.  We encourage users to consider upgrading to this version in the near future as it contains numerous enhancements over older versions: logging of full file paths, clearer parser and job summary output, addition of a BG/Q specific instrumentation module, future proofing for any upcoming instrumentation modules, etc.
The following changes have been made between this official release and the last pre-release (darshan-3.0.0-pre3):

• fix bug causing compile errors when building external utilities that use the darshan-util API/library
• update docs to give debugging tips for cases where Darshan logs are not generated
• update darshan-runtime docs to give information on upgrading Darshan
• fix shared library regression test script to check for potential errors with Darshan symbols rather than failing silently in these cases
• bug fix for determining minimum non-zero counters in shared file reductions in all modules
• loosen Darshan’s PMPI symbol check to prevent inadvertent disabling of Darshan for some MPICH builds
• bug fix for resolving MPI_Gather and MPI_Barrier when LDPRELOADing Darshan’s shared libraries
• add more helpful error handling when opening 2.x version log files
• port darshan-diff utility over to new 3.0.0 log file format
• fix numerous configure bugs on Cray systems
• add synthetic benchmarking hooks for testing Darshan’s shutdown procedure

We have tested this version extensively on a number of platforms over the last few months and are satisfied with the new features and Darshan’s overall performance. Included in the darshan-runtime documentation (a link is provided below) is some information on how to best upgrade from a Darshan 2.x version to Darshan 3.0.0. As always, please let us know if you have any issues or questions (using either the Darshan users mailing list or by opening an issue on the Darshan GitLab page).
The release tarball can be downloaded HERE.
Documentation for this version of Darshan can be found HERE.

There is a new point release available for the modularized 3.0.0 version of Darshan. This point release contains the following changes:

• add module-specific version fields to header to allow utilities to handle different versions of a module’s I/O data for backwards compatibility — NOTE: this breaks the log file parsing for logs obtained using Darshan-3.0.0-pre2 & Darshan-3.0.0-pre1
• bug fix in regression test scripts for setting proper environment variables to use MPI profiling configuration for Fortran apps
• bug fix in bzip2 log writing implementation in darshan-logutils
• possible race conditions resolved in each module’s shutdown code
• general code, comment, and documentation cleanup
• addition of module-specific counter descriptions printed prior to parsing a modules I/O data in darshan-parser

As a reminder, this is still an experimental release and is not appropriate for production use at the moment, though we are getting close to an official release! We hope you can take the time to try the new version in preparation for the upcoming release, and look forward to any feedback you may have.
Documentation for this version of Darshan can be found HERE, under the Darshan 3.0.0 heading.

Presentations, project links, and contact information from the presenters at the SC15 Analyzing parallel I/O BoF are now available online at http://wr.informatik.uni-hamburg.de/events/2015/analyzingparallelio.  Thanks you to all of the attendees and presenters!

Julian Kunkel and Philip Carns are co-organizing a Birds of a Feather session at SC15 in Austin, Texas, entitled “Analyzing Parallel I/O”.  The BoF will feature a variety of guest speakers to share their perspectives on the field as well as a discussion session.  This is a great opportunity to find out more about I/O analysis tools and best practices.  The BoF will be held in room 13B from 1:30 to 3:00pm on Thursday, Nov 19.  The tentative agenda is available online.

A new point release is available for the new modularized version of Darshan (Darshan 3.0). This point release includes the following additions to the initial Darshan 3.0.0 pre-release:

• the ability to configure Darshan’s max memory usage per instrumentation module at configure time or runtime
  • use `–with-mod-mem=` configure parameter to set at configure time
  • use DARSHAN_MODMEM environment variable to set at runtime
• updated Ruby bindings for the darshan-util library to simplify Darshan log analysis using Ruby (contributed by Matthieu Dorier)

As a reminder, this version of Darshan is experimental and still going through alpha testing, and is thus not suitable for deployment on production systems, yet. We do encourage users to consider using/testing this version as we will be switching over to this branch in the future. Please feel free to provide us with questions, feedback, feature requests, etc.
Relevant docs:
– darshan-runtime installation and usage: darshan-runtime
– darshan-util installation and usage: darshan-util
– docs on new modularized architecture and how to add new instrumentation modules: darshan modularization

A publication on modeling HPC I/O workloads using a variety of workload sources, including Darshan logs, has been accepted at PMBS 2015. This work utilizes the CODES exascale storage system simulation toolkit to analyze the merits of using distinct workload sources (I/O traces, synthetic I/O descriptions, and Darshan I/O characterizations) as input to storage system simulations, as well as replays of I/O workloads on real HPC systems. Documentation on how to configure the CODES workload generator to use Darshan logs as input is given in the CODES repository.
This post will be updated with a link to the paper once the final version is made available online.

There is a pre-release available for download today for a new experimental version of Darshan! You can download it here.
Darshan 3.0.0 provides much of the same functionality of prior versions, but with the following new enhancements and features:
* hooks for developers to add their own instrumentation module
implementations to capture new I/O characterization data
– these instrumentation modules can be used to instrument new
I/O interfaces or gather system-specific parameters, for instance
* modularized log format allows new module-specific utilities to
access their I/O characterization data independently
– this new format also allows new counters to be added to existing
instrumentation modules without breaking existing utilities
* Darshan logs now contain a mapping of Darshan’s unique record
identifiers to full file names, instead of fix-sized file name
suffixes
* a new instrumentation module for capturing BG/Q-specific parameters
(BG/Q environment is automatically detected at configure time)
* new darshan-parser and darshan-job-summary output to utilize the
new modularized log format
This version is not intended for use in a production environment, yet, so please just use for testing for now. We have had success in the cross-platform testing we have done so far, but more thorough testing is necessary before we are ready for to officially release the new modularized implementation. Issues can be reported directly to the mailing list or using the new issue tracking feature at the Darshan GitLab page (https://xgitlab.cels.anl.gov/darshan/darshan).
As always, we welcome any feedback you may have!
Relevant docs:
– darshan-runtime installation and usage: darshan-runtime
– darshan-util installation and usage: darshan-util
– docs on new modularized architecture and how to add new instrumentation modules: darshan modularization

The Darshan source code, issue tracking, and milestone management has been moved to the Darshan gitlab page at ANL.  You can find instructions there for how to clone the repository.
If you have an existing clone of the old Darshan repository, then you can connect it to the new repository with the following command:
git remote set-url origin https://xgitlab.cels.anl.gov/darshan/darshan.git
If you are interested in contributing to Darshan, the gitlab page brings several new features to the project including the ability for external users to sign up for accounts, simplified forking and pull requests, and the ability to track issues and milestones.

The documentation page has been updated to include links to site-specific Darshan documentation for ALCF, NERSC, NCSA, and OLCF facilities.  Please let us know if you would like to have documentation for your deployment added to the list.

Darshan (along with MPI-IO, PnetCDF, HDF5, and Globus) was featured in the “HPC I/O for Computational Scientists” portion of the Argonne Training Program on Extreme Scale Computing (ATPESC) 2015 agenda on August 13, 2015.  See the tutorial page for details and links to curriculum material.