Uncategorized

There is a new point release available for the modularized 3.0.0 version of Darshan. This point release contains the following changes:

• add module-specific version fields to header to allow utilities to handle different versions of a module’s I/O data for backwards compatibility — NOTE: this breaks the log file parsing for logs obtained using Darshan-3.0.0-pre2 & Darshan-3.0.0-pre1
• bug fix in regression test scripts for setting proper environment variables to use MPI profiling configuration for Fortran apps
• bug fix in bzip2 log writing implementation in darshan-logutils
• possible race conditions resolved in each module’s shutdown code
• general code, comment, and documentation cleanup
• addition of module-specific counter descriptions printed prior to parsing a modules I/O data in darshan-parser

As a reminder, this is still an experimental release and is not appropriate for production use at the moment, though we are getting close to an official release! We hope you can take the time to try the new version in preparation for the upcoming release, and look forward to any feedback you may have.
Documentation for this version of Darshan can be found HERE, under the Darshan 3.0.0 heading.

Presentations, project links, and contact information from the presenters at the SC15 Analyzing parallel I/O BoF are now available online at http://wr.informatik.uni-hamburg.de/events/2015/analyzingparallelio.  Thanks you to all of the attendees and presenters!

Julian Kunkel and Philip Carns are co-organizing a Birds of a Feather session at SC15 in Austin, Texas, entitled “Analyzing Parallel I/O”.  The BoF will feature a variety of guest speakers to share their perspectives on the field as well as a discussion session.  This is a great opportunity to find out more about I/O analysis tools and best practices.  The BoF will be held in room 13B from 1:30 to 3:00pm on Thursday, Nov 19.  The tentative agenda is available online.

A new point release is available for the new modularized version of Darshan (Darshan 3.0). This point release includes the following additions to the initial Darshan 3.0.0 pre-release:

• the ability to configure Darshan’s max memory usage per instrumentation module at configure time or runtime
  • use `–with-mod-mem=` configure parameter to set at configure time
  • use DARSHAN_MODMEM environment variable to set at runtime
• updated Ruby bindings for the darshan-util library to simplify Darshan log analysis using Ruby (contributed by Matthieu Dorier)

As a reminder, this version of Darshan is experimental and still going through alpha testing, and is thus not suitable for deployment on production systems, yet. We do encourage users to consider using/testing this version as we will be switching over to this branch in the future. Please feel free to provide us with questions, feedback, feature requests, etc.
Relevant docs:
– darshan-runtime installation and usage: darshan-runtime
– darshan-util installation and usage: darshan-util
– docs on new modularized architecture and how to add new instrumentation modules: darshan modularization

A publication on modeling HPC I/O workloads using a variety of workload sources, including Darshan logs, has been accepted at PMBS 2015. This work utilizes the CODES exascale storage system simulation toolkit to analyze the merits of using distinct workload sources (I/O traces, synthetic I/O descriptions, and Darshan I/O characterizations) as input to storage system simulations, as well as replays of I/O workloads on real HPC systems. Documentation on how to configure the CODES workload generator to use Darshan logs as input is given in the CODES repository.
This post will be updated with a link to the paper once the final version is made available online.

There is a pre-release available for download today for a new experimental version of Darshan! You can download it here.
Darshan 3.0.0 provides much of the same functionality of prior versions, but with the following new enhancements and features:
* hooks for developers to add their own instrumentation module
implementations to capture new I/O characterization data
– these instrumentation modules can be used to instrument new
I/O interfaces or gather system-specific parameters, for instance
* modularized log format allows new module-specific utilities to
access their I/O characterization data independently
– this new format also allows new counters to be added to existing
instrumentation modules without breaking existing utilities
* Darshan logs now contain a mapping of Darshan’s unique record
identifiers to full file names, instead of fix-sized file name
suffixes
* a new instrumentation module for capturing BG/Q-specific parameters
(BG/Q environment is automatically detected at configure time)
* new darshan-parser and darshan-job-summary output to utilize the
new modularized log format
This version is not intended for use in a production environment, yet, so please just use for testing for now. We have had success in the cross-platform testing we have done so far, but more thorough testing is necessary before we are ready for to officially release the new modularized implementation. Issues can be reported directly to the mailing list or using the new issue tracking feature at the Darshan GitLab page (https://xgitlab.cels.anl.gov/darshan/darshan).
As always, we welcome any feedback you may have!
Relevant docs:
– darshan-runtime installation and usage: darshan-runtime
– darshan-util installation and usage: darshan-util
– docs on new modularized architecture and how to add new instrumentation modules: darshan modularization

The Darshan source code, issue tracking, and milestone management has been moved to the Darshan gitlab page at ANL.  You can find instructions there for how to clone the repository.
If you have an existing clone of the old Darshan repository, then you can connect it to the new repository with the following command:
git remote set-url origin https://xgitlab.cels.anl.gov/darshan/darshan.git
If you are interested in contributing to Darshan, the gitlab page brings several new features to the project including the ability for external users to sign up for accounts, simplified forking and pull requests, and the ability to track issues and milestones.

The documentation page has been updated to include links to site-specific Darshan documentation for ALCF, NERSC, NCSA, and OLCF facilities.  Please let us know if you would like to have documentation for your deployment added to the list.

Darshan (along with MPI-IO, PnetCDF, HDF5, and Globus) was featured in the “HPC I/O for Computational Scientists” portion of the Argonne Training Program on Extreme Scale Computing (ATPESC) 2015 agenda on August 13, 2015.  See the tutorial page for details and links to curriculum material.

Darshan version 2.3.1 is now available for download.  This release includes several bug fixes and enhancements, including improvements for Cray, IBM Blue Gene, and Linux cluster environments.   One of the most notable new features is that Darshan now supports the MPICH profiling configuration file method for instrumentation, which simplifies installation on some platforms.  This release also includes a new regression testing framework.
The full changelog since 2.3.0:
darshan-2.3.1
=============
* added documentation and example configuration files for using the -profile or $MPICC_PROFILE hooks to add instrumentation to MPICH-based MPI implementations without generating custom wrapper scripts
* Add wrappers for mkstemp(), mkostemp(), mkstemps(), and mkostemps() (reported by Tom Peterka)
* Change OPEN_TIMESTAMP field to report timestamp right before open() is invoked rather than after timestamp after open is completed. NOTE: updated log format version to 2.06 to reflect this change.
* Change start_time and end_time fields in job record to use min and max (respectively) across all ranks
* Fix bug in write volume data reported in file system table in darshan-job-summary.pl (reported by Matthieu Dorier)
* Clean up autoconf test for zlib and make zlib mandatory (reported by Kalyana Chadalavada)
* add –start-group and –end-group notation to Darshan libraries for Cray PE 2.x environment to fix link-time corner cases (Yushu Yao)
* improve y axis labels on time interval graphs in darshan-job-summary.pl (reported by Tom Peterka)
* misc. improvements to darshan-parser –perf output (reported by Shane Snyder)
– indicate which rank was slowest in unique file results
– label I/O vs. meta time more clearly
– include unique file meta time in agg_perf_by_slowest calculation
* added regression test script framework in darshan-test/regression/
– currently support platforms include:
– Linux with static linking and generated compiler wrappers
– Linux with static linking and profiler configuration files
– Linux with dynamic linking and LD_PRELOAD
– Blue Gene/Q with static linking and profiler configuration files
* update darshan-gen-fortran.pl and darshan-gen-cxx.pl to support new library naming conventions in MPICH 3.1.1 and higher
* update documentation to reflect known issues with some versions of MPICH
* Cray platforms: modify darshan-runtime so that link-time instrumentation options are only used when statically linking via Libs.private. (reported by Kalyana Chadalavada)