The Darshan team recently encountered a sequence of bugs that produced corrupt log files when Darshan was linked against OpenMPI. This article provides some broader background, how we used Darshan itself to diagnose the problem, and what we learned from the experience.
Background
Darshan is an I/O characterization tool that produces concise summaries of how applications use a variety of different I/O interfaces. One of the interfaces that it instruments is MPI-IO (the part of the MPI specification that provides an abstraction for accessing files in parallel). MPI-IO is unique among the interfaces that Darshan instruments, however, because Darshan is itself also a user of MPI-IO. When an application terminates, Darshan writes the final compressed version of its instrumentation to a log file using MPI-IO.
MPI-IO is crucial to Darshan’s efficiency and portability because it has the discretion to reshape I/O traffic into more efficient access patterns for whatever platform you are running your code on. For example, data from collective writes is often aggregated into intermediate buffers that can satisfy the optimal concurrency, access size, and lock boundaries of an underlying storage system. This optimization is called “collective buffering”. The Darshan code is greatly simplified because it does not have to detect the underlying file system parameters or implement this optimization itself; it simply describes the data to be written, and MPI-IO does the rest.
Darshan also uses hints to further optimize how collective buffering is performed in Darshan’s specific use case. MPI-IO must optimize for the general case, but Darshan log files have specific properties: they are usually very small (far less than a megabyte), and they usually contain small amounts of data contributed by every process. For most file systems, this means that the time to write a Darshan log at scale is dominated by the cost of concurrent open() traffic, not by the cost of the actual data transfer. We therefore set hints when the log file is opened to give MPI-IO some clues. Specifically, Darshan sets cb_nodes=4 to suggest that no more than 4 processes are needed to aggregate data, and romio_no_indep_rw=true to indicate that most Darshan ranks will not perform independent I/O operations. Taken together, an MPI implementation can use this information to activate a “deferred open” mode, in which at most 4 processes open the Darshan log file and write data on behalf of all of the other processes. This greatly reduces the cost of writing small log files onto a parallel file system at scale.
Darshan was originally developed at Argonne National Laboratory using MPI implementations derived from MPICH, which internally uses an MPI-IO implementation called “ROMIO.” Darshan also works equally well with MPI implementations derived from OpenMPI, which internally uses an MPI-IO implementation called “OMPIO”. Different MPI-IO implementations do not support the same hints, but there is no harm in attempting to set them for any MPI implementation. They are advisory parameters that implementations are not obligated to honor, and they do not impact data correctness.
The problem
Over time, the Darshan team has occasionally received bug reports (via Slack, mailing list, or GitHub) of corrupted Darshan logs. Darshan does not produce any runtime error messages in these cases, but subsequent analysis tools are unable to parse the log files. The most frustrating part of this problem has been our inability to independently reproduce it. The reports all involved OpenMPI, but that didn’t really tell us anything (or even indicate that OpenMPI had anything to do with the problem; OpenMPI is broadly used, and most of Darshan log files produced with it look perfectly fine). The applications were different, the scales were different, the OpenMPI versions were different, and our attempts to locally reproduce the problem always failed.
The breakthrough
Wei-keng Liao of Northwestern University (a recent addition to the core Darshan development team after a long history of contributions) recently implemented expanded coverage of the MPI-IO API to account for large integer types in https://github.com/darshan-hpc/darshan/pull/1060. As part of this work, Wei-keng added GitHub CI tests to exercise the instrumentation and validate corresponding Darshan counters.
Although it had nothing to do with the feature being implemented, Wei-keng happened to find a permutation that reliably caused Darshan log file corruption every time we executed the GitHub CI action! The problem (as in previous reports) occurred with OpenMPI, but this time within a CI environment with 4 processes executing on a single virtual node.
We weren’t quite done yet, because despite having the code and configuration clearly documented, the same problem still didn’t necessarily occur on other (non-GitHub) environments.
Isolating the problem
Wei-keng looked at the precise offsets and sizes being written by Darshan and re-created these in a standalone MPI program (without the Darshan library) that self-validated the data that it wrote using a predetermined pattern so that we could more easily try different permutations. We used strace to observe the system calls that it produced, and were surprised to see that it generated read/modify/write operations at the file system level. This meant that OpenMPI was performing “data sieving”. Data sieving is another popular MPI-IO optimization; if you need to write multiple discontiguous regions in a file, sometimes it is faster to read in the full span, modify the regions of interest, and then write the full span out rather than issues many small write operations. However, we confirmed that the Darshan log file was densely populated (no gaps) and was written with non-overlapping writes. There was no apparent reason for data sieving to be activated.
With this knowledge in hand, we identified that the problem only occurs in OpenMPI 5.0.5 or earlier, and that it was likely resolved by the bug fix to the ompio file locking strategy in https://github.com/open-mpi/ompi/pull/12759. Our CI action happened to use an old enough version of OpenMPI and issued a particular combination of writes operations that triggered the bug.
This still left us with two issues: a) Why was OpenMPI performing data sieving in the first place? (regardless of the locking strategy) and b) What could we do in Darshan to mitigate the problem? Darshan must operate correctly even on production systems that have deployed older versions of OpenMPI.
OpenMPI issues
Using the information we learned above, we further simplified the reproducer, used Darshan to observe the access patterns it generated, and opened an issue on the OpenMPI GitHub repository describing some precise scenarios in which adversarial access patterns are generated (https://github.com/open-mpi/ompi/issues/13376). We do not have sufficient expertise in the code base to attempt a fix, but it is clear that under certain configurations, ompio distributes data to aggregators in a way that results in interleaved discontiguous regions at some ranks. This, in conjunction with an automatic data sieving optimization, forces ompio to write data using conflicting write operations that are serialized with advisory locks. While recent versions of OpenMPI have an improved file locking algorithm that avoids data corruption in this scenario, it is still not an efficient access pattern.
While exploring potential mitigations to use in Darshan, we considered that maybe we should simply turn off collective buffering (via another MPI_Info hint) in some situations to avoid the problem. While checking the impact of this potential mitigation, we discovered a second problem, reported in https://github.com/open-mpi/ompi/issues/13377, in that the logic for the collective_buffering= hint appears to be inverted. If that hint is provided, then it must be set to false to enable collective buffering. This counterintuitive behavior makes usage of this hint particularly dangerous, because the semantics could be corrected in the future. We had to continue looking for a different mitigation.
Darshan mitigation
In addition to alerting the OpenMPI developers of the ompio problems described above, we did eventually identify a Darshan mitigation that allows Darshan to work correctly with any OpenMPI version. If we detect that OpenMPI is present, then we simply set our default hints to cb_nodes=1 rather than cb_nodes=4 (the cb_nodes hint is honored by both MPICH/ROMIO and OpenMPI/OMPIO). When we set it to 1, OpenMPI will still perform collective buffering, but the data is buffered at a single process so that there is no risk of interleaved/overlapping/conflicting write operations.
This mitigation will be included in the next Darshan minor release (probably Darshan 3.5.0).
Takeaways
Although the OpenMPI bugs we observed were originally triggered by Darshan’s internal logging code, we found Darshan itself to be very helpful in understanding the nature of the bugs. In particular, we used Darshan’s optional DXT tracing mode, which can be activated in any Darshan build by setting an environment variable, to compare the access patterns expressed at the MPI-IO level to the access patterns enacted at the POSIX level of the HPC I/O stack.
We also learned not to take MPI-IO behavior for granted. The API and semantics are defined by the MPI specification, but implementors have the discretion to implement I/O transformations how they see fit. In our case, the Darshan log write access pattern happened to produce an adversarial access pattern in OMPIO that requires special handling to mitigate.
Finally, continuous integration proved itself to be an invaluable tool for supporting complex system software. We would not have been able to isolate this problem (much less had any clue what the underlying causes were) without a pull request that introduced a thorough CI test for an entirely unrelated capability. Darshan is a mature software project that has gained CI capability in a piecemeal fashion; this case study encourages us to pursue a more comprehensive CI strategy in the future.
