Archive for February, 2015

lustre, pread/pwrite and caddr_t

February 26th, 2015
Comments Off on lustre, pread/pwrite and caddr_t

I just pushed some changes to MPICH master to help ROMIO deal with useful  features that might not be available on all platforms and configurations.
The system calls pread(2) and pwrite(2)  act just like read(2) and write(2), but they take an additional offset parameter.  The intent was to make life easier for threaded applications, but these system calls help with ROMIO scalablity, too.  The lseek(2) system call isn’t terribly expensive most of the time, but on some platforms with system call forwarding (e.g. Blue Gene/ Q), the extra system call can contribute to congestion at the system call forwarding interface.
Perhaps foolishly, I defined _XOPEN_SOURCE to 600 in ROMIO.   this worked fine on all platforms I could test, but some of our Lustre using friends have run into problems.    ROMIO’s lustre driver includes lustre.h.  Lustre.h includes quota.h and quota.h tries to use a caddr_t datatype — one that is not defined if _XOPEN_SOURCE is set to 600.
This problem can show up if you are building ROMIO, MPICH, MVAPICH, or OpenMPI with lustre, and looks like this:

In file included from /usr/include/linux/lustre_user.h:46,
                 from /usr/include/lustre/lustre_user.h:54,
                 from adio/ad_lustre/ad_lustre.h:30,
                 from adio/ad_lustre/ad_lustre_rwcontig.c:20:
/usr/include/sys/quota.h:221: error: expected declaration specifiers or
'...' before 'caddr_t'

In this case, what I decided to do was simply provide a re-implemented pread/pwrite in ROMIO if the pread/pwrite prototypes don’t exist.
If you ran into this problem, sorry about that.  If you found this page because you are facing a similar problem, please try the latest MPICH.  The two changes are the following: and


ROMIO changes in MPICH-3.1.4

February 24th, 2015
Comments Off on ROMIO changes in MPICH-3.1.4

The MPICH team released MPICH-3.1.4 , and with it some ROMIO bug fixes.  Not much in the way of new features:

  • continue to improve handing of bogus arguments
  • several GPFS and Blue Gene optimizations
  • continued improvements with IBM PE compatibility
  • picked up a few build-related changes from OpenMPI
  • worked around a strange MPICH type handling bug related to zero-length struct members