lustre, pread/pwrite and caddr_t

February 26, 2015 by Latham, Robert J.

I just pushed some changes to MPICH master to help ROMIO deal with useful features that might not be available on all platforms and configurations.
The system calls pread(2) and pwrite(2) act just like read(2) and write(2), but they take an additional offset parameter. The intent was to make life easier for threaded applications, but these system calls help with ROMIO scalablity, too. The lseek(2) system call isn’t terribly expensive most of the time, but on some platforms with system call forwarding (e.g. Blue Gene/ Q), the extra system call can contribute to congestion at the system call forwarding interface.
Perhaps foolishly, I defined _XOPEN_SOURCE to 600 in ROMIO. this worked fine on all platforms I could test, but some of our Lustre using friends have run into problems. ROMIO’s lustre driver includes lustre.h. Lustre.h includes quota.h and quota.h tries to use a caddr_t datatype — one that is not defined if _XOPEN_SOURCE is set to 600.
This problem can show up if you are building ROMIO, MPICH, MVAPICH, or OpenMPI with lustre, and looks like this:

In file included from /usr/include/linux/lustre_user.h:46,
                 from /usr/include/lustre/lustre_user.h:54,
                 from adio/ad_lustre/ad_lustre.h:30,
                 from adio/ad_lustre/ad_lustre_rwcontig.c:20:
/usr/include/sys/quota.h:221: error: expected declaration specifiers or
'...' before 'caddr_t'

In this case, what I decided to do was simply provide a re-implemented pread/pwrite in ROMIO if the pread/pwrite prototypes don’t exist.
If you ran into this problem, sorry about that. If you found this page because you are facing a similar problem, please try the latest MPICH. The two changes are the following:
http://git.mpich.org/mpich.git/commit/4e80e1d2b9e and http://git.mpich.org/mpich.git/commit/5a10283bf7