{"id":27,"date":"2013-08-05T16:12:25","date_gmt":"2013-08-05T16:12:25","guid":{"rendered":"http:\/\/press3.mcs.anl.gov\/romio\/?p=27"},"modified":"2013-08-05T16:12:25","modified_gmt":"2013-08-05T16:12:25","slug":"bglockless","status":"publish","type":"post","link":"https:\/\/wordpress.cels.anl.gov\/romio\/2013\/08\/05\/bglockless\/","title":{"rendered":"bglockless"},"content":{"rendered":"

Update: in MPICH-3.1.1 we finally scrapped bglockless, (see this writeup on 3.1.1 and Blue Gene enhancements<\/a>)\u00a0 but it’s still part of the system software on any BG \/L BG \/P or BG \/Q machines.\u00a0 The following writeup is perhaps of historical interest, but it will be a while (maybe never) before mpich-3.1.1 is the default MPI on Bue Gene \/Q.
\n<\/em>
\nThe IBM BGP MPI-IO implementation is designed to the “lowest common denominator”: NFS. So they’re performing some very conservative locking in their ADIO file system driver in order to try to get correct MPI-IO semantics out of what might be an NFS volume underneath.\u00a0 It’s possible, though, to select an alternate driver that gives better performance in most cases — and terrible, terrible performance in one specific case.
\nThe MPI routine MPI_File_open takes a string “filename” argument. Normally, ROMIO does a stat of the file system to figure out what kind of file system that file lives on, and then selects a “file system driver” (one of the ADIO modules) that might contain file system specific optimizations.
\nIf you provide a prefix, like “ufs:” for traditional unix files, or “pvfs2:” or even “gridftp:”, then that prefix overrides whatever magic detection routines ROMIO would run, and the corresponding “ADIO driver” will be selected.
\nFor Blue Gene \/L (L, I tell you!) I wrote a ROMIO driver that made no explicit fcntl() lock calls.\u00a0 Those lock calls are normally not a big deal, but PVFS v2 did not support fcntl() locks.\u00a0\u00a0 I called this driver ‘bglockless’.
\nour friends at IBM, in a conservative effort to ensure correctness for all possible file systems, wrapped every I\/O operation in an fcntl() lock.\u00a0 90% of these locks were unnecessary and served only to slow down I\/O.
\nso, the half-day “driver with no locks” project I wrote for PVFS takes on a second life as the “make I\/O go fast” driver.
\nNow here’s the catch, and why we can’t just make “bglockless” the default: certain I\/O workloads, if locks are not available, must be carried out in a extremely inefficient manner.\u00a0 Specifically, strided independent writes to\u00a0 a file. \u00a0 Certain rarely used functionality, like shared file pointers and ordered mode operations, are not implemented when locks are disabled.
\nFor Blue Gene \/P and \/Q, one can set the environment variable BGLOCKLESSMPIO_F_TYPE to 0x47504653 (the GPFS file system magic number). ROMIO will then pretend GPFS is like PVFS and not issue any fcntl() lock commands.<\/p>\n","protected":false},"excerpt":{"rendered":"

Update: in MPICH-3.1.1 we finally scrapped bglockless, (see this writeup on 3.1.1 and Blue Gene enhancements)\u00a0 but it’s still part of the system software on any BG \/L BG \/P or BG \/Q machines.\u00a0 The following writeup is perhaps of historical interest, but it will be a while (maybe never) before mpich-3.1.1 is the default …<\/p>\n","protected":false},"author":362,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-27","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"acf":[],"_links":{"self":[{"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/posts\/27","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/users\/362"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/comments?post=27"}],"version-history":[{"count":0,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/posts\/27\/revisions"}],"wp:attachment":[{"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/media?parent=27"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/categories?post=27"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/tags?post=27"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}