{"id":11,"date":"2014-06-05T15:23:18","date_gmt":"2014-06-05T15:23:18","guid":{"rendered":"http:\/\/press3.mcs.anl.gov\/romio\/?p=11"},"modified":"2014-06-05T15:23:18","modified_gmt":"2014-06-05T15:23:18","slug":"new-romio-optimizations-for-blue-gene-q","status":"publish","type":"post","link":"https:\/\/wordpress.cels.anl.gov\/romio\/2014\/06\/05\/new-romio-optimizations-for-blue-gene-q\/","title":{"rendered":"New ROMIO optimizations for Blue Gene \/Q"},"content":{"rendered":"<p>The IBM and Argonne teams have been digging into ROMIO&#8217;s collective I\/O performance on the <a href=\"http:\/\/www.alcf.anl.gov\/mira\">Mira<\/a> supercomputer. These optimizations made it into the <a href=\"http:\/\/www.mpich.org\/2014\/06\/04\/mpich-3-1-1-released\/\">MPICH-3.1.1<\/a> release, so it seemed like a good time to write up a bit about these optimizations.<br \/>\n<strong><em>no more &#8220;<a title=\"bglockless\" href=\"http:\/\/press3.mcs.anl.gov\/romio\/2013\/08\/05\/bglockless\/\">bglockless<\/a>&#8220;<\/em><\/strong>: for Blue Gene \/L and Blue Gene \/P we wrote a ROMIO driver that never called fcntl-style user-space locks.\u00a0 This approach worked great for PVFS, which did not support locks anyway, but had a pleasant side effect of improving performance on GPFS too (as long as you did not care about specific workloads and MPI-IO features).\u00a0 Now, we removed all the extraneous locks from the default I\/O driver.\u00a0 Even better, we kept the locks in the few cases they were needed: shared file pointers and data sieving writes.\u00a0 Now one does not need to prefix the file name with &#8216;bglockless:&#8217; or set the BGLOCKLESSMPIO_F_TYPE\u00a0 environment variable.\u00a0\u00a0 It&#8217;s the way it should have been 5 years ago.<br \/>\n<em><strong>Alternate Aggregator<\/strong><\/em><strong> Selection:\u00a0 <\/strong>Collective I\/O on Blue Gene has long been the primary way to extract maximum performance.\u00a0 One good optimization is how ROMIO&#8217;s two-phase optimization will deal with GPFS file system block alignment.\u00a0\u00a0 Even better is how it selects a subset of MPI processes to carry out I\/O.\u00a0 The other MPI processes route their I\/O through these &#8220;I\/O aggregators&#8221;.\u00a0\u00a0\u00a0 On Blue Gene, there are some new ways to select which MPI processes should be aggregators:<strong><br \/>\n<\/strong><\/p>\n<ul>\n<li>Default: the N I\/O aggregators are assigned depth-first based on connections to the I\/O forwarding node.\u00a0\u00a0\u00a0 If a file is not very large, we can end up with many active I\/O aggregators assigned to one of these I\/O nodes, and some I\/O nodes with only idle I\/O aggregators.<\/li>\n<li>&#8220;Balanced&#8221;:\u00a0 set the environment variable GPFSMPIO_BALANCECONTIG to 1 and the I\/O aggregators will be selected in a more balanced fashion.\u00a0 With this setting, even small files will be assigned I\/O aggregators across as many I\/O nodes as possible.\u00a0 (there&#8217;s a limit: we don&#8217;t split file domains any smaller than the GPFS block size)<\/li>\n<li>&#8220;Point-to-point&#8221;:\u00a0 The general two-phase algorithm is built to handle the case where any process might want to send data to or receive data from\u00a0 any I\/O aggregator.\u00a0 For simple I\/O cases we want the benefits of collective I\/O &#8212; aggregation to a subset of processes, file system alignment &#8212; but don&#8217;t need the full overhead of potential &#8220;all to all&#8221; traffic.\u00a0\u00a0 Set the environment variable &#8220;GPFSMPIO_P2PCONTIG&#8221;\u00a0 to &#8220;1&#8221; and if certain workload conditions are met &#8212; contiguous data, ranks are writing to the file in order (lower mpi ranks write to earlier parts of the file), and data has no holes &#8212; then ROMIO will carry out point-to-point communication among an I\/O aggregator and the much smaller subset of processes assigned to it.<\/li>\n<\/ul>\n<p>We don&#8217;t have MPI Info hints for these yet, since they are so new.\u00a0 Once we have some more experience using them, we can provide hints and guidance on when the hints might make sense.\u00a0\u00a0 For now, they are only used if\u00a0 environment variables are set.<br \/>\n<em><strong>Deferred Open<\/strong> <strong>revisited<\/strong><\/em>: The old &#8220;<a title=\"Deferred Open\" href=\"http:\/\/press3.mcs.anl.gov\/romio\/2003\/08\/05\/deferred-open\/\">deferred open<\/a>&#8221; optimization, where specifying some hints would have only the I\/O aggregators open the file, has not seen a lot of testing over the years.\u00a0 Turns out it was not working on Blue Gene. We re-worked the deferred open logic, and now it works again.\u00a0\u00a0 Codes that open a file only to do a small amount of I\/O should see an improvement in open times with this approach.\u00a0 Oddly, IOR does not show any benefit.\u00a0 We&#8217;re still trying to figure that one out.<br \/>\n<em><strong>no more seeks:<\/strong><\/em><strong> <\/strong>An individual lseek() system call is not so expensive on Blue Gene \/Q.\u00a0 However, if you have tens of thousands of lseek() system calls, they\u00a0 interact with the outstanding read() and write() calls and can sometimes stall for a long time.\u00a0 We have replaced &#8216;lseek() + read()&#8217; and &#8216;lseek() + write()&#8217; with pread() and pwrite().<br \/>\n&nbsp;<br \/>\n&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The IBM and Argonne teams have been digging into ROMIO&#8217;s collective I\/O performance on the Mira supercomputer. These optimizations made it into the MPICH-3.1.1 release, so it seemed like a good time to write up a bit about these optimizations. no more &#8220;bglockless&#8220;: for Blue Gene \/L and Blue Gene \/P we wrote a ROMIO &hellip;<\/p>\n","protected":false},"author":362,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":"","_members_access_role":[],"_members_access_error":""},"categories":[1],"tags":[12],"class_list":["post-11","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-bluegene"],"acf":[],"_links":{"self":[{"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/posts\/11","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/users\/362"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/comments?post=11"}],"version-history":[{"count":0,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/posts\/11\/revisions"}],"wp:attachment":[{"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/media?parent=11"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/categories?post=11"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/romio\/wp-json\/wp\/v2\/tags?post=11"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}