{"id":1367,"date":"2016-07-29T22:45:04","date_gmt":"2016-07-29T21:45:04","guid":{"rendered":"https:\/\/scienceclouds.org\/?p=1367"},"modified":"2016-07-29T22:45:04","modified_gmt":"2016-07-29T21:45:04","slug":"building-a-great-testbed-for-cloud-computing-research","status":"publish","type":"post","link":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/2016\/07\/29\/building-a-great-testbed-for-cloud-computing-research\/","title":{"rendered":"Building a Great Testbed for Cloud Computing Research"},"content":{"rendered":"

We have been silent for a while \u2013 not for the lack of something to say but lack of time to say it in ;-). But today is a very special day: yesterday was the first anniversary of the day Chameleon<\/a>, a cloud computing experimental instrument project that Nimbus team is proud to lead, <\/span>went public<\/span><\/a>. Considering how busy we are, breaking the silence is a bit of a treat but after all that\u2019s what anniversaries are for!<\/span><\/p>\n

One lesson we all learned in over a decade of working on cloud computing research \u2013 or any type of systems research in fact \u2013 it is that without a reliable testbed such research is hard with gusts to impossible. Experimenting with new solutions in operating systems, virtualization, power management all require significant levels of control where the user has access to a broad range of system level tasks: everything from customizing the system kernel, accessing IPMI or even reconfiguring BIOS. They also frequently require that experiments be run in a controlled environment, without being impacted by other users. This was always the crux of the matter: such resources are hard to obtain \u2013 and even harder to obtain at scale required by research on Big Data and Big Compute. <\/span><\/p>\n

No longer. Thanks to NSF\u2019s vision and support for experimental Computer Science we now have the large-scale, deeply reconfigurable testbed that cloud computing research requires. The bulk of <\/span>Chameleon hardware<\/span><\/a> so far consists of over 550 nodes and 5 PBs of storage distributed over University of Chicago and TACC with 100 Gbps network between them. The nodes are distributed over 12 homogenous racks, each consisting of 42 compute nodes (Intel Haswell processors) and 4 storage nodes with 16 2TB disks each \u2013 that\u2019s with very high bandwidth I\/O for your Big Data experiments. The remaining active storage is configured as object store for experimental data and image server. On this relatively homogenous framework were grafted heterogeneous elements: one of the racks has Infiniband network in addition to Ethernet, two nodes have higher memory, SSD, and HDD elements to facilitate experiments with storage hierarchies, we have two K80 GPU nodes, and two M40 GPU nodes. To this we are planning to add NVRAM, FPGAs, as well as a new cluster of ARMs and Atoms. <\/span><\/p>\n

The really exciting bit is in the configuration though. We built this testbed for our colleagues and ourselves \u2013 anybody who needs \u201cas if it were in my lab\u201d level of access to a system \u2013 requirements from detailed interviews with almost 20 Computer Science research teams went into developing a vision. As a result of their insights, our <\/span>Resource Discovery portal<\/span><\/a> has painstakingly detailed hardware configuration information, ranging from cache levels of nodes to serial numbers of individual components, that is automatically discovered and updated as hardware and firmware on the testbed changes. Moreover, every time anything on the testbed changes, we release a new testbed version (over 30 versions over the last year!) so that you can tell at a glance if the testbed today is the same as the testbed yesterday. You can ask for resources interactively (on demand) or place advance reservations \u2013 which you may have to do if you have an eye on hundreds of nodes for your Big Compute experiment! You can reconfigure nodes on a \u201c<\/span>bare metal<\/span><\/a>\u201d level \u2013 we provide a range of <\/span>appliances<\/span><\/a> to make it easier but you can also roll your own \u2013 and from there you can add configuration, reboot into a custom kernel, or reconfigure to a completely new system as needed. Then you can <\/span>snapshot<\/span><\/a>, i.e., save your appliance so that you can move it to a different site, revisit later, or point to the exact version of your environment in your paper. <\/span><\/p>\n

And there is one more interesting thing: we built the cloud research testbed on top of a cloud, pulling ourselves up by our bootstraps as it were. While many pioneering research infrastructures, such as <\/span>GENI<\/span><\/a> and <\/span>Grid\u20195000<\/span><\/a>, gave us ideas, we took a gamble that today this type of infrastructure can be built using off-the-shelf software components \u2013 and it paid off with dividends! Most of Chameleon is built using OpenStack: we are using Ironic for bare metal reconfiguration, Nova, Glance, Swift, and all the other familiar OpenStack components. To be sure, we had to extend it to fit our requirements \u2013 add advance reservations and snapshotting for example \u2013 but we are very happy with the payoff. Using OpenStack allows us to leverage its ecosystem of tools and new features as they come out, make contributions that then serve a community broader than our users, and make our model very accessible to others. OpenStack is not all of course: our resource description and versioning system was borrowed from Grid\u20195000, we added and populated the Appliance Catalog as well as other features making things easier for the users. <\/span><\/p>\n

It has been a busy, adventurous, and very exciting year: we pioneered a new way of building experimental systems and of this we are very proud. But not as proud as of the fact that in this one year of operation Chameleon served as an experimental instrument for 190+ exciting and innovative research projects and was used for 800+ users working on everything from developing <\/span>exascale operating systems<\/span><\/a>, through <\/span>security<\/span><\/a>, to <\/span>education<\/span><\/a> projects. And not as proud as of the fact that\u00a0<\/span>five fantastic institutions<\/span><\/a>, each contributing complementary expertise, could come together working as one team to build a \u00a0new experimental instrument for other cloud computing researchers.<\/span><\/p>\n

Happy Birthday Chameleon! <\/span><\/p>\n

P.S.\u00a0If you are working on cloud computing research and need a testbed, check us out at www.chameleoncloud.org<\/a> — we support Computer Science research nationwide and international collaborations as well. We will be happy to support your research!<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

We have been silent for a while \u2013 not for the lack of something to say but lack of time to say it in ;-). But today is a very special day: yesterday was the first anniversary of the day Chameleon, a cloud computing experimental instrument project that Nimbus team is proud to lead, went… Read more »<\/a><\/p>\n","protected":false},"author":90,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[3,1],"tags":[],"class_list":["post-1367","post","type-post","status-publish","format-standard","hentry","category-general","category-uncategorized"],"acf":[],"_links":{"self":[{"href":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/wp-json\/wp\/v2\/posts\/1367","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/wp-json\/wp\/v2\/users\/90"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/wp-json\/wp\/v2\/comments?post=1367"}],"version-history":[{"count":0,"href":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/wp-json\/wp\/v2\/posts\/1367\/revisions"}],"wp:attachment":[{"href":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/wp-json\/wp\/v2\/media?parent=1367"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/wp-json\/wp\/v2\/categories?post=1367"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cels.anl.gov\/scienceclouds\/wp-json\/wp\/v2\/tags?post=1367"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}