{"id":15088,"date":"2015-12-12T20:45:30","date_gmt":"2015-12-12T18:45:30","guid":{"rendered":"http:\/\/hgpu.org\/?p=15088"},"modified":"2018-09-02T14:14:37","modified_gmt":"2018-09-02T11:14:37","slug":"large-scale-compute-intensive-analysis-via-a-combined-in-situ-and-co-scheduling-workflow-approach","status":"publish","type":"post","link":"https:\/\/hgpu.org\/?p=15088","title":{"rendered":"Large-Scale Compute-Intensive Analysis via a Combined In-Situ and Co-Scheduling Workflow Approach"},"content":{"rendered":"<p>Large-scale simulations can produce hundreds of terabytes to petabytes of data, complicating and limiting the efficiency of work-flows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in-situ, utilizing the same resources as the simulation, and\/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in-situ and co-scheduling approaches for handling petabyte-scale outputs. We compare different analysis set-ups ranging from purely off-line, to purely in-situ to insitu\/co-scheduling. The analysis routines are implemented using the PISTON\/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multicore, and many-core architectures.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Large-scale simulations can produce hundreds of terabytes to petabytes of data, complicating and limiting the efficiency of work-flows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in-situ, utilizing the [&hellip;]<\/p>\n","protected":false},"author":351,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[11,90,3],"tags":[1782,106,258,20,1793,854,1390],"class_list":["post-15088","post","type-post","status-publish","format-standard","hentry","category-computer-science","category-opencl","category-paper","tag-computer-science","tag-gpu-cluster","tag-n-body-simulation","tag-nvidia","tag-opencl","tag-task-scheduling","tag-tesla-k20"],"views":2898,"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/posts\/15088","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/users\/351"}],"replies":[{"embeddable":true,"href":"https:\/\/hgpu.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=15088"}],"version-history":[{"count":1,"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/posts\/15088\/revisions"}],"predecessor-version":[{"id":18431,"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/posts\/15088\/revisions\/18431"}],"wp:attachment":[{"href":"https:\/\/hgpu.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=15088"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hgpu.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=15088"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hgpu.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=15088"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}