{"id":3313,"date":"2014-06-19T12:05:55","date_gmt":"2014-06-19T19:05:55","guid":{"rendered":"https:\/\/developer.nvidia.com\/blog\/parallelforall\/?p=3313"},"modified":"2022-08-21T16:37:06","modified_gmt":"2022-08-21T23:37:06","slug":"cuda-pro-tip-profiling-mpi-applications","status":"publish","type":"post","link":"https:\/\/developer.nvidia.com\/blog\/cuda-pro-tip-profiling-mpi-applications\/","title":{"rendered":"CUDA Pro Tip: Profiling MPI Applications"},"content":{"rendered":"

When I profile MPI+CUDA applications, sometimes performance issues only occur for certain MPI ranks. To fix these, it’s necessary to identify the MPI rank where the performance issue occurs. Before CUDA 6.5 it was hard to do this because the CUDA profiler only shows the PID of the processes and leaves the developer to figure out the mapping from PIDs to MPI ranks. Although the mapping can be done manually, for example for OpenMPI via the command-line option --display-map<\/code>, it’s tedious and error prone. A solution which solves this for the command-line output of nvprof<\/code> is described here http:\/\/www.parallel-computing.pro\/index.php\/9-cuda\/5-sorting-cuda-profiler-output-of-the-mpi-cuda-program<\/a> . In this post I will describe how the new output file naming of nvprof to be introduced with CUDA 6.5 can be used to conveniently analyze the performance of a MPI+CUDA application with nvprof<\/code> and the NVIDIA Visual Profiler (nvvp<\/code>).<\/p>\n

Profiling MPI applications with nvprof and nvvp<\/i><\/a><\/h2>\n

Collecting data with nvprof<\/i><\/a><\/h3>\n

nvprof<\/code> supports dumping the profile to a file which can be later imported into nvvp<\/code>. To generate a profile for a MPI+CUDA application I simply start nvprof<\/code> with the MPI launcher and up to CUDA 6 I used the string “%p<\/code>” in the output file name. nvprof<\/code> automatically replaces that string with the PID and generates a separate file for each MPI rank. With CUDA 6.5, the string “%q{ENV}<\/code>” can be used to name the output file of nvprof<\/code>. This allows us to include the MPI rank in the output file name by utilizing environment variables automatically set by the MPI launcher (mpirun<\/code> or mpiexec<\/code>). E.g. for OpenMPI OMPI_COMM_WORLD_RANK<\/code> is set to the MPI rank for each launched process.<\/p>\n

$ mpirun -np 2 nvprof -o simpleMPI.%q{OMPI_COMM_WORLD_RANK}.nvprof ./simpleMPI\nRunning on 2 nodes\n==18811== NVPROF is profiling process 18811, command: ./simpleMPI\n==18813== NVPROF is profiling process 18813, command: ./simpleMPI\nAverage of square roots is: 0.667279\nPASSED\n==18813== Generated result file: simpleMPI.1.nvprof\n==18811== Generated result file: simpleMPI.0.nvprof<\/pre>\n
<\/p>\n
Analyzing profiles with nvvp<\/i><\/a><\/h3>\n
The output files produced by nvprof<\/code> can be either read by nvprof<\/code> to analyze the profile one rank at a time (using\u00a0--import-profile<\/code>) or imported into nvvp<\/code>. Since CUDA 6 it’s possible to import multiple files into the same time-line as described here<\/a>. This significantly improves the usability of nvvp<\/code> for MPI applications.<\/p>\n
\"NVVP<\/a>
NVVP time line showing the GPU activity of two MPI processes.<\/figcaption><\/figure>\n
Enhancing profiles with NVTX<\/i><\/a><\/h2>\n
The analysis process can be further improved by using NVTX<\/a> and naming the CPU threads and CUDA devices according to the MPI rank associated to them.\u00a0With CUDA 7.5 you can name threads just as you name\u00a0output files with the command line options --context-name<\/code> and --process-name<\/code>, by passing a string like \u201cMPI Rank %q{OMPI_COMM_WORLD_RANK}\u201d<\/code> as a parameter. Before CUDA 7.5 you can achieve the same result by using NVTX explicitly from your application:<\/p>\n
char name[256];\nsprintf( name, "MPI Rank %d", rank );\n\nnvtxNameOsThread(pthread_self(), name);\nnvtxNameCudaDeviceA(rank, name);<\/pre>\n
\"NVVP<\/a>
NVVP time line with named OS thread and CUDA device showing the GPU activity of two MPI processes.<\/figcaption><\/figure>\n
Instead of naming the CUDA devices it’s also possible to name the GPU context:<\/p>\n
char name[256];\nsprintf( name, "MPI Rank %d", rank );\nnvtxNameOsThread(pthread_self(), name);\n\nCUcontext ctx;\ncuCtxGetCurrent( &ctx );\nnvtxNameCuContextA( ctx, name );<\/pre>\n
\"NVVP<\/a>
NVVP time line with named OS thread and CUDA context showing the GPU activity of two MPI processes.<\/figcaption><\/figure>\n
To guarantee that cuCtxGetCurrent<\/code> picks the right context, it’s required that a CUDA Runtime call is made between the calls to cudaSetDevice<\/code> and cuCtxGetCurrent<\/code>.<\/p>\n
Other Tools<\/i><\/a><\/h2>\n
To collect application traces and analyze the performance of MPI applications, well established and much more sophisticated tools like Score-P<\/a>, Vampir<\/a> or TAU<\/a> exists. These tools use our profiling interface CUPTI to assess MPI+CUDA applications and also offer advanced support to detect MPI and CPU related performance issues.<\/p>\n
Conclusion<\/i><\/a><\/h2>\n
Following the above approach many performance issues of MPI+CUDA applications can be identified with NVIDIA tools and NVTX can be used to improve working with these profiles. Besides the NVTX resource naming, everything described here works equally well with MPI+OpenACC applications.<\/p>\n","protected":false},"excerpt":{"rendered":"
When I profile MPI+CUDA applications, sometimes performance issues only occur for certain MPI ranks. To fix these, it’s necessary to identify the MPI rank where the performance issue occurs. Before CUDA 6.5 it was hard to do this because the CUDA profiler only shows the PID of the processes and leaves the developer to figure … Continued<\/a><\/p>\n","protected":false},"author":245,"featured_media":9152,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"publish_to_discourse":"","publish_post_category":"318","wpdc_auto_publish_overridden":"","wpdc_topic_tags":"","wpdc_pin_topic":"","wpdc_pin_until":"","discourse_post_id":"602000","discourse_permalink":"https:\/\/forums.developer.nvidia.com\/t\/cuda-pro-tip-profiling-mpi-applications\/148444","wpdc_publishing_response":"success","wpdc_publishing_error":"","nv_subtitle":"","ai_post_summary":"","footnotes":"","_links_to":"","_links_to_target":""},"categories":[503],"tags":[21,48,52,449,59],"coauthors":[139],"class_list":["post-3313","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-simulation-modeling-design","tag-cuda","tag-mpi","tag-openacc","tag-pro-tip","tag-profiling"],"acf":{"post_industry":[],"post_products":["CUDA","OpenACC"],"post_learning_levels":[],"post_content_types":[],"post_collections":[]},"jetpack_featured_media_url":"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2017\/02\/GPU-Pro-Tip-e1753800348480.png","primary_category":{"category":"Simulation \/ Modeling \/ Design","link":"https:\/\/developer.nvidia.com\/blog\/category\/simulation-modeling-design\/","id":503,"data_source":""},"nv_translations":[],"jetpack_shortlink":"https:\/\/wp.me\/pcCQAL-Rr","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/posts\/3313","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/users\/245"}],"replies":[{"embeddable":true,"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/comments?post=3313"}],"version-history":[{"count":1,"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/posts\/3313\/revisions"}],"predecessor-version":[{"id":9492,"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/posts\/3313\/revisions\/9492"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/media\/9152"}],"wp:attachment":[{"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/media?parent=3313"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/categories?post=3313"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/tags?post=3313"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/developer-blogs.nvidia.com\/wp-json\/wp\/v2\/coauthors?post=3313"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}