[J3] [EXTERNAL] Re: Question to compiler developers about compilation speed

Tue Oct 29 16:14:20 UTC 2024

HI Jeff,

Thank you for the additional information.

I too am generally happy even on my apple laptop.  Fewer cores, but the serial build performance dramatically outperforms the mainframes.   I had thought that this was due to the use of SSDs, but recent experiments suggest that is not the culprit. (And yes, we used SSD for tmp as well.)   For the mainframe there is the metadata management (GPFS), but we also have standalone nodes (smaller) that still build about as slow per core.

We do use CMake, and I do not consider it to be a foolish decision.  (Let’s not rehash that one.)    And we use Ninja for data collection as it gives useful thread-based profiles.    To my knowledge the total build time with ninja and gnumake are comparable but I will have my person double check this.

Actual sources are divided into many semi-independent libraries so it is difficult to try to game the build to do certain files first.   However, outside the infrastructure layer the model components are generally independent (in terms of build anyway).   To that end it is a matter of minimizing dependencies within the infrastructure layer and cmake seems to be reasonably smart (within a library) to identify the deepest bits of the dependency chain and building those first.

Cheers,

  *   Tom

From: Jeff Hammond <jehammond at nvidia.com>
Date: Tuesday, October 29, 2024 at 12:03 PM
To: General J3 interest list <j3 at mailman.j3-fortran.org>
Cc: Clune, Thomas L. (GSFC-6101) <thomas.l.clune at nasa.gov>, Kouatchou, Jules (GSFC-606.0)[SCIENCE SYSTEMS AND APPLICATIONS INC] <jules.kouatchou-1 at nasa.gov>, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson at nasa.gov>
Subject: [EXTERNAL] Re: [J3] Question to compiler developers about compilation speed
CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.

I am not surprised.  I have tried to optimize NWChem builds in the past and found that fewer big source files is better than many small ones, but we have no modules, so the comparison is not ideal.

Make sure /tmp is backed by memory, since many compilers (e.g. NVIDIA’s) generate temp files there.

I am happiest with compilation times on my (water-cooled) AMD 7950X desktop with NVMe drives.  It has only 16 cores but hits >5 GHz and has zero overhead from parallel filesystems, etc.  Parallel filesystem metadata overhead has always been a huge pain for builds.

If you have a few files that are expensive, make sure your build system schedules those first.

If you are so foolish as to use CMake, at least use Ninja as the back-end.  It’s much better.

Jeff

On 29. Oct 2024, at 17.32, Clune, Thomas L. (GSFC-6101) via J3 <j3 at mailman.j3-fortran.org> wrote:

External email: Use caution opening links or attachments

A bit of background,    The  Earth system model that I support has millions of lines of source code, and as such build times can be a major factor in software development.   The scientists generally don’t care so much as they usually build the model in batch and spend their time monitoring the execution and performing analysis of the results.  But the model developers, and esp. my team of infrastructure developers spend a considerable amount of time building the model in order to verify changes.    In particular this creates significant latency for our CI tests.

A raw optimized build takes O(15 minutes) on a current generation server node.   This time includes ~10x speedup from parallelization.  Unoptimized debug compiles a bit faster (30%), but not as much as you might think due to the parallelization bottlenecks.     Our CI generally is slower because it only runs on 2 cores in the cloud, and routinely takes > 30 minutes to pass.   (And of course infrastructure layers and their tests require less time, but we need to verify the model as a whole still works, and that dominates.)

Current generations of HPC nodes now have O(100 cores),  and that leads me to ask the question how much faster could we build the model.    We know from the profile of the build that module dependencies (unsurprisingly) play a significant role in producing bottlenecks during the build.   Further we have a relatively small number of files that are disproportionately expensive to compile on their own.   We can usually split those into separate modules and achieve superlinear speedup.

My thought was to push the envelope on what submodules can do.     Consider the extreme approach where every module procedure is placed into its own submodule.  (And yes, > 99% of our source code is in the form of modules.)   My expectation was that in this scenario, the “depopulated” modules would compile quickly, and we could keep dozens of threads busy building the independent submodules.    And indeed we do see that scalability is improved with this approach in that we can keep more threads busy than before.    But I was quite surprised to find the actual build timedid not significantly improve.   First, we have found that each new submodules often takes a similar amount of time to compile as the original module.   Second, we also found that the depopulated module was often nearly as expensive to compile as the original.     The details vary, but the conclusions generally apply across multiple compilers (intel, gnu, nag).

Note that it has still been beneficial to create the submodules to avoid cascade bottlenecks, but for that we can put all of the module procedures into a single submodule.

My questions to compiler developers:

·         Do these results surprise you?

·         Is there a general explanation for why what I’m attempting is limited?

·         Is there other advice on how to optimize for build time?

Thanks in advance,

·         Tom

PS We’ve also looked at placing our source files on solid-state disk, but disk access does not appear to be a significant part of the build cost.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.j3-fortran.org/pipermail/j3/attachments/20241029/97734a88/attachment.htm>