Documentation/Labs/BuildSystem ImproveCMakeConfigurationTime

From SlicerWiki
Jump to: navigation, search
Home < Documentation < Labs < BuildSystem ImproveCMakeConfigurationTime

CMake performance Analysis

During the 31st 3D Slicer Project Week, we found a loss of performance in CMake for the generation of Slicer targets in the operating system Windows. The performance loss means the time for the generation of the target is larger than the operating system Linux. This page analysises the performance loss in Windows in several computers: two desktops and a laptop. The computer configurations are shown in the table below.

Desktop1 Desktop2 Surface
Processor Intel i7-6700 CPU 3.40GHz Intel i7-6700 CPU 3.40GHz Intel i7-8650U CPU 2.11GHz
RAM 16GB 16GB 16GB
Disk storage Hard disk Drive (HDD) Solid State Drive (SSD) Solid State Drive (SSD)
Operating system Windows 10 x64 Windows 10 x64 Windows 10 x64


Enviroment

The test enviroment uses the CMake versions 3.14.4 and nightly (3.15.20190910-g39c58dd), and the master branch of Slicer repository (commit: e091a2b016eef4a621fce1dfecc6faa7f6433b51).

The 21 tests were generated in different configuration variables. All tests were executed five times in the folder Slicer-SuperBuild/Slicer-build after the Slicer had been built.

The modified variables are: Slicer_BUILD_CL, Slicer_BUILD_I18N_SUPPORT, Slicer_USE_PYTHONQT, Slicer_BUILD_QTLOADABLEMODULES, Slicer_BUILD_QTSCRIPTEDMODULES, Slicer_EP_LABEL_REMOTE_MODULE, CMAKE_CONFIGURATION_TYPES. The internationalization(I18n) is not activated any tests.

Results

The below table shows the command lines and the time averages for each test in each CMake version.

Test name Command Line CMake 3.14.4 CMake nightly (3.15.20190910-g39c58dd)
Desktop1 Desktop2 Surface Desktop1 Desktop2 Surface
test1 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="MultiVolumeExplorer_SOURCE_DIR;MultiVolumeImporter_SOURCE_DIR;SimpleFilters_SOURCE_DIR;BRAINSTools_SOURCE_DIR;DataStore_SOURCE_DIR;CompareVolumes_SOURCE_DIR;LandmarkRegistration_SOURCE_DIR" -DCMAKE_CONFIGURATION_TYPES="Debug;Release;MinSizeRel;RelWithDebInfo" . 340.84 305.30 314.50 156.44 118.46 112.60
test2 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="MultiVolumeExplorer_SOURCE_DIR;MultiVolumeImporter_SOURCE_DIR;SimpleFilters_SOURCE_DIR;BRAINSTools_SOURCE_DIR;DataStore_SOURCE_DIR;CompareVolumes_SOURCE_DIR;LandmarkRegistration_SOURCE_DIR" -DCMAKE_CONFIGURATION_TYPES="Debug;Release;MinSizeRel" . 254.40 238.58 265.64 118.43 99.90 93.55
test3 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="MultiVolumeExplorer_SOURCE_DIR;MultiVolumeImporter_SOURCE_DIR;SimpleFilters_SOURCE_DIR;BRAINSTools_SOURCE_DIR;DataStore_SOURCE_DIR;CompareVolumes_SOURCE_DIR;LandmarkRegistration_SOURCE_DIR" -DCMAKE_CONFIGURATION_TYPES="Debug;Release" . 192.34 177.33 191.21 95.37 82.88 76.39
test4 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="MultiVolumeExplorer_SOURCE_DIR;MultiVolumeImporter_SOURCE_DIR;SimpleFilters_SOURCE_DIR;BRAINSTools_SOURCE_DIR;DataStore_SOURCE_DIR;CompareVolumes_SOURCE_DIR;LandmarkRegistration_SOURCE_DIR" -DCMAKE_CONFIGURATION_TYPES="Debug" . 129.68 118.47 118.83 72.39 65.43 59.20
test5 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="MultiVolumeExplorer_SOURCE_DIR;MultiVolumeImporter_SOURCE_DIR;SimpleFilters_SOURCE_DIR;BRAINSTools_SOURCE_DIR;DataStore_SOURCE_DIR;CompareVolumes_SOURCE_DIR;LandmarkRegistration_SOURCE_DIR" -DCMAKE_CONFIGURATION_TYPES="Release" . 129.98 119.35 117.70 75.40 65.99 60.25
test6 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug;Release;MinSizeRel;RelWithDebInfo" . 313.28 280.61 286.78 120.15 109.32 105.31
test7 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="MultiVolumeExplorer_SOURCE_DIR;MultiVolumeImporter_SOURCE_DIR;SimpleFilters_SOURCE_DIR;BRAINSTools_SOURCE_DIR;DataStore_SOURCE_DIR;CompareVolumes_SOURCE_DIR;LandmarkRegistration_SOURCE_DIR" -DCMAKE_CONFIGURATION_TYPES="Debug;Release;MinSizeRel;RelWithDebInfo" . 314.07 309.06 292.90 119.62 133.38 108.71
test8 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="MultiVolumeExplorer_SOURCE_DIR;MultiVolumeImporter_SOURCE_DIR;SimpleFilters_SOURCE_DIR;BRAINSTools_SOURCE_DIR;DataStore_SOURCE_DIR;CompareVolumes_SOURCE_DIR;LandmarkRegistration_SOURCE_DIR" -DCMAKE_CONFIGURATION_TYPES="Debug;Release;MinSizeRel;RelWithDebInfo" . 295.44 287.10 280.79 108.57 111.68 96.79
test9 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=0 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug;Release;MinSizeRel;RelWithDebInfo" . 266.17 250.50 252.05 107.73 99.07 91.28
test10 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug;Release;MinSizeRel;RelWithDebInfo" . 99.19 93.70 92.71 46.81 45.98 41.30
test11 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=0 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug;Release;MinSizeRel;RelWithDebInfo" . 5.53 5.35 4.89 5.62 5.18 4.66
test12 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=0 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug;Release;MinSizeRel" . 5.18 5.21 4.85 5.44 4.98 4.61
test13 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=0 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug;Release" . 5.38 5.15 4.94 5.35 5.00 4.61
test14 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=0 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug" . 5.47 5.18 4.82 5.34 4.99 4.59
test15 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=0 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Release" . 5.56 5.16 4.84 5.16 5.00 4.59
test16 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=0 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="RelWithDebInfo" . 5.25 5.13 4.88 5.60 4.94 4.60
test17 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=0 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug" . 36.99 32.16 31.96 23.80 21.64 20.38
test18 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug" . 6.47 6.18 5.86 6.43 5.93 5.46
test19 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=0 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug" . 32.11 28.32 26.55 15.45 16.66 13.33
test20 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=0 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=0 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=0 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug" . 97.88 83.01 83.64 41.90 41.53 36.40
test21 cmake -G "Visual Studio 14 2015 Win64" -DSlicer_BUILD_CLI=1 -DSlicer_BUILD_I18N_SUPPORT=0 -DSlicer_USE_PYTHONQT=1 -DSlicer_BUILD_QTLOADABLEMODULES=1 -DSlicer_BUILD_QTSCRIPTEDMODULES=1 -DSlicer_EP_LABEL_REMOTE_MODULE="" -DCMAKE_CONFIGURATION_TYPES="Debug" . 117.22 119.79 111.90 68.50 69.11 56.11

The below figures show the previous information in a graphic mode.

CMakePerformance4144.png CMakePerformance415Nightly.png

We observe several behaviours:

  • When the number of the targets is decreased, the time saving is meaningful in CMake versions. In the desktop 1, the time reduction is around 62% (version 4.14.4) and 52% (version 3.15.20190910-g39c58dd) from four targets to one target. In the desktop2, the time reduction is around 62% (version 4.14.4) and 44% (version 3.15.20190910-g39c58dd) and in the surface, it is around 63% (version 4.14.4) and 46% (version 3.15.20190910-g39c58dd). Moreover, the time reduction averages from version 4.14 to version 3.15.20190910-g39c58dd are 51%, 54%, and 58% in the desktop 1, desktop 2, and surface, respectively.
  • The hard disk technologies affect the time reduction. If the hard disk technology is SSD, the time reduction average is 7% (version 4.14.4) and 18% (version 3.15.20190910-g39c58dd).
  • The variable Slicer_BUILD_QTLOADABLEMODULES has a relevant impact on generate time. The time reduction average is 70% (version 4.14.4), and 65% (version 3.15.20190910-g39c58dd) when the variable Slicer_BUILD_QTLOADABLEMODULES is not active and four targets. All remaining variables affect lesser in the reduction time.

Conclusions

The number of the targets, the hard disk technologies and the variable Slicer_BUILD_QTLOADABLEMODULES have an important impact on the the CMake configuration time in both CMake versions. The hard technologies in the CMake version 4.14.4 affects less than the version nightly (3.15.20190910-g39c58dd).

The CMake version nightly (3.15.20190910-g39c58dd) performs better than the version 4.14.4 in the configuration time. The time reduction average is 46% in one target.