Difference between revisions of "Documentation/Nightly/Developers/Factory"

From Slicer Wiki
Jump to: navigation, search
Tag: 2017 source edit
 
(22 intermediate revisions by 2 users not shown)
Line 5: Line 5:
  
 
These builds are generated every day on three different operating systems: Windows 7, Mac OS X, and Linux.  Because it is not legal to run any version of Mac OS X on a virtual machine (unless its host also runs Mac OS X), the host OS is Mac OS X, and the Windows and Linux builds are run within this machine as virtual machines.
 
These builds are generated every day on three different operating systems: Windows 7, Mac OS X, and Linux.  Because it is not legal to run any version of Mac OS X on a virtual machine (unless its host also runs Mac OS X), the host OS is Mac OS X, and the Windows and Linux builds are run within this machine as virtual machines.
 
'''Dashboard build scripts are located on GitHub: https://github.com/Slicer/DashboardScripts'''
 
  
 
==Required factory components==
 
==Required factory components==
Line 12: Line 10:
 
See [[Documentation/{{documentation/version}}/Developers/Tutorials/DashboardSetup|dashboard prerequisites]].
 
See [[Documentation/{{documentation/version}}/Developers/Tutorials/DashboardSetup|dashboard prerequisites]].
  
=Host(s)=
+
== Build scripts ==
  
== factory.kitware (RETIRED) ==
+
Dashboard build scripts are located on GitHub: https://github.com/Slicer/DashboardScripts
  
=== Software ===
+
== Remote Access ==
The factory machine runs Mac OS X v10.6.8, Snow Leopard. 
 
* CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3 with [https://github.com/Kitware/CMake/compare/master...jcfr:osx-framework this patch])
 
** Since CMake release >= 3.6 is compiled only for  >= 10.7, a custom build of cmake was done with <code>-DCMAKE_INSTALL_PREFIX:PATH=/Users/kitware/Dashboards/Support/CMake-3.9.0.app/Contents</code> 
 
* Compiler llvm-3.1. Note that the default compiled shipped with XCode 4.2 is not used. See [http://slicer-devel.65872.n3.nabble.com/MacOSX-factory-build-Transitioning-from-hybrid-quot-llvm-g-quot-to-quot-clang-quot-Update-to-Qt-4-8-6-tt4032198.html#a4032199 here] for more details.
 
* Tasks scheduled using [http://en.wikipedia.org/wiki/Cron cron]
 
  
* Nightly generation of HomebrewCasks: [[Documentation/Nightly/Developers/Factory/HomeBrewInstall|Installation aborted]] (newer system is required for <tt>homebrew/cask-repair</tt>).
+
To be granted remote access to the Slicer factories maintained and hosted by Kitware, contact https://discourse.slicer.org/u/sam_horvath and https://discourse.slicer.org/u/jcfr
  
Remote access: Via SSH or VNC from Kitware internal network (192.168.113.230).
+
=Host(s)=
<!--
 
The output of <pre>crontab -l</pre> on the factory machine is as follows:
 
<pre>
 
55 23 * * * cd /Users/kitware/DashboardScripts && /usr/bin/cvs -q up -dAP
 
0 0 * * * /Users/kitware/DashboardScripts/factory.sh > /Users/kitware/Dashboards/Logs/factory.log 2>&1
 
0 9 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_CTKAppLauncher_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_CTKAppLauncher_release_continuous.log 2>&1
 
0 9 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_slicer4_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_slicer4_release_continuous.log 2>&1
 
0 10 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_slicerextensions_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_slicerextensions_release_continuous.log 2>&1
 
0 10 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_slicerextensions_411_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_slicerextensions_411_release_continuous.log 2>&1
 
</pre>
 
  
The first tab is a command to update the cvs repository storing all of Kitware's dashboard scripts.  The second is a bash script which runs all of the nightly builds done by the Mac OS X factory.  The rest correspond to all the continuous builds.  An example of an entry in the bash script is as follows:
+
== computron.kitware ==
<pre>
 
open -a x11
 
export DISPLAY=:0.0 # just DISPLAY=:0.0 without export is not enough
 
export PATH=:/usr/local/git/bin:$PATH
 
  
CTEST=/Applications/CMake-3.5.0-rc3.app/Contents/bin/ctest
+
=== Software ===
LOG_DIR=/Users/kitware/Dashboards/Logs
 
DASHBOARD_SCRIPTS_DIR=/Users/kitware/DashboardScripts
 
  
# Nightly build of CTKAppLauncher
+
The factory machine runs Mac OS X 10.15.7 (19H1323)
"$CTEST" -S $DASHBOARD_SCRIPTS_DIR/factory-64bits_CTKAppLauncher_release_nightly.cmake -VV -O $LOG_DIR/factory-64bits_CTKAppLauncher_release_nightly.log
 
  
# Nightly build of slicer
+
==== Remote access ====
"$CTEST" -S $DASHBOARD_SCRIPTS_DIR/factory-64bits_slicer4_release_nightly.cmake -VV -O $LOG_DIR/factory-64bits_slicer4_release_nightly.log
 
  
</pre>
+
Remote access: Via SSH or VNC from Kitware internal network (10.171.2.179). Username is <code>svc-dashboard</code>
-->
 
 
 
<!--
 
The virtual  machines are run using the Parallels tool.
 
-->
 
  
 
=== Hardware ===
 
=== Hardware ===
  
* "Mac Pro (Mid 2010)" which has the 64-bit kernel enabled by default. See [http://support.apple.com/kb/HT3770 here] for details.
+
<pre>
 +
Model Name: Mac mini
 +
Model Identifier: Macmini8,1
 +
Processor Name: 6-Core Intel Core i7
 +
Processor Speed: 3.2 GHz
 +
Memory: 64 GB
  
==== Processor ====
+
Chipset Model: Intel UHD Graphics 630
* 2 x 2.93 GHz 6-Core Intel Xeon
+
Type: GPU
 
+
Bus: Built-In
==== Graphics Card ====
+
VRAM (Dynamic, Max): 1536 MB
* Chipset Model: ATI Radeon HD 5770
+
</pre>
* Type: GPU
 
* Bus PCIe
 
* VRAM (total): 1024MB
 
* EFI Driver Version: 01.00.436
 
 
 
==== Memory ====
 
* 8 x 4GB = 32GB
 
* Type: DDR3 ECC
 
* Speed: 1333 MHz
 
  
 
== factory-south.kitware ==
 
== factory-south.kitware ==
Line 86: Line 52:
 
* Tasks scheduled using [http://en.wikipedia.org/wiki/Cron cron]
 
* Tasks scheduled using [http://en.wikipedia.org/wiki/Cron cron]
  
The compiler is the following:
+
The compiler is the one provided by default after installing XCode:
 
<pre>
 
<pre>
 
Apple LLVM version 8.0.0 (clang-800.0.42.1)
 
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Line 92: Line 58:
 
</pre>
 
</pre>
  
 +
<!--
 
The virtual machines are run using the [http://www.parallels.com/ Parallels Desktop 11].
 
The virtual machines are run using the [http://www.parallels.com/ Parallels Desktop 11].
 
* Note that Parallels Desktop supports [http://kb.parallels.com/en/115487 OpenGL 2.1 only] (not higher)
 
* Note that Parallels Desktop supports [http://kb.parallels.com/en/115487 OpenGL 2.1 only] (not higher)
 +
-->
  
Remote access: Via SSH or VNC from Kitware internal network (10.171.2.166).
+
==== Remote access ====
 +
 
 +
Remote access: Via SSH or VNC from Kitware internal network (10.171.2.166). Username is <code>kitware</code>
  
 
=== Hardware ===
 
=== Hardware ===
Line 148: Line 118:
  
 
* Windows 7 Professional - Service Pack 1 (Build 7601)
 
* Windows 7 Professional - Service Pack 1 (Build 7601)
* CMake version: 3.9.0 (before 2017/07/04 was CMake 3.7.1)
+
* Compiler:
* Compiler: VS2013 Community Edition
+
** VS2013 Community Edition
 +
** VS2015 Pro Edition
 +
 
 +
==== Remote access ====
 +
 
 +
Remote access: Via VNC from Kitware internal network (192.168.113.173). Consider using the [https://www.realvnc.com/en/connect/download/viewer/ RealVNC] client. Username corresponds to your Kitware domain credentials.
 +
 
 +
=== metroplex.kitware ===
 +
 
 +
* Red Hat Enterprise Linux Workstation
 +
 
 +
<pre>
 +
Product ID:    71
 +
Version:        7.2
 +
Arch:          x86_64
 +
</pre>
 +
 
 +
* Docker community edition
 +
 
 +
==== Remote access ====
 +
 
 +
Remote access: Via SSH from Kitware internal network (192.168.113.208). Username is <code>kitware</code>
 +
 
 +
=Maintenance=
 +
 
 +
== CUDA Install notes ==
 +
 
 +
=== Linux ===
 +
 
 +
<pre>
 +
sudo yum -y install kernel-devel
 +
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
 +
 
 +
wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm
 +
sudo rpm -i cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm
 +
sudo yum clean all
 +
sudo yum -y install cuda
 +
</pre>
 +
 
 +
Refs:
 +
* https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=CentOS&target_version=7&target_type=rpmlocal
 +
* https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
 +
 
 +
=== Mac OS ===
 +
 
 +
Beware.  Not tested.  I don't have a mac.
 +
 
 +
<pre>
 +
wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_mac-dmg
 +
wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/patches/2/cuda_8.0.61.2_mac-dmg
 +
 
 +
dmgFilePath=cuda_8.0.61_mac-dmg
 +
MOUNTDEV=$(hdiutil mount $dmgFilePath | awk '/dev.disk/{print$1}')
 +
MOUNTDIR="$(mount | grep $MOUNTDEV | awk '{$1=$2="";sub(" [(].*","");sub("^  ","");print}')"
 +
sudo installer -pkg "${MOUNTDIR}/"*.pkg -target /
 +
hdiutil unmount "$MOUNTDIR"
 +
 
 +
dmgFilePath=cuda_8.0.61.2_mac-dmg
 +
MOUNTDEV=$(hdiutil mount $dmgFilePath | awk '/dev.disk/{print$1}')
 +
MOUNTDIR="$(mount | grep $MOUNTDEV | awk '{$1=$2="";sub(" [(].*","");sub("^  ","");print}')"
 +
sudo installer -pkg "${MOUNTDIR}/"*.pkg -target /
 +
hdiutil unmount "$MOUNTDIR"
 +
</pre>
 +
 
 +
Refs:
 +
* https://developer.nvidia.com/cuda-80-ga2-download-archive
 +
* https://docs.nvidia.com/cuda/archive/8.0/cuda-installation-guide-mac-os-x/index.html
 +
* https://stackoverflow.com/questions/22934083/install-dmg-package-on-mac-os-from-terminal
 +
 
 +
=== Windows ===
 +
 
 +
According to the docs, the "-s" option should allow unattended install.  However, I was unable to get this to work.
 +
 
 +
<pre>
 +
wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.168_425.25_windows.exe
 +
cuda_10.1.168_425.25_windows.exe -s
 +
</pre>
 +
 
 +
Refs:
 +
* https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64&target_version=7&target_type=exelocal
 +
* https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html
 +
 
 +
== Maintenance Guides ==
 +
 
 +
See https://github.com/Slicer/DashboardScripts#maintenance-guides
 +
 
 +
== Check Virtual Machines (factory-south-ubuntu) ==
 +
 
 +
<b>factory-south-win7 VM is shutdown, build have been moved to overload.kitware</b>
  
 +
=== When ===
  
=VM: Linux=
+
Every day (or as soon as issues are suspected)
 +
 
 +
=== What ===
 +
 
 +
* Try to:
 +
** SSH into factory-south-ubuntu
 
<!--
 
<!--
== factory.kitware ==
+
** RDP into factory-south-win7
* Ubuntu 11.04
+
-->
* CMake version: 2.8.8
+
 
* GCC version: Ubuntu 4.4.3-4ubuntu5.1
+
* If failure:
 +
** Connect to <code>factory.south</code> and check the state of the virtual machines.
 +
 
 +
* Are the virtual machine unresponsive ?
 +
 
 +
* '''No''' -> No further action required
 +
* '''Yes''' -> Read below
 +
** Kill both VMs
 +
** Restart the host (factory-south)
 +
** Manually start the virtual machines
 +
** If needed, manually restarts the nightly build (See below)
 +
 
 +
== Manually restart the night builds ==
 +
 
 +
If the factory machines were restarted after the regular Nightly startup time, the nightly scripts will have to be explicitly started.
 +
 
 +
* Linux: Run <code>/home/kitware/Dashboards/DashboardScripts/factory-south-ubuntu.sh</code>
 +
 
 +
* MacOSX: Run <code>/Users/kitware/DashboardScripts/factory.sh</code>
 +
 
 +
* Windows: Open "Task Scheduled" -> slicer -> right click on "Nightly" -> Start
 +
 
 +
== Nightly Build startup times ==
 +
 
 +
* Linux: 11:10pm
 +
 
 +
* MacOSx: 11:10pm
 +
 
 +
* Windows: Midnight
 +
 
 +
= Retired Hosts =
 +
 
 +
== factory.kitware (RETIRED) ==
 +
 
 +
=== Software ===
 +
The factory machine runs Mac OS X v10.6.8, Snow Leopard.
 +
* CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3 with [https://github.com/Kitware/CMake/compare/master...jcfr:osx-framework this patch])
 +
** Since CMake release >= 3.6 is compiled only for  >= 10.7, a custom build of cmake was done with <code>-DCMAKE_INSTALL_PREFIX:PATH=/Users/kitware/Dashboards/Support/CMake-3.9.0.app/Contents</code> 
 +
* Compiler llvm-3.1. Note that the default compiled shipped with XCode 4.2 is not used. See [http://slicer-devel.65872.n3.nabble.com/MacOSX-factory-build-Transitioning-from-hybrid-quot-llvm-g-quot-to-quot-clang-quot-Update-to-Qt-4-8-6-tt4032198.html#a4032199 here] for more details.
 
* Tasks scheduled using [http://en.wikipedia.org/wiki/Cron cron]
 
* Tasks scheduled using [http://en.wikipedia.org/wiki/Cron cron]
  
The scripts on this VM are very similar to those on the Host.
+
* Nightly generation of HomebrewCasks: [[Documentation/Nightly/Developers/Factory/HomeBrewInstall|Installation aborted]] (newer system is required for <tt>homebrew/cask-repair</tt>).
 +
 
 +
Remote access: Via SSH or VNC from Kitware internal network (192.168.113.230).
 +
<!--
 +
The output of <pre>crontab -l</pre> on the factory machine is as follows:
 +
<pre>
 +
55 23 * * * cd /Users/kitware/DashboardScripts && /usr/bin/cvs -q up -dAP
 +
0 0 * * * /Users/kitware/DashboardScripts/factory.sh > /Users/kitware/Dashboards/Logs/factory.log 2>&1
 +
0 9 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_CTKAppLauncher_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_CTKAppLauncher_release_continuous.log 2>&1
 +
0 9 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_slicer4_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_slicer4_release_continuous.log 2>&1
 +
0 10 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_slicerextensions_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_slicerextensions_release_continuous.log 2>&1
 +
0 10 * * * /Applications/CMake\ 2.8-8.app/Contents/bin/ctest -V -VV -S /Users/kitware/DashboardScripts/factory-64bits_slicerextensions_411_release_continuous.cmake > /Users/kitware/Dashboards/Logs/factory-64bits_slicerextensions_411_release_continuous.log 2>&1
 +
</pre>
 +
 
 +
The first tab is a command to update the cvs repository storing all of Kitware's dashboard scripts.  The second is a bash script which runs all of the nightly builds done by the Mac OS X factory.  The rest correspond to all the continuous builds.  An example of an entry in the bash script is as follows:
 +
<pre>
 +
open -a x11
 +
export DISPLAY=:0.0 # just DISPLAY=:0.0 without export is not enough
 +
export PATH=:/usr/local/git/bin:$PATH
 +
 
 +
CTEST=/Applications/CMake-3.5.0-rc3.app/Contents/bin/ctest
 +
LOG_DIR=/Users/kitware/Dashboards/Logs
 +
DASHBOARD_SCRIPTS_DIR=/Users/kitware/DashboardScripts
 +
 
 +
# Nightly build of CTKAppLauncher
 +
"$CTEST" -S $DASHBOARD_SCRIPTS_DIR/factory-64bits_CTKAppLauncher_release_nightly.cmake -VV -O $LOG_DIR/factory-64bits_CTKAppLauncher_release_nightly.log
  
=== Remote access ===
+
# Nightly build of slicer
This machine is accessible through SSH and remote VNC. It has been configured with its own static IP, and should be accessed by VNC through this IP instead of through a VNC connection on the Mac Host.  Do not try to connect through VNC to this machine on the Host.
+
"$CTEST" -S $DASHBOARD_SCRIPTS_DIR/factory-64bits_slicer4_release_nightly.cmake -VV -O $LOG_DIR/factory-64bits_slicer4_release_nightly.log
 +
 
 +
</pre>
 +
-->
  
===Known issues===
+
<!--
* X crash / GLX" missing on display. See [http://www.na-mic.org/Bug/view.php?id=2468 #2468]: '''FIXED'''
+
The virtual  machines are run using the Parallels tool.
* Beside the fact Parallel has been updated, the X server still crash and auto-login does NOT succeed. See [http://www.na-mic.org/Bug/view.php?id=2624 #2624]: '''FIXED'''
 
 
-->
 
-->
  
 +
=== Hardware ===
 +
 +
* "Mac Pro (Mid 2010)" which has the 64-bit kernel enabled by default. See [http://support.apple.com/kb/HT3770 here] for details.
 +
 +
==== Processor ====
 +
* 2 x 2.93 GHz 6-Core Intel Xeon
 +
 +
==== Graphics Card ====
 +
* Chipset Model: ATI Radeon HD 5770
 +
* Type: GPU
 +
* Bus PCIe
 +
* VRAM (total): 1024MB
 +
* EFI Driver Version: 01.00.436
 +
 +
==== Memory ====
 +
* 8 x 4GB = 32GB
 +
* Type: DDR3 ECC
 +
* Speed: 1333 MHz
 +
 +
== factory-south-win7.kitware (RETIRED VM) ==
 +
 +
* Tasks are scheduled using the Windows Task Scheduler (Start->All Programs->Accessories->System Tools->Task Scheduler)
 +
* On Windows, instead of bash, we use batch scripts comprised of commands to call ctest to run all the nightly scripts. 
 +
 +
<b>This VM is shutdown, build have been moved to overload.kitware</b>
 +
 +
* Windows 7 Ultimate N - Service Pack 1 (Build 7601) on an x86 platform.
 +
* CMake version: 3.5.0-rc3
 +
* Compiler: VS2013 Community Edition
 +
* Processors: 6
 +
* Memory: 16GB
 +
 +
 +
 +
Also, keep in mind the [[Documentation/{{documentation/version}}/Developers/Tutorials/DashboardSetup#Remarks remarks]] about building a dashboard on a Windows machine.  These all apply here.
 +
 +
Remote access: Via Remote Desktop from Kitware internal network (10.171.2.245).
  
 
== factory-south-ubuntu.kitware (RETIRED) ==
 
== factory-south-ubuntu.kitware (RETIRED) ==
 +
 +
<b>This VM is shutdown, build have been moved to metroplex.kitware</b>
 +
 
* Ubuntu 10.04
 
* Ubuntu 10.04
 
* CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3)
 
* CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3)
Line 238: Line 407:
  
 
Remote access: Via SSH or VNC from Kitware internal network (10.171.2.113).
 
Remote access: Via SSH or VNC from Kitware internal network (10.171.2.113).
 
=VM: Windows 7=
 
 
* Tasks are scheduled using the Windows Task Scheduler (Start->All Programs->Accessories->System Tools->Task Scheduler)
 
* On Windows, instead of bash, we use batch scripts comprised of commands to call ctest to run all the nightly scripts. 
 
 
== factory-south-win7.kitware ==
 
 
<b>This VM is shutdown, build have been moved to overload.kitware</b>
 
 
* Windows 7 Ultimate N - Service Pack 1 (Build 7601) on an x86 platform.
 
* CMake version: 3.5.0-rc3
 
* Compiler: VS2013 Community Edition
 
* Processors: 6
 
* Memory: 16GB
 
 
 
 
Also, keep in mind the [[Documentation/{{documentation/version}}/Developers/Tutorials/DashboardSetup#Remarks remarks]] about building a dashboard on a Windows machine.  These all apply here.
 
 
Remote access: Via Remote Desktop from Kitware internal network (10.171.2.245).
 
 
=Maintenance=
 
 
== Check Virtual Machines (factory-south-ubuntu) ==
 
 
<b>factory-south-win7 VM is shutdown, build have been moved to overload.kitware</b>
 
 
=== When ===
 
 
Every day (or as soon as issues are suspected)
 
 
=== What ===
 
 
* Try to:
 
** SSH into factory-south-ubuntu
 
<!--
 
** RDP into factory-south-win7
 
-->
 
 
* If failure:
 
** Connect to <code>factory.south</code> and check the state of the virtual machines.
 
 
* Are the virtual machine unresponsive ?
 
 
* '''No''' -> No further action required
 
* '''Yes''' -> Read below
 
** Kill both VMs
 
** Restart the host (factory-south)
 
** Manually start the virtual machines
 
** If needed, manually restarts the nightly build (See below)
 
 
== Manually restart the night builds ==
 
 
If the factory machines were restarted after the regular Nightly startup time, the nightly scripts will have to be explicitly started.
 
 
* Linux: Run <code>/home/kitware/Dashboards/DashboardScripts/factory-south-ubuntu.sh</code>
 
 
* MacOSX: Run <code>/Users/kitware/DashboardScripts/factory.sh</code>
 
 
* Windows: Open "Task Scheduled" -> slicer -> right click on "Nightly" -> Start
 
 
== Nightly Build startup times ==
 
 
* Linux: 11:10pm
 
 
* MacOSx: 11:10pm
 
 
* Windows: Midnight
 

Latest revision as of 18:05, 30 September 2021

Home < Documentation < Nightly < Developers < Factory


For the latest Slicer documentation, visit the read-the-docs.


Overview

The Slicer factory machine is maintained by Kitware and produces dashboard builds of many software projects, including nightly, continuous, and packaged versions of Slicer and its approved extensions. See the Slicer4 dashboard.

These builds are generated every day on three different operating systems: Windows 7, Mac OS X, and Linux. Because it is not legal to run any version of Mac OS X on a virtual machine (unless its host also runs Mac OS X), the host OS is Mac OS X, and the Windows and Linux builds are run within this machine as virtual machines.

Required factory components

See dashboard prerequisites.

Build scripts

Dashboard build scripts are located on GitHub: https://github.com/Slicer/DashboardScripts

Remote Access

To be granted remote access to the Slicer factories maintained and hosted by Kitware, contact https://discourse.slicer.org/u/sam_horvath and https://discourse.slicer.org/u/jcfr

Host(s)

computron.kitware

Software

The factory machine runs Mac OS X 10.15.7 (19H1323)

Remote access

Remote access: Via SSH or VNC from Kitware internal network (10.171.2.179). Username is svc-dashboard

Hardware

Model Name: Mac mini
Model Identifier: Macmini8,1
Processor Name: 6-Core Intel Core i7
Processor Speed: 3.2 GHz
Memory: 64 GB

Chipset Model: Intel UHD Graphics 630
Type: GPU
Bus: Built-In
VRAM (Dynamic, Max): 1536 MB

factory-south.kitware

Software

The factory machine runs Mac OS X 10.11.6 (15G31), El Capitan

  • Tasks scheduled using cron

The compiler is the one provided by default after installing XCode:

Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0


Remote access

Remote access: Via SSH or VNC from Kitware internal network (10.171.2.166). Username is kitware

Hardware

Processor

  • 2 x 2.4 GHz 6-Core Intel Xeon

Graphics Card

  • Chipset Model: ATI Radeon HD 5770
  • Type: GPU
  • Bus PCIe
  • VRAM (total): 1024MB
  • EFI Driver Version: 01.00.436

Memory

  • 8 x 8GB = 64 GB
  • Type: DDR3 ECC
  • Speed: 1333 MHz

overload.kitware and metroplex.kitware

Kitware build servers metroplex-and-overload.jpg


Software

  • overload: Windows 7 Professional 64-bit
  • metroplex: Red Hat Enterprise Linux WS v7.0

Hardware

  • Dell Precision R7910 Motherboard,BW

Processor

  • Dual Intel® Xeon® Processor E5-2630 v3 (8C, 2.4GHz, Turbo, HT, 20M, 85W)

Graphics Card

  • NVIDIA® Quadro® K620 2GB (DP, DL-DVI-I) (1 DP to SL-DVI adapter)

Memory

  • 32GB (8x4GB) 2133MHz DDR4 RDIMM ECC

Storage

  • 2 X Integrated Intel 4 port AHCI SATA chipset controllers (tot 8 X 6.0Gb/s) SW RAID 0,1,5,10
  • 1 x 1TB Dell 4*Drive PCIe x16 M.2 Solid State Drive Card (boot)

overload.kitware

  • Windows 7 Professional - Service Pack 1 (Build 7601)
  • Compiler:
    • VS2013 Community Edition
    • VS2015 Pro Edition

Remote access

Remote access: Via VNC from Kitware internal network (192.168.113.173). Consider using the RealVNC client. Username corresponds to your Kitware domain credentials.

metroplex.kitware

  • Red Hat Enterprise Linux Workstation
Product ID:     71
Version:        7.2
Arch:           x86_64
  • Docker community edition

Remote access

Remote access: Via SSH from Kitware internal network (192.168.113.208). Username is kitware

Maintenance

CUDA Install notes

Linux

sudo yum -y install kernel-devel
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm
sudo rpm -i cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm
sudo yum clean all
sudo yum -y install cuda

Refs:

Mac OS

Beware. Not tested. I don't have a mac.

wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_mac-dmg
wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/patches/2/cuda_8.0.61.2_mac-dmg

dmgFilePath=cuda_8.0.61_mac-dmg
MOUNTDEV=$(hdiutil mount $dmgFilePath | awk '/dev.disk/{print$1}')
MOUNTDIR="$(mount | grep $MOUNTDEV | awk '{$1=$2="";sub(" [(].*","");sub("^  ","");print}')"
sudo installer -pkg "${MOUNTDIR}/"*.pkg -target /
hdiutil unmount "$MOUNTDIR"

dmgFilePath=cuda_8.0.61.2_mac-dmg
MOUNTDEV=$(hdiutil mount $dmgFilePath | awk '/dev.disk/{print$1}')
MOUNTDIR="$(mount | grep $MOUNTDEV | awk '{$1=$2="";sub(" [(].*","");sub("^  ","");print}')"
sudo installer -pkg "${MOUNTDIR}/"*.pkg -target /
hdiutil unmount "$MOUNTDIR"

Refs:

Windows

According to the docs, the "-s" option should allow unattended install. However, I was unable to get this to work.

wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.168_425.25_windows.exe
cuda_10.1.168_425.25_windows.exe -s

Refs:

Maintenance Guides

See https://github.com/Slicer/DashboardScripts#maintenance-guides

Check Virtual Machines (factory-south-ubuntu)

factory-south-win7 VM is shutdown, build have been moved to overload.kitware

When

Every day (or as soon as issues are suspected)

What

  • Try to:
    • SSH into factory-south-ubuntu
  • If failure:
    • Connect to factory.south and check the state of the virtual machines.
  • Are the virtual machine unresponsive ?
  • No -> No further action required
  • Yes -> Read below
    • Kill both VMs
    • Restart the host (factory-south)
    • Manually start the virtual machines
    • If needed, manually restarts the nightly build (See below)

Manually restart the night builds

If the factory machines were restarted after the regular Nightly startup time, the nightly scripts will have to be explicitly started.

  • Linux: Run /home/kitware/Dashboards/DashboardScripts/factory-south-ubuntu.sh
  • MacOSX: Run /Users/kitware/DashboardScripts/factory.sh
  • Windows: Open "Task Scheduled" -> slicer -> right click on "Nightly" -> Start

Nightly Build startup times

  • Linux: 11:10pm
  • MacOSx: 11:10pm
  • Windows: Midnight

Retired Hosts

factory.kitware (RETIRED)

Software

The factory machine runs Mac OS X v10.6.8, Snow Leopard.

  • CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3 with this patch)
    • Since CMake release >= 3.6 is compiled only for >= 10.7, a custom build of cmake was done with -DCMAKE_INSTALL_PREFIX:PATH=/Users/kitware/Dashboards/Support/CMake-3.9.0.app/Contents
  • Compiler llvm-3.1. Note that the default compiled shipped with XCode 4.2 is not used. See here for more details.
  • Tasks scheduled using cron
  • Nightly generation of HomebrewCasks: Installation aborted (newer system is required for homebrew/cask-repair).

Remote access: Via SSH or VNC from Kitware internal network (192.168.113.230).


Hardware

  • "Mac Pro (Mid 2010)" which has the 64-bit kernel enabled by default. See here for details.

Processor

  • 2 x 2.93 GHz 6-Core Intel Xeon

Graphics Card

  • Chipset Model: ATI Radeon HD 5770
  • Type: GPU
  • Bus PCIe
  • VRAM (total): 1024MB
  • EFI Driver Version: 01.00.436

Memory

  • 8 x 4GB = 32GB
  • Type: DDR3 ECC
  • Speed: 1333 MHz

factory-south-win7.kitware (RETIRED VM)

  • Tasks are scheduled using the Windows Task Scheduler (Start->All Programs->Accessories->System Tools->Task Scheduler)
  • On Windows, instead of bash, we use batch scripts comprised of commands to call ctest to run all the nightly scripts.

This VM is shutdown, build have been moved to overload.kitware

  • Windows 7 Ultimate N - Service Pack 1 (Build 7601) on an x86 platform.
  • CMake version: 3.5.0-rc3
  • Compiler: VS2013 Community Edition
  • Processors: 6
  • Memory: 16GB


Also, keep in mind the Documentation/Nightly/Developers/Tutorials/DashboardSetup#Remarks remarks about building a dashboard on a Windows machine. These all apply here.

Remote access: Via Remote Desktop from Kitware internal network (10.171.2.245).

factory-south-ubuntu.kitware (RETIRED)

This VM is shutdown, build have been moved to metroplex.kitware

  • Ubuntu 10.04
  • CMake version: 3.9.0 (before 2017/07/04 was 3.5.0-rc3)
  • GCC version: Ubuntu 4.4.3-4ubuntu5.1
  • Tasks scheduled using cron
  • Processors: 5
  • Memory: 10GB
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 10.04.4 LTS
Release:	10.04
Codename:	lucid

These two compilers are available:

gcc/g++ 4.4.3

/usr/bin/cpp-4.4
/usr/bin/g++-4.4
/usr/bin/gcc-4.4
/usr/bin/gcov-4.4
/usr/bin/x86_64-linux-gnu-cpp-4.4
/usr/bin/x86_64-linux-gnu-g++-4.4
/usr/bin/x86_64-linux-gnu-gcc-4.4
  • In the PATH:
$ gcc --version
gcc (Ubuntu 4.4.3-4ubuntu5.1) 4.4.3
$ g++ --version
g++ (Ubuntu 4.4.3-4ubuntu5.1) 4.4.3

gcc/g++ 4.6.4

/usr/bin/cpp-4.6
/usr/bin/g++-4.6
/usr/bin/gcc-4.6
/usr/bin/gcov-4.6
/usr/bin/x86_64-linux-gnu-cpp-4.6
/usr/bin/x86_64-linux-gnu-g++-4.6
/usr/bin/x86_64-linux-gnu-gcc-4.6

Installed following these instructions:

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get update
sudo apt-get install gcc-4.6
sudo apt-get install g++-4.6

Remote access

Remote access: Via SSH or VNC from Kitware internal network (10.171.2.113).