Difference between revisions of "Documentation/Labs/SlicerCondaIntegration"

From Slicer Wiki
Jump to: navigation, search
 
(33 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
Slicer currently builds a custom python distribution from source, but not all useful packages are available.  But [https://store.continuum.io/cshop/anaconda/ Anaconda] provides a set of binaries that might provide a more comprehensive ecosystem.
 
Slicer currently builds a custom python distribution from source, but not all useful packages are available.  But [https://store.continuum.io/cshop/anaconda/ Anaconda] provides a set of binaries that might provide a more comprehensive ecosystem.
  
Before attempting to use the whole Anaconda package, the first approach has been to use Miniconda, which is a reduced version of Anaconda.  
+
Instead of attempting to use the whole Anaconda package, our approach will be to use Miniconda, a reduced version of Anaconda, and (1) install a pre-defined set of packages and (2) provide a way to easily install new ones.  
  
Note: the platform used is MacOSX 10.9.5 with XCode 6
+
== Update 2016-01-22 (Jorge) ==
 +
After a follow up with JC about this topic, and due to the apparent complexity of this task, it has been decided to make a small survey to have a better idea of which would be the most needed packages for the community from those ones available in Anaconda.
 +
 
 +
The link to indicate your preferences is: https://docs.google.com/spreadsheets/d/1j4iugjs0GkHEn1elUbceMW2KJ21FM-QrYoFtZ4P9yOc/edit?usp=sharing
 +
 
 +
== Update 2015-10-10 ==
 +
 
 +
We (Kitware) did few experiments where we copied the anaconda numpy and SciPy packages into a Slicer build tree and run the associated tests. The link below summarizes the results.
 +
 
 +
It turns out that there are a lot of issues with this approach.
 +
 
 +
See https://goo.gl/VNy7yV
  
==Compile Slicer against miniconda==
+
Next step: We suggest to continue with the initial approach where Slicer is built against miniconda Python libraries.
  
=== Mac ===
+
== Virtual Hackfest: August 24, 9am ==
**First approach: install Miniconda manually and then build Slicer against it
 
  
Assuming that we already installed Miniconda in the default folder ({HOME}/miniconda), we could build Slicer against it:
+
=== Info ===
  
<pre>
+
Location: Google hangout (connection details will be shared on the Slicer developer list)
cmake -DCMAKE_BUILD_TYPE:STRING=Debug \
 
-DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
 
-DPYTHON_EXECUTABLE:FILEPATH=${HOME}/miniconda/bin/python \
 
-DPYTHON_INCLUDE_DIR:PATH=${HOME}/miniconda/include/python2.7 \
 
-DPYTHON_LIBRARY:FILEPATH=${HOME}/miniconda/lib/libpython2.7.dylib \
 
-DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 \
 
-DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk  \
 
-DSlicer_BUILD_CLI:BOOL=OFF \
 
-DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 
../Slicer
 
</pre>
 
  
At the moment there are also some additional actions to take, like setting some environment variables:
+
What: Work on Slicer and Conda integration
  export PYTHONHOME="${HOME}/miniconda/"
 
  
It is also possible that you have to set another ones like:
+
=== Proposed tasks ===
 +
* <s>Review and consolidate contributions of Jorge and Adam</s> {{done}}
 +
* <s>Implement miniconda external project files as outlined [[#Next_steps|below]]</s> {{done}}
 +
* Create a Slicer module providing:
 +
** (1) scripted interface to install additional conda package
 +
** (2) UI to provide similar functionality
 +
* Update [[Documentation/Nightly/Developers/Extensions/DescriptionFile|extension description format]] to specify python dependencies that could be installed either at build/package time or at run time.
  
  export PYTHONPATH="{HOME}/miniconda/:{HOME}/miniconda/lib/:{HOME}/miniconda/lib/python2.7/site-packages/:{HOME}/miniconda/lib/python2.7/lib-dynload/
+
=== Links ===
  :{HOME}/miniconda/lib/python2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg/SimpleITK/:{HOME}/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/"
 
 
 
  export DYLD_LIBRARY_PATH="{HOME}/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/SimpleITK-build/lib/:{HOME}/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/"
 
  
Besides, it is necessary to fix rpath in the library:
+
* Jorge:
 +
** https://github.com/jonieva/Slicer/tree/Conda
 +
** http://slicer-devel.65872.n3.nabble.com/Compiling-Slicer-with-Anaconda-td4033321.html
 +
* Adam:
 +
** https://github.com/adamrankin/SlicerTAVI
 +
** http://slicer-devel.65872.n3.nabble.com/Re-Slicer-build-with-custom-python-td4034662.html
  
  install_name_tool -id {HOME}/miniconda/lib/libpython2.7.dylib {HOME}/miniconda/lib/libpython2.7.dylib
 
  Note: the paths have to be absolute, so you want to replace {HOME} for your home folder.
 
  
**Second approach: to build a "SlicerPython" using Miniconda
+
==Compile Slicer against miniconda==
I created a fork from Slicer to implement this approach, packaging Miniconda and installing it in a similar way that SlicerPython is used now. You can download the code in https://github.com/jonieva/Slicer/tree/Conda
 
  
To build with this code:
+
=== Mac ===
  
  cmake -DCMAKE_BUILD_TYPE:STRING=Debug -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9
+
Note: the platform used is MacOSX 10.9.5 with XCode 6
  -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk
 
  -DSlicer_BUILD_CLI:BOOL=OFF -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF -DSlicer_USE_CONDA_python:BOOL=ON ../Slicer
 
  
The only changes are in Superbuild/External_python.cmake file. There is a commented block that you can use to implement the first approach.
+
* Download the source code hosted in https://github.com/jonieva/Slicer/tree/Conda
 +
* Build (assuming MacOSX 10.9):
 +
  cmake \
 +
  '''-DSlicer_USE_CONDA_python:BOOL=ON''' \
 +
  -DCMAKE_BUILD_TYPE:STRING=Debug \
 +
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
 +
  -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 \
 +
  -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk \
 +
  -DSlicer_BUILD_CLI:BOOL=OFF \
 +
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
  
However, at the moment I get a build error:
+
The only changes in the source code are in Superbuild/External_python.cmake and in Superbuild/External_NUMPY.cmake files (in the current version Slicer will use Anaconda Numpy).
 +
For the sake of testing, the next Python Conda modules are installed by default: numpy, scikit-learn.
  
 +
====Issues====
 +
* libsqlite3.dylib library in miniconda has a conflict with CTK:
 +
<pre>
 
   Compiling python scripts: CTKScriptingPythonCore
 
   Compiling python scripts: CTKScriptingPythonCore
 
   Compiling python scripts: ctkSimplePythonShell
 
   Compiling python scripts: ctkSimplePythonShell
Line 64: Line 79:
 
     Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
 
     Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
 
   in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 
   in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 +
</pre>
  
'''NOTE''': This problem can be "solved" if we replace the libsqlite3.dylib by a link to the /usr/lib/libsqlite3.dylib library, or even just removing the library from Miniconda. There is a hack included in the External_python.cmake file to this end.
+
  Workaround: delete or rename the file (there is a hack included in the External_python.cmake file to this end).
  
Also, when I run Slicer.app the program is not using Miniconda, but my System Python, so it doesn't find the right libraries:
+
* When running Slicer.app (not the binary file in Slicer-Superbuild-Conda/Slicer-build) the program is not using Miniconda, but the System Python, so it doesn't find the right libraries:
 
<pre>
 
<pre>
 
Python 2.7.5 (default, Mar  9 2014, 22:15:05)  
 
Python 2.7.5 (default, Mar  9 2014, 22:15:05)  
Line 86: Line 102:
 
</pre>
 
</pre>
  
** "Pre-release"
+
So, in order that the system works, '''Slicer must be started from a terminal running the file Slicer-Superbuild-Conda/Slicer-build/Slicer'''
With all the mentioned fixes, Slicer should build against Miniconda using the branch "Conda" in https://github.com/jonieva/Slicer/tree/Conda and using this command (with your right OSX version):
+
 
 +
 
 +
=== Linux ===
 +
 
 +
* Download the source code in https://github.com/jonieva/Slicer/tree/Conda
 +
* Build Slicer normally, except for the setting Slicer_USE_CONDA_python. Example:
 +
 
 
<pre>
 
<pre>
cmake -DCMAKE_BUILD_TYPE:STRING=Debug -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9
+
cmake \
-DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk
+
  '''-DSlicer_USE_CONDA_python:BOOL=ON''' \
-DSlicer_BUILD_CLI:BOOL=OFF -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF -DSlicer_USE_CONDA_python:BOOL=ON ../Slicer  
+
  -DCMAKE_BUILD_TYPE:STRING=Debug \
 +
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake
 +
  -DSlicer_BUILD_CLI:BOOL=OFF \
 +
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
 
</pre>
 
</pre>
  
The application should be run from the binary in Slicer-build/Slicer, and as an example, Conda numpy and scikit-learn modules are installed.
 
  
=== Linux ===
+
====Different approach====
 +
 
 +
===== 2015-02-04 - Steve Pieper =====
  
 
The following worked for me (Steve Pieper, 2015-02-04) using ubuntu 13.10 and miniconda  3.8.4-py27_0
 
The following worked for me (Steve Pieper, 2015-02-04) using ubuntu 13.10 and miniconda  3.8.4-py27_0
Line 152: Line 179:
 
</pre>
 
</pre>
  
-- Configuring incomplet
+
-- Configuring incomplete
  
** Building a Slicer-Python with Miniconda
+
===== 2015-03-02 - Jc =====
It should work without problems if you download the branch "Conda" in https://github.com/jonieva/Slicer/tree/Conda and running this command:
+
 
<pre>
+
2015-03-02(Jc): To simplify even further, using Slicer >= r24007, the following is expected to work. It is not needed to manually remove Numpy.
cmake -DCMAKE_BUILD_TYPE:STRING=Debug -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake -DSlicer_BUILD_CLI:BOOL=OFF
 
-DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF -DSlicer_USE_CONDA_python:BOOL=ON ../Slicer
 
</pre>
 
  
=== Windows ===
+
This is achieved by specifying <code>-DSlicer_USE_SYSTEM_NUMPY:BOOL=1</code> and also by specifying all dependencies of NUMPY (python, zlib, OpenSSL and CTKAPPLAUNCHER) should not be expected on the system.
  
TBD
+
Note also that since I built Qt against the latest version of OpenSSL in the 1.0.1 series, I explicitly add <code>-DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l</code>
  
== Compile Slicer using System Python ==
 
In order to compare the behaviors, I also compiled another version of Slicer using my System Python:
 
  
  ccmake -DCMAKE_BUILD_TYPE:STRING=Debug -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake  -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.
+
<pre>
  -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk 
+
QT_QMAKE_EXECUTABLE=/home/jcfr/Support/qt-everywhere-opensource-release-build-4.8.6/bin/qmake
  -G  "Eclipse CDT4 - Unix Makefiles" -DSlicer_USE_SYSTEM_python:BOOL=TRUE ../Slicer
+
CONDA_ROOT=/home/jcfr/Projects/sandbox
  
Checking my System Python folder, everything seems to be properly installed.
+
cmake \
 +
  -DCMAKE_BUILD_TYPE:STRING=Release \
 +
  -DQT_QMAKE_EXECUTABLE:FILEPATH=${QT_QMAKE_EXECUTABLE} \
 +
  -DPYTHON_EXECUTABLE:FILEPATH=${CONDA_ROOT}/miniconda/bin/python \
 +
  -DPYTHON_INCLUDE_DIR:PATH=${CONDA_ROOT}/miniconda/include/python2.7 \
 +
  -DPYTHON_LIBRARY:FILEPATH=${CONDA_ROOT}/miniconda/lib/libpython2.7.so  \
 +
  -DSlicer_USE_PYTHONQT_WITH_OPENSSL:BOOL=1 \
 +
  -DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l \
 +
  -DSlicer_USE_SYSTEM_CTKAPPLAUNCHER:BOOL=0 \
 +
  -DSlicer_USE_SYSTEM_OpenSSL:BOOL=0 \
 +
  -DSlicer_USE_SYSTEM_zlib:BOOL=0 \
 +
  -DSlicer_USE_SYSTEM_python:BOOL=0 \
 +
  -DSlicer_USE_SYSTEM_NUMPY:BOOL=1 \
 +
  -DSlicer_BUILD_CLI:BOOL=OFF \
 +
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
  
== Common issues in Miniconda and System python ==
+
nice make -j4
There is a difference between executing '''Slicer-Superbuild/Slicer-build/bin/Slicer.app''' and '''Slicer-Superbuild/Slicer-build/Slicer''' when I open a Slicer Python console. In the former case, Slicer cannot find some libraries. When I open the console I get:
 
  
  Python 2.7.5 (default, Mar  9 2014, 22:15:05)
+
</pre>
  [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
 
  >>> Traceback (most recent call last):
 
  File "<string>", line 1, in <module>
 
  File "/Users/Jorge/Projects/BWH/Slicer-Superbuild-SystemPython/Slicer-build/lib/Slicer-4.4/qt-scripted-modules/SimpleFilters.py", line 3, in <module>
 
    import SimpleITK as sitk
 
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/__init__.py", line 1, in <module>
 
    from .SimpleITK import *
 
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 28, in <module>
 
    _SimpleITK = swig_import_helper()
 
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 24, in swig_import_helper
 
    _mod = imp.load_module('_SimpleITK', fp, pathname, description)
 
  ImportError: dlopen(/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so, 2): Library not loaded: libSimpleITKBasicFilters12-0.9.1.dylib
 
  Referenced from: /Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so
 
  Reason: image not found
 
  
If I run Slicer-Superbuild/Slicer-build/Slicer, everything seems to work fine.
+
=== Windows ===
  
However, '''in both cases Slicer is loading my System python'''!
+
Although the code in https://github.com/jonieva/Slicer/tree/Conda should work with Windows too (using the setting Slicer_USE_CONDA_python=ON), it has not been tested at the moment.
  
== Next step: isolating the Python console problem ==
 
Following the advice of Steve Pieper, I tried to replicate the problem just in the CTK Python console, without running Slicer. To do that, I enabled the setting CTK_APP_ctkSimplePythonShell, inside Slicer-Superbuild/CTK-build/CTK-build.
 
I had exactly the same behavior that I had within Slicer.
 
  
I tried to set exactly the same settings in my "Slicer Python" version and in my "Miniconda" version, replacing just the settings PYTHON_EXECUTABLE, PYTHON_INCLUDE_DIR and PYTHON_LIBRARY. Still some behavior...
+
==Compile Slicer and python 2.7.10 + drop in conda binary packages ==
  
In this process, I found out two things that are different in the "Slicer Python" and "Miniconda":
+
=== Initial experiments ===
:If I run otool in my ctkSimplePythonShell, there is a difference in libpython2.7.dylib
 
~/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin$ otool -L ctkSimplePythonShell
 
  ctkSimplePythonShell:
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKScriptingPythonWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKVisualizationVTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/QtTesting-build/libQtTesting.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKScriptingPythonCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkInteraction-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkFiltersHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkOpenGL-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkImagingHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkIO-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 20.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkTestingRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkFilters-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkalglib-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkverdict-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkftgl-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkfreetype-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 48.0.0)
 
  /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 59.0.0)
 
  /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkImaging-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CMakeExternals/Install/lib/libPythonQt.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkWrappingPython27Core-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkCommon-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  '''/Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/python-install/lib/libpython2.7.dylib (compatibility version 0.0.0, current version 0.0.0)'''
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtksys-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  QtOpenGL.framework/Versions/4/QtOpenGL (compatibility version 4.8.0, current version 4.8.6)
 
  QtXmlPatterns.framework/Versions/4/QtXmlPatterns (compatibility version 4.8.0, current version 4.8.6)
 
  QtGui.framework/Versions/4/QtGui (compatibility version 4.8.0, current version 4.8.6)
 
  QtXml.framework/Versions/4/QtXml (compatibility version 4.8.0, current version 4.8.6)
 
  QtSql.framework/Versions/4/QtSql (compatibility version 4.8.0, current version 4.8.6)
 
  QtNetwork.framework/Versions/4/QtNetwork (compatibility version 4.8.0, current version 4.8.6)
 
  QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.6)
 
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
 
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
 
  
 +
TBD
  
 +
== Next steps ==
  
~/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin$ otool -L ctkSimplePythonShell
+
=== Phase 1 ===
ctkSimplePythonShell:
+
Copied from the conclusion of the [[Developer_Meetings/20150324#Conclusions|Developer hangout of March 24th 2015]]
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKScriptingPythonWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKVisualizationVTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/QtTesting-build/libQtTesting.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKScriptingPythonCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkInteraction-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkFiltersHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkOpenGL-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkImagingHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkIO-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 20.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkTestingRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkFilters-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkalglib-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkverdict-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkftgl-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkfreetype-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 48.0.0)
 
  /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 59.0.0)
 
  /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkImaging-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CMakeExternals/Install/lib/libPythonQt.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkWrappingPython27Core-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkCommon-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  '''libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)'''
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtksys-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  QtOpenGL.framework/Versions/4/QtOpenGL (compatibility version 4.8.0, current version 4.8.6)
 
  QtXmlPatterns.framework/Versions/4/QtXmlPatterns (compatibility version 4.8.0, current version 4.8.6)
 
  QtGui.framework/Versions/4/QtGui (compatibility version 4.8.0, current version 4.8.6)
 
  QtXml.framework/Versions/4/QtXml (compatibility version 4.8.0, current version 4.8.6)
 
  QtSql.framework/Versions/4/QtSql (compatibility version 4.8.0, current version 4.8.6)
 
  QtNetwork.framework/Versions/4/QtNetwork (compatibility version 4.8.0, current version 4.8.6)
 
  QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.6)
 
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
 
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
 
  
So, it looks like there is a problem with the rpath of the library. Even when I tried to copy the python executable and the library itself to the same bin folder where the ctkSimplePython is (Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin), I still got the same behavior.  
+
<s>
Following JC advice (thanks!), I used '''instal_name_tool''' to add manually the rpath to libpython2.7.dylib:
+
* Python conda - next steps:
  install_name_tool -id /Users/Jorge/miniconda/lib/libpython2.7.dylib /Users/Jorge/miniconda/lib/libpython2.7.dylib
+
** Create External_miniconda.cmake
 +
** Add a variable named EXTERNAL_PYTHON_NAME that could take two values: python or miniconda
 +
*** In the SuperBuild.cmake, an option named  EXTERNAL_PYTHON_NAME could be added. By default, it would be set to "python" but could switch to "miniconda"
 +
** Projects like External_{CTK, VTK}.cmake would be updated to depend on ${EXTERNAL_PYTHON_NAME}
 +
** For now, let's consider only the case OpenSSL disabled.
 +
</s>
 +
{{done}} See https://github.com/jcfr/Slicer/tree/slicer-miniconda
  
Voila! Now we can rebuild and the ctkSimplePythonShell will use the correct Python! However, we will have to set up some environment variables in order than all the libraries are found. I tried some of the most important and checked that they were properly loaded in the Python console, but we will need the same ones that we are adding when building the current "SlicerPython":
+
=== Phase 2 ===
  export PYTHONHOME="/Users/Jorge/miniconda/"
 
  export PYTHONPATH="/Users/Jorge/miniconda/:/Users/Jorge/miniconda/lib/:/Users/Jorge/miniconda/lib/python2.7/site-packages/:
 
  /Users/Jorge/miniconda/lib/python2.7/lib-dynload/
 
  :/Users/Jorge/miniconda/lib/python2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg/SimpleITK/
 
  :/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/"
 
  export DYLD_LIBRARY_PATH="/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/SimpleITK-build/lib/
 
  :/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/"
 
  
I also checked that the Python console works inside Slicer, as long as I launch the Slicer binary from a terminal that has all these environment variables initialized.
+
TBD

Latest revision as of 22:19, 22 March 2016

Home < Documentation < Labs < SlicerCondaIntegration

Slicer currently builds a custom python distribution from source, but not all useful packages are available. But Anaconda provides a set of binaries that might provide a more comprehensive ecosystem.

Instead of attempting to use the whole Anaconda package, our approach will be to use Miniconda, a reduced version of Anaconda, and (1) install a pre-defined set of packages and (2) provide a way to easily install new ones.

Update 2016-01-22 (Jorge)

After a follow up with JC about this topic, and due to the apparent complexity of this task, it has been decided to make a small survey to have a better idea of which would be the most needed packages for the community from those ones available in Anaconda.

The link to indicate your preferences is: https://docs.google.com/spreadsheets/d/1j4iugjs0GkHEn1elUbceMW2KJ21FM-QrYoFtZ4P9yOc/edit?usp=sharing

Update 2015-10-10

We (Kitware) did few experiments where we copied the anaconda numpy and SciPy packages into a Slicer build tree and run the associated tests. The link below summarizes the results.

It turns out that there are a lot of issues with this approach.

See https://goo.gl/VNy7yV

Next step: We suggest to continue with the initial approach where Slicer is built against miniconda Python libraries.

Virtual Hackfest: August 24, 9am

Info

Location: Google hangout (connection details will be shared on the Slicer developer list)

What: Work on Slicer and Conda integration

Proposed tasks

  • Review and consolidate contributions of Jorge and Adam Check.svg
  • Implement miniconda external project files as outlined below Check.svg
  • Create a Slicer module providing:
    • (1) scripted interface to install additional conda package
    • (2) UI to provide similar functionality
  • Update extension description format to specify python dependencies that could be installed either at build/package time or at run time.

Links


Compile Slicer against miniconda

Mac

Note: the platform used is MacOSX 10.9.5 with XCode 6

 cmake \
 -DSlicer_USE_CONDA_python:BOOL=ON \
 -DCMAKE_BUILD_TYPE:STRING=Debug \
 -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
 -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 \
 -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk \	
 -DSlicer_BUILD_CLI:BOOL=OFF \
 -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \ 
 ../Slicer

The only changes in the source code are in Superbuild/External_python.cmake and in Superbuild/External_NUMPY.cmake files (in the current version Slicer will use Anaconda Numpy). For the sake of testing, the next Python Conda modules are installed by default: numpy, scikit-learn.

Issues

  • libsqlite3.dylib library in miniconda has a conflict with CTK:
  Compiling python scripts: CTKScriptingPythonCore
  Compiling python scripts: ctkSimplePythonShell
  Linking CXX shared module ../../bin/CTKCorePythonQt.so
    dyld: Symbol not found: _sqlite3_intarray_bind
  Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
  Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
    in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
  dyld: Symbol not found: _sqlite3_intarray_bind
    Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
    Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
  in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 Workaround: delete or rename the file (there is a hack included in the External_python.cmake file to this end).
  • When running Slicer.app (not the binary file in Slicer-Superbuild-Conda/Slicer-build) the program is not using Miniconda, but the System Python, so it doesn't find the right libraries:
Python 2.7.5 (default, Mar  9 2014, 22:15:05) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
>>> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda/Slicer-build/lib/Slicer-4.4/qt-scripted-modules/SimpleFilters.py", line 3, in <module>
    import SimpleITK as sitk
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/__init__.py", line 1, in <module>
    from .SimpleITK import *
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 28, in <module>
    _SimpleITK = swig_import_helper()
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 24, in swig_import_helper
    _mod = imp.load_module('_SimpleITK', fp, pathname, description)
ImportError: dlopen(/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so, 2): Library not loaded: libSimpleITKBasicFilters12-0.9.1.dylib
  Referenced from: /Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so
  Reason: image not found

So, in order that the system works, Slicer must be started from a terminal running the file Slicer-Superbuild-Conda/Slicer-build/Slicer


Linux

cmake \
  '''-DSlicer_USE_CONDA_python:BOOL=ON''' \
  -DCMAKE_BUILD_TYPE:STRING=Debug \
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \  
  -DSlicer_BUILD_CLI:BOOL=OFF \
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
  ../Slicer


Different approach

2015-02-04 - Steve Pieper

The following worked for me (Steve Pieper, 2015-02-04) using ubuntu 13.10 and miniconda 3.8.4-py27_0

Slicer-conda-linux-2015-02-04.png
cmake \
  -DCMAKE_BUILD_TYPE:STRING=Debug \
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
  -DPYTHON_EXECUTABLE:FILEPATH=${HOME}/miniconda/bin/python \
  -DPYTHON_INCLUDE_DIR:PATH=${HOME}/miniconda/include/python2.7 \
  -DPYTHON_LIBRARY:FILEPATH=${HOME}/miniconda/lib/libpython2.7.so  \
  -DSlicer_BUILD_CLI:BOOL=OFF \
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
  ../Slicer

make -j20

export PYTHONHOME=${USER}/miniconda

./Slicer-build/Slicer


But after it ran there was slicer's build of numpy inside miniconda, so I needed to do this:

bin/conda uninstall numpy
bin/conda install numpy

I tried using this:

 -DSlicer_USE_SYSTEM_NUMPY:BOOL=ON

but got this error:

...
-- SuperBuild -         CTKAPPLAUNCHER[OK] (SYSTEM)
CMake Error at SuperBuild/External_CTKAPPLAUNCHER.cmake:21 (message):
  Enabling Slicer_USE_SYSTEM_CTKAPPLAUNCHER is not supported !
Call Stack (most recent call first):
  CMake/ExternalProjectDependency.cmake:561 (include)
  SuperBuild/External_python.cmake:21 (ExternalProject_Include_Dependencies)
  CMake/ExternalProjectDependency.cmake:561 (include)
  SuperBuild/External_VTKv6.cmake:11 (ExternalProject_Include_Dependencies)
  CMake/ExternalProjectDependency.cmake:561 (include)
  SuperBuild/External_teem.cmake:11 (ExternalProject_Include_Dependencies)
  CMake/ExternalProjectDependency.cmake:561 (include)
  CMake/ExternalProjectDependency.cmake:607 (ExternalProject_Include_Dependencies)
  SuperBuild.cmake:320 (ExternalProject_Include_Dependencies)
  CMakeLists.txt:599 (include)

-- Configuring incomplete

2015-03-02 - Jc

2015-03-02(Jc): To simplify even further, using Slicer >= r24007, the following is expected to work. It is not needed to manually remove Numpy.

This is achieved by specifying -DSlicer_USE_SYSTEM_NUMPY:BOOL=1 and also by specifying all dependencies of NUMPY (python, zlib, OpenSSL and CTKAPPLAUNCHER) should not be expected on the system.

Note also that since I built Qt against the latest version of OpenSSL in the 1.0.1 series, I explicitly add -DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l


QT_QMAKE_EXECUTABLE=/home/jcfr/Support/qt-everywhere-opensource-release-build-4.8.6/bin/qmake
CONDA_ROOT=/home/jcfr/Projects/sandbox

cmake \
  -DCMAKE_BUILD_TYPE:STRING=Release \
  -DQT_QMAKE_EXECUTABLE:FILEPATH=${QT_QMAKE_EXECUTABLE} \
  -DPYTHON_EXECUTABLE:FILEPATH=${CONDA_ROOT}/miniconda/bin/python \
  -DPYTHON_INCLUDE_DIR:PATH=${CONDA_ROOT}/miniconda/include/python2.7 \
  -DPYTHON_LIBRARY:FILEPATH=${CONDA_ROOT}/miniconda/lib/libpython2.7.so  \
  -DSlicer_USE_PYTHONQT_WITH_OPENSSL:BOOL=1 \
  -DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l \
  -DSlicer_USE_SYSTEM_CTKAPPLAUNCHER:BOOL=0 \
  -DSlicer_USE_SYSTEM_OpenSSL:BOOL=0 \
  -DSlicer_USE_SYSTEM_zlib:BOOL=0 \
  -DSlicer_USE_SYSTEM_python:BOOL=0 \
  -DSlicer_USE_SYSTEM_NUMPY:BOOL=1 \
  -DSlicer_BUILD_CLI:BOOL=OFF \
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
  ../Slicer

nice make -j4

Windows

Although the code in https://github.com/jonieva/Slicer/tree/Conda should work with Windows too (using the setting Slicer_USE_CONDA_python=ON), it has not been tested at the moment.


Compile Slicer and python 2.7.10 + drop in conda binary packages

Initial experiments

TBD

Next steps

Phase 1

Copied from the conclusion of the Developer hangout of March 24th 2015

  • Python conda - next steps:
    • Create External_miniconda.cmake
    • Add a variable named EXTERNAL_PYTHON_NAME that could take two values: python or miniconda
      • In the SuperBuild.cmake, an option named EXTERNAL_PYTHON_NAME could be added. By default, it would be set to "python" but could switch to "miniconda"
    • Projects like External_{CTK, VTK}.cmake would be updated to depend on ${EXTERNAL_PYTHON_NAME}
    • For now, let's consider only the case OpenSSL disabled.

Check.svg See https://github.com/jcfr/Slicer/tree/slicer-miniconda

Phase 2

TBD