Difference between revisions of "Documentation/Labs/SlicerCondaIntegration"

From Slicer Wiki
Jump to: navigation, search
 
(43 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
Slicer currently builds a custom python distribution from source, but not all useful packages are available.  But [https://store.continuum.io/cshop/anaconda/ Anaconda] provides a set of binaries that might provide a more comprehensive ecosystem.
 
Slicer currently builds a custom python distribution from source, but not all useful packages are available.  But [https://store.continuum.io/cshop/anaconda/ Anaconda] provides a set of binaries that might provide a more comprehensive ecosystem.
  
Before attempting to use the whole Anaconda package, the first approach has been to use Miniconda, which is a reduced version of Anaconda.  
+
Instead of attempting to use the whole Anaconda package, our approach will be to use Miniconda, a reduced version of Anaconda, and (1) install a pre-defined set of packages and (2) provide a way to easily install new ones.
 +
 
 +
== Update 2016-01-22 (Jorge) ==
 +
After a follow up with JC about this topic, and due to the apparent complexity of this task, it has been decided to make a small survey to have a better idea of which would be the most needed packages for the community from those ones available in Anaconda.
 +
 
 +
The link to indicate your preferences is: https://docs.google.com/spreadsheets/d/1j4iugjs0GkHEn1elUbceMW2KJ21FM-QrYoFtZ4P9yOc/edit?usp=sharing
 +
 
 +
== Update 2015-10-10 ==
 +
 
 +
We (Kitware) did few experiments where we copied the anaconda numpy and SciPy packages into a Slicer build tree and run the associated tests. The link below summarizes the results.
 +
 
 +
It turns out that there are a lot of issues with this approach.
 +
 
 +
See https://goo.gl/VNy7yV
 +
 
 +
Next step: We suggest to continue with the initial approach where Slicer is built against miniconda Python libraries.
 +
 
 +
== Virtual Hackfest: August 24, 9am ==
 +
 
 +
=== Info ===
 +
 
 +
Location: Google hangout (connection details will be shared on the Slicer developer list)
 +
 
 +
What: Work on Slicer and Conda integration
 +
 
 +
=== Proposed tasks ===
 +
* <s>Review and consolidate contributions of Jorge and Adam</s> {{done}}
 +
* <s>Implement miniconda external project files as outlined [[#Next_steps|below]]</s> {{done}}
 +
* Create a Slicer module providing:
 +
** (1) scripted interface to install additional conda package
 +
** (2) UI to provide similar functionality
 +
* Update [[Documentation/Nightly/Developers/Extensions/DescriptionFile|extension description format]] to specify python dependencies that could be installed either at build/package time or at run time.
 +
 
 +
=== Links ===
 +
 
 +
* Jorge:
 +
** https://github.com/jonieva/Slicer/tree/Conda
 +
** http://slicer-devel.65872.n3.nabble.com/Compiling-Slicer-with-Anaconda-td4033321.html
 +
* Adam:
 +
** https://github.com/adamrankin/SlicerTAVI
 +
** http://slicer-devel.65872.n3.nabble.com/Re-Slicer-build-with-custom-python-td4034662.html
  
Note: the platform used is MacOSX 10.9.5 with XCode 6
 
  
 
==Compile Slicer against miniconda==
 
==Compile Slicer against miniconda==
  
 
=== Mac ===
 
=== Mac ===
 +
 +
Note: the platform used is MacOSX 10.9.5 with XCode 6
 +
 +
* Download the source code hosted in https://github.com/jonieva/Slicer/tree/Conda
 +
* Build (assuming MacOSX 10.9):
 +
  cmake \
 +
  '''-DSlicer_USE_CONDA_python:BOOL=ON''' \
 +
  -DCMAKE_BUILD_TYPE:STRING=Debug \
 +
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
 +
  -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 \
 +
  -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk \
 +
  -DSlicer_BUILD_CLI:BOOL=OFF \
 +
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
 +
 +
The only changes in the source code are in Superbuild/External_python.cmake and in Superbuild/External_NUMPY.cmake files (in the current version Slicer will use Anaconda Numpy).
 +
For the sake of testing, the next Python Conda modules are installed by default: numpy, scikit-learn.
 +
 +
====Issues====
 +
* libsqlite3.dylib library in miniconda has a conflict with CTK:
 +
<pre>
 +
  Compiling python scripts: CTKScriptingPythonCore
 +
  Compiling python scripts: ctkSimplePythonShell
 +
  Linking CXX shared module ../../bin/CTKCorePythonQt.so
 +
    dyld: Symbol not found: _sqlite3_intarray_bind
 +
  Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 +
  Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
 +
    in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 +
  dyld: Symbol not found: _sqlite3_intarray_bind
 +
    Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 +
    Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
 +
  in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 +
</pre>
 +
 +
  Workaround: delete or rename the file (there is a hack included in the External_python.cmake file to this end).
 +
 +
* When running Slicer.app (not the binary file in Slicer-Superbuild-Conda/Slicer-build) the program is not using Miniconda, but the System Python, so it doesn't find the right libraries:
 
<pre>
 
<pre>
cmake -DCMAKE_BUILD_TYPE:STRING=Debug \
+
Python 2.7.5 (default, Mar  9 2014, 22:15:05)
-DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
+
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
-DPYTHON_EXECUTABLE:FILEPATH=${HOME}/miniconda/bin/python \
+
>>> Traceback (most recent call last):
-DPYTHON_INCLUDE_DIR:PATH=${HOME}/miniconda/include/python2.7 \
+
  File "<string>", line 1, in <module>
-DPYTHON_LIBRARY:FILEPATH=${HOME}/miniconda/lib/libpython2.7.dylib \
+
  File "/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda/Slicer-build/lib/Slicer-4.4/qt-scripted-modules/SimpleFilters.py", line 3, in <module>
-DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 \
+
    import SimpleITK as sitk
-DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk  \
+
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/__init__.py", line 1, in <module>
-DSlicer_BUILD_CLI:BOOL=OFF \
+
    from .SimpleITK import *
-DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
+
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 28, in <module>
../Slicer
+
    _SimpleITK = swig_import_helper()
 +
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 24, in swig_import_helper
 +
    _mod = imp.load_module('_SimpleITK', fp, pathname, description)
 +
ImportError: dlopen(/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so, 2): Library not loaded: libSimpleITKBasicFilters12-0.9.1.dylib
 +
  Referenced from: /Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so
 +
  Reason: image not found
 
</pre>
 
</pre>
  
At the moment there are also some additional actions to take, like setting some environment variables: 
+
So, in order that the system works, '''Slicer must be started from a terminal running the file Slicer-Superbuild-Conda/Slicer-build/Slicer'''
  export PYTHONHOME="${HOME} /miniconda/"
+
 
 
 
  export PYTHONPATH="{HOME}/miniconda/:{HOME}/miniconda/lib/:{HOME}/miniconda/lib/python2.7/site-packages/:{HOME}/miniconda/lib/python2.7/lib-dynload/
 
  :{HOME}/miniconda/lib/python2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg/SimpleITK/:{HOME}/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/"
 
 
 
  export DYLD_LIBRARY_PATH="{HOME}/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/SimpleITK-build/lib/:{HOME}/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/"
 
  
 
=== Linux ===
 
=== Linux ===
 +
 +
* Download the source code in https://github.com/jonieva/Slicer/tree/Conda
 +
* Build Slicer normally, except for the setting Slicer_USE_CONDA_python. Example:
 +
 +
<pre>
 +
cmake \
 +
  '''-DSlicer_USE_CONDA_python:BOOL=ON''' \
 +
  -DCMAKE_BUILD_TYPE:STRING=Debug \
 +
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \ 
 +
  -DSlicer_BUILD_CLI:BOOL=OFF \
 +
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
 +
</pre>
 +
 +
 +
====Different approach====
 +
 +
===== 2015-02-04 - Steve Pieper =====
  
 
The following worked for me (Steve Pieper, 2015-02-04) using ubuntu 13.10 and miniconda  3.8.4-py27_0
 
The following worked for me (Steve Pieper, 2015-02-04) using ubuntu 13.10 and miniconda  3.8.4-py27_0
Line 85: Line 179:
 
</pre>
 
</pre>
  
-- Configuring incomplet
+
-- Configuring incomplete
  
=== Windows ===
+
===== 2015-03-02 - Jc =====
  
TBD
+
2015-03-02(Jc): To simplify even further, using Slicer >= r24007, the following is expected to work. It is not needed to manually remove Numpy.
  
== Compile Slicer using System Python ==
+
This is achieved by specifying <code>-DSlicer_USE_SYSTEM_NUMPY:BOOL=1</code> and also by specifying all dependencies of NUMPY (python, zlib, OpenSSL and CTKAPPLAUNCHER) should not be expected on the system.
In order to compare the behaviors, I also compiled another version of Slicer using my System Python:
 
  
  ccmake -DCMAKE_BUILD_TYPE:STRING=Debug -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake  -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 
+
Note also that since I built Qt against the latest version of OpenSSL in the 1.0.1 series, I explicitly add <code>-DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l</code>
  -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk 
 
  -G  "Eclipse CDT4 - Unix Makefiles" -DSlicer_USE_SYSTEM_python:BOOL=TRUE ../Slicer
 
  
Checking my System Python folder, everything seems to be properly installed.
 
  
== Common issues in Miniconda and System python ==
+
<pre>
There is a difference between executing '''Slicer-Superbuild/Slicer-build/bin/Slicer.app''' and '''Slicer-Superbuild/Slicer-build/Slicer''' when I open a Slicer Python console. In the former case, Slicer cannot find some libraries. When I open the console I get:
+
QT_QMAKE_EXECUTABLE=/home/jcfr/Support/qt-everywhere-opensource-release-build-4.8.6/bin/qmake
 +
CONDA_ROOT=/home/jcfr/Projects/sandbox
  
   Python 2.7.5 (default, Mar  9 2014, 22:15:05)
+
cmake \
   [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
+
   -DCMAKE_BUILD_TYPE:STRING=Release \
   >>> Traceback (most recent call last):
+
   -DQT_QMAKE_EXECUTABLE:FILEPATH=${QT_QMAKE_EXECUTABLE} \
  File "<string>", line 1, in <module>
+
   -DPYTHON_EXECUTABLE:FILEPATH=${CONDA_ROOT}/miniconda/bin/python \
  File "/Users/Jorge/Projects/BWH/Slicer-Superbuild-SystemPython/Slicer-build/lib/Slicer-4.4/qt-scripted-modules/SimpleFilters.py", line 3, in <module>
+
  -DPYTHON_INCLUDE_DIR:PATH=${CONDA_ROOT}/miniconda/include/python2.7 \
    import SimpleITK as sitk
+
   -DPYTHON_LIBRARY:FILEPATH=${CONDA_ROOT}/miniconda/lib/libpython2.7.so  \
   File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/__init__.py", line 1, in <module>
+
  -DSlicer_USE_PYTHONQT_WITH_OPENSSL:BOOL=1 \
    from .SimpleITK import *
+
   -DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l \
   File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 28, in <module>
+
  -DSlicer_USE_SYSTEM_CTKAPPLAUNCHER:BOOL=0 \
    _SimpleITK = swig_import_helper()
+
   -DSlicer_USE_SYSTEM_OpenSSL:BOOL=0 \
   File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 24, in swig_import_helper
+
  -DSlicer_USE_SYSTEM_zlib:BOOL=0 \
    _mod = imp.load_module('_SimpleITK', fp, pathname, description)
+
   -DSlicer_USE_SYSTEM_python:BOOL=0 \
   ImportError: dlopen(/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so, 2): Library not loaded: libSimpleITKBasicFilters12-0.9.1.dylib
+
  -DSlicer_USE_SYSTEM_NUMPY:BOOL=1 \
   Referenced from: /Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so
+
   -DSlicer_BUILD_CLI:BOOL=OFF \
  Reason: image not found
+
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
 +
 
 +
nice make -j4
 +
 
 +
</pre>
  
If I run Slicer-Superbuild/Slicer-build/Slicer, everything seems to work fine.
+
=== Windows ===
  
However, '''in both cases Slicer is loading my System python'''!
+
Although the code in https://github.com/jonieva/Slicer/tree/Conda should work with Windows too (using the setting Slicer_USE_CONDA_python=ON), it has not been tested at the moment.
  
== Next step: isolating the Python console problem ==
 
Following the advice of Steve Pieper, I tried to replicate the problem just in the CTK Python console, without running Slicer. To do that, I enabled the setting CTK_APP_ctkSimplePythonShell, inside Slicer-Superbuild/CTK-build/CTK-build.
 
I had exactly the same behavior that I had within Slicer.
 
  
I tried to set exactly the same settings in my "Slicer Python" version and in my "Miniconda" version, replacing just the settings PYTHON_EXECUTABLE, PYTHON_INCLUDE_DIR and PYTHON_LIBRARY. Still some behavior...
+
==Compile Slicer and python 2.7.10 + drop in conda binary packages ==
  
In this process, I found out two things that are different in the "Slicer Python" and "Miniconda":
+
=== Initial experiments ===
:If I run otool in my ctkSimplePythonShell, there is a difference in libpython2.7.dylib
 
~/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin$ otool -L ctkSimplePythonShell
 
  ctkSimplePythonShell:
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKScriptingPythonWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKVisualizationVTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/QtTesting-build/libQtTesting.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKScriptingPythonCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkInteraction-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkFiltersHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkOpenGL-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkImagingHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkIO-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 20.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkTestingRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkFilters-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkalglib-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkverdict-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkftgl-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkfreetype-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 48.0.0)
 
  /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 59.0.0)
 
  /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkImaging-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CMakeExternals/Install/lib/libPythonQt.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkWrappingPython27Core-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkCommon-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  '''/Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/python-install/lib/libpython2.7.dylib (compatibility version 0.0.0, current version 0.0.0)'''
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtksys-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  QtOpenGL.framework/Versions/4/QtOpenGL (compatibility version 4.8.0, current version 4.8.6)
 
  QtXmlPatterns.framework/Versions/4/QtXmlPatterns (compatibility version 4.8.0, current version 4.8.6)
 
  QtGui.framework/Versions/4/QtGui (compatibility version 4.8.0, current version 4.8.6)
 
  QtXml.framework/Versions/4/QtXml (compatibility version 4.8.0, current version 4.8.6)
 
  QtSql.framework/Versions/4/QtSql (compatibility version 4.8.0, current version 4.8.6)
 
  QtNetwork.framework/Versions/4/QtNetwork (compatibility version 4.8.0, current version 4.8.6)
 
  QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.6)
 
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
 
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
 
  
 +
TBD
  
 +
== Next steps ==
  
~/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin$ otool -L ctkSimplePythonShell
+
=== Phase 1 ===
ctkSimplePythonShell:
+
Copied from the conclusion of the [[Developer_Meetings/20150324#Conclusions|Developer hangout of March 24th 2015]]
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKScriptingPythonWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKVisualizationVTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/QtTesting-build/libQtTesting.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKScriptingPythonCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkInteraction-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkFiltersHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkOpenGL-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkImagingHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkIO-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 20.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkTestingRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkFilters-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkalglib-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkverdict-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkftgl-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkfreetype-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 48.0.0)
 
  /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 59.0.0)
 
  /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkImaging-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CMakeExternals/Install/lib/libPythonQt.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkWrappingPython27Core-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkCommon-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  '''libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)'''
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtksys-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  QtOpenGL.framework/Versions/4/QtOpenGL (compatibility version 4.8.0, current version 4.8.6)
 
  QtXmlPatterns.framework/Versions/4/QtXmlPatterns (compatibility version 4.8.0, current version 4.8.6)
 
  QtGui.framework/Versions/4/QtGui (compatibility version 4.8.0, current version 4.8.6)
 
  QtXml.framework/Versions/4/QtXml (compatibility version 4.8.0, current version 4.8.6)
 
  QtSql.framework/Versions/4/QtSql (compatibility version 4.8.0, current version 4.8.6)
 
  QtNetwork.framework/Versions/4/QtNetwork (compatibility version 4.8.0, current version 4.8.6)
 
  QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.6)
 
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
 
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
 
  
So, it looks like there is a problem with the rpath of the library. Even when I tried to copy the python executable and the library itself to the same bin folder where the ctkSimplePython is (Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin), I still got the same behavior.  
+
<s>
Following JC advice (thanks!), I used '''instal_name_tool''' to add manually the rpath to libpython2.7.dylib:
+
* Python conda - next steps:
  install_name_tool -id /Users/Jorge/miniconda/lib/libpython2.7.dylib /Users/Jorge/miniconda/lib/libpython2.7.dylib
+
** Create External_miniconda.cmake
 +
** Add a variable named EXTERNAL_PYTHON_NAME that could take two values: python or miniconda
 +
*** In the SuperBuild.cmake, an option named  EXTERNAL_PYTHON_NAME could be added. By default, it would be set to "python" but could switch to "miniconda"
 +
** Projects like External_{CTK, VTK}.cmake would be updated to depend on ${EXTERNAL_PYTHON_NAME}
 +
** For now, let's consider only the case OpenSSL disabled.
 +
</s>
 +
{{done}} See https://github.com/jcfr/Slicer/tree/slicer-miniconda
  
Voila! Now we can rebuild and the ctkSimplePythonShell will use the correct Python! However, we will have to set up some environment variables in order than all the libraries are found. I tried some of the most important and checked that they were properly loaded in the Python console, but we will need the same ones that we are adding when building the current "SlicerPython":
+
=== Phase 2 ===
  export PYTHONHOME="/Users/Jorge/miniconda/"
 
  export PYTHONPATH="/Users/Jorge/miniconda/:/Users/Jorge/miniconda/lib/:/Users/Jorge/miniconda/lib/python2.7/site-packages/:
 
  /Users/Jorge/miniconda/lib/python2.7/lib-dynload/
 
  :/Users/Jorge/miniconda/lib/python2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg/SimpleITK/
 
  :/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/"
 
  export DYLD_LIBRARY_PATH="/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/SimpleITK-build/lib/
 
  :/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/"
 
  
I also checked that the Python console works inside Slicer, as long as I launch the Slicer binary from a terminal that has all these environment variables initialized.
+
TBD

Latest revision as of 22:19, 22 March 2016

Home < Documentation < Labs < SlicerCondaIntegration

Slicer currently builds a custom python distribution from source, but not all useful packages are available. But Anaconda provides a set of binaries that might provide a more comprehensive ecosystem.

Instead of attempting to use the whole Anaconda package, our approach will be to use Miniconda, a reduced version of Anaconda, and (1) install a pre-defined set of packages and (2) provide a way to easily install new ones.

Update 2016-01-22 (Jorge)

After a follow up with JC about this topic, and due to the apparent complexity of this task, it has been decided to make a small survey to have a better idea of which would be the most needed packages for the community from those ones available in Anaconda.

The link to indicate your preferences is: https://docs.google.com/spreadsheets/d/1j4iugjs0GkHEn1elUbceMW2KJ21FM-QrYoFtZ4P9yOc/edit?usp=sharing

Update 2015-10-10

We (Kitware) did few experiments where we copied the anaconda numpy and SciPy packages into a Slicer build tree and run the associated tests. The link below summarizes the results.

It turns out that there are a lot of issues with this approach.

See https://goo.gl/VNy7yV

Next step: We suggest to continue with the initial approach where Slicer is built against miniconda Python libraries.

Virtual Hackfest: August 24, 9am

Info

Location: Google hangout (connection details will be shared on the Slicer developer list)

What: Work on Slicer and Conda integration

Proposed tasks

  • Review and consolidate contributions of Jorge and Adam Check.svg
  • Implement miniconda external project files as outlined below Check.svg
  • Create a Slicer module providing:
    • (1) scripted interface to install additional conda package
    • (2) UI to provide similar functionality
  • Update extension description format to specify python dependencies that could be installed either at build/package time or at run time.

Links


Compile Slicer against miniconda

Mac

Note: the platform used is MacOSX 10.9.5 with XCode 6

 cmake \
 -DSlicer_USE_CONDA_python:BOOL=ON \
 -DCMAKE_BUILD_TYPE:STRING=Debug \
 -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
 -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 \
 -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk \	
 -DSlicer_BUILD_CLI:BOOL=OFF \
 -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \ 
 ../Slicer

The only changes in the source code are in Superbuild/External_python.cmake and in Superbuild/External_NUMPY.cmake files (in the current version Slicer will use Anaconda Numpy). For the sake of testing, the next Python Conda modules are installed by default: numpy, scikit-learn.

Issues

  • libsqlite3.dylib library in miniconda has a conflict with CTK:
  Compiling python scripts: CTKScriptingPythonCore
  Compiling python scripts: ctkSimplePythonShell
  Linking CXX shared module ../../bin/CTKCorePythonQt.so
    dyld: Symbol not found: _sqlite3_intarray_bind
  Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
  Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
    in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
  dyld: Symbol not found: _sqlite3_intarray_bind
    Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
    Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
  in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 Workaround: delete or rename the file (there is a hack included in the External_python.cmake file to this end).
  • When running Slicer.app (not the binary file in Slicer-Superbuild-Conda/Slicer-build) the program is not using Miniconda, but the System Python, so it doesn't find the right libraries:
Python 2.7.5 (default, Mar  9 2014, 22:15:05) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
>>> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda/Slicer-build/lib/Slicer-4.4/qt-scripted-modules/SimpleFilters.py", line 3, in <module>
    import SimpleITK as sitk
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/__init__.py", line 1, in <module>
    from .SimpleITK import *
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 28, in <module>
    _SimpleITK = swig_import_helper()
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 24, in swig_import_helper
    _mod = imp.load_module('_SimpleITK', fp, pathname, description)
ImportError: dlopen(/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so, 2): Library not loaded: libSimpleITKBasicFilters12-0.9.1.dylib
  Referenced from: /Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so
  Reason: image not found

So, in order that the system works, Slicer must be started from a terminal running the file Slicer-Superbuild-Conda/Slicer-build/Slicer


Linux

cmake \
  '''-DSlicer_USE_CONDA_python:BOOL=ON''' \
  -DCMAKE_BUILD_TYPE:STRING=Debug \
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \  
  -DSlicer_BUILD_CLI:BOOL=OFF \
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
  ../Slicer


Different approach

2015-02-04 - Steve Pieper

The following worked for me (Steve Pieper, 2015-02-04) using ubuntu 13.10 and miniconda 3.8.4-py27_0

Slicer-conda-linux-2015-02-04.png
cmake \
  -DCMAKE_BUILD_TYPE:STRING=Debug \
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
  -DPYTHON_EXECUTABLE:FILEPATH=${HOME}/miniconda/bin/python \
  -DPYTHON_INCLUDE_DIR:PATH=${HOME}/miniconda/include/python2.7 \
  -DPYTHON_LIBRARY:FILEPATH=${HOME}/miniconda/lib/libpython2.7.so  \
  -DSlicer_BUILD_CLI:BOOL=OFF \
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
  ../Slicer

make -j20

export PYTHONHOME=${USER}/miniconda

./Slicer-build/Slicer


But after it ran there was slicer's build of numpy inside miniconda, so I needed to do this:

bin/conda uninstall numpy
bin/conda install numpy

I tried using this:

 -DSlicer_USE_SYSTEM_NUMPY:BOOL=ON

but got this error:

...
-- SuperBuild -         CTKAPPLAUNCHER[OK] (SYSTEM)
CMake Error at SuperBuild/External_CTKAPPLAUNCHER.cmake:21 (message):
  Enabling Slicer_USE_SYSTEM_CTKAPPLAUNCHER is not supported !
Call Stack (most recent call first):
  CMake/ExternalProjectDependency.cmake:561 (include)
  SuperBuild/External_python.cmake:21 (ExternalProject_Include_Dependencies)
  CMake/ExternalProjectDependency.cmake:561 (include)
  SuperBuild/External_VTKv6.cmake:11 (ExternalProject_Include_Dependencies)
  CMake/ExternalProjectDependency.cmake:561 (include)
  SuperBuild/External_teem.cmake:11 (ExternalProject_Include_Dependencies)
  CMake/ExternalProjectDependency.cmake:561 (include)
  CMake/ExternalProjectDependency.cmake:607 (ExternalProject_Include_Dependencies)
  SuperBuild.cmake:320 (ExternalProject_Include_Dependencies)
  CMakeLists.txt:599 (include)

-- Configuring incomplete

2015-03-02 - Jc

2015-03-02(Jc): To simplify even further, using Slicer >= r24007, the following is expected to work. It is not needed to manually remove Numpy.

This is achieved by specifying -DSlicer_USE_SYSTEM_NUMPY:BOOL=1 and also by specifying all dependencies of NUMPY (python, zlib, OpenSSL and CTKAPPLAUNCHER) should not be expected on the system.

Note also that since I built Qt against the latest version of OpenSSL in the 1.0.1 series, I explicitly add -DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l


QT_QMAKE_EXECUTABLE=/home/jcfr/Support/qt-everywhere-opensource-release-build-4.8.6/bin/qmake
CONDA_ROOT=/home/jcfr/Projects/sandbox

cmake \
  -DCMAKE_BUILD_TYPE:STRING=Release \
  -DQT_QMAKE_EXECUTABLE:FILEPATH=${QT_QMAKE_EXECUTABLE} \
  -DPYTHON_EXECUTABLE:FILEPATH=${CONDA_ROOT}/miniconda/bin/python \
  -DPYTHON_INCLUDE_DIR:PATH=${CONDA_ROOT}/miniconda/include/python2.7 \
  -DPYTHON_LIBRARY:FILEPATH=${CONDA_ROOT}/miniconda/lib/libpython2.7.so  \
  -DSlicer_USE_PYTHONQT_WITH_OPENSSL:BOOL=1 \
  -DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l \
  -DSlicer_USE_SYSTEM_CTKAPPLAUNCHER:BOOL=0 \
  -DSlicer_USE_SYSTEM_OpenSSL:BOOL=0 \
  -DSlicer_USE_SYSTEM_zlib:BOOL=0 \
  -DSlicer_USE_SYSTEM_python:BOOL=0 \
  -DSlicer_USE_SYSTEM_NUMPY:BOOL=1 \
  -DSlicer_BUILD_CLI:BOOL=OFF \
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
  ../Slicer

nice make -j4

Windows

Although the code in https://github.com/jonieva/Slicer/tree/Conda should work with Windows too (using the setting Slicer_USE_CONDA_python=ON), it has not been tested at the moment.


Compile Slicer and python 2.7.10 + drop in conda binary packages

Initial experiments

TBD

Next steps

Phase 1

Copied from the conclusion of the Developer hangout of March 24th 2015

  • Python conda - next steps:
    • Create External_miniconda.cmake
    • Add a variable named EXTERNAL_PYTHON_NAME that could take two values: python or miniconda
      • In the SuperBuild.cmake, an option named EXTERNAL_PYTHON_NAME could be added. By default, it would be set to "python" but could switch to "miniconda"
    • Projects like External_{CTK, VTK}.cmake would be updated to depend on ${EXTERNAL_PYTHON_NAME}
    • For now, let's consider only the case OpenSSL disabled.

Check.svg See https://github.com/jcfr/Slicer/tree/slicer-miniconda

Phase 2

TBD