Difference between revisions of "Documentation/Labs/SlicerCondaIntegration"

From Slicer Wiki
Jump to: navigation, search
(Status at 2015-01-30)
 
(59 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
Slicer currently builds a custom python distribution from source, but not all useful packages are available.  But [https://store.continuum.io/cshop/anaconda/ Anaconda] provides a set of binaries that might provide a more comprehensive ecosystem.
 
Slicer currently builds a custom python distribution from source, but not all useful packages are available.  But [https://store.continuum.io/cshop/anaconda/ Anaconda] provides a set of binaries that might provide a more comprehensive ecosystem.
  
Before attempting to use the whole Anaconda package, the first approach has been to use Miniconda, which is a reduced version of Anaconda.  
+
Instead of attempting to use the whole Anaconda package, our approach will be to use Miniconda, a reduced version of Anaconda, and (1) install a pre-defined set of packages and (2) provide a way to easily install new ones.  
  
Note: the platform used is MacOSX 10.9.5 with XCode 6
+
== Update 2016-01-22 (Jorge) ==
 +
After a follow up with JC about this topic, and due to the apparent complexity of this task, it has been decided to make a small survey to have a better idea of which would be the most needed packages for the community from those ones available in Anaconda.
 +
 
 +
The link to indicate your preferences is: https://docs.google.com/spreadsheets/d/1j4iugjs0GkHEn1elUbceMW2KJ21FM-QrYoFtZ4P9yOc/edit?usp=sharing
 +
 
 +
== Update 2015-10-10 ==
 +
 
 +
We (Kitware) did few experiments where we copied the anaconda numpy and SciPy packages into a Slicer build tree and run the associated tests. The link below summarizes the results.
 +
 
 +
It turns out that there are a lot of issues with this approach.
 +
 
 +
See https://goo.gl/VNy7yV
 +
 
 +
Next step: We suggest to continue with the initial approach where Slicer is built against miniconda Python libraries.
 +
 
 +
== Virtual Hackfest: August 24, 9am ==
 +
 
 +
=== Info ===
 +
 
 +
Location: Google hangout (connection details will be shared on the Slicer developer list)
 +
 
 +
What: Work on Slicer and Conda integration
 +
 
 +
=== Proposed tasks ===
 +
* <s>Review and consolidate contributions of Jorge and Adam</s> {{done}}
 +
* <s>Implement miniconda external project files as outlined [[#Next_steps|below]]</s> {{done}}
 +
* Create a Slicer module providing:
 +
** (1) scripted interface to install additional conda package
 +
** (2) UI to provide similar functionality
 +
* Update [[Documentation/Nightly/Developers/Extensions/DescriptionFile|extension description format]] to specify python dependencies that could be installed either at build/package time or at run time.
 +
 
 +
=== Links ===
  
==1: compile Slicer against miniconda==
+
* Jorge:
  ccmake -DCMAKE_BUILD_TYPE:STRING=Debug -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake -DPYTHON_EXECUTABLE:FILEPATH=/Users/Jorge/miniconda/bin/python
+
** https://github.com/jonieva/Slicer/tree/Conda
  -DPYTHON_INCLUDE_DIR:PATH=/Users/Jorge/miniconda/include/python2.7 -DPYTHON_LIBRARY:FILEPATH=/Users/Jorge/miniconda/lib/libpython2.7.dylib
+
** http://slicer-devel.65872.n3.nabble.com/Compiling-Slicer-with-Anaconda-td4033321.html
  -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer
+
* Adam:
  /SDKs/MacOSX10.9.sdk  -G  "Eclipse CDT4 - Unix Makefiles" ../Slicer
+
** https://github.com/adamrankin/SlicerTAVI
 +
** http://slicer-devel.65872.n3.nabble.com/Re-Slicer-build-with-custom-python-td4034662.html
  
Doing this, it seems that the Slicer superbuild works ok, and some packages are installed in /Users/Jorge/miniconda/lib/python2.7/site-packages. Ex: gitdb, numpy, SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg
 
  
SimpleITK is not properly installed, when I run miniconda interpreter and try to import it, I get an error:
+
==Compile Slicer against miniconda==
  
  import SimpleITK
+
=== Mac ===
  Traceback (most recent call last):
 
  File "<stdin>", line 1, in <module>
 
  File "/Users/Jorge/miniconda/lib/python2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg/SimpleITK/__init__.py", line 1, in <module>
 
    from .SimpleITK import *
 
  File "/Users/Jorge/miniconda/lib/python2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg/SimpleITK/SimpleITK.py", line 28, in <module>
 
    _SimpleITK = swig_import_helper()
 
  File "/Users/Jorge/miniconda/lib/python2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg/SimpleITK/SimpleITK.py", line 24, in swig_import_helper
 
    _mod = imp.load_module('_SimpleITK', fp, pathname, description)
 
  ImportError: dlopen(/Users/Jorge/miniconda/lib/python2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg/SimpleITK/_SimpleITK.so, 2): Library not loaded: libSimpleITKBasicFilters12-0.9.1.dylib
 
  Referenced from: /Users/Jorge/miniconda/lib/python2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.5-x86_64.egg/SimpleITK/_SimpleITK.so
 
  Reason: image not found
 
  
Other libraries like gitdb or numpy seem to work properly.
+
Note: the platform used is MacOSX 10.9.5 with XCode 6
  
== 2: compile Slicer using System Python ==
+
* Download the source code hosted in https://github.com/jonieva/Slicer/tree/Conda
In order to compare the behaviors, I also compiled another version of Slicer using my System Python:
+
* Build (assuming MacOSX 10.9):
 +
  cmake \
 +
  '''-DSlicer_USE_CONDA_python:BOOL=ON''' \
 +
  -DCMAKE_BUILD_TYPE:STRING=Debug \
 +
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
 +
  -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 \
 +
  -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk \
 +
  -DSlicer_BUILD_CLI:BOOL=OFF \
 +
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
  
  ccmake -DCMAKE_BUILD_TYPE:STRING=Debug -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake  -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.
+
The only changes in the source code are in Superbuild/External_python.cmake and in Superbuild/External_NUMPY.cmake files (in the current version Slicer will use Anaconda Numpy).
  -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk 
+
For the sake of testing, the next Python Conda modules are installed by default: numpy, scikit-learn.
  -G  "Eclipse CDT4 - Unix Makefiles" -DSlicer_USE_SYSTEM_python:BOOL=TRUE ../Slicer
 
  
Checking my System Python folder, everything seems to be properly installed.
+
====Issues====
 +
* libsqlite3.dylib library in miniconda has a conflict with CTK:
 +
<pre>
 +
  Compiling python scripts: CTKScriptingPythonCore
 +
  Compiling python scripts: ctkSimplePythonShell
 +
  Linking CXX shared module ../../bin/CTKCorePythonQt.so
 +
    dyld: Symbol not found: _sqlite3_intarray_bind
 +
  Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 +
  Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
 +
    in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 +
  dyld: Symbol not found: _sqlite3_intarray_bind
 +
    Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 +
    Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
 +
  in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 +
</pre>
  
== Common issues in Miniconda and System python ==
+
  Workaround: delete or rename the file (there is a hack included in the External_python.cmake file to this end).
There is a difference between executing '''Slicer-Superbuild/Slicer-build/bin/Slicer.app''' and '''Slicer-Superbuild/Slicer-build/Slicer''' when I open a Slicer Python console. In the former case, Slicer cannot find some libraries. When I open the console I get:
 
  
  Python 2.7.5 (default, Mar  9 2014, 22:15:05)  
+
* When running Slicer.app (not the binary file in Slicer-Superbuild-Conda/Slicer-build) the program is not using Miniconda, but the System Python, so it doesn't find the right libraries:
  [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
+
<pre>
  >>> Traceback (most recent call last):
+
Python 2.7.5 (default, Mar  9 2014, 22:15:05)  
 +
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
 +
>>> Traceback (most recent call last):
 
   File "<string>", line 1, in <module>
 
   File "<string>", line 1, in <module>
   File "/Users/Jorge/Projects/BWH/Slicer-Superbuild-SystemPython/Slicer-build/lib/Slicer-4.4/qt-scripted-modules/SimpleFilters.py", line 3, in <module>
+
   File "/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda/Slicer-build/lib/Slicer-4.4/qt-scripted-modules/SimpleFilters.py", line 3, in <module>
 
     import SimpleITK as sitk
 
     import SimpleITK as sitk
 
   File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/__init__.py", line 1, in <module>
 
   File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/__init__.py", line 1, in <module>
Line 54: Line 97:
 
   File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 24, in swig_import_helper
 
   File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 24, in swig_import_helper
 
     _mod = imp.load_module('_SimpleITK', fp, pathname, description)
 
     _mod = imp.load_module('_SimpleITK', fp, pathname, description)
  ImportError: dlopen(/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so, 2): Library not loaded: libSimpleITKBasicFilters12-0.9.1.dylib
+
ImportError: dlopen(/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so, 2): Library not loaded: libSimpleITKBasicFilters12-0.9.1.dylib
 
   Referenced from: /Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so
 
   Referenced from: /Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so
 
   Reason: image not found
 
   Reason: image not found
 +
</pre>
 +
 +
So, in order that the system works, '''Slicer must be started from a terminal running the file Slicer-Superbuild-Conda/Slicer-build/Slicer'''
 +
 +
 +
=== Linux ===
 +
 +
* Download the source code in https://github.com/jonieva/Slicer/tree/Conda
 +
* Build Slicer normally, except for the setting Slicer_USE_CONDA_python. Example:
 +
 +
<pre>
 +
cmake \
 +
  '''-DSlicer_USE_CONDA_python:BOOL=ON''' \
 +
  -DCMAKE_BUILD_TYPE:STRING=Debug \
 +
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \ 
 +
  -DSlicer_BUILD_CLI:BOOL=OFF \
 +
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
 +
</pre>
 +
 +
 +
====Different approach====
 +
 +
===== 2015-02-04 - Steve Pieper =====
 +
 +
The following worked for me (Steve Pieper, 2015-02-04) using ubuntu 13.10 and miniconda  3.8.4-py27_0
 +
 +
[[File:Slicer-conda-linux-2015-02-04.png|300px|thumb|right]]
 +
 +
<pre>
 +
cmake \
 +
  -DCMAKE_BUILD_TYPE:STRING=Debug \
 +
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
 +
  -DPYTHON_EXECUTABLE:FILEPATH=${HOME}/miniconda/bin/python \
 +
  -DPYTHON_INCLUDE_DIR:PATH=${HOME}/miniconda/include/python2.7 \
 +
  -DPYTHON_LIBRARY:FILEPATH=${HOME}/miniconda/lib/libpython2.7.so  \
 +
  -DSlicer_BUILD_CLI:BOOL=OFF \
 +
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
 +
 +
make -j20
 +
 +
export PYTHONHOME=${USER}/miniconda
 +
 +
./Slicer-build/Slicer
 +
 +
</pre>
 +
 +
 +
 +
But after it ran there was slicer's build of numpy inside miniconda, so I needed to do this:
 +
 +
bin/conda uninstall numpy
 +
bin/conda install numpy
 +
 +
I tried using this:
 +
 +
  -DSlicer_USE_SYSTEM_NUMPY:BOOL=ON
 +
 +
but got this error:
 +
 +
<pre>
 +
...
 +
-- SuperBuild -        CTKAPPLAUNCHER[OK] (SYSTEM)
 +
CMake Error at SuperBuild/External_CTKAPPLAUNCHER.cmake:21 (message):
 +
  Enabling Slicer_USE_SYSTEM_CTKAPPLAUNCHER is not supported !
 +
Call Stack (most recent call first):
 +
  CMake/ExternalProjectDependency.cmake:561 (include)
 +
  SuperBuild/External_python.cmake:21 (ExternalProject_Include_Dependencies)
 +
  CMake/ExternalProjectDependency.cmake:561 (include)
 +
  SuperBuild/External_VTKv6.cmake:11 (ExternalProject_Include_Dependencies)
 +
  CMake/ExternalProjectDependency.cmake:561 (include)
 +
  SuperBuild/External_teem.cmake:11 (ExternalProject_Include_Dependencies)
 +
  CMake/ExternalProjectDependency.cmake:561 (include)
 +
  CMake/ExternalProjectDependency.cmake:607 (ExternalProject_Include_Dependencies)
 +
  SuperBuild.cmake:320 (ExternalProject_Include_Dependencies)
 +
  CMakeLists.txt:599 (include)
 +
</pre>
 +
 +
-- Configuring incomplete
 +
 +
===== 2015-03-02 - Jc =====
 +
 +
2015-03-02(Jc): To simplify even further, using Slicer >= r24007, the following is expected to work. It is not needed to manually remove Numpy.
 +
 +
This is achieved by specifying <code>-DSlicer_USE_SYSTEM_NUMPY:BOOL=1</code> and also by specifying all dependencies of NUMPY (python, zlib, OpenSSL and CTKAPPLAUNCHER) should not be expected on the system.
 +
 +
Note also that since I built Qt against the latest version of OpenSSL in the 1.0.1 series, I explicitly add <code>-DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l</code>
 +
 +
 +
<pre>
 +
QT_QMAKE_EXECUTABLE=/home/jcfr/Support/qt-everywhere-opensource-release-build-4.8.6/bin/qmake
 +
CONDA_ROOT=/home/jcfr/Projects/sandbox
 +
 +
cmake \
 +
  -DCMAKE_BUILD_TYPE:STRING=Release \
 +
  -DQT_QMAKE_EXECUTABLE:FILEPATH=${QT_QMAKE_EXECUTABLE} \
 +
  -DPYTHON_EXECUTABLE:FILEPATH=${CONDA_ROOT}/miniconda/bin/python \
 +
  -DPYTHON_INCLUDE_DIR:PATH=${CONDA_ROOT}/miniconda/include/python2.7 \
 +
  -DPYTHON_LIBRARY:FILEPATH=${CONDA_ROOT}/miniconda/lib/libpython2.7.so  \
 +
  -DSlicer_USE_PYTHONQT_WITH_OPENSSL:BOOL=1 \
 +
  -DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l \
 +
  -DSlicer_USE_SYSTEM_CTKAPPLAUNCHER:BOOL=0 \
 +
  -DSlicer_USE_SYSTEM_OpenSSL:BOOL=0 \
 +
  -DSlicer_USE_SYSTEM_zlib:BOOL=0 \
 +
  -DSlicer_USE_SYSTEM_python:BOOL=0 \
 +
  -DSlicer_USE_SYSTEM_NUMPY:BOOL=1 \
 +
  -DSlicer_BUILD_CLI:BOOL=OFF \
 +
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
 +
  ../Slicer
 +
 +
nice make -j4
 +
 +
</pre>
 +
 +
=== Windows ===
  
If I run Slicer-Superbuild/Slicer-build/Slicer, everything seems to work fine.
+
Although the code in https://github.com/jonieva/Slicer/tree/Conda should work with Windows too (using the setting Slicer_USE_CONDA_python=ON), it has not been tested at the moment.
  
However, '''in both cases Slicer is loading my System python'''!
 
  
== Next step: isolating the Python console problem ==
+
==Compile Slicer and python 2.7.10 + drop in conda binary packages ==
Following the advice of Steve Pieper, I tried to replicate the problem just in the CTK Python console, without running Slicer. To do that, I enabled the setting CTK_APP_ctkSimplePythonShell, inside Slicer-Superbuild/CTK-build/CTK-build.
 
I had exactly the same behavior that I had within Slicer.
 
  
I tried to set exactly the same settings in my "Slicer Python" version and in my "Miniconda" version, replacing just the settings PYTHON_EXECUTABLE, PYTHON_INCLUDE_DIR and PYTHON_LIBRARY. Still some behavior...
+
=== Initial experiments ===
  
In this process, I found out two things that are different in the "Slicer Python" and "Miniconda":
+
TBD
:If I run otool in my ctkSimplePythonShell, there is a difference in libpython2.7.dylib
 
~/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin$ otool -L ctkSimplePythonShell
 
  ctkSimplePythonShell:
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKScriptingPythonWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKVisualizationVTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/QtTesting-build/libQtTesting.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKScriptingPythonCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkInteraction-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkFiltersHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkOpenGL-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkImagingHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkIO-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 20.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkTestingRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkFilters-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkalglib-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkverdict-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkftgl-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkfreetype-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 48.0.0)
 
  /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 59.0.0)
 
  /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkImaging-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CMakeExternals/Install/lib/libPythonQt.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkWrappingPython27Core-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtkCommon-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  '''/Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/python-install/lib/libpython2.7.dylib (compatibility version 0.0.0, current version 0.0.0)'''
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/VTKv6-build/lib/libvtksys-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Debug/CTK-build/CTK-build/bin/libCTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  QtOpenGL.framework/Versions/4/QtOpenGL (compatibility version 4.8.0, current version 4.8.6)
 
  QtXmlPatterns.framework/Versions/4/QtXmlPatterns (compatibility version 4.8.0, current version 4.8.6)
 
  QtGui.framework/Versions/4/QtGui (compatibility version 4.8.0, current version 4.8.6)
 
  QtXml.framework/Versions/4/QtXml (compatibility version 4.8.0, current version 4.8.6)
 
  QtSql.framework/Versions/4/QtSql (compatibility version 4.8.0, current version 4.8.6)
 
  QtNetwork.framework/Versions/4/QtNetwork (compatibility version 4.8.0, current version 4.8.6)
 
  QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.6)
 
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
 
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
 
  
 +
== Next steps ==
  
 +
=== Phase 1 ===
 +
Copied from the conclusion of the [[Developer_Meetings/20150324#Conclusions|Developer hangout of March 24th 2015]]
  
~/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin$ otool -L ctkSimplePythonShell
+
<s>
ctkSimplePythonShell:
+
* Python conda - next steps:
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKScriptingPythonWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
+
** Create External_miniconda.cmake
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKVisualizationVTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
+
** Add a variable named EXTERNAL_PYTHON_NAME that could take two values: python or miniconda
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKWidgets.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
+
*** In the SuperBuild.cmake, an option named  EXTERNAL_PYTHON_NAME could be added. By default, it would be set to "python" but could switch to "miniconda"
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/QtTesting-build/libQtTesting.dylib (compatibility version 0.0.0, current version 0.0.0)
+
** Projects like External_{CTK, VTK}.cmake would be updated to depend on ${EXTERNAL_PYTHON_NAME}
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKScriptingPythonCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
+
** For now, let's consider only the case OpenSSL disabled.
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkInteraction-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
+
</s>
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkFiltersHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
+
{{done}} See https://github.com/jcfr/Slicer/tree/slicer-miniconda
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkOpenGL-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkImagingHybrid-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkIO-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 20.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkTestingRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkRendering-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkFilters-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkalglib-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkverdict-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkftgl-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkfreetype-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 48.0.0)
 
  /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 59.0.0)
 
  /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkImaging-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CMakeExternals/Install/lib/libPythonQt.dylib (compatibility version 0.0.0, current version 0.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkWrappingPython27Core-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtkCommon-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  '''libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)'''
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/VTKv6-build/lib/libvtksys-6.2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 
  /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin/libCTKCore.0.1.dylib (compatibility version 0.1.0, current version 0.1.0)
 
  QtOpenGL.framework/Versions/4/QtOpenGL (compatibility version 4.8.0, current version 4.8.6)
 
  QtXmlPatterns.framework/Versions/4/QtXmlPatterns (compatibility version 4.8.0, current version 4.8.6)
 
  QtGui.framework/Versions/4/QtGui (compatibility version 4.8.0, current version 4.8.6)
 
  QtXml.framework/Versions/4/QtXml (compatibility version 4.8.0, current version 4.8.6)
 
  QtSql.framework/Versions/4/QtSql (compatibility version 4.8.0, current version 4.8.6)
 
  QtNetwork.framework/Versions/4/QtNetwork (compatibility version 4.8.0, current version 4.8.6)
 
  QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.6)
 
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
 
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
 
  
So, it looks like there is a problem with the rpath of the library. However, I tried to copy the python executable and the library itself to the same bin folder where the ctkSimplePython is (Slicer-Superbuild-Conda-Eclipse/CTK-build/CTK-build/bin) and still I get the same behaviour, so although it may be necessary to work on it, it doesn't seem to be the main cause of the problem.
+
=== Phase 2 ===
  
== Future steps ==
+
TBD
I observed there is a big difference in the structure of the folders in the Slicer Python and Miniconda that could be affecting. There is a '''Libs''' folder inside the /Slicer-Superbuild-Debug/CTK-build/CTK-build that contain many classes that seem required to run a console that uses a Python version different than the system.
 
I hope that someone can guide me a little bit in this...
 

Latest revision as of 22:19, 22 March 2016

Home < Documentation < Labs < SlicerCondaIntegration

Slicer currently builds a custom python distribution from source, but not all useful packages are available. But Anaconda provides a set of binaries that might provide a more comprehensive ecosystem.

Instead of attempting to use the whole Anaconda package, our approach will be to use Miniconda, a reduced version of Anaconda, and (1) install a pre-defined set of packages and (2) provide a way to easily install new ones.

Update 2016-01-22 (Jorge)

After a follow up with JC about this topic, and due to the apparent complexity of this task, it has been decided to make a small survey to have a better idea of which would be the most needed packages for the community from those ones available in Anaconda.

The link to indicate your preferences is: https://docs.google.com/spreadsheets/d/1j4iugjs0GkHEn1elUbceMW2KJ21FM-QrYoFtZ4P9yOc/edit?usp=sharing

Update 2015-10-10

We (Kitware) did few experiments where we copied the anaconda numpy and SciPy packages into a Slicer build tree and run the associated tests. The link below summarizes the results.

It turns out that there are a lot of issues with this approach.

See https://goo.gl/VNy7yV

Next step: We suggest to continue with the initial approach where Slicer is built against miniconda Python libraries.

Virtual Hackfest: August 24, 9am

Info

Location: Google hangout (connection details will be shared on the Slicer developer list)

What: Work on Slicer and Conda integration

Proposed tasks

  • Review and consolidate contributions of Jorge and Adam Check.svg
  • Implement miniconda external project files as outlined below Check.svg
  • Create a Slicer module providing:
    • (1) scripted interface to install additional conda package
    • (2) UI to provide similar functionality
  • Update extension description format to specify python dependencies that could be installed either at build/package time or at run time.

Links


Compile Slicer against miniconda

Mac

Note: the platform used is MacOSX 10.9.5 with XCode 6

 cmake \
 -DSlicer_USE_CONDA_python:BOOL=ON \
 -DCMAKE_BUILD_TYPE:STRING=Debug \
 -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
 -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 \
 -DCMAKE_OSX_SYSROOT:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk \	
 -DSlicer_BUILD_CLI:BOOL=OFF \
 -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \ 
 ../Slicer

The only changes in the source code are in Superbuild/External_python.cmake and in Superbuild/External_NUMPY.cmake files (in the current version Slicer will use Anaconda Numpy). For the sake of testing, the next Python Conda modules are installed by default: numpy, scikit-learn.

Issues

  • libsqlite3.dylib library in miniconda has a conflict with CTK:
  Compiling python scripts: CTKScriptingPythonCore
  Compiling python scripts: ctkSimplePythonShell
  Linking CXX shared module ../../bin/CTKCorePythonQt.so
    dyld: Symbol not found: _sqlite3_intarray_bind
  Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
  Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
    in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
  dyld: Symbol not found: _sqlite3_intarray_bind
    Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
    Expected in: /Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda-Eclipse/python-miniconda/lib/libsqlite3.dylib
  in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 Workaround: delete or rename the file (there is a hack included in the External_python.cmake file to this end).
  • When running Slicer.app (not the binary file in Slicer-Superbuild-Conda/Slicer-build) the program is not using Miniconda, but the System Python, so it doesn't find the right libraries:
Python 2.7.5 (default, Mar  9 2014, 22:15:05) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
>>> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/Jorge/Projects/BWH/Slicer-Superbuild-Conda/Slicer-build/lib/Slicer-4.4/qt-scripted-modules/SimpleFilters.py", line 3, in <module>
    import SimpleITK as sitk
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/__init__.py", line 1, in <module>
    from .SimpleITK import *
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 28, in <module>
    _SimpleITK = swig_import_helper()
  File "/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/SimpleITK.py", line 24, in swig_import_helper
    _mod = imp.load_module('_SimpleITK', fp, pathname, description)
ImportError: dlopen(/Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so, 2): Library not loaded: libSimpleITKBasicFilters12-0.9.1.dylib
  Referenced from: /Library/Python/2.7/site-packages/SimpleITK-0.9.0.dev656-py2.7-macosx-10.9-intel.egg/SimpleITK/_SimpleITK.so
  Reason: image not found

So, in order that the system works, Slicer must be started from a terminal running the file Slicer-Superbuild-Conda/Slicer-build/Slicer


Linux

cmake \
  '''-DSlicer_USE_CONDA_python:BOOL=ON''' \
  -DCMAKE_BUILD_TYPE:STRING=Debug \
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \  
  -DSlicer_BUILD_CLI:BOOL=OFF \
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
  ../Slicer


Different approach

2015-02-04 - Steve Pieper

The following worked for me (Steve Pieper, 2015-02-04) using ubuntu 13.10 and miniconda 3.8.4-py27_0

Slicer-conda-linux-2015-02-04.png
cmake \
  -DCMAKE_BUILD_TYPE:STRING=Debug \
  -DQT_QMAKE_EXECUTABLE:FILEPATH=/usr/bin/qmake \
  -DPYTHON_EXECUTABLE:FILEPATH=${HOME}/miniconda/bin/python \
  -DPYTHON_INCLUDE_DIR:PATH=${HOME}/miniconda/include/python2.7 \
  -DPYTHON_LIBRARY:FILEPATH=${HOME}/miniconda/lib/libpython2.7.so  \
  -DSlicer_BUILD_CLI:BOOL=OFF \
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
  ../Slicer

make -j20

export PYTHONHOME=${USER}/miniconda

./Slicer-build/Slicer


But after it ran there was slicer's build of numpy inside miniconda, so I needed to do this:

bin/conda uninstall numpy
bin/conda install numpy

I tried using this:

 -DSlicer_USE_SYSTEM_NUMPY:BOOL=ON

but got this error:

...
-- SuperBuild -         CTKAPPLAUNCHER[OK] (SYSTEM)
CMake Error at SuperBuild/External_CTKAPPLAUNCHER.cmake:21 (message):
  Enabling Slicer_USE_SYSTEM_CTKAPPLAUNCHER is not supported !
Call Stack (most recent call first):
  CMake/ExternalProjectDependency.cmake:561 (include)
  SuperBuild/External_python.cmake:21 (ExternalProject_Include_Dependencies)
  CMake/ExternalProjectDependency.cmake:561 (include)
  SuperBuild/External_VTKv6.cmake:11 (ExternalProject_Include_Dependencies)
  CMake/ExternalProjectDependency.cmake:561 (include)
  SuperBuild/External_teem.cmake:11 (ExternalProject_Include_Dependencies)
  CMake/ExternalProjectDependency.cmake:561 (include)
  CMake/ExternalProjectDependency.cmake:607 (ExternalProject_Include_Dependencies)
  SuperBuild.cmake:320 (ExternalProject_Include_Dependencies)
  CMakeLists.txt:599 (include)

-- Configuring incomplete

2015-03-02 - Jc

2015-03-02(Jc): To simplify even further, using Slicer >= r24007, the following is expected to work. It is not needed to manually remove Numpy.

This is achieved by specifying -DSlicer_USE_SYSTEM_NUMPY:BOOL=1 and also by specifying all dependencies of NUMPY (python, zlib, OpenSSL and CTKAPPLAUNCHER) should not be expected on the system.

Note also that since I built Qt against the latest version of OpenSSL in the 1.0.1 series, I explicitly add -DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l


QT_QMAKE_EXECUTABLE=/home/jcfr/Support/qt-everywhere-opensource-release-build-4.8.6/bin/qmake
CONDA_ROOT=/home/jcfr/Projects/sandbox

cmake \
  -DCMAKE_BUILD_TYPE:STRING=Release \
  -DQT_QMAKE_EXECUTABLE:FILEPATH=${QT_QMAKE_EXECUTABLE} \
  -DPYTHON_EXECUTABLE:FILEPATH=${CONDA_ROOT}/miniconda/bin/python \
  -DPYTHON_INCLUDE_DIR:PATH=${CONDA_ROOT}/miniconda/include/python2.7 \
  -DPYTHON_LIBRARY:FILEPATH=${CONDA_ROOT}/miniconda/lib/libpython2.7.so  \
  -DSlicer_USE_PYTHONQT_WITH_OPENSSL:BOOL=1 \
  -DOPENSSL_DOWNLOAD_VERSION:STRING=1.0.1l \
  -DSlicer_USE_SYSTEM_CTKAPPLAUNCHER:BOOL=0 \
  -DSlicer_USE_SYSTEM_OpenSSL:BOOL=0 \
  -DSlicer_USE_SYSTEM_zlib:BOOL=0 \
  -DSlicer_USE_SYSTEM_python:BOOL=0 \
  -DSlicer_USE_SYSTEM_NUMPY:BOOL=1 \
  -DSlicer_BUILD_CLI:BOOL=OFF \
  -DSlicer_USE_PYTHONQT_WITH_TCL:BOOL=OFF \
  ../Slicer

nice make -j4

Windows

Although the code in https://github.com/jonieva/Slicer/tree/Conda should work with Windows too (using the setting Slicer_USE_CONDA_python=ON), it has not been tested at the moment.


Compile Slicer and python 2.7.10 + drop in conda binary packages

Initial experiments

TBD

Next steps

Phase 1

Copied from the conclusion of the Developer hangout of March 24th 2015

  • Python conda - next steps:
    • Create External_miniconda.cmake
    • Add a variable named EXTERNAL_PYTHON_NAME that could take two values: python or miniconda
      • In the SuperBuild.cmake, an option named EXTERNAL_PYTHON_NAME could be added. By default, it would be set to "python" but could switch to "miniconda"
    • Projects like External_{CTK, VTK}.cmake would be updated to depend on ${EXTERNAL_PYTHON_NAME}
    • For now, let's consider only the case OpenSSL disabled.

Check.svg See https://github.com/jcfr/Slicer/tree/slicer-miniconda

Phase 2

TBD