Difference between revisions of "Documentation/Nightly/Developers/ReleaseProcess"

From Slicer Wiki
Jump to: navigation, search
Tag: 2017 source edit
 
(229 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== Release planning ==
 
== Release planning ==
  
Usually ~1 month before release.
+
* Decide what will be the version number for the next stable release
 
+
* Update issue tracker: https://issues.slicer.org/roadmap_page.php
* Sept 13: Discussed next release during the dev hangout
+
* Define a release date
 +
* Announce the release: https://discourse.slicer.org/t/upcoming-slicer-4-8-release/1120
  
 
== Prerequisites ==
 
== Prerequisites ==
  
* Make sure to patch VS2008 using the script [https://gist.github.com/jcfr/3c7bef3f8b32f9f6ad4b fix-vc9-redist-manifests.ps1]
+
Account on the following systems are required:
 +
* Slicer wiki
 +
* Slicer Packages and Data server.
 +
** (1) [[Documentation/Nightly/Developers/Tutorials/ObtainExtensionServerApiKey|Set up an account and obtaining an API key]]
 +
** (2) Request to be associated with at least <tt>Slicer App Packages Uploader</tt> NA-MIC community group
 +
* macOS signing server
 +
** Contact Kitware sysadmins providing your public SSH key. You will then be given a PIN code required to unlock the keychain.
 +
<!--
 +
* Svn server
 +
** http://viewvc.slicer.org/viewvc.cgi/Slicer4/
 +
-->
 +
* Build machines
 +
** See [[Documentation/Nightly/Developers/Factory]]
 +
* Slicer Github
 +
** https://github.com/Slicer/Slicer
 +
* Issue tracker
 +
** https://issues.slicer.org
 +
** Admin account is needed
 +
* nitrc
 +
** See https://www.nitrc.org/account/register.php
 +
** Provide your user name to contact below to be added to http://www.nitrc.org/projects/slicer/
 +
* discourse
 +
** request to be added to the <tt>Group working on draft posts</tt>
  
* <strike>Update modules documentation link in help section (loadable, scripted and cli)</strike>
+
For more details, see [[Resources]]
** And some html links in Welcome module (Modules/Loadable/SlicerWelcome/Resources/HTML)
 
** <strike>Update CLI XML description files</strike>
 
  
Commit message: <code>Update Documentation to X.Y</code>
+
Contact <email>jcfr@kitware.com</email> or <email>sam.horvath@kitware.com</email> to request access.
  
For example: [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=22407 r22407]
+
== Day 1: Release ==
  
Note: Some of this updates are not needed anymore. Indeed, following [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=22423 r22423] and [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=22424 r22424] documentation url should already reference the <code>Nightly</code> namespace.
+
Since there all development occurs on <code>master</code>, each time version is updated, two commits will be required.
  
== Release-candidate ==
+
=== Update Slicer.crt and commit change ===
  
Since there all development occurs on <code>master</code>, each time version is updated, two commits will be required.
+
See https://github.com/Slicer/Slicer/blob/master/Base/QTCore/Resources/Certs/README
  
<code><RC></code> corresponds to the release candidate number. It is greater or equal to one.
+
=== CMakeLists.txt: Update the Slicer version information for the release ===
  
 
<ol start="1" style="list-style-type: decimal;">
 
<ol start="1" style="list-style-type: decimal;">
  
<li><p>Update the Slicer version information for the release candidate:</p>
+
<li><p>Update the Slicer version information for the release:</p>
 
<ol style="list-style-type: lower-roman;">
 
<ol style="list-style-type: lower-roman;">
<li><p>In <code>CMakeLists.txt</code>, uncomment and set:</p>
+
<li><p>In <code>CMakeLists.txt</code>, update at least one these variables: <code>Slicer_VERSION_MAJOR</code>, <code>Slicer_VERSION_MINOR</code>, <code>Slicer_VERSION_PATCH</code></p>
*<code>Slicer_VERSION_TWEAK</code> to <code>0</code>
+
</li>
*<code>Slicer_VERSION_RC</code> to <code><RC></code>
+
<li>
<p>...and if this is the first release candidate, update at least one these variables: <code>Slicer_VERSION_MAJOR</code>, <code>Slicer_VERSION_MINOR</code>, <code>Slicer_VERSION_PATCH</code></p>
+
  <p>Re-run CMake with <code>-DSlicer_RELEASE_TYPE:STRING=Stable</code>. This is required to update <code>Utilities/Scripts/SlicerWizard/__version__.py</code>.</p>
 +
  <p><small>{{note}}If not doing a clean build, make sure to explicitly specify the version re-configuring the inner build with <code>-DSlicer_VERSION_MAJOR:STRING=X -DSlicer_VERSION_MINOR:STRING=Y -DSlicer_VERSION_PATCH:STRING=Z -DSlicerApp_VERSION_MAJOR:STRING=X -DSlicerApp_VERSION_MINOR:STRING=Y -DSlicerApp_VERSION_PATCH:STRING=Z.</code></small></p>
 +
  <p>
 +
</li>
 +
<li><p>Commit the above changes using this message like:</p>
 +
ENH: Slicer X.Y.Z
 
</li>
 
</li>
<li>Re-run CMake in order to update <code>Utilities/Scripts/SlicerWizard/__version__.py</code>.</li>
+
</ol>
<li><p>Commit the above changes using this message:</p>
 
ENH: Slicer X.Y.Z-rc<RC>
 
 
</li>
 
</li>
 +
 
</ol>
 
</ol>
 +
 +
=== Tag the repository ===
 +
 +
<ol start="9" style="list-style-type: decimal;">
 +
<li><p>Tag the repository:</p>
 +
Set version variables:
 +
X=<major-version>
 +
Y=<minor-version>
 +
Z=<patch-version>
 +
echo "X.Y.Z is ${X}.${Y}.${Z}"
 +
 +
GIT_TAG=v${X}.${Y}.${Z}
 +
echo "GIT_TAG is ${GIT_TAG}"
 +
 +
git checkout master
 +
git tag -s -m "ENH: Slicer ${X}.${Y}.${Z}" ${GIT_TAG}
 +
git push origin ${GIT_TAG}
 
</li>
 
</li>
 +
<li>Keep track of the <tt>GIT_TAG</tt> for the next steps</li>
 +
</ol>
  
<li>Generate packages based on REVISION associated with step 1.</li>
+
=== CMakeLists.txt: Update the Slicer version information for the development ===
  
 +
<ol start="2" style="list-style-type: decimal;">
 
<li><p>Update the Slicer version information for the development:</p>
 
<li><p>Update the Slicer version information for the development:</p>
 
<ol style="list-style-type: lower-roman;">
 
<ol style="list-style-type: lower-roman;">
<li>In <code>CMakeLists.txt</code>, comment <code>Slicer_VERSION_TWEAK</code> so that the next builds will contain the date associated with the last commit.</li>
+
<li><p>In <code>CMakeLists.txt</code>,</p>
<li>Re-run CMake in order to update <code>Utilities/Scripts/SlicerWizard/__version__.py</code>.</li>
+
* Update <code>Slicer_VERSION_MAJOR</code> and/or <code>Slicer_VERSION_MINOR</code>
 
</li>
 
</li>
<li><p>Commit the above changes using this message:</p>
+
<li>
  ENH: Begin post-X.Y.Z-rc<RC> development
+
  <p>Re-run CMake with <code>-DSlicer_RELEASE_TYPE:STRING=Experimental</code>. This is required to update <code>Utilities/Scripts/SlicerWizard/__version__.py</code>.</p>
 +
  <p><small>{{note}}If not doing a clean build, make sure to explicitly specify the version re-configuring with <code>-DSlicer_VERSION_MAJOR:STRING=X -DSlicer_VERSION_MINOR:STRING=Y -DSlicer_VERSION_PATCH:STRING=Z .</code></small></p>
 +
</li>
 +
<li><p>Commit the above changes using this message like:</p>
 +
  cd ~/Projects/Slicer
 +
git add -A
 +
git commit -m "ENH: Begin X.Y.Z development"
 +
</li>
 +
</ol>
 +
</li>
 +
 
 +
</ol>
 +
 
 +
=== Tag and publish SlicerBuildEnvironment docker image ===
 +
 
 +
<ol start="3" style="list-style-type: decimal;">
 +
<li>
 +
  <p>Tag the docker build environment image. See https://github.com/Slicer/SlicerBuildEnvironment/blob/master/README.rst#maintainers</p>
 +
  <p><small>{{note}}If the environment did not change, you may skip this step for patch release.</small></p>
 +
  <p>Keep track of the selected <tt>TAG</tt> for updating the release scripts</p>
 
</li>
 
</li>
 
</ol>
 
</ol>
  
 +
=== Update release scripts ===
 +
 +
<ol start="4" style="list-style-type: decimal;">
 +
<li>Update release scripts. See https://github.com/Slicer/DashboardScripts#maintenance-guides</li>
 
</ol>
 
</ol>
  
== Release ==
+
=== Disable regular nightly builds ===
  
Since there all development occurs on <code>master</code>, each time version is updated, two commits will be required.
+
<ol start="5" style="list-style-type: decimal;">
 +
<li>On each factory machines, disable regular nightly build in crontab and task scheduler.</li>
 +
</ol>
 +
 
 +
=== Update ExtensionsIndex ===
  
<ol start="1" style="list-style-type: decimal;">
+
<i>This applies to https://github.com/Slicer/ExtensionsIndex and you should skip this step for patch release.</i>
  
<li><p>Update the Slicer version information for the release:</p>
+
<ol start="6" style="list-style-type: decimal;">
<ol style="list-style-type: lower-roman;">
+
<li><p>Create branch X.Y based of master:</p>
<li><p>In <code>CMakeLists.txt</code>, uncomment and set <code>Slicer_VERSION_TWEAK</code> to <code>0</code></p>
+
  DEST_VERSION=X.Y
<p>...and if this no release candidate has been made, update at least one these variables: <code>Slicer_VERSION_MAJOR</code>, <code>Slicer_VERSION_MINOR</code>, <code>Slicer_VERSION_PATCH</code></p>
+
 
</li>
+
  cd /tmp
<li>Re-run CMake in order to update <code>Utilities/Scripts/SlicerWizard/__version__.py</code>.</li>
+
  git clone git@github.com:Slicer/ExtensionsIndex
<li><p>Commit the above changes using this message:</p>
+
  cd ExtensionsIndex
ENH: Slicer X.Y.Z
+
  git checkout master
 +
  git push origin master:${DEST_VERSION}
 
</li>
 
</li>
 
</ol>
 
</ol>
</li>
 
  
<li>Generate packages based on REVISION associated with step 1.
+
=== Update CDash ===
 +
 
 +
<i>This applies to the CDash instance associated with http://slicer.cdash.org/index.php?project=Slicer4 and you should skip this step for patch release.</i>
  
<li><p>Tag the repository:</p>
+
<ol start="7" style="list-style-type: decimal;">
svn copy -r<REVISION> http://svn.slicer.org/Slicer4/trunk  \
+
<li><p>Create new CDash groups for extension submissions associated with <code>X.Y</code> release:</p>
      http://svn.slicer.org/Slicer4/tags/Slicer-X-Y  \
+
<pre>
    -m "ENH: Tag of X.Y.Z release based on r<REVISION>."
+
Extensions-X.Y-Nightly
</li>
+
Extensions-X.Y-Continuous
 +
</pre></li>
  
<li><p>Update the Slicer version information for the development:</p>
 
<ol style="list-style-type: lower-roman;">
 
<li><p>In <code>CMakeLists.txt</code>, comment and set:</p>
 
* <code>Slicer_VERSION_TWEAK</code> to <code>0</code> so that the next builds will contain the date associated with the last commit.
 
* <code>Slicer_VERSION_RC</code> to <code>0</code>
 
</li>
 
<li>Re-run CMake in order to update <code>Utilities/Scripts/SlicerWizard/__version__.py</code>.</li>
 
<li><p>Commit the above changes using this message:</p>
 
ENH: Begin post-X.Y.Z development
 
</li>
 
 
</ol>
 
</ol>
</li>
 
  
 +
=== Generate application and extension packages ===
 +
 +
<ol start="8" style="list-style-type: decimal;">
 +
<li>Generate packages running Slicer package scripts on each factory machines. (These are the scripts updated in the previous step)</li>
 
</ol>
 
</ol>
  
== Post release ==
+
=== Create release branch ===
 +
 
 +
<i>You should skip this step for patch release.</i>
  
=== Create a release branch ===
+
<ol start="10" style="list-style-type: decimal;">
 +
<li><p>Create release branch:</p>
 +
Set version variables (same value as in the previous step):
 +
  X=<major-version>
 +
  Y=<minor-version>
 +
  Z=<patch-version>
 +
  echo "X.Y.Z is ${X}.${Y}.${Z}"
  
svn copy -r <REVISION> http://svn.slicer.org/Slicer4/trunk http://svn.slicer.org/Slicer4/branches/Slicer-X-Y \
+
  GIT_TAG=v${X}.${Y}.${Z}
   -m "ENH: Branching from trunk to Slicer-X-Y at <REVISION>"
+
   echo "GIT_TAG is ${GIT_TAG}"
  
 +
  cd ~/Projects/Slicer
 +
  git checkout -b master-${X}.${Y} ${GIT_TAG}
 +
  git push origin master-${X}.${Y}
 +
</li>
 +
</ol>
  
  svn checkout http://svn.slicer.org/Slicer4/branches/Slicer-X-Y
+
=== Update Release Details on the wiki ===
  
=== Generate ChangeLog ===
+
<ol start="11" style="list-style-type: decimal;">
 +
<li>Update [[Release Details]] with release information</li>
 +
</ol>
  
* See [https://raw.github.com/cryos/avogadro/master/scripts/gitlog2changelog.py gitlog2changelog.py]. More details [http://blog.cryos.net/archives/202-Git-and-Automatic-ChangeLog-Generation.html here].
+
== Day 2: Post release ==
  
* Update [[Release Details]] page using generated ChangeLog
+
=== Re-enable regular nightly builds ===
  
 +
On each factory machines, re-enable build in crontab and task scheduler.
 +
 +
<!--
 
=== Update Mantis ===
 
=== Update Mantis ===
 +
 +
<i>This applies to the Mantis instance associated with https://issues.slicer.org</i>
  
 
* "Release" current target
 
* "Release" current target
 
* Create new target
 
* Create new target
 
* Check the "fixed in" fields
 
* Check the "fixed in" fields
 +
-->
 +
 +
=== Update Slicer wiki ===
 +
 +
<i>This applies to the MediaWiki instance associated with https://wiki.slicer.org</i>
  
=== Midas ===
+
There are two steps involved:
 +
* copy of the pages associated with the <code>Nightly</code> namespace into the <code>X.Y</code> namespace.
 +
* update of permanent pages referencing the current Slicer version
  
==== Tag release packages ====
+
As of November 2016, the process is fully automated and can be done using [https://github.com/Slicer/Slicer/tree/master/Base/Python/slicer/release/wiki.py Base/Python/slicer/release/wiki.py] script.
  
* If needed, create a X.Y.Z folder in [http://slicer.kitware.com/midas3/folder/274 Slicer/Packages/Application/Release]
+
1) Install prerequisites:
** Copy uploaded packages into the folder created above
+
 
** Identify the item_id associated with uploaded packages. For example: 11926, 11925, 11927, 11992
+
* Check with <email>jchris.fillionr@kitware.com</email> to get the credential associated with <code>UpdateBot</code> user.
** SSH connect to <code>jcfr@slicer.kitware.com</code>
+
* Install [https://pypi.python.org/pypi/mwdoc mwdoc]:
** Connect to mysql using <code>mysql -u midas -p</code>
+
 
*** See file <code>/var/www/midas3/core/configs/database.local.ini</code> for password
+
<pre>
*** Choose midas database: <code>use midas</code>
+
$ pip install mwdoc
** List packages associated with identified items and check they are the appropriate ones:
+
</pre>
*** <code>select *  from slicerpackages_package as p , item as i where i.item_id = p.item_id and p.item_id in (11926, 11925, 11927, 11992);</code>
+
 
** Set release field
+
2) Query current version information:
*** <code>update slicerpackages_package set `release`="4.2.2-1" where item_id in (11926, 11925, 11927, 11992);</code>
+
 
 +
<pre>
 +
$ python Base/Python/slicer/release/wiki.py query
 +
</pre>
 +
 
 +
3) Copy and update pages
 +
 
 +
<pre>
 +
$ DEST_VERSION=X.Y
 +
 
 +
$ python Base/Python/slicer/release/wiki.py update ${DEST_VERSION}
 +
$ python Base/Python/slicer/release/wiki.py copy ${DEST_VERSION}
 +
</pre>
 +
 
 +
The following pages have been updated:
 +
 
 +
* [[Template:Documentation/prevversion]]
 +
* [[Template:Documentation/nextversion]]
 +
* [[Template:Documentation/currentversion]]
 +
* [[Template:Documentation/versionlist]]
 +
* [[Template:Documentation/acknowledgments-versionlist]]
 +
* [[Documentation]]
 +
* [[Documentation/Release/Acknowledgments]]
 +
 
 +
The following pages with #REDIRECT have been updated:
 +
 
 +
* [[FAQ]]
 +
* [[Documentation/Release]]
 +
* [[Documentation/Release/Announcements]]
 +
* [[Documentation/Release/Report a problem]]
 +
* [[Documentation/UserTraining]]
 +
* [[Documentation/UserFeedback]]
 +
* [[Documentation/Release/SlicerApplication/HardwareConfiguration]]
 +
 
 +
=== Prepare Slicer Announcement ===
 +
 
 +
<ul>
 +
  <li>
 +
    <p>(1) Reusing previous documents available in this [https://drive.google.com/drive/u/0/folders/0B6cFo-C6Ob4-Q1NjRk1PRU5COFk shared Google folder], create and update the following new Google documents:</p>
 +
    <ul>
 +
      <li><tt>Slicer {{documentation/nextversion}}: Kitware Social Media (Shared Publicly)</tt></li>
 +
      <li>
 +
        <p><tt>Slicer {{documentation/nextversion}}: Summary, Highlights and Changelog (Shared Publicly)</tt></p>
 +
        <p><small>{{note}}Updating this document is currently a tedious manual process. It will soon be streamlined by including descriptive text with each contribution to the code base.</small></p>
 +
      </li>
 +
      <li><tt>Slicer {{documentation/nextversion}}: Kitware Annoucement blog</tt></li>
 +
    </ul>
 +
  </li>
 +
  <li>
 +
    <p>(2) Once the <tt>Slicer {{documentation/nextversion}} Release notes</tt> document is mature, post it on discourse for including images.</p>
 +
    <ul>
 +
      <li>(a) Convert it by following [[#Converting_Google_document_to_markdown]] guide.</li>
 +
      <li>(b) Then, update the generated markdown document to be compliant with discourse. See [[#Generating_a_Discourse_compliant_table_of_content_for_a_Markdown_document]] guide.</li>
 +
      <li>(c) Finally, create a [https://meta.discourse.org/t/shared-drafts-allow-staff-to-collaborate-on-posts-before-publishing/83493 draft discourse post] selecting <tt>Draft</tt> category.</li>
 +
      [[File:ReleaseProcess-Create-Slicer-discourse-summary-highlight-and-changelog.png]]
 +
      <li>
 +
        <p>(d) Select the destination category but do <b>NOT</b> Publish the shared draft.</p>
 +
        <p>''Note: Due to limitation of discourse, you may not be able to see the post listed in the draft category or publish it. It this is the case, ask @jcfr, @pieper or @lassoan to help out.'' </p>
 +
      </li>
 +
      [[File:ReleaseProcess-Draft-Discourse-Post-Select-Dest-Category.png]]
 +
    </ul>
 +
  </li>
 +
</ul>
 +
 
 +
=== Clean-up older nightly packages ===
 +
 
 +
<i>This applies to the Midas instance associated with http://slicer.kitware.com</i>
 +
 
 +
See https://gist.github.com/jcfr/ea9ef199bd5a3e071b8f
 +
 
 +
=== Manually sign packages ===
 +
 
 +
{{remark|red|These instructions apply only to <b>macOS</b> and <b>Windows</b> packages}}
 +
 
 +
 
 +
<ol start="1" style="list-style-type: decimal;">
 +
 
 +
<li>If needed, create a X.Y.Z folder in [http://slicer.kitware.com/midas3/folder/274 Slicer/Packages/Application/Release]</li>
 +
 
 +
<li>Download package from [http://slicer.cdash.org/index.php?project=Slicer4&filtercount=1&showfilters=1&field1=groupname&compare1=61&value1=Experimental-Packages http://slicer.cdash.org/index.php?project=Slicer4]. You may have to click Previous to list the packages generated during Day1.</li>
 +
 
 +
[[File:2018.10.19 slicer cdash build.png]]
 +
 
 +
<li>After downloading, locally compute the md5 sums and check that they match with the sums associated with download link displayed on the download page:</li>
 +
 
 +
[[File:2018.10.19 slicer download package link.png|border]]
 +
 
 +
<li><p>Sign the package following these instructions:
 +
<ul>
 +
  <li>Linux: <small><i>No signing process, follow [[#Tag_release_packages]] instructions</i></small></li>
 +
  <li>macOS: See [[#macOS_Code_Signing]]</li>
 +
  <li>Windows: See [[#Windows_Code_Signing]]</li>
 +
</ul></p></li>
 +
 
 +
<li>'''macOS only''': Confirm that the signed package is valid by trying to install and run Slicer. Consider using a computer different from the build machine.</li>
 +
 
 +
<li>Upload the signed package using [[Documentation/Nightly/Developers/ReleaseProcess#Script_to_upload_Slicer_packages|upload-Slicer-to-slicer-kitware-com.cmake]].</li>
 +
 
 +
<li>The newly uploaded package will be listed in the Midas feed. See http://slicer.kitware.com/midas3/feed</li>
 +
 
 +
[[File:ReleaseProcess-Package-manually-uploaded-midas.png]]
 +
 
 +
<li>Manually update the name of the item removing the <code> (1)</code> suffix and appending <code> [signed]</code>. Make sure to <b>not</b> update the bitstream name. For example: see <tt>Slicer-4.8.0-win-amd64.exe [signed]</tt> available [http://slicer.kitware.com/midas3/item/316852 here]</li>
 +
 
 +
<li>For future reference, copy the unsigned packages into the folder created above (Keep the default <b>Create a reference to the existing item</b>)</li>
 +
 
 +
</ol>
 +
 
 +
=== Tag release packages ===
 +
 
 +
{{remark|red|<i>These instructions apply only to the '''Linux''' package</i>}}
 +
 
 +
Due to [[#Limitations_of_download.slicer.org|limitations]] of http://download.slicer.org, the <b>Linux</b> package must be downloaded and re-uploaded. On the other hand, the <b>macOS</b> and <b>Windows</b> packages uploaded during the manual signing are already associated with the release tag.
 +
 
 +
<ol>
 +
 
 +
<li>Download packages from [http://slicer.cdash.org/index.php?project=Slicer4&filtercount=1&showfilters=1&field1=groupname&compare1=61&value1=Experimental-Packages http://slicer.cdash.org/index.php?project=Slicer4]. You may have to click Previous to list the packages generated during Day1.</li>
 +
 
 +
[[File:2018.10.19 slicer cdash build.png]]
 +
 
 +
<li>After downloading, locally compute the md5 sums and check that they match with the sums associated with download link displayed on the download page:</li>
 +
 
 +
[[File:2018.10.19 slicer download package link.png|border]]
 +
 
 +
<li>
 +
  <p>Re-upload each packages using [[#Script_to_upload_Slicer_packages|upload-Slicer-to-slicer-kitware-com.cmake]] script.</p>
 +
  <p>For convenience, the script can be found in these directories:</p>
 +
  <ul>
 +
    <li>overload: <pre>D:\Support\slicer-tagged-release-upload</pre></li>
 +
    <li>metroplex: <pre>/home/kitware/Support/slicer-tagged-release-upload</pre></li>
 +
    <li>factory-south-macos: <pre>/Volumes/D/Support/slicer-tagged-release-upload</pre></li>
 +
  </ul>
 +
</li>
 +
 
 +
 
 +
<li>The newly uploaded packages will be listed in the Midas feed. See http://slicer.kitware.com/midas3/feed</li>
 +
 
 +
<li>After each upload, manually update the name of the item removing the <code> (1)</code> suffix. Make sure to <b>not</b> update the bitstream name.</li>
 +
 
 +
<li>For future reference, copy the packages into the folder created above (Keep the default <b>Create a reference to the existing item</b>)</li>
 +
 
 +
[[File:2018.10.18 copy-package-into-release-folder.png]]
 +
 
 +
</ol>
 +
 
 +
=== Confirm that packages are tagged ===
 +
 
 +
To confirm that all three packages are tagged. The following URL should return three packages:
 +
 
 +
http://slicer.kitware.com/midas3/api/json?method=midas.slicerpackages.package.list&release={{documentation/currentversion}}.0
 +
 
 +
=== Version NA-MIC data tree  ===
 +
 
 +
<i>This applies to the Midas instance associated with http://slicer.kitware.com</i>
  
==== Version NA-MIC data tree ====
 
  
 
<ol start="1" style="list-style-type: decimal;">
 
<ol start="1" style="list-style-type: decimal;">
 +
 +
<li><p>If you do not have an account and/or if you do not belong to the <code>DataManager</code> group:</p>
 +
 +
<ul>
  
 
<li><p>[[Documentation/{{documentation/version}}/Developers/Tutorials/ObtainExtensionServerApiKey|Create an account on the extension server and obtain an API Key]]. You will then use your midas login and api key to substitute <code><YOUR-MIDAS-LOGIN></code> and <code><YOUR-MIDAS-APIKEY></code> in the examples.</p></li>
 
<li><p>[[Documentation/{{documentation/version}}/Developers/Tutorials/ObtainExtensionServerApiKey|Create an account on the extension server and obtain an API Key]]. You will then use your midas login and api key to substitute <code><YOUR-MIDAS-LOGIN></code> and <code><YOUR-MIDAS-APIKEY></code> in the examples.</p></li>
Line 143: Line 394:
  
 
<li><p>Send an email on the developer list asking to be added to the <code>DataManager</code> group on NA-MIC community. That will grant you read/write permissions to the <code>Data</code> folder and sub-folders.</p></li>
 
<li><p>Send an email on the developer list asking to be added to the <code>DataManager</code> group on NA-MIC community. That will grant you read/write permissions to the <code>Data</code> folder and sub-folders.</p></li>
 +
 +
</ul></li>
  
 
<li><p>Install prerequisites</p>
 
<li><p>Install prerequisites</p>
 
<pre>
 
<pre>
 +
$ mkvirtualenv -p python2.7 version_namic_datatree
 
$ pip install pydas
 
$ pip install pydas
 
</pre></li>
 
</pre></li>
  
<li><p>Identify <code>id</code> of the <code>Data</code> folder. For example [http://slicer.kitware.com/midas3/folder/301 301]</p></li>
+
<li><p>Confirm that the <code>id</code> of the <code>Data</code> folder is still [http://slicer.kitware.com/midas3/folder/301 301].</p></li>
  
 
<li><p>Simulate creation of <code>X.Y</code> data folders based <code>Nightly</code> ones</p>
 
<li><p>Simulate creation of <code>X.Y</code> data folders based <code>Nightly</code> ones</p>
 +
<pre>
 +
MIDAS_EMAIL=<YOUR-MIDAS-LOGIN>
 +
MIDAS_API_KEY=<YOUR-MIDAS-APIKEY>
 +
DEST_VERSION=X.Y
 +
DATA_FOLDER_ID=301
 +
</pre>
 +
 
<pre>
 
<pre>
 
$ cd /path/to/Slicer/Base/Python/slicer/release
 
$ cd /path/to/Slicer/Base/Python/slicer/release
$ python midasdata.py --url=http://slicer.kitware.com/midas3 --data_id=301 \
+
$ python midasdata.py --url=http://slicer.kitware.com/midas3 --data_id=$DATA_FOLDER_ID \
   --email=<YOUR-MIDAS-LOGIN> --apikey=<YOUR-MIDAS-APIKEY> --source_version=Nightly --dest_version=X.Y --dry-run
+
   --email=$MIDAS_EMAIL --apikey=$MIDAS_API_KEY --source_version=Nightly --dest_version=$DEST_VERSION --dry-run
 +
</pre>
 +
 
 +
Example of output:
 +
<pre>
 
Application ( folder_id: 302 )
 
Application ( folder_id: 302 )
 
'-Nightly ( folder_id: 831 )
 
'-Nightly ( folder_id: 831 )
Line 180: Line 445:
 
<pre>
 
<pre>
 
$ python midasdata.py --url=http://slicer.kitware.com/midas3 --data_id=301 \
 
$ python midasdata.py --url=http://slicer.kitware.com/midas3 --data_id=301 \
   --email=<YOUR-MIDAS-LOGIN> --apikey=<YOUR-MIDAS-APIKEY> --source_version=Nightly --dest_version=X.Y
+
   --email=$MIDAS_EMAIL --apikey=$MIDAS_API_KEY --source_version=Nightly --dest_version=$DEST_VERSION
  
 
Versioning of the NA-MIC Data tree for release X.Y...
 
Versioning of the NA-MIC Data tree for release X.Y...
Line 197: Line 462:
 
</ol>
 
</ol>
  
=== Update Slicer wiki ===
+
=== Update external website ===
  
The copy of the pages associated with the <code>Nightly</code> namespace into the <code>X.Y</code> namespace is done using the convenience python module [https://github.com/jcfr/mwdoc#readme mwdoc].
+
* http://en.wikipedia.org/wiki/3DSlicer
 +
** Update version. See [https://en.wikipedia.org/w/index.php?title=3DSlicer&type=revision&diff=866829109&oldid=863082700 here] for an example.
 +
** Review page (list of external dependencies, ... ) and edit as needed.
 +
<s>
 +
* http://www.nitrc.org/projects/slicer/
 +
** Login
 +
** Go to Downloads -> Admin.
 +
** Click on "Edit Release" associated with slicer package.
 +
** Click on "Edit" associated the release named "Latest".
 +
** Update <tt>Release Date</tt> field.
 +
** Click on <tt>Submit/Refresh</tt>
 +
</s>
  
<ol start="1" style="list-style-type: decimal;">
+
=== Update ExtensionStats extension ===
  
<li><p>Check with <email>jchris.fillionr@kitware.com</email> to get the credential associated with <code>UpdateBot</code> user.</p></li>
+
* See https://github.com/fbudin69500/SlicerDeveloperToolsForExtensions/blob/master/ExtensionStats/ExtensionStats.py#L209
 +
** Update list of extension names
 +
** Update list of Slicer revisions
  
<li>
+
=== Publish Slicer Announcement ===
  <p>Copy <code>Nightly</code> pages into <code>X.Y</code> pages.</p>
 
  <pre>
 
$ cd ~/Projects
 
  
$ sudo pip install mwclient==0.6.5
+
* Update https://slicer.org website landing page. See https://github.com/Slicer/slicer.org
  
$ git clone git://github.com/jcfr/mwdoc
+
* Publish the discourse post created in [[#Prepare_Slicer_Announcement]].
  
$ cd mwdoc
+
* Update [[Documentation/{{documentation/currentversion}}/Announcements]] based of [[Documentation/{{documentation/prevversion}}/Announcements]]
  
$ python
+
* Update [[Documentation/{{documentation/currentversion}}/ReleaseNotes]] based of [[Documentation/{{documentation/prevversion}}/ReleaseNotes]]
  
>>> import mwdoc
+
* Publish news item to [https://www.nitrc.org/news/?group_id=50 Slicer NITRC project].
>>> doc = mwdoc.Documentation('slicer.org', '/slicerWiki/')
 
>>> doc.login('UpdateBot', 'XXXXXXX')
 
>>> doc.versionPages('Nightly', '4.3', ['Documentation', 'Template:Documentation'])
 
[INFO] Page successfully created: 'Documentation/4.3/Extensions/ErodeDilateLabel'
 
[...]
 
[INFO] Page successfully created: 'Template:Documentation/4.3/module-header'
 
[INFO] Page successfully created: 'Template:Documentation/4.3/module-section'
 
[INFO] Page successfully created: 'Template:Documentation/4.3/module/footer'
 
  </pre>
 
</li>
 
 
 
<li><p>Update [[Template:Documentation/prevversion]]</p></li>
 
 
 
<li><p>Update [[Template:Documentation/nextversion]]</p></li>
 
 
 
<li><p>Update [[Template:Documentation/currentversion]]</p></li>
 
 
 
<li><p>Update [[Template:Documentation/versionlist]]</p></li>
 
 
 
<li><p>Update [[Template:Documentation/acknowledgments-versionlist]]</p></li>
 
 
 
<li><p>Update [[FAQ]]</p></li>
 
 
 
<li><p>Update [[Documentation]]</p></li>
 
 
 
<li><p>Update [[Documentation/Release]]</p></li>
 
 
 
<li><p>Update [[Documentation/Release/Announcements]]</p></li>
 
 
 
<li><p>Update [[Documentation/Release/Acknowledgments]]</p></li>
 
 
 
<li><p>Update [[Documentation/Release/Report a problem]]</p></li>
 
 
 
<li><p>Update [[Documentation/UserTraining]]</p></li>
 
 
 
<li><p>Update [[Documentation/UserOrientation]]</p></li>
 
 
 
<li><p>Update [[Documentation/UserFeedback]]</p></li>
 
 
 
<li><p>Update [[Documentation/Release/SlicerApplication/HardwareConfiguration]]</p></li>
 
 
 
<li><p><del>Update [[Documentation/DeveloperOrientation]]</del></p></li>
 
</ol>
 
 
 
=== CDash ===
 
 
 
<ol start="1" style="list-style-type: decimal;">
 
<li><p>Create new CDash groups for extension submissions associated with <code>X.Y</code> release:</p>
 
<pre>
 
Extensions-X.Y-Nightly
 
Extensions-X.Y-Continuous
 
</pre></li>
 
 
 
</ol>
 
 
 
== ExtensionsIndex ==
 
 
 
See https://github.com/Slicer/ExtensionsIndex
 
 
 
1) Create branch X.Y based of master
 
 
 
=== Update external website ===
 
 
 
* http://en.wikipedia.org/wiki/3DSlicer
 
* http://www.nitrc.org/projects/slicer/
 
  
 +
<!--
 
=== Tag associated projects ===
 
=== Tag associated projects ===
 
* Slicer/SlicerExecutionModel
 
* Slicer/SlicerExecutionModel
Line 291: Line 502:
 
* commontk/CTK ?
 
* commontk/CTK ?
 
* commontk/PythonQt ?
 
* commontk/PythonQt ?
 +
-->
  
=== Update User Survey ===
+
== Appendix ==
  
Title of the survey should be updated.
+
=== Sign packages ===
See https://docs.google.com/forms/d/1rggxoTV5KL_vt3gX9BNNAcKH4pnL5lkMqPnklOThINg
 
  
=== Sign packages ===
+
==== macOS Code Signing ====
  
==== MacOSX ====
+
Packages have to be signed manually.
As discussed in issue [http://na-mic.org/Bug/view.php?id=3965 #3965], the code signing command is of the form:
 
  
 +
<ul>
 +
  <li>(1) connecting using SSH to <tt>sign-srv-mac.kitware.com</tt> (Available only from within Kitware internal network)</li>
 +
  <li>
 +
    <p>(2) Download signing script</p>
 +
    <pre>mkdir -p ~/tmp
 +
cd ~/tmp
 +
rm -rf macos-codesign-scripts
 +
git clone https://github.com/jcfr/macos-codesign-scripts</pre>
 +
  </li>
 +
  <li>
 +
    <p>(3) Download unsigned package using curl (using the URL reported on CDash). For example:</p>
 +
    <pre>curl -L "http://slicer.kitware.com/midas3/api/rest?method=midas.bitstream.download&name=Slicer-4.10.1-macosx-amd64.dmg&checksum=0b7b5603a3b8ec9a65a31e980703c984" -o Slicer-4.10.1-macosx-amd64.dmg</pre>
 +
  </li>
 +
<!--
 +
  <li>
 +
    <p>(4) unlock the keychain</p>
 +
    <pre>security unlock-keychain "Developer ID Application: Kitware Inc. (W38PE5Y733)"</pre>
 +
    <p>Expected output:
 
<pre>
 
<pre>
codesign --deep --force --sign "Developer ID Application: <Name-of-certificate-in-keychain>" ./Slicer.app
+
"Developer ID Application: Kitware Inc. (W38PE5Y733)"
 +
"/Users/jchris.fillionr/Library/Keychains/login.keychain-db"
 +
"/Library/Keychains/System.keychain"
 
</pre>
 
</pre>
 +
    </p>
 +
  </li>
 +
-->
 +
  <li>
 +
    <p>(4) Run signing script</p>
 +
    <pre>version=X.Y.Z
 +
cd ~/tmp
 +
./macos-codesign-scripts/sign.bash \
 +
  org.slicer.slicer ${version} \
 +
  "Developer ID Application: Kitware Inc. (W38PE5Y733)" \
 +
  "Developer ID Installer: Kitware Inc. (W38PE5Y733)" \
 +
  ./Slicer-${version}-macosx-amd64.dmg</pre>
 +
  </li>
 +
</ul>
  
where the argument to <code>--sign</code> is the name of the developer's certificate in Mac OS X Keychain.app .
+
Useful Commands:
  
'''Note: The signing of MacOSX package is not yet implemented in our nightly and release build workflows.''' See [[Documentation/Nightly/Developers/Mac_OS_X_Code_Signing]] for more information.
+
* To list of the keychain: <tt>security list-keychain</tt>
  
==== Windows ====
+
References:
 +
* [[Documentation/Nightly/Developers/Mac_OS_X_Code_Signing]]
 +
* http://na-mic.org/Bug/view.php?id=3965
 +
 
 +
==== Windows Code Signing ====
 +
 
 +
 
 +
Packages have to be signed manually.
 +
 
 +
See [[Documentation/Nightly/Developers/Windows_Code_Signing]] for more information.
  
 
TBD - See [http://na-mic.org/Bug/view.php?id=2697 #2697]
 
TBD - See [http://na-mic.org/Bug/view.php?id=2697 #2697]
  
Also see [[Documentation/Nightly/Developers/Windows_Code_Signing]] for more information.
+
<!--
 
 
 
=== Backport commit into release branch ===
 
=== Backport commit into release branch ===
  
Line 348: Line 600:
 
==== Step 4: Backport ====
 
==== Step 4: Backport ====
  
We can now cherry pick commit associated with master (trunk) into <code>master-XY</code> (Slicer-X-Y)
+
# cherry pick commits associated with master (trunk) into <code>master-XY</code> (Slicer-X-Y)
 +
# amend the commit to reference the original commit. See [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=28221 r28221] for an example.
 +
# commit to SVN using <code>git svn dcommit</code>
 +
# update associated GitHub branch using <code>git push origin master-XY</code>
  
 +
-->
 +
 +
<!--
 
==== Step 5: Create a patch release ====
 
==== Step 5: Create a patch release ====
  
 
{{wip}}
 
{{wip}}
  
<!--
 
 
Slicer trunk: Update Major/Minor/Patch version to match latest release
 
Slicer trunk: Update Major/Minor/Patch version to match latest release
 
-->
 
-->
 +
 +
=== Script to upload Slicer packages ===
 +
 +
 +
<pre>
 +
File: upload-Slicer-to-slicer-kitware-com.cmake
 +
</pre>
 +
 +
<pre>
 +
cmake_minimum_required(VERSION 3.9.0)
 +
 +
#
 +
# To run this script:
 +
#
 +
#  cmake -P upload-Slicer-to-slicer-kitware-com.cmake
 +
#
 +
 +
#
 +
# Variables to update:
 +
#
 +
#  package_dir
 +
#  cmake_dir
 +
#
 +
#  release
 +
#  source_revision
 +
#  source_checkoutdate
 +
#  MIDAS_PACKAGE_EMAIL
 +
#  MIDAS_PACKAGE_API_KEY
 +
#
 +
 +
set(package_dir        "C:/Users/dashboard/Downloads")
 +
set(cmake_dir          "D:/D/N/Slicer-1/CMake")
 +
 +
set(operating_system    "win") # linux, macosx, win
 +
 +
set(release            "4.8.0")
 +
set(source_revision    "26489")
 +
set(source_checkoutdate "2017-10-18 05:11:33") # This corresponds to the start date reported on CDash (consider converting time from UDT to EST)
 +
set(MIDAS_PACKAGE_EMAIL "<YOUR-MIDAS-LOGIN>")
 +
set(MIDAS_PACKAGE_API_KEY "<YOUR-MIDAS-APIKEY>")
 +
 +
set(ext_linux  "tar.gz")
 +
set(ext_macosx "dmg")
 +
set(ext_win    "exe")
 +
 +
set(package_filepath    "${package_dir}/Slicer-${release}-${operating_system}-amd64.${ext_${operating_system}}")
 +
 +
message(STATUS "Upload ${package_filepath}")
 +
 +
set(MIDAS_PACKAGE_URL  "http://slicer.kitware.com/midas3")
 +
set(submission_type    "experimental")
 +
set(architecture        "amd64")
 +
set(package_type        "installer")
 +
 +
list(APPEND CMAKE_MODULE_PATH
 +
  ${cmake_dir}
 +
  )
 +
include(MIDASAPIUploadPackage)
 +
 +
midas_api_upload_package(
 +
  SERVER_URL ${MIDAS_PACKAGE_URL}
 +
  SERVER_EMAIL ${MIDAS_PACKAGE_EMAIL}
 +
  SERVER_APIKEY ${MIDAS_PACKAGE_API_KEY}
 +
  SUBMISSION_TYPE ${submission_type}
 +
  SOURCE_REVISION ${source_revision}
 +
  SOURCE_CHECKOUTDATE ${source_checkoutdate}
 +
  OPERATING_SYSTEM ${operating_system}
 +
  ARCHITECTURE ${architecture}
 +
  PACKAGE_FILEPATH ${package_filepath}
 +
  PACKAGE_TYPE ${package_type}
 +
  RELEASE ${release}
 +
  RESULT_VARNAME output
 +
  )
 +
set(expected_output "ok")
 +
if(NOT "${output}" STREQUAL "${expected_output}")
 +
  message(FATAL_ERROR "Problem with midas_api_upload_package()\n"
 +
                      "output:${output}\n"
 +
                      "expected_output:${expected_output}")
 +
endif()
 +
</pre>
 +
 +
=== Limitations of download.slicer.org ===
 +
 +
Considering that
 +
 +
<p>(1) few minutes after a package is uploaded on http://slicer.kitware.com, it is indexed in the database associated with http://download.slicer.org, and </p>
 +
<p>(2) the system indexing the package does <b>NOT</b> rewrite its database entries,</p>
 +
 +
using the [[#Explicit_tagging_of_release_packages|steps below]] will <b>NOT</b> result in an update of the packages reported on http://download.slicer.org.
 +
 +
 +
Source code available at https://github.com/mhalle/slicer4-download.
 +
 +
 +
 +
=== Explicit tagging of release packages ===
 +
 +
Steps:
 +
 +
* If needed, create a X.Y.Z folder in [http://slicer.kitware.com/midas3/folder/274 Slicer/Packages/Application/Release]
 +
** Copy uploaded packages into the folder created above (Keep the default <b>Create a reference to the existing item</b>)
 +
** Identify the item_id associated with uploaded packages. For example: 316530, 316529
 +
** From Kitware internal network, SSH connect to <code>jcfr@slicer.kitware.com</code> (or <code>jcfr@192.168.120.210</code>)
 +
** Connect to mysql
 +
*** <code>password=$(cat /var/www/midas3/core/configs/database.local.ini | grep database.params.password | cut -d"=" -f2 | head -n1 | sed 's/"//g')</code>
 +
*** <code>mysql -u midas -p$password -D midas</code>
 +
*** See file <code>/var/www/midas3/core/configs/database.local.ini</code> for password
 +
*** Selected database is midas (<code>use midas</code>)
 +
** List packages associated with identified items and check they are the appropriate ones:
 +
*** <code>select *  from slicerpackages_package as p , item as i where i.item_id = p.item_id and p.item_id in (316530, 316529);</code>
 +
** Set release field
 +
*** <code>update slicerpackages_package set `release`="4.8.0" where item_id in (316530, 316529);</code>
 +
 +
=== User Survey (not used anymore) ===
 +
 +
See https://docs.google.com/forms/d/1rggxoTV5KL_vt3gX9BNNAcKH4pnL5lkMqPnklOThINg
 +
 +
With each release, the title of the survey had to be updated.
 +
 +
Now, the Slicer welcome module has been updated to ask user to report feedback on the Slicer forum
 +
 +
 +
=== Converting Google document to markdown ===
 +
 +
* (1) Open https://github.com/mangini/gdocs2md/blob/master/converttomarkdown.gapps?raw=1
 +
 +
* (2) Open Google document in an other tab
 +
 +
* (3) Go to Tools -> Script Editor
 +
 +
* (4) Replace text with content of script opened in (1)
 +
 +
* (5) Save script specifying name (e.g <tt>slicer-{{documentation/nextversion}}-converttomarkdown</tt>). <small>Note that the script is associated with the document and will have to be imported with new version of the document.</small>
 +
 +
* (6) From with the <tt>Script Editor</tt>, select <tt>Run -> Run Function -> ConvertToMarkdown</tt>
 +
 +
* (7) When asked, authorize the script to access your account and review permissions.
 +
 +
* (8) Check you email, you should receive a message entitled <tt>[MARKDOWN_MAKER] Title Of Document</tt> with an attachment named <tt>Title Of Document.md</tt>.
 +
 +
 +
References:
 +
* https://lifehacker.com/this-script-converts-google-documents-to-markdown-for-e-511746113
 +
* https://github.com/mangini/gdocs2md
 +
 +
=== Generating a Discourse compliant table of content for a Markdown document ===
 +
 +
<b>Prerequisites:</b>
 +
 +
* Install executable <tt>gh-md-toc</tt> available at https://github.com/ekalinin/github-markdown-toc in the <tt>PATH</tt>
 +
 +
 +
<b>Step-by-steps:</b>
 +
 +
Assuming the document for which a table of content should be generated is saved as <tt>/tmp/document.md</tt>:
 +
 +
* (1) Remove the <tt><nowiki>"[[TOC]]"</nowiki></tt> string left over from gdoc to markdown conversion
 +
 +
<pre>
 +
sed -i -E 's/^\[\[TOC\]\]//g' /tmp/document.md
 +
</pre>
 +
 +
* (2) Generate table of content
 +
 +
<pre>
 +
$ gh-md-toc /tmp/document.md > /tmp/toc.md
 +
</pre>
 +
 +
* (3) Create script to convert table of content and document to a format usable by discourse.
 +
 +
<pre>
 +
cat > /tmp/update-markdown-document-and-toc-for-discourse.sh << 'EOF'
 +
#!/usr/bin/env bash
 +
 +
set -eo pipefail
 +
 +
if [ $# -ne 2 ]
 +
then
 +
  echo "Usage: $0 /path/to/toc.md /path/to/document.md"
 +
  exit 0
 +
fi
 +
 +
toc_document=$1
 +
document=$2
 +
 +
for idx in $(seq 1 10); do
 +
  # Compute number of spaces expected for each level
 +
  (( space_count = ${idx} * 3 ))
 +
  spaces=$(printf %${space_count}s)
 +
 +
  echo
 +
  echo "${spaces}level ${idx}"
 +
 +
  # Extract title and anchor from each TOC entries associated with level ${idx}
 +
  IFS=''
 +
  cat ${toc_document} | ack "^${spaces}\* \[" | sed -E 's/^\s+\* \[//g' | sed -E 's/\]\(\#([0-9A-Za-z\-]+)\)/|\1/g' |
 +
  while read line; do
 +
    title=$(echo $line | cut -d"|" -f1)
 +
    anchor=$(echo $line | cut -d"|" -f2)
 +
    echo "${spaces}${title} - ${anchor}"
 +
 +
    # Replace title with html one
 +
    header=$(printf %${idx}s | tr " " "#")
 +
    sed -i -E "s|^${header} ${title}|\\<h${idx} id\\=\"heading--${anchor}\"\\>${title}\\<\\/h${idx}\\>|g" "${document}"
 +
 +
    # Replace anchor in table of content
 +
    sed -i -E "s/#${anchor}/#heading--${anchor}/g" ${toc_document}
 +
  done
 +
done
 +
EOF
 +
 +
chmod u+x /tmp/update-markdown-document-and-toc-for-discourse.sh
 +
</pre>
 +
 +
* (4) Execute script to update in place the document
 +
 +
<pre>
 +
/tmp/update-markdown-document-and-toc-for-discourse.sh /tmp/toc.md /tmp/document.md
 +
</pre>
 +
 +
* (5) Copy content of <tt>/tmp/toc.md</tt> and <tt>/tmp/document.md</tt> into a discourse post. Depending on the length of the table of content combined with the document, you may have to update the discourse settings <tt>max post length</tt>.

Latest revision as of 17:47, 23 March 2021

Home < Documentation < Nightly < Developers < ReleaseProcess

Release planning

Prerequisites

Account on the following systems are required:

For more details, see Resources

Contact <email>jcfr@kitware.com</email> or <email>sam.horvath@kitware.com</email> to request access.

Day 1: Release

Since there all development occurs on master, each time version is updated, two commits will be required.

Update Slicer.crt and commit change

See https://github.com/Slicer/Slicer/blob/master/Base/QTCore/Resources/Certs/README

CMakeLists.txt: Update the Slicer version information for the release

  1. Update the Slicer version information for the release:

    1. In CMakeLists.txt, update at least one these variables: Slicer_VERSION_MAJOR, Slicer_VERSION_MINOR, Slicer_VERSION_PATCH

    2. Re-run CMake with -DSlicer_RELEASE_TYPE:STRING=Stable. This is required to update Utilities/Scripts/SlicerWizard/__version__.py.

      Note If not doing a clean build, make sure to explicitly specify the version re-configuring the inner build with -DSlicer_VERSION_MAJOR:STRING=X -DSlicer_VERSION_MINOR:STRING=Y -DSlicer_VERSION_PATCH:STRING=Z -DSlicerApp_VERSION_MAJOR:STRING=X -DSlicerApp_VERSION_MINOR:STRING=Y -DSlicerApp_VERSION_PATCH:STRING=Z.

    3. Commit the above changes using this message like:

      ENH: Slicer X.Y.Z
      

Tag the repository

  1. Tag the repository:

    Set version variables:

    X=<major-version>
    Y=<minor-version>
    Z=<patch-version>
    echo "X.Y.Z is ${X}.${Y}.${Z}"
    
    GIT_TAG=v${X}.${Y}.${Z}
    echo "GIT_TAG is ${GIT_TAG}"
    
    git checkout master
    git tag -s -m "ENH: Slicer ${X}.${Y}.${Z}" ${GIT_TAG}
    git push origin ${GIT_TAG}
    
  2. Keep track of the GIT_TAG for the next steps

CMakeLists.txt: Update the Slicer version information for the development

  1. Update the Slicer version information for the development:

    1. In CMakeLists.txt,

      • Update Slicer_VERSION_MAJOR and/or Slicer_VERSION_MINOR
    2. Re-run CMake with -DSlicer_RELEASE_TYPE:STRING=Experimental. This is required to update Utilities/Scripts/SlicerWizard/__version__.py.

      Note If not doing a clean build, make sure to explicitly specify the version re-configuring with -DSlicer_VERSION_MAJOR:STRING=X -DSlicer_VERSION_MINOR:STRING=Y -DSlicer_VERSION_PATCH:STRING=Z .

    3. Commit the above changes using this message like:

      cd ~/Projects/Slicer
      git add -A
      git commit -m "ENH: Begin X.Y.Z development"
      

Tag and publish SlicerBuildEnvironment docker image

  1. Tag the docker build environment image. See https://github.com/Slicer/SlicerBuildEnvironment/blob/master/README.rst#maintainers

    Note If the environment did not change, you may skip this step for patch release.

    Keep track of the selected TAG for updating the release scripts

Update release scripts

  1. Update release scripts. See https://github.com/Slicer/DashboardScripts#maintenance-guides

Disable regular nightly builds

  1. On each factory machines, disable regular nightly build in crontab and task scheduler.

Update ExtensionsIndex

This applies to https://github.com/Slicer/ExtensionsIndex and you should skip this step for patch release.

  1. Create branch X.Y based of master:

     DEST_VERSION=X.Y
     
     cd /tmp
     git clone git@github.com:Slicer/ExtensionsIndex 
     cd ExtensionsIndex
     git checkout master
     git push origin master:${DEST_VERSION}
    

Update CDash

This applies to the CDash instance associated with http://slicer.cdash.org/index.php?project=Slicer4 and you should skip this step for patch release.

  1. Create new CDash groups for extension submissions associated with X.Y release:

    Extensions-X.Y-Nightly
    Extensions-X.Y-Continuous
    

Generate application and extension packages

  1. Generate packages running Slicer package scripts on each factory machines. (These are the scripts updated in the previous step)

Create release branch

You should skip this step for patch release.

  1. Create release branch:

    Set version variables (same value as in the previous step):

     X=<major-version>
     Y=<minor-version>
     Z=<patch-version>
     echo "X.Y.Z is ${X}.${Y}.${Z}"
    
     GIT_TAG=v${X}.${Y}.${Z}
     echo "GIT_TAG is ${GIT_TAG}"
    
     cd ~/Projects/Slicer
     git checkout -b master-${X}.${Y} ${GIT_TAG}
     git push origin master-${X}.${Y}
    

Update Release Details on the wiki

  1. Update Release Details with release information

Day 2: Post release

Re-enable regular nightly builds

On each factory machines, re-enable build in crontab and task scheduler.


Update Slicer wiki

This applies to the MediaWiki instance associated with https://wiki.slicer.org

There are two steps involved:

  • copy of the pages associated with the Nightly namespace into the X.Y namespace.
  • update of permanent pages referencing the current Slicer version

As of November 2016, the process is fully automated and can be done using Base/Python/slicer/release/wiki.py script.

1) Install prerequisites:

  • Check with <email>jchris.fillionr@kitware.com</email> to get the credential associated with UpdateBot user.
  • Install mwdoc:
$ pip install mwdoc

2) Query current version information:

$ python Base/Python/slicer/release/wiki.py query

3) Copy and update pages

$ DEST_VERSION=X.Y

$ python Base/Python/slicer/release/wiki.py update ${DEST_VERSION}
$ python Base/Python/slicer/release/wiki.py copy ${DEST_VERSION}

The following pages have been updated:

The following pages with #REDIRECT have been updated:

Prepare Slicer Announcement

  • (1) Reusing previous documents available in this shared Google folder, create and update the following new Google documents:

    • Slicer 4.12: Kitware Social Media (Shared Publicly)
    • Slicer 4.12: Summary, Highlights and Changelog (Shared Publicly)

      Note Updating this document is currently a tedious manual process. It will soon be streamlined by including descriptive text with each contribution to the code base.

    • Slicer 4.12: Kitware Annoucement blog
  • (2) Once the Slicer 4.12 Release notes document is mature, post it on discourse for including images.

Clean-up older nightly packages

This applies to the Midas instance associated with http://slicer.kitware.com

See https://gist.github.com/jcfr/ea9ef199bd5a3e071b8f

Manually sign packages

  1. These instructions apply only to macOS and Windows packages


  1. If needed, create a X.Y.Z folder in Slicer/Packages/Application/Release
  2. Download package from http://slicer.cdash.org/index.php?project=Slicer4. You may have to click Previous to list the packages generated during Day1.
  3. 2018.10.19 slicer cdash build.png
  4. After downloading, locally compute the md5 sums and check that they match with the sums associated with download link displayed on the download page:
  5. 2018.10.19 slicer download package link.png
  6. Sign the package following these instructions:

  7. macOS only: Confirm that the signed package is valid by trying to install and run Slicer. Consider using a computer different from the build machine.
  8. Upload the signed package using upload-Slicer-to-slicer-kitware-com.cmake.
  9. The newly uploaded package will be listed in the Midas feed. See http://slicer.kitware.com/midas3/feed
  10. ReleaseProcess-Package-manually-uploaded-midas.png
  11. Manually update the name of the item removing the (1) suffix and appending [signed]. Make sure to not update the bitstream name. For example: see Slicer-4.8.0-win-amd64.exe [signed] available here
  12. For future reference, copy the unsigned packages into the folder created above (Keep the default Create a reference to the existing item)

Tag release packages

  1. These instructions apply only to the Linux package

Due to limitations of http://download.slicer.org, the Linux package must be downloaded and re-uploaded. On the other hand, the macOS and Windows packages uploaded during the manual signing are already associated with the release tag.

  1. Download packages from http://slicer.cdash.org/index.php?project=Slicer4. You may have to click Previous to list the packages generated during Day1.
  2. 2018.10.19 slicer cdash build.png
  3. After downloading, locally compute the md5 sums and check that they match with the sums associated with download link displayed on the download page:
  4. 2018.10.19 slicer download package link.png
  5. Re-upload each packages using upload-Slicer-to-slicer-kitware-com.cmake script.

    For convenience, the script can be found in these directories:

    • overload:
      D:\Support\slicer-tagged-release-upload
    • metroplex:
      /home/kitware/Support/slicer-tagged-release-upload
    • factory-south-macos:
      /Volumes/D/Support/slicer-tagged-release-upload
  6. The newly uploaded packages will be listed in the Midas feed. See http://slicer.kitware.com/midas3/feed
  7. After each upload, manually update the name of the item removing the (1) suffix. Make sure to not update the bitstream name.
  8. For future reference, copy the packages into the folder created above (Keep the default Create a reference to the existing item)
  9. 2018.10.18 copy-package-into-release-folder.png

Confirm that packages are tagged

To confirm that all three packages are tagged. The following URL should return three packages:

http://slicer.kitware.com/midas3/api/json?method=midas.slicerpackages.package.list&release=4.10.0

Version NA-MIC data tree

This applies to the Midas instance associated with http://slicer.kitware.com


  1. If you do not have an account and/or if you do not belong to the DataManager group:

    • Create an account on the extension server and obtain an API Key. You will then use your midas login and api key to substitute <YOUR-MIDAS-LOGIN> and <YOUR-MIDAS-APIKEY> in the examples.

    • If not already done, go to NA-MIC community and click on Join community

    • Send an email on the developer list asking to be added to the DataManager group on NA-MIC community. That will grant you read/write permissions to the Data folder and sub-folders.

  2. Install prerequisites

    $ mkvirtualenv -p python2.7 version_namic_datatree
    $ pip install pydas
    
  3. Confirm that the id of the Data folder is still 301.

  4. Simulate creation of X.Y data folders based Nightly ones

    MIDAS_EMAIL=<YOUR-MIDAS-LOGIN>
    MIDAS_API_KEY=<YOUR-MIDAS-APIKEY>
    DEST_VERSION=X.Y
    DATA_FOLDER_ID=301
    
    $ cd /path/to/Slicer/Base/Python/slicer/release
    $ python midasdata.py --url=http://slicer.kitware.com/midas3 --data_id=$DATA_FOLDER_ID \
      --email=$MIDAS_EMAIL --apikey=$MIDAS_API_KEY --source_version=Nightly --dest_version=$DEST_VERSION --dry-run
    

    Example of output:

    Application ( folder_id: 302 )
    '-Nightly ( folder_id: 831 )
    '-'-Testing ( folder_id: 832 )
    '-'-'-Baseline ( folder_id: 889 )
    '-'-'-'-DiffusionTensorImagingTutorial.png ( item_id: 12067 )
    '-'-'-'-NeurosurgicalPlanningTutorial.png ( item_id: 12066 )
    '-'-'-'-SlicerTestingTest.png ( item_id: 17760 )
    '-'-'-Input ( folder_id: 833 )
    '-'-'-'-AtlasTests ( folder_id: 834 )
    '-'-'-'-'-2012-10-26-BrainAtlas.mrb ( item_id: 10276 )
    [...]
    Module: VotingBinaryHoleFillingImageFilter ( folder_id: 1491 )
    '-Nightly ( folder_id: 1491 )
    '-'-Testing ( folder_id: 1492 )
    '-'-'-Baseline ( folder_id: 1493 )
    '-'-'-'-VotingBinaryHoleFillingImageFilterTest.nhdr ( item_id: 103418 )
    '-'-'-'-VotingBinaryHoleFillingImageFilterTest.raw.gz ( item_id: 103419 )
    
    
  5. Create X.Y data tree based Nightly ones

    $ python midasdata.py --url=http://slicer.kitware.com/midas3 --data_id=301 \
      --email=$MIDAS_EMAIL --apikey=$MIDAS_API_KEY --source_version=Nightly --dest_version=$DEST_VERSION
    
    Versioning of the NA-MIC Data tree for release X.Y...
    Versioning Modules...
    [...]
    Versioning Modules...[DONE]
    Versioning Application...
    Creating folder X.Y under Application directory
    Duplicating subfolders from Nightly to X.Y...
    Duplicating subfolders from Nightly to X.Y...[DONE]
    Versioning Application...[DONE]
    Versioning of the NA-MIC Data tree for release X.Y...[DONE]
    
    

Update external website

  • http://www.nitrc.org/projects/slicer/
    • Login
    • Go to Downloads -> Admin.
    • Click on "Edit Release" associated with slicer package.
    • Click on "Edit" associated the release named "Latest".
    • Update Release Date field.
    • Click on Submit/Refresh

Update ExtensionStats extension

Publish Slicer Announcement


Appendix

Sign packages

macOS Code Signing

Packages have to be signed manually.

  • (1) connecting using SSH to sign-srv-mac.kitware.com (Available only from within Kitware internal network)
  • (2) Download signing script

    mkdir -p ~/tmp
    cd ~/tmp
    rm -rf macos-codesign-scripts
    git clone https://github.com/jcfr/macos-codesign-scripts
  • (3) Download unsigned package using curl (using the URL reported on CDash). For example:

    curl -L "http://slicer.kitware.com/midas3/api/rest?method=midas.bitstream.download&name=Slicer-4.10.1-macosx-amd64.dmg&checksum=0b7b5603a3b8ec9a65a31e980703c984" -o Slicer-4.10.1-macosx-amd64.dmg
  • (4) Run signing script

    version=X.Y.Z
    cd ~/tmp
    ./macos-codesign-scripts/sign.bash \
      org.slicer.slicer ${version} \
      "Developer ID Application: Kitware Inc. (W38PE5Y733)" \
      "Developer ID Installer: Kitware Inc. (W38PE5Y733)" \
      ./Slicer-${version}-macosx-amd64.dmg

Useful Commands:

  • To list of the keychain: security list-keychain

References:

Windows Code Signing

Packages have to be signed manually.

See Documentation/Nightly/Developers/Windows_Code_Signing for more information.

TBD - See #2697


Script to upload Slicer packages

File: upload-Slicer-to-slicer-kitware-com.cmake
cmake_minimum_required(VERSION 3.9.0)

#
# To run this script:
#
#  cmake -P upload-Slicer-to-slicer-kitware-com.cmake
#

#
# Variables to update:
#
#   package_dir
#   cmake_dir
#
#   release
#   source_revision
#   source_checkoutdate
#   MIDAS_PACKAGE_EMAIL
#   MIDAS_PACKAGE_API_KEY
#

set(package_dir         "C:/Users/dashboard/Downloads")
set(cmake_dir           "D:/D/N/Slicer-1/CMake")

set(operating_system    "win") # linux, macosx, win

set(release             "4.8.0")
set(source_revision     "26489")
set(source_checkoutdate "2017-10-18 05:11:33") # This corresponds to the start date reported on CDash (consider converting time from UDT to EST)
set(MIDAS_PACKAGE_EMAIL "<YOUR-MIDAS-LOGIN>")
set(MIDAS_PACKAGE_API_KEY "<YOUR-MIDAS-APIKEY>")

set(ext_linux  "tar.gz")
set(ext_macosx "dmg")
set(ext_win    "exe")

set(package_filepath    "${package_dir}/Slicer-${release}-${operating_system}-amd64.${ext_${operating_system}}")

message(STATUS "Upload ${package_filepath}")

set(MIDAS_PACKAGE_URL   "http://slicer.kitware.com/midas3")
set(submission_type     "experimental")
set(architecture        "amd64")
set(package_type        "installer")

list(APPEND CMAKE_MODULE_PATH
  ${cmake_dir}
  )
include(MIDASAPIUploadPackage)

midas_api_upload_package(
  SERVER_URL ${MIDAS_PACKAGE_URL}
  SERVER_EMAIL ${MIDAS_PACKAGE_EMAIL}
  SERVER_APIKEY ${MIDAS_PACKAGE_API_KEY}
  SUBMISSION_TYPE ${submission_type}
  SOURCE_REVISION ${source_revision}
  SOURCE_CHECKOUTDATE ${source_checkoutdate}
  OPERATING_SYSTEM ${operating_system}
  ARCHITECTURE ${architecture}
  PACKAGE_FILEPATH ${package_filepath}
  PACKAGE_TYPE ${package_type}
  RELEASE ${release}
  RESULT_VARNAME output
  )
set(expected_output "ok")
if(NOT "${output}" STREQUAL "${expected_output}")
  message(FATAL_ERROR "Problem with midas_api_upload_package()\n"
                      "output:${output}\n"
                      "expected_output:${expected_output}")
endif()

Limitations of download.slicer.org

Considering that

(1) few minutes after a package is uploaded on http://slicer.kitware.com, it is indexed in the database associated with http://download.slicer.org, and

(2) the system indexing the package does NOT rewrite its database entries,

using the steps below will NOT result in an update of the packages reported on http://download.slicer.org.


Source code available at https://github.com/mhalle/slicer4-download.


Explicit tagging of release packages

Steps:

  • If needed, create a X.Y.Z folder in Slicer/Packages/Application/Release
    • Copy uploaded packages into the folder created above (Keep the default Create a reference to the existing item)
    • Identify the item_id associated with uploaded packages. For example: 316530, 316529
    • From Kitware internal network, SSH connect to jcfr@slicer.kitware.com (or jcfr@192.168.120.210)
    • Connect to mysql
      • password=$(cat /var/www/midas3/core/configs/database.local.ini | grep database.params.password | cut -d"=" -f2 | head -n1 | sed 's/"//g')
      • mysql -u midas -p$password -D midas
      • See file /var/www/midas3/core/configs/database.local.ini for password
      • Selected database is midas (use midas)
    • List packages associated with identified items and check they are the appropriate ones:
      • select * from slicerpackages_package as p , item as i where i.item_id = p.item_id and p.item_id in (316530, 316529);
    • Set release field
      • update slicerpackages_package set `release`="4.8.0" where item_id in (316530, 316529);

User Survey (not used anymore)

See https://docs.google.com/forms/d/1rggxoTV5KL_vt3gX9BNNAcKH4pnL5lkMqPnklOThINg

With each release, the title of the survey had to be updated.

Now, the Slicer welcome module has been updated to ask user to report feedback on the Slicer forum


Converting Google document to markdown

  • (2) Open Google document in an other tab
  • (3) Go to Tools -> Script Editor
  • (4) Replace text with content of script opened in (1)
  • (5) Save script specifying name (e.g slicer-4.12-converttomarkdown). Note that the script is associated with the document and will have to be imported with new version of the document.
  • (6) From with the Script Editor, select Run -> Run Function -> ConvertToMarkdown
  • (7) When asked, authorize the script to access your account and review permissions.
  • (8) Check you email, you should receive a message entitled [MARKDOWN_MAKER] Title Of Document with an attachment named Title Of Document.md.


References:

Generating a Discourse compliant table of content for a Markdown document

Prerequisites:


Step-by-steps:

Assuming the document for which a table of content should be generated is saved as /tmp/document.md:

  • (1) Remove the "[[TOC]]" string left over from gdoc to markdown conversion
sed -i -E 's/^\[\[TOC\]\]//g' /tmp/document.md
  • (2) Generate table of content
$ gh-md-toc /tmp/document.md > /tmp/toc.md
  • (3) Create script to convert table of content and document to a format usable by discourse.
cat > /tmp/update-markdown-document-and-toc-for-discourse.sh << 'EOF'
#!/usr/bin/env bash

set -eo pipefail

if [ $# -ne 2 ]
then
  echo "Usage: $0 /path/to/toc.md /path/to/document.md"
  exit 0
fi

toc_document=$1
document=$2

for idx in $(seq 1 10); do
  # Compute number of spaces expected for each level
  (( space_count = ${idx} * 3 ))
  spaces=$(printf %${space_count}s)

  echo
  echo "${spaces}level ${idx}"

  # Extract title and anchor from each TOC entries associated with level ${idx}
  IFS=''
  cat ${toc_document} | ack "^${spaces}\* \[" | sed -E 's/^\s+\* \[//g' | sed -E 's/\]\(\#([0-9A-Za-z\-]+)\)/|\1/g' |
  while read line; do
    title=$(echo $line | cut -d"|" -f1)
    anchor=$(echo $line | cut -d"|" -f2)
    echo "${spaces}${title} - ${anchor}"

    # Replace title with html one
    header=$(printf %${idx}s | tr " " "#")
    sed -i -E "s|^${header} ${title}|\\<h${idx} id\\=\"heading--${anchor}\"\\>${title}\\<\\/h${idx}\\>|g" "${document}"

    # Replace anchor in table of content
    sed -i -E "s/#${anchor}/#heading--${anchor}/g" ${toc_document}
  done
done
EOF

chmod u+x /tmp/update-markdown-document-and-toc-for-discourse.sh
  • (4) Execute script to update in place the document
/tmp/update-markdown-document-and-toc-for-discourse.sh /tmp/toc.md /tmp/document.md
  • (5) Copy content of /tmp/toc.md and /tmp/document.md into a discourse post. Depending on the length of the table of content combined with the document, you may have to update the discourse settings max post length.