Difference between revisions of "Documentation/Labs/ExtensionsServer"

From Slicer Wiki
Jump to: navigation, search
Tag: 2017 source edit
Tag: 2017 source edit
 
(15 intermediate revisions by the same user not shown)
Line 2: Line 2:
 
This document summarizes the plan to update the infrastructure used to organize the Slicer application and extension packages.
 
This document summarizes the plan to update the infrastructure used to organize the Slicer application and extension packages.
  
== Transition Plan ==
+
==Status==
 +
See https://github.com/KitwareMedical/slicer-extensions-webapp/issues
  
* finalize design and implementation of new extension server (see details below)
+
<!--
* update download.slicer.org to use updated API
+
'''In progress''':
* work with sysadmin to setup server instance
 
* transition data from current system to new system
 
* update integration with Slicer app
 
  
=== Extension server ===
+
*Update frontend to integrated with Slicer: https://github.com/KitwareMedical/slicer-extensions-webapp/pull/10
 +
**Estimated completion: <s>Sept 5th</s> <s>Sept 15th</s> <S>Jan 15th</s> May 31st
  
==== Frontend ====
+
*Update Slicer to integrate with new frontend
 +
**Estimated completion: <s>July 23th</s> <s>July 30th</s> <s>Sept 15th</s> <s>Jan 15th</s> May 31st
 +
 
 +
'''Todo''':
 +
 
 +
*Setup monitoring of backend services
 +
**Estimated completion: ??
 +
 
 +
*Setup backup of backend database and assets
 +
**Estimated completion: ??
 +
 
 +
*Update download.slicer.org to interface with new backend
 +
**Estimated completion: ??
 +
 
 +
*Setup recurring tasks for uploading of backend webserver log to GitHub project for later processing
 +
**Estimated completion: <s>August 4th</s> <s>Oct 1st</s> ??
 +
 
 +
*Engage with our UI team so that we have a consistent theme between the <tt>slicer.org</tt> landing page and <tt>extensions.slicer.org</tt>
 +
**Estimated completion: <s>August 15th</s> <s>Oct 1st</s> ??
 +
 
 +
'''Completed''':
 +
 
 +
*Preliminary version of the frontend implemented. See preview [https://github.com/KitwareMedical/slicer-extensions-webapp/issues/2#issuecomment-650374057 here]
 +
*Transition backend from Girder 2.x to 3.x. See https://github.com/girder/slicer_package_manager/pull/78
 +
*Finalize provisioning scripts for deploying backend
 +
*Setup netlify project to deploy the frontend (available as <tt>https://extensions-slicer-org.netlify.app</tt>)
 +
*Work with Kitware sysadmin to deploy backend (will be available as <tt>https://slicer-packages.kitware.com</tt>)
 +
* Setup redirect from <tt>https://extensions.slicer.org</tt> to <tt>https://extensions-slicer-org.netlify.app</tt>
 +
-->
 +
 
 +
==Source code==
 +
 
 +
*Frontend: https://github.com/KitwareMedical/slicer-extensions-webapp/
 +
*Backend: https://github.com/girder/slicer_package_manager
 +
*Provisioning scripts (internal to Kitware): [https://kwgitlab.kitware.com/data-analytics-deployments/slicer-packages.kitware.com here]
 +
 
 +
==Transition Plan==
 +
 
 +
*finalize design and implementation of new extension server (see details below)
 +
*update download.slicer.org to use updated API
 +
*work with sysadmin to setup server instance
 +
*transition data from current system to new system
 +
*update integration with Slicer app
 +
 
 +
===Extension server===
 +
 
 +
====Frontend====
  
 
Requirements:
 
Requirements:
  
* display only extensions (it will <b>not</b> replace download.slicer.org)
+
*source code will be available at https://github.com/KitwareMedical/slicer-extensions-webapp
* use standard web/js library (
+
*display only extensions (it will <b>not</b> replace download.slicer.org)
* no direct dependency on Girder
+
*use standard web/js library (
* readonly
+
*no direct dependency on Girder
* leverage the backend API
+
*readonly
* deployed using netlify - this will allow anyone in the community  to submit PR and look at preview
+
*leverage the backend API
* URL will be https://extensions.slicer.org  
+
*deployed using netlify - this will allow anyone in the community  to submit PR and look at preview
 +
*URL will be https://extensions.slicer.org
  
 
Preliminary work:
 
Preliminary work:
* [https://github.com/Pierre-Assemat/Vuejs-Project/issues/4#issue-428895494 draft design]
 
* [https://discourse.slicer.org/t/slicer-extension-manager-user-interface-feedback/2475 user-feedback])
 
  
==== backend ====
+
*[https://github.com/Pierre-Assemat/Vuejs-Project/issues/4#issue-428895494 draft design]
 +
*[https://discourse.slicer.org/t/slicer-extension-manager-user-interface-feedback/2475 user-feedback])
 +
 
 +
====backend====
  
 
Requirements:
 
Requirements:
* leverage the existing work. See https://slicer-package-manager.readthedocs.io/en/latest/user_guide.html
+
 
* deployed by Kitware Sysadmin and publicly available as https://slicer-packages.kitware.com
+
*leverage the existing work. See https://slicer-package-manager.readthedocs.io/en/latest/user_guide.html
** Also discussed the possibility of hosting this on Digital Ocean to allow maintenance with community. But this would likely make backup of the database and restoring of the system more complex.
+
*deployed by Kitware Sysadmin and publicly available as https://slicer-packages.kitware.com
* used to manage both application and extension packages
+
**Also discussed the possibility of hosting this on Digital Ocean to allow maintenance with community. But this would likely make backup of the database and restoring of the system more complex.
* extension download stats will be simple download count per Extension per release
+
*used to manage both application and extension packages
* web server logs associated with the backend would be uploaded daily to a private Github project. This would allow the community to generate detailed stats. Similarly to what is done with mhalle/slicer4-download/slicer4-parselogs
+
*extension download stats will be simple download count per Extension per release
 +
*web server logs associated with the backend would be uploaded daily to a private Github project. This would allow the community to generate detailed stats. Similarly to what is done with mhalle/slicer4-download/slicer4-parselogs
  
 
Optional:
 
Optional:
* ideally Slicer package will be uploaded as GitHub release assets by the build machine and only a URL would be uploaded to Girder. The backend would return a redirect code
 
  
== Current infrastructure ==
+
*ideally Slicer package will be uploaded as GitHub release assets by the build machine and only a URL would be uploaded to Girder. The backend would return a redirect code
 +
 
 +
==Current infrastructure==
  
 
The current infrastructure is on "life-support":
 
The current infrastructure is on "life-support":
  
* the operating system is out-dated, we are working with sysadmin to update it while keeping an older php version (which itself will go end-of-life by the end of the year)
+
*the operating system is out-dated, we are working with sysadmin to update it while keeping an older php version (which itself will go end-of-life by the end of the year)
* we are having more and more failed upload because the server require more and more memory.  
+
*we are having more and more failed upload because the server require more and more memory.  
** For the past 5 years, we kept increasing the memory. The root cause is a design flow in Midas. See https://github.com/midasplatform/Midas/issues/69
+
**For the past 5 years, we kept increasing the memory. The root cause is a design flow in Midas. See https://github.com/midasplatform/Midas/issues/69
** Recent changes helped mitigate this. See [https://github.com/Slicer/Midas/commit/0d609469f69e464080ee02dd0eaf6f615bc17ff9 Slicer/Midas@0d60946]
+
**Recent changes helped mitigate this. See [https://github.com/Slicer/Midas/commit/0d609469f69e464080ee02dd0eaf6f615bc17ff9 Slicer/Midas@0d60946]
* more and more user are [https://discourse.slicer.org/t/multiple-issues-with-extension-builds/6160 reporting] problems with failed upload and download.
+
*more and more user are [https://discourse.slicer.org/t/multiple-issues-with-extension-builds/6160 reporting] problems with failed upload and download.
  
== Background ==
+
==Background==
  
 
During the past two years, we have been working toward phasing out the aging Midas based server used to host, serve and manage Slicer application and extension packages.
 
During the past two years, we have been working toward phasing out the aging Midas based server used to host, serve and manage Slicer application and extension packages.
  
* (1) We now have a Girder plugin for the backend (meaning API endpoints and database interaction). See https://slicer-package-manager.readthedocs.io
+
*(1) We now have a Girder plugin for the backend (meaning API endpoints and database interaction). See https://slicer-package-manager.readthedocs.io
* (2) We also experimented with few prototype regarding the extension manager but have nothing production ready.
+
*(2) We also experimented with few prototype regarding the extension manager but have nothing production ready.
  
 
(1) and (2) were done by one of our former intern Pierre Assemat and was funded by the D&A group.
 
(1) and (2) were done by one of our former intern Pierre Assemat and was funded by the D&A group.

Latest revision as of 02:11, 2 September 2021

Home < Documentation < Labs < ExtensionsServer

This document summarizes the plan to update the infrastructure used to organize the Slicer application and extension packages.

Status

See https://github.com/KitwareMedical/slicer-extensions-webapp/issues


Source code

Transition Plan

  • finalize design and implementation of new extension server (see details below)
  • update download.slicer.org to use updated API
  • work with sysadmin to setup server instance
  • transition data from current system to new system
  • update integration with Slicer app

Extension server

Frontend

Requirements:

Preliminary work:

backend

Requirements:

  • leverage the existing work. See https://slicer-package-manager.readthedocs.io/en/latest/user_guide.html
  • deployed by Kitware Sysadmin and publicly available as https://slicer-packages.kitware.com
    • Also discussed the possibility of hosting this on Digital Ocean to allow maintenance with community. But this would likely make backup of the database and restoring of the system more complex.
  • used to manage both application and extension packages
  • extension download stats will be simple download count per Extension per release
  • web server logs associated with the backend would be uploaded daily to a private Github project. This would allow the community to generate detailed stats. Similarly to what is done with mhalle/slicer4-download/slicer4-parselogs

Optional:

  • ideally Slicer package will be uploaded as GitHub release assets by the build machine and only a URL would be uploaded to Girder. The backend would return a redirect code

Current infrastructure

The current infrastructure is on "life-support":

  • the operating system is out-dated, we are working with sysadmin to update it while keeping an older php version (which itself will go end-of-life by the end of the year)
  • we are having more and more failed upload because the server require more and more memory.
  • more and more user are reporting problems with failed upload and download.

Background

During the past two years, we have been working toward phasing out the aging Midas based server used to host, serve and manage Slicer application and extension packages.

  • (1) We now have a Girder plugin for the backend (meaning API endpoints and database interaction). See https://slicer-package-manager.readthedocs.io
  • (2) We also experimented with few prototype regarding the extension manager but have nothing production ready.

(1) and (2) were done by one of our former intern Pierre Assemat and was funded by the D&A group.