Difference between revisions of "Documentation/Labs/TransitionToGit"

From Slicer Wiki
Jump to: navigation, search
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
This page keep tracks of the progress done toward the conversion from (1) Slicer svn repository mirror onto github into (2) a only git repository.
 
This page keep tracks of the progress done toward the conversion from (1) Slicer svn repository mirror onto github into (2) a only git repository.
  
= Related issues =
+
= Removing binary data from history =
 +
 
 +
Motivation:
 +
* Speedup time required to checkout Slicer source code.
 +
* Quote "Because of the decentralized nature of Git, which means every developer has the full change history on his or her computer, changes in large binary files cause Git repositories to grow by the size of the file in question every time the file is changed and the change is committed. The growth directly affects the amount of data end users need to retrieve when they need to clone the repository." See https://www.perforce.com/blog/storing-large-binary-files-in-git-repositories
 +
 
 +
Tasks:
 +
* Filter the history
 +
** Fix [[#Authorship]] and [[#Commit_message]]
 +
** experiment with history filter and compare with above stats
 +
* Remove file above <= X MB that have a poor compression ratio. See [https://gist.github.com/jcfr/4348af13d2c8931daeab4ff9ab73e14b git_list_largest_file_from_history.sh] and [https://gist.github.com/jcfr/93fe51974d9db8ef55a6d3172c1de68d slicer_git_history_350_largest_files.txt]
 +
* Integrate a NOTE with location of files and checksum
 +
* Distribution of binary files across revision.
 +
** Leverage CMake ExternalData module. Existing modules need to be updated to leverage it. Work-in-progress in [https://nbviewer.jupyter.org/github/jcfr/jupyter-notebooks/blob/master/02_Update_Slicer_CLI_buildsystem_to_download_test_data_from_midas.ipynb 02_Update_Slicer_CLI_buildsystem_to_download_test_data_from_midas.ipynb] notebook to update buildsystem.
 +
** Improve the SampleData module. See https://discourse.slicer.org/t/improving-testing-data-management-for-self-test/
 +
** Copy testing data to https://github.com/Slicer/SlicerTestingData and work with Kitware system admin to set redirect for existing datasets.
 +
 
 +
Some stats:
 +
* Hina, Jc (from Kitware Carrboro): 3min30sec
 +
* Andrey (SPL): 2m13s
 +
* Steve (Cambridge): 21s (full history),  8s (with --depth=1)
 +
 
 +
== Related issues ==
 
* {{done}} [http://na-mic.org/Mantis/view.php?id=2834 2834: Consolidate svn and git hooks]
 
* {{done}} [http://na-mic.org/Mantis/view.php?id=2834 2834: Consolidate svn and git hooks]
 
* {{wip}} [http://na-mic.org/Mantis/view.php?id=2060 2060: Move testing data onto Midas]
 
* {{wip}} [http://na-mic.org/Mantis/view.php?id=2060 2060: Move testing data onto Midas]
Line 7: Line 29:
 
* [http://na-mic.org/Mantis/view.php?id=2059 2059: Transition to Git]
 
* [http://na-mic.org/Mantis/view.php?id=2059 2059: Transition to Git]
  
= Authorship =
+
== Authorship ==
  
 
Since contribution have been [[Slicer:git-svn#Integrating_topic_from_external_contributor|integrated]] using the <code>svn dcommit --add-from-author</code>, the history should be filtered to change the commit author as specified in the <code>From:</code> statement.
 
Since contribution have been [[Slicer:git-svn#Integrating_topic_from_external_contributor|integrated]] using the <code>svn dcommit --add-from-author</code>, the history should be filtered to change the commit author as specified in the <code>From:</code> statement.
  
 
In addition, the following update should be considered:
 
In addition, the following update should be considered:
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=22869 r22869] -> Franklinwk <franklin.king@queensu.ca>
+
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=22869 r22869] -> <code>Franklinwk <franklin.king@queensu.ca></code>
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=23080 r23080] -><Code>Co-authored-by: Andras Lasso <lasso@queensu.ca></code>
+
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=23080 r23080] -> <Code>Co-authored-by: Andras Lasso <lasso@queensu.ca></code>
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=23506 r23506] -> Andras Lasso <lasso@queensu.ca>
+
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=23506 r23506] -> <code>Andras Lasso <lasso@queensu.ca></code>
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=23758 r23758] -> Andras Lasso <lasso@queensu.ca>
+
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=23758 r23758] -> <code>Andras Lasso <lasso@queensu.ca></code>
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=24061 r24061] -> Alireza Mehrtash <mehrtash@bwh.harvard.edu>
+
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=24061 r24061] -> <code>Alireza Mehrtash <mehrtash@bwh.harvard.edu></code>
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=24309 r24309] -> Andras Lasso <lasso@queensu.ca>
+
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=24309 r24309] -> <code>Andras Lasso <lasso@queensu.ca></code>
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=24631 r24631] -><Code>Co-authored-by: Max Smolens <max.smolens@kitware.com></code>
+
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=24631 r24631] -> <Code>Co-authored-by: Max Smolens <max.smolens@kitware.com></code>
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=24720 r24720] -> Change author email to zhangfanmark@gmail.com
+
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=24720 r24720] -> Change author email to <code>zhangfanmark@gmail.com</code>
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=25140 r25140] -> Hastings Greer <hastings.greer@kitware.com>
+
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=25140 r25140] -> <code>Hastings Greer <hastings.greer@kitware.com></code>
 
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=25744 r25744] -> Add <code>Suggested-by: Alexis Girault <alexis.girault@kitware.com></code>
 
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=25744 r25744] -> Add <code>Suggested-by: Alexis Girault <alexis.girault@kitware.com></code>
 +
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=26841 r26841] -> <code>From: Gregory C. Sharp <gregsharp.geo@yahoo.com></code>, <code>Co-authored-by: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com></code>
 +
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=26856 r26856] -> <code>Adam Rankin <adam.rankin@gmail.com></code>
 +
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=26987 r26987] -> <code>Fernando Perez-Garcia <fepegar@gmail.com></code>
 +
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=28318 r28318] -> <code>lopezmt <lpzmateo@gmail.com></code>, <tt>lopezmt</tt> -> <tt>Mateo Lopez</tt>
 +
* [http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=28328 r28328] -> <Code>Kyle Sunderland <sunderlandkyl@gmail.com></code>, <code>Co-authored-by: Andras Lasso <lasso@queensu.ca></code>
  
= Commit message =
+
== Commit message ==
  
 
The following commit message could be fixed:
 
The following commit message could be fixed:
Line 52: Line 79:
 
</pre>
 
</pre>
  
== Removing binary data from history ==
+
= Redirect to setup =
 +
 
 +
* URL like http://svn.slicer.org/Slicer4/trunk/CMake/SlicerDashboardDriverScript.cmake should redirect to the corresponding GitHub URL
 +
 
 +
 
 +
 
 +
= Setup infrastructure to streamline writing of release notes =
 +
 
 +
Motivation:
 +
* Reduce [[Documentation/Nightly/Developers/ReleaseProcess|Release Process]] overall time by streamlining the generation of the list of improvements and fixes associated with Release announcement. Compiling this list currently consists in scraping the list of commits and PRs since the last release.
 +
* Involve developers of features and fixes in the writing and review of release note
 +
* Always have an up-to-date list documenting the list of features and fixes associated with the upcoming release
 +
 
 +
Possible Approach:
 +
* Developers should add a `name-of-the-note.rst` file for each topic branch making a noteworthy change. Each document should be named and titled to match the topic name to avoid merge conflicts. This is similar to approach used by CMake (see https://github.com/Kitware/CMake/tree/master/Help/release/dev). When releasing the collection of files is combined into one document.
 +
* When relevant, images should be associated.
 +
* Release highlight would still be manually crafted.
  
* Filter the history
+
= Useful References =
* Remove file above <= X MB
 
* Integrate a NOTE with location of files and crypto hash
 
  
 +
* Ruby recently transitioned to Git (2019-04-23)
 +
** https://www.ruby-lang.org/en/news/2019/04/23/move-to-git-from-svn/
 +
** https://bugs.ruby-lang.org/issues/14470: Use Commit together with co-authors Github feature in svn commits
 +
** https://bugs.ruby-lang.org/issues/14632: Transition notes
  
Some stats:
+
* Python usage of GitHub issues
* Hina, Jc (from Kitware Carrboro): 3min30sec
+
** PEP 581 - Using GitHub Issues: https://discuss.python.org/t/pep-581-using-github-issues/535/
* Andrey (SPL): 2m13s
+
** Bot for GitHub labels: https://github.com/python/core-workflow/issues/239
* Steve (Cambridge): 21s (full history),  8s (with --depth=1)
 

Revision as of 19:12, 9 August 2019

Home < Documentation < Labs < TransitionToGit

This page keep tracks of the progress done toward the conversion from (1) Slicer svn repository mirror onto github into (2) a only git repository.

Removing binary data from history

Motivation:

  • Speedup time required to checkout Slicer source code.
  • Quote "Because of the decentralized nature of Git, which means every developer has the full change history on his or her computer, changes in large binary files cause Git repositories to grow by the size of the file in question every time the file is changed and the change is committed. The growth directly affects the amount of data end users need to retrieve when they need to clone the repository." See https://www.perforce.com/blog/storing-large-binary-files-in-git-repositories

Tasks:

Some stats:

  • Hina, Jc (from Kitware Carrboro): 3min30sec
  • Andrey (SPL): 2m13s
  • Steve (Cambridge): 21s (full history), 8s (with --depth=1)

Related issues

Authorship

Since contribution have been integrated using the svn dcommit --add-from-author, the history should be filtered to change the commit author as specified in the From: statement.

In addition, the following update should be considered:

  • r22869 -> Franklinwk <franklin.king@queensu.ca>
  • r23080 -> Co-authored-by: Andras Lasso <lasso@queensu.ca>
  • r23506 -> Andras Lasso <lasso@queensu.ca>
  • r23758 -> Andras Lasso <lasso@queensu.ca>
  • r24061 -> Alireza Mehrtash <mehrtash@bwh.harvard.edu>
  • r24309 -> Andras Lasso <lasso@queensu.ca>
  • r24631 -> Co-authored-by: Max Smolens <max.smolens@kitware.com>
  • r24720 -> Change author email to zhangfanmark@gmail.com
  • r25140 -> Hastings Greer <hastings.greer@kitware.com>
  • r25744 -> Add Suggested-by: Alexis Girault <alexis.girault@kitware.com>
  • r26841 -> From: Gregory C. Sharp <gregsharp.geo@yahoo.com>, Co-authored-by: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com>
  • r26856 -> Adam Rankin <adam.rankin@gmail.com>
  • r26987 -> Fernando Perez-Garcia <fepegar@gmail.com>
  • r28318 -> lopezmt <lpzmateo@gmail.com>, lopezmt -> Mateo Lopez
  • r28328 -> Kyle Sunderland <sunderlandkyl@gmail.com>, Co-authored-by: Andras Lasso <lasso@queensu.ca>

Commit message

The following commit message could be fixed:

STYLE: Add test for CLI Node status update

This commit adds a test checking that CLI status has the expected
value following StatusModifiedEvent.

Possible status are the following:

    Idle
    Scheduled
    Running
    Cancelling
    Cancelled
    Completing
    Completed
    ErrorsMask
    CompletedWithErrors
    BusyMask

Note that the status is inconsistently update between synchronous and asynchronous mode. This
will be addressed in a follow up commit.

Redirect to setup


Setup infrastructure to streamline writing of release notes

Motivation:

  • Reduce Release Process overall time by streamlining the generation of the list of improvements and fixes associated with Release announcement. Compiling this list currently consists in scraping the list of commits and PRs since the last release.
  • Involve developers of features and fixes in the writing and review of release note
  • Always have an up-to-date list documenting the list of features and fixes associated with the upcoming release

Possible Approach:

  • Developers should add a `name-of-the-note.rst` file for each topic branch making a noteworthy change. Each document should be named and titled to match the topic name to avoid merge conflicts. This is similar to approach used by CMake (see https://github.com/Kitware/CMake/tree/master/Help/release/dev). When releasing the collection of files is combined into one document.
  • When relevant, images should be associated.
  • Release highlight would still be manually crafted.

Useful References