Slicer3:UIDesign:WorkingProblems:SlicerInformatics:Draft1 - Revision history

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-03-08T15:11:27Z

FetchMI ToDo List for Slicer3.4 code freeze

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-03-07T23:09:38Z

FetchMI ToDo List for Slicer3.4 code freeze

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-03-07T23:06:42Z

FetchMI ToDo List for Slicer3.4 code freeze

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-03-07T21:51:28Z

FetchMI ToDo List for Slicer3.4 code freeze

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-03-07T21:41:21Z

FetchMI ToDo List for Slicer3.4 code freeze

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-02-26T18:58:40Z

FetchMI ToDo List for Slicer3.4 code freeze

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-02-26T18:56:31Z

FetchMI ToDo List for Slicer3.4 code freeze

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-02-18T16:10:58Z

FetchMI ToDo List for Slicer3.4 code freeze

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-02-18T15:06:43Z

FetchMI ToDo List for Slicer3.4 code freeze

Wjp: /* FetchMI ToDo List for Slicer3.4 code freeze */

2009-02-16T19:55:25Z

FetchMI ToDo List for Slicer3.4 code freeze

← Older revision		Revision as of 15:11, 8 March 2009
Line 110:		Line 110:
	*** For instance: Save Data pipeline calls a storageNode's WriteData method first, which writes data to disk. If a uri is specified for the data, the call to WriteData writes to cache, and a superclass method called 'StageWriteData' writes from cache to the remote host using the informatics tools. FreeSurfer storable nodes have no writers! So there's no way to get the to write to cache;		*** For instance: Save Data pipeline calls a storageNode's WriteData method first, which writes data to disk. If a uri is specified for the data, the call to WriteData writes to cache, and a superclass method called 'StageWriteData' writes from cache to the remote host using the informatics tools. FreeSurfer storable nodes have no writers! So there's no way to get the to write to cache;
	*** Approach: vtkMRMLFreeSurferModelStorageNode and vtkMRMLFreeSuferModelOverlayStorageNode get an optional CopyData method which acts like write data, but does a disk copy from source to cache, and then an upload from cache to remote host. Not so great, but we'll test to see if it works.		*** Approach: vtkMRMLFreeSurferModelStorageNode and vtkMRMLFreeSuferModelOverlayStorageNode get an optional CopyData method which acts like write data, but does a disk copy from source to cache, and then an upload from cache to remote host. Not so great, but we'll test to see if it works.
		+	** Would like to save known servers into the application registry so they are available when user starts up...

@@ Line 85: / Line 85: @@
 *Logic & MRML
 ** trim off last ampersand in query
-** make sure all query, delete, DL, UL commands exist in logic (so can be called from commandline)
+** ask user to save data locally first in case the remote save fails (while debugging anyway).
 ** after a scene close event, remove all tags from scene's UserTagTable
 ** If scene is renamed, ResourceUpload widget doesn't update.
-** Restore after MID-UPLoad/Download crapout -- need to reset write or read state on all selected resources; make sure filenames point to cache files or to final/original destination. (DONE)
+** Restore after MID-UPLoad/Download crapout -- need to reset write or read state on all selected resources; make sure filenames point to cache files or to final/original destination. (DONE needs more testing tho)
 * GUI
 ** put up a 'wait' message (or write text in status bar)
 *** figure out what's overriding the status text when server is selected & query is made (still hmmm)
 ** when a scene is closed, remove all tags from the empty scene's tag table.
 ** when a tag is deleted from datasets, raise the tagViewer with updated info
@@ Line 116: / Line 97: @@
 ** find and fix debug leaks: appears to be part of the TagTableCollection (done).
 ** want to be able to download any kind of Volume or Model or MRML file. Problem: inferring what kind of node to create from the uri of a remote resource. Currently, when a filename is passed in on the command line, loader.tcl parses out the filename and sends it to the AddDataWidget's logic; this figures out what kind of nodes to create for the data. When a filename is loaded through AddVolumes, AddModel, AddTransform, etc., the node is created directly. Probably need to build logic like the AddDataWidget's logic, that infers the node type from the filename -- if the uri contains a valid filename.
 * DebugNotes:
 ** Slightly confusing lack of parallelism in logic: In a storage node, the first FileName is repeated as the first entry in FileNameList, but the first URI is *not* repeated in URIList. To get a uri that corresponds to a filename, need to do a little index jigger. Principle code changes for this are in vtkDataIOManagerLogic in Base/Logic. I'm not familiar enough with the ArchetypeStorageNode to be confident in making a change -- Maybe Nicole can address?
 ** Added a modification to set up FileNameList initially when a file series is read -- (previously only getting set up in the write.) Seems ok now -- will wait for Nicole to test. Principle changes occur in vtkMRMLVolumeArchetypeStorageNode.cxx/h
 ** Fix: We are requiring a SlicerDataType tag on all storable nodes that can get uploaded to a remote server, and downloaded from a remote server. Why necessary? when datasets are encapsulated by mrml scene description and a scene is downloaded, we can use the scene context to decide what kind of storable node to create to hold the data about to be downloaded. When individual datasets come down (we want to be able to accommodate this), Slicer needs to create a storage node of a particular type, but has no way to know which (and cannot reliably infer this from the filename or extension.)
 ** CHALLENGE: We are trying to do all tagging of storabel nodes from an external informatics module by looking for node created events on the scene, testing to see what the node is, and then tagging its storage node with an appropriate value for SlicerDataType. However, it's possible to miss nodes entirely. If they don't get tagged, they don't get uploaded by logic, since we won't be able to reliably download them later. First, FreeSurfer models can have scalar overlays; when these nodes are created and added to a model display node, no events are available to tell us how to tag this from inside an external informatics module.  Second, fiducialList nodes are created without storage nodes if no fiducail points are immediately added, so no events triggered when user adds fiducials to a list after some time.

@@ Line 129: / Line 129: @@
 ** CHALLENGE: We are trying to do all tagging of storabel nodes from an external informatics module by looking for node created events on the scene, testing to see what the node is, and then tagging its storage node with an appropriate value for SlicerDataType. However, it's possible to miss nodes entirely. If they don't get tagged, they don't get uploaded by logic, since we won't be able to reliably download them later. First, FreeSurfer models can have scalar overlays; when these nodes are created and added to a model display node, no events are available to tell us how to tag this from inside an external informatics module.  Second, fiducialList nodes are created without storage nodes if no fiducail points are immediately added, so no events triggered when user adds fiducials to a list after some time.
 *** Approach: MRMLScene gets a new event MetadataAddedEvent
-*** StorableNode gets new members, SlicerDataType and methods to set it and invoke an event, and methods to get it. We do this in SlicerBase, instead of in an external informatics module. Values for these types should be unambiguous and semantically meaningful to someone looking at their markup..
+*** StorableNode gets new member ''SlicerDataType'' and methods to set it and invoke an event, and methods to get it. We do this in SlicerBase, instead of in an external informatics module. Values for these types should be unambiguous and semantically meaningful to someone looking at their markup..
-*** Currently Supported SlicerDataTypes: MRML, ScalarVolume, LabelMap, VTKModel, FreeSurferModel, FreeSurferModelOverlay, DTIVolume, DWIVolume, UnstructuredGrid, FiducialList, ColorTable.
+*** Currently Supported SlicerDataTypes: MRML, ScalarVolume, LabelMap, VTKModel, FreeSurferModel, FreeSurferModelWithOverlay, DTIVolume, DWIVolume, UnstructuredGrid, FiducialList, ColorTable.
-*** For now, we'll set SlicerDataType on the nodes from FetchMILogic for everyone we can; but for Fiducials, and FreeSurferScalarOverlays, we'll set those in the nodes themselves to ensure they get set.
+*** For now, we'll set SlicerDataType on the nodes from FetchMILogic for everyone we can; but for FreeSurferScalarOverlays, we'll set those in the nodes themselves to ensure they get set. We tried to catch them all with ''NodeAddedEvents'' posted by the scene, but not all storage nodes are created when the storable node is created, and if they are created ''later'', they are usually added under ''no notify'' conditions.
 *** NEED A CONSISTENT WAY TO DO THIS that hopefully doesn't require touching all these node classes...
-** CHALLENGE 2: Save Data pipeline calls a storageNode's WriteData method first, which writes data to disk. If a uri is specified for the data, the call to WriteData writes to cache, and a superclass method called 'StageWriteData' writes from cache to the remote host using the informatics tools. FreeSurfer storable nodes have no writers! So there's no way to get the to write to cache;
+** Once all storable nodes in the scene are tagged, FetchMILogic can post them to a remote host, handling any differences in the save data pipeline for differnt types of nodes.
 *** Approach: vtkMRMLFreeSurferModelStorageNode and vtkMRMLFreeSuferModelOverlayStorageNode get an optional CopyData method which acts like write data, but does a disk copy from source to cache, and then an upload from cache to remote host. Not so great, but we'll test to see if it works.

@@ Line 129: / Line 129: @@
 ** CHALLENGE: We are trying to do all tagging of storabel nodes from an external informatics module by looking for node created events on the scene, testing to see what the node is, and then tagging its storage node with an appropriate value for SlicerDataType. However, it's possible to miss nodes entirely. If they don't get tagged, they don't get uploaded by logic, since we won't be able to reliably download them later. First, FreeSurfer models can have scalar overlays; when these nodes are created and added to a model display node, no events are available to tell us how to tag this from inside an external informatics module.  Second, fiducialList nodes are created without storage nodes if no fiducail points are immediately added, so no events triggered when user adds fiducials to a list after some time.
 *** Approach: MRMLScene gets a new event MetadataAddedEvent
-*** StorableNode gets new members, SlicerDataType and methods to set it and invoke an event, and methods to get it. We do this in SlicerBase, instead of in an external informatics module. This seems fair, since this functionality is intended to move into slicer base load/save eventually anyway.
+*** StorableNode gets new members, SlicerDataType and methods to set it and invoke an event, and methods to get it. We do this in SlicerBase, instead of in an external informatics module. Values for these types should be unambiguous and semantically meaningful to someone looking at their markup..
 ** CHALLENGE 2: Save Data pipeline calls a storageNode's WriteData method first, which writes data to disk. If a uri is specified for the data, the call to WriteData writes to cache, and a superclass method called 'StageWriteData' writes from cache to the remote host using the informatics tools. FreeSurfer storable nodes have no writers! So there's no way to get the to write to cache;
 *** Approach: vtkMRMLFreeSurferModelStorageNode and vtkMRMLFreeSuferModelOverlayStorageNode get an optional CopyData method which acts like write data, but does a disk copy from source to cache, and then an upload from cache to remote host. Not so great, but we'll test to see if it works.

← Older revision		Revision as of 21:41, 7 March 2009
Line 125:		Line 125:
	** Slightly confusing lack of parallelism in logic: In a storage node, the first FileName is repeated as the first entry in FileNameList, but the first URI is not repeated in URIList. To get a uri that corresponds to a filename, need to do a little index jigger. Principle code changes for this are in vtkDataIOManagerLogic in Base/Logic. I'm not familiar enough with the ArchetypeStorageNode to be confident in making a change -- Maybe Nicole can address?		** Slightly confusing lack of parallelism in logic: In a storage node, the first FileName is repeated as the first entry in FileNameList, but the first URI is not repeated in URIList. To get a uri that corresponds to a filename, need to do a little index jigger. Principle code changes for this are in vtkDataIOManagerLogic in Base/Logic. I'm not familiar enough with the ArchetypeStorageNode to be confident in making a change -- Maybe Nicole can address?
	** Added a modification to set up FileNameList initially when a file series is read -- (previously only getting set up in the write.) Seems ok now -- will wait for Nicole to test. Principle changes occur in vtkMRMLVolumeArchetypeStorageNode.cxx/h		** Added a modification to set up FileNameList initially when a file series is read -- (previously only getting set up in the write.) Seems ok now -- will wait for Nicole to test. Principle changes occur in vtkMRMLVolumeArchetypeStorageNode.cxx/h
		+	* NEW
		+	** Fix: We are requiring a SlicerDataType tag on all storable nodes that can get uploaded to a remote server, and downloaded from a remote server. Why necessary? when datasets are encapsulated by mrml scene description and a scene is downloaded, we can use the scene context to decide what kind of storable node to create to hold the data about to be downloaded. When individual datasets come down (we want to be able to accommodate this), Slicer needs to create a storage node of a particular type, but has no way to know which (and cannot reliably infer this from the filename or extension.)
		+	** CHALLENGE: We are trying to do all tagging of storabel nodes from an external informatics module by looking for node created events on the scene, testing to see what the node is, and then tagging its storage node with an appropriate value for SlicerDataType. However, it's possible to miss nodes entirely. If they don't get tagged, they don't get uploaded by logic, since we won't be able to reliably download them later. First, FreeSurfer models can have scalar overlays; when these nodes are created and added to a model display node, no events are available to tell us how to tag this from inside an external informatics module. Second, fiducialList nodes are created without storage nodes if no fiducail points are immediately added, so no events triggered when user adds fiducials to a list after some time.
		+	*** Approach: MRMLScene gets a new event MetadataAddedEvent
		+	*** StorableNode gets new members, SlicerDataType and methods to set it and invoke an event, and methods to get it. We do this in SlicerBase, instead of in an external informatics module. This seems fair, since this functionality is intended to move into slicer base load/save eventually anyway.
		+	** CHALLENGE 2: Save Data pipeline calls a storageNode's WriteData method first, which writes data to disk. If a uri is specified for the data, the call to WriteData writes to cache, and a superclass method called 'StageWriteData' writes from cache to the remote host using the informatics tools. FreeSurfer storable nodes have no writers! So there's no way to get the to write to cache;
		+	*** Approach: vtkMRMLFreeSurferModelStorageNode and vtkMRMLFreeSuferModelOverlayStorageNode get an optional CopyData method which acts like write data, but does a disk copy from source to cache, and then an upload from cache to remote host. Not so great, but we'll test to see if it works.

@@ Line 105: / Line 105: @@
 ** Restore after MID-UPLoad/Download crapout -- need to reset write or read state on all selected resources; make sure filenames point to cache files or to final/original destination. (DONE)
 ** prompt user to save data locally prior to upload.
 * GUI
 ** improve readability of current keyword/value pair in tagging panel (done)

@@ Line 91: / Line 91: @@
 *** make parser base class and derive XNDParser HIDParser, move parsing methods there (done)
 *** make xml writer base class and derive XNDWriter HIDWriter, move xml writing methods there (done)
-*** make xml query base class and derive XNDQueryConstructor HIDQueryConstructor, move query generation there.
+*** make xml query base class and derive XNDClientCaller HIDClientCaller, move query generation there.
 ** efficiency: create a 'NewTagList' and add only new tags to DB instead of all tags at each upload.
 ** escape all special characters in new tags & values before posting.