Difference between revisions of "Slicer3:UIDesign:WorkingProblems:SlicerInformatics:Draft1"

From Slicer Wiki
Jump to: navigation, search
 
(11 intermediate revisions by the same user not shown)
Line 85: Line 85:
 
*Logic & MRML
 
*Logic & MRML
 
** trim off last ampersand in query
 
** trim off last ampersand in query
** make sure all query, delete, DL, UL commands exist in logic (so can be called from commandline)
+
** ask user to save data locally first in case the remote save fails (while debugging anyway).
** modularize architecture for multi-services (set appropriate handler in node)
 
*** make tcl procs into c++ logic (done)
 
*** move requestDL, UL into logic (done)
 
*** make parser base class and derive XNDParser HIDParser, move parsing methods there (done)
 
*** make xml writer base class and derive XNDWriter HIDWriter, move xml writing methods there (done)
 
*** make xml query base class and derive XNDQueryConstructor HIDQueryConstructor, move query generation there.
 
*** add a SelectedURIHandler to node.
 
** efficiency: create a 'NewTagList' and add only new tags to DB instead of all tags at each upload.
 
** escape all special characters in new tags & values before posting.
 
** decode all escaped special characters in new tags
 
** during a scene upload, keep track of all files uploaded so far. If something fails before MRML scene goes up, post message about what resource failed, then delete all component resources from the remote host (prevent partial upload footprint).
 
** when user selects to delete a scene, query if they want to delete scenefile + all data it references. if no, query whether they just want to delete scene file alone. If opt to delete all, then download the scene, get out all refs, delete scene and refs from remote host.
 
 
** after a scene close event, remove all tags from scene's UserTagTable
 
** after a scene close event, remove all tags from scene's UserTagTable
** AllValuesForAllTagsOnServer -->ServerMetadata. (done)
 
** Node --> ServerAndUserMetadata.
 
** Node --> CurrentTagDefinitions.
 
** Node --> CurrentTagSelections.
 
 
** If scene is renamed, ResourceUpload widget doesn't update.
 
** If scene is renamed, ResourceUpload widget doesn't update.
 +
** Restore after MID-UPLoad/Download crapout -- need to reset write or read state on all selected resources; make sure filenames point to cache files or to final/original destination. (DONE needs more testing tho)
 
* GUI
 
* GUI
** improve readability of current keyword/value pair in tagging panel (done)
 
 
** put up a 'wait' message (or write text in status bar)
 
** put up a 'wait' message (or write text in status bar)
 
*** figure out what's overriding the status text when server is selected & query is made (still hmmm)
 
*** figure out what's overriding the status text when server is selected & query is made (still hmmm)
** implement show tags for individual dataset
 
 
** when a scene is closed, remove all tags from the empty scene's tag table.
 
** when a scene is closed, remove all tags from the empty scene's tag table.
 
** when a tag is deleted from datasets, raise the tagViewer with updated info
 
** when a tag is deleted from datasets, raise the tagViewer with updated info
Line 114: Line 97:
 
** find and fix debug leaks: appears to be part of the TagTableCollection (done).
 
** find and fix debug leaks: appears to be part of the TagTableCollection (done).
 
** want to be able to download any kind of Volume or Model or MRML file. Problem: inferring what kind of node to create from the uri of a remote resource. Currently, when a filename is passed in on the command line, loader.tcl parses out the filename and sends it to the AddDataWidget's logic; this figures out what kind of nodes to create for the data. When a filename is loaded through AddVolumes, AddModel, AddTransform, etc., the node is created directly. Probably need to build logic like the AddDataWidget's logic, that infers the node type from the filename -- if the uri contains a valid filename.
 
** want to be able to download any kind of Volume or Model or MRML file. Problem: inferring what kind of node to create from the uri of a remote resource. Currently, when a filename is passed in on the command line, loader.tcl parses out the filename and sends it to the AddDataWidget's logic; this figures out what kind of nodes to create for the data. When a filename is loaded through AddVolumes, AddModel, AddTransform, etc., the node is created directly. Probably need to build logic like the AddDataWidget's logic, that infers the node type from the filename -- if the uri contains a valid filename.
** when server metadata is refreshed, tag selection state changes on ResourceUpload widget. decouple.
 
** debugging assist: beef up datatransfer tools so we can interrogate each one.
 
* Test
 
** test: save locally, then upload.
 
** test: download, local save, then upload.
 
** test: download, remove a dataset, then upload.
 
 
* DebugNotes:
 
* DebugNotes:
** Loading data and uploading first time:
+
** Slightly confusing lack of parallelism in logic: In a storage node, the first FileName is repeated as the first entry in FileNameList, but the first URI is *not* repeated in URIList. To get a uri that corresponds to a filename, need to do a little index jigger. Principle code changes for this are in vtkDataIOManagerLogic in Base/Logic. I'm not familiar enough with the ArchetypeStorageNode to be confident in making a change -- Maybe Nicole can address?
 
+
** Added a modification to set up FileNameList initially when a file series is read -- (previously only getting set up in the write.) Seems ok now -- will wait for Nicole to test. Principle changes occur in vtkMRMLVolumeArchetypeStorageNode.cxx/h
If i read a volume from disk that includes tinkle.hdr and  tinkle.img,
+
** Fix: We are requiring a SlicerDataType tag on all storable nodes that can get uploaded to a remote server, and downloaded from a remote server. Why necessary? when datasets are encapsulated by mrml scene description and a scene is downloaded, we can use the scene context to decide what kind of storable node to create to hold the data about to be downloaded. When individual datasets come down (we want to be able to accommodate this), Slicer needs to create a storage node of a particular type, but has no way to know which (and cannot reliably infer this from the filename or extension.)  
then try to do the remote upload, here's what happens:
+
** CHALLENGE: We are trying to do all tagging of storabel nodes from an external informatics module by looking for node created events on the scene, testing to see what the node is, and then tagging its storage node with an appropriate value for SlicerDataType. However, it's possible to miss nodes entirely. If they don't get tagged, they don't get uploaded by logic, since we won't be able to reliably download them later. First, FreeSurfer models can have scalar overlays; when these nodes are created and added to a model display node, no events are available to tell us how to tag this from inside an external informatics moduleSecond, fiducialList nodes are created without storage nodes if no fiducail points are immediately added, so no events triggered when user adds fiducials to a list after some time.  
 
+
*** Approach: MRMLScene gets a new event MetadataAddedEvent
The first time I go into the loop to write metadata for upload in FetchMILogic,
+
*** StorableNode gets new member ''SlicerDataType'' and methods to set it and invoke an event, and methods to get it. We do this in SlicerBase, instead of in an external informatics module. Values for these types should be unambiguous and semantically meaningful to someone looking at their markup..
I get the storage node, write and post  metadata for FileName, and get a uri for tinkle.hdr.
+
*** Currently Supported SlicerDataTypes: MRML, ScalarVolume, LabelMap, VTKModel, FreeSurferModel, FreeSurferModelWithOverlay, DTIVolume, DWIVolume, UnstructuredGrid, FiducialList, ColorTable.
Then I try to process the storage node's fileListMembers; I find:
+
*** For now, we'll set SlicerDataType on the nodes from FetchMILogic for everyone we can; but for FreeSurferScalarOverlays, we'll set those in the nodes themselves to ensure they get set. We tried to catch them all with ''NodeAddedEvents'' posted by the scene, but not all storage nodes are created when the storable node is created, and if they are created ''later'', they are usually added under ''no notify'' conditions.
storageNode->GetNumberOfFileNames() = 0
+
*** NEED A CONSISTENT WAY TO DO THIS that hopefully doesn't require touching all these node classes...
I would expect the first to be 2. Seems these things are not set on the
+
** Once all storable nodes in the scene are tagged, FetchMILogic can post them to a remote host, handling any differences in the save data pipeline for differnt types of nodes.
node when it is created.
+
*** For instance: Save Data pipeline calls a storageNode's WriteData method first, which writes data to disk. If a uri is specified for the data, the call to WriteData writes to cache, and a superclass method called 'StageWriteData' writes from cache to the remote host using the informatics tools. FreeSurfer storable nodes have no writers! So there's no way to get the to write to cache;
 
+
*** Approach: vtkMRMLFreeSurferModelStorageNode and vtkMRMLFreeSuferModelOverlayStorageNode get an optional CopyData method which acts like write data, but does a disk copy from source to cache, and then an upload from cache to remote host. Not so great, but we'll test to see if it works.
As a result, I don't go into the loop that writes/posts metadata for, and
+
** Would like to save known servers into the application registry so they are available when user starts up...
gets a uri back for tinkle.img (which would have been fileListMember1).
 
 
 
  Then, data gets written to cache by vtkMRMLVolumeArchetypeStorageNode->WriteData
 
as tinkle.hdr and tinkle.img and your TempWrite dir is there and filled too.
 
 
 
The scene file gets written to cache. It's included as SlicerSceneFirst.mrml.
 
At least one issue is that it doesn't yet contain any uriListMembers, tho it does
 
contain fileListMembers.
 
 
 
resulting upload is incorrect.
 
 
 
** trying the same upload the second time:
 
 
 
When I just try to upload the same thing second time in a row (now that some things
 
have been set on the storage node), now in FetchMILogic I find
 
storageNode->GetNumberOfFileNames() = 2
 
storageNode->GetNumberOfURIs = 0
 
which is as I expect. yay.
 
 
 
I get the storage node, write and post metadata for FileName, and get a uri
 
back for tinkle.hdr as before. Then, this time I fall into the loop that writes and
 
posts metadata for, and gets a uri back for each fileListMember -- so first, I get
 
a double-post for tinkle.hdr, and two uris. (I can skip the first fileListMember
 
in my logic to avoid this repeat).
 
 
 
Data gets written to cache as tinkle.hdr, tinkle.img and here's the MRML scene
 
(SlicerSceneSecond.mrml). Note the filenames and uris and members. the
 
uri and uriListMember0 are different for tinkle.hdr.
 
 
 
upload again is incorrect.
 
 
 
** Second guy is fixed; just need to take care NOT to reprocess the first filename in the series twice; and be mindful of the fact that tho the first filename is repeated as the first entry in FileNameList, but the first URI is *not* repeated in URIList. Requires a little jigger in the code. Principle code changes occur in vtkDataIOManagerLogic in Base/Logic.
 
** Second issue is fixed but requires a test before checking in; node was not getting set up initially when a file series was read -- only getting set up in the write. Seems ok now. Principle changes occur in vtkMRMLVolumeArchetypeStorageNode.cxx/h
 

Latest revision as of 15:11, 8 March 2009

Home < Slicer3:UIDesign:WorkingProblems:SlicerInformatics:Draft1

Discussion at the NA-MIC 2009 AHM in Salt Lake City

Planning XNAT and Slicer integration next steps

Rough timeline for the next several months


XNATSlicerTimeline-010909.png


Discussion points and action items

Deleting:

  • resources (Use http delete),
  • tags on resources, and
  • tags from whole repository

Kevin will email wendy the API for the last two of these.

QA: detecting whether uploaded data is valid. WUSTL is thinking about QA; right now no provision. Future, maybe look at checksum on server side... Maybe there's a short-term soln: API provision for client to query for remote file size, and check this against local file size, the request file be deleted on remote host if no match...

Server ID: is there a way to validate a server as XND? We want to expose capability to 'add a new server', and a way to query the resulting servername for its identity, like

 http://xnd.slicer.org:8000/identify

Kevin will look into this and update Wendy.

Security: no security yet. A first pass will come soon, probably http basic.

Tag/value length limits: There are no intentional limits on tag/value lengths. Long names and values may slow search time.

Special character restrictions: excaping all characters as for xml/html should be sufficient.

Note: WUSTL implemented a file browser metaphor for looking at hierarchy of resources, found that intuitive for users.

Note: XND current version is challenged by enormous datasets (like large collections of DICOM). Fix to this coming in version 0.8, with "collection support".

Note: Current XND version 0.7 is compatible with Slicer's existing implementation. Slicer will have to be modified to take advantage of Version 0.8's collection support: collections may not be supported by XND FileServer until a month or two after XND 0.8 is released. Misha will let Wendy know when draft API is available and advise.

Format tag: In the event that URIs don't contain an explicit extension (like those for XNAT-E, which also encode ticket info) we need some kind of tag on the dataset that Slicer can use to know how to open/load it. Current Slicer implementation automatically generates a "SlicerDataType" tag that it uses for that purpose. Better if we change that logic to use a "Format" tag that Slicer- and non-Slicer-people use. But we need to encourage the value assignments to be as consistent as possible. Can we create an XND default tag with a pre-populated set of values that takes a good stab at this? If so, Slicer can be modified to use "Format" instead of (or in addition to) "SlicerDataType".

Annotations, Thumbnails, and other derived data: We discussed the best way to accommodate annotations, and other data (thumbnails) associated with a particular dataset or collection of datasets. We would want to be able to:

  • create these 'documents or datasets',
  • upload them and associate them with a resource,
  • query for them, and
  • download them separately or bundled with data.

FetchMI release into nightly builds: This will happen soon after bug fixing and testing. Wendy will update XNAT team when module is released.

Cache Manager, Remote Data I/O ToDo List for Slicer 3.4 code freeze

  • Fix Raise and rendering update behavior on CacheAndDataIOManager. When any data transfer takes place, it should raise, render itself, and then continue. After each data transfer, it should update.
  • Disable Asynchronous IO option for now since that sometimes causes problems.
  • Make sure "always redownload" is selected by default.
  • Upload and download of MRML Scene files don't generate a DataTransfer entry -- not sure why. Only data files. Find and fix.
  • Remove "Publish to XNAT" option from the File menu
  • Make sure all path constructions are created with join, and not std::string "+"

FetchMI ToDo List for Slicer3.4 code freeze

  • Logic & MRML
    • trim off last ampersand in query
    • ask user to save data locally first in case the remote save fails (while debugging anyway).
    • after a scene close event, remove all tags from scene's UserTagTable
    • If scene is renamed, ResourceUpload widget doesn't update.
    • Restore after MID-UPLoad/Download crapout -- need to reset write or read state on all selected resources; make sure filenames point to cache files or to final/original destination. (DONE needs more testing tho)
  • GUI
    • put up a 'wait' message (or write text in status bar)
      • figure out what's overriding the status text when server is selected & query is made (still hmmm)
    • when a scene is closed, remove all tags from the empty scene's tag table.
    • when a tag is deleted from datasets, raise the tagViewer with updated info
  • Overall
    • find and fix debug leaks: appears to be part of the TagTableCollection (done).
    • want to be able to download any kind of Volume or Model or MRML file. Problem: inferring what kind of node to create from the uri of a remote resource. Currently, when a filename is passed in on the command line, loader.tcl parses out the filename and sends it to the AddDataWidget's logic; this figures out what kind of nodes to create for the data. When a filename is loaded through AddVolumes, AddModel, AddTransform, etc., the node is created directly. Probably need to build logic like the AddDataWidget's logic, that infers the node type from the filename -- if the uri contains a valid filename.
  • DebugNotes:
    • Slightly confusing lack of parallelism in logic: In a storage node, the first FileName is repeated as the first entry in FileNameList, but the first URI is *not* repeated in URIList. To get a uri that corresponds to a filename, need to do a little index jigger. Principle code changes for this are in vtkDataIOManagerLogic in Base/Logic. I'm not familiar enough with the ArchetypeStorageNode to be confident in making a change -- Maybe Nicole can address?
    • Added a modification to set up FileNameList initially when a file series is read -- (previously only getting set up in the write.) Seems ok now -- will wait for Nicole to test. Principle changes occur in vtkMRMLVolumeArchetypeStorageNode.cxx/h
    • Fix: We are requiring a SlicerDataType tag on all storable nodes that can get uploaded to a remote server, and downloaded from a remote server. Why necessary? when datasets are encapsulated by mrml scene description and a scene is downloaded, we can use the scene context to decide what kind of storable node to create to hold the data about to be downloaded. When individual datasets come down (we want to be able to accommodate this), Slicer needs to create a storage node of a particular type, but has no way to know which (and cannot reliably infer this from the filename or extension.)
    • CHALLENGE: We are trying to do all tagging of storabel nodes from an external informatics module by looking for node created events on the scene, testing to see what the node is, and then tagging its storage node with an appropriate value for SlicerDataType. However, it's possible to miss nodes entirely. If they don't get tagged, they don't get uploaded by logic, since we won't be able to reliably download them later. First, FreeSurfer models can have scalar overlays; when these nodes are created and added to a model display node, no events are available to tell us how to tag this from inside an external informatics module. Second, fiducialList nodes are created without storage nodes if no fiducail points are immediately added, so no events triggered when user adds fiducials to a list after some time.
      • Approach: MRMLScene gets a new event MetadataAddedEvent
      • StorableNode gets new member SlicerDataType and methods to set it and invoke an event, and methods to get it. We do this in SlicerBase, instead of in an external informatics module. Values for these types should be unambiguous and semantically meaningful to someone looking at their markup..
      • Currently Supported SlicerDataTypes: MRML, ScalarVolume, LabelMap, VTKModel, FreeSurferModel, FreeSurferModelWithOverlay, DTIVolume, DWIVolume, UnstructuredGrid, FiducialList, ColorTable.
      • For now, we'll set SlicerDataType on the nodes from FetchMILogic for everyone we can; but for FreeSurferScalarOverlays, we'll set those in the nodes themselves to ensure they get set. We tried to catch them all with NodeAddedEvents posted by the scene, but not all storage nodes are created when the storable node is created, and if they are created later, they are usually added under no notify conditions.
      • NEED A CONSISTENT WAY TO DO THIS that hopefully doesn't require touching all these node classes...
    • Once all storable nodes in the scene are tagged, FetchMILogic can post them to a remote host, handling any differences in the save data pipeline for differnt types of nodes.
      • For instance: Save Data pipeline calls a storageNode's WriteData method first, which writes data to disk. If a uri is specified for the data, the call to WriteData writes to cache, and a superclass method called 'StageWriteData' writes from cache to the remote host using the informatics tools. FreeSurfer storable nodes have no writers! So there's no way to get the to write to cache;
      • Approach: vtkMRMLFreeSurferModelStorageNode and vtkMRMLFreeSuferModelOverlayStorageNode get an optional CopyData method which acts like write data, but does a disk copy from source to cache, and then an upload from cache to remote host. Not so great, but we'll test to see if it works.
    • Would like to save known servers into the application registry so they are available when user starts up...