Documentation/Labs/Improving Markups

Overview

The goal of this page is to support improvements to the Markups Module for fiducial placement. This page will document improvements suggested by users and developers and coordinate efforts to address them.

History

Markups, also known as annotations (and almost the same as VTK widgets) proved to be very difficult to design well. First-generation VTK widgets were simple but too monolithic, their behavior and appearance was not customizable enough. Second-generation VTK widgets took a completely new approach, separated "widget" from "representation" allowing much more customization, but also resulting in very complex widgets, with very poor performance in some cases (mainly because many classes are used to implement simple widgets and there is a complex and costly synchronization of states between them). Widgets in Slicer are similarly problematic. First attempt resulted in "Annotations" module, which was very complex to use and develop and also very unstable and so most widgets had to be disabled and only ruler (line) and ROI widgets remained. "Markups" module was the second attempt. The project ran out of time and only markup fiducials were implemented. Implementation is complicated, which makes adding new features very difficult, it has serious issues (sometimes points cannot be clicked or hidden points can be clicked), and it becomes unusably slowe after placing 100-200 points. There have been additional attempts to rework widgets, but so far they have all failed. We don't give up, so here we are again.

Design considerations

Data synchronization between multiple classes (VTK and MRML nodes, instances for each viewer, widget and representation) has proven to be one of the fundamental issues. It would have been nice to create generic VTK-only widgets and use them in Slicer, but the additional layer of abstraction would have made implementation more complicated and degraded performance. Therefore, we do not use VTK widgets and implement Slicer-specific classes, which directly use information stored in MRML nodes.

Separation between "widget" (behavior) and "representation" (appearance) is a good concept, but in second-generation VTK widgets, widgets and representations are too much separated (they don't access each other's state but communicate with state transitions and events). This scheme may be useful for client-server visualization, but it is inefficient and too complicated to use within a simple application. Therefore, we need to split classes in a way that minimizes need for manual synchronization. Separation of "data" (shared globally in the whole scene, such as widget point positions) must be separated from "display" (properties that control how a widget should appear in each view) works very well for displayable MRML nodes and chosen to be used for widgets as well. Data adds thin wrapper over VTK data object (e.g., vtkPointSet), which allows the same data object used in multiple renderers and alleviate any need for synchronizing point lists. State information (even low-level detail, such as markup that the user is currently hovered over) is stored in markup data and display nodes.

Performance of markups module was very slow because each clickable point was implemented by a handle widget. Each handle widget requires instantiation of several classes per views (widget, representation, actors, pickers, etc.), many view observations, and trigger various expensive synchronization mechanisms when anything is changed. Instead, a single glyph actor is created to display all markups.

Point picking (determining what is under the mouse cursor position) and decision about what to do with is performed in VTK using pickers and a picking manager. Picking manager introduced to allow prioritization of events between multiple widgets. However, this solution was quite complicated (it hijacked events in each picker and ran it through the picking manager to decide if the widget may process the event) and due to its distributed nature, difficult to control. In Slicer widgets, each displayable manager can be queried if it can process a particular event (CanProcessEvent), returning a "distance" value. The displayable manager with the closest distance gets the right to process an event. Within a displayable manager, various flexible strategies can be implemented to decide which particular widget may process the event (taking into account the currently active widget, the interaction state of the application, etc.). This centralized solution allows reducing the number of picking operations, for example the interactor style can do a quick approximate picking using the Z buffer, then displayable managers may decide to use a more accurate picker (e.g., vtkCellPicker) and use the result in all the widgets that they manage.

In the old markups module, labels were rendered as polygons in 3D. This made their appearance very low quality (since anti-aliasing is problematic) and they often very hard to see because they are partially occluded by various 3D objects. We considered rendering the text in a texture and display as a semi-transparent rectangle by a 3D actor, but image quality still would have been an issue (texture resolution would need to match screen size and resolution) and occlusion by other 3D objects could still would have made it difficult to see some labels. Therefore, we decided to render labels as 2D text actors using existing VTK mappers. 2D actors can be displayed in high quality but they appear as a separate layer over 3D objects (they could appear very disconnected from the 3D glyphs), which is especially confusing when the markup point is occluded by other objects. Using VTK's fast point visibility checks (based on Z buffer) and label placement mapper features, the labels can be rendered acceptably well (and with the improvement of the mapper in VTK, label display will improve in Slicer, too).

Persistence: it should be possible to save all annotations in the scene (without storage node), to avoid having lots of tiny files (with just a few numbers in each). On the other hand, it would be nice to be able to save annotations in a standard file format and not just in MRML scene. It would be nice to be able to save group of markups to a single file, but unfortunately storing multiple nodes in a single file is problematic. We could add importers/exporters for standard annotation file formats (DICOM, AIM, ...). For now, all markups still use markup fiducial's storage node

Class roles & responsibilities (https://github.com/Kitware/vtk-js/issues/682)

• MRML data and display nodes (MVC: model, VTK: source) - vtkMRMLMarkupsNode & vtkMRMLMarkupsDisplayNode
  • Stores point positions, curve interpolation type, etc.
  • One instance shared between all renderers.
• MRML display node (MVC: model, VTK: source) - vtkMRMLMarkupsNode
  • Stores point positions, curve interpolation type, etc.
  • One instance shared between all renderers.
  • One markups may appear in various views in different ways, by defining various display nodes.
• Widget (MVC: controller) - vtkSlicerAbstractWidget (similar to vtkAbstractWidget)
  • It can decide if it can process a user interface event (mouse move, click, etc.)
  • It can process a user interface event by translating it to a widget event and then executing that (add/delete point, jump to a point, etc.).
  • There is one widget per display node in each renderer.
  • Maintains a very simple internal state (idle, places a point, being over some component of a widget)
  • Keeps a reference to a widget representation.
• WidgetRepresentation (MVC: view, VTK: actor) - vtkSlicerAbstractWidgetRepresentation (similar to vtkWidgetRepresentation)
  • VTK actor that can be added to a renderer.
  • It can tell if an event position is near some component of the widget.
  • Stores VTK rendering pipeline (actors, mappers, filters).
  • There is one widget representation per widget.
  • There are different representation for 2D and 3D views.

Notes on vtkAnnotationROIWidget/vtkAnnotationROIRepresentation

Annotation ROI widget would need to be moved to Markups module, too.

vtkAnnotationROIWidget/vtkAnnotationROIRepresentation are clones of vtkBoxWidget2/vtkBoxRepresentation that have been modified and extended. Ideally the necessary changes will be integrated back into the VTK classes. The changes are:

• Commented out face picking, and therefore rotation by dragging on faces. The VTK classes now offer this control.
• Added control over the color of individual handles. The only concern would be maintaining a backwards compatible API.
• Added members and getters related to handling when the MRML node has a parent transform. For example, WorldToLocalMatrix, GetExtents(), GetCenter(), and GetActors().

The transform-related additions seem intended to either simplify or optimize how vtkMRMLAnnotationROIDisplayableManager synchronizes the MRML node and the widget. For example, instead of calling SetTransform() on the widget, which would rebuild the representation, the displayable manager sets the UserMatrix on all the vtkActors. Another example is how GetExtents() and GetCenter() return computations based directly on the point data, which are now always in local coordinates, whereas calling GetBounds() would return world coordinates.

Reworking the displayable manager to not use these extra functions seems possible. One concern would be that doing so would affect performance, because likely the extra functions allow optimization by avoiding excessive transforming of the points and rebuilding of the widget representation. Another concern would be verifying that the proper coordinate frames are used in the reworked the display manager, since the code is currently not well-documented and some method calls to convert coordinate frames are simply commented out.

Some picking might need to be reworked. In the test branch I can't pick handles on the back of the cube. See topic sync-vtkAnnotationROIWidget-with-vtkBoxWidget2.

Features

Widget types:

• point set: DONE
• angle: DONE
• curve (open and closed, with different interpolations): DONE
• line (arrow, ruler): partially done (it is still just a simple line, no length or arrow tip is displayed)
• ROI (may be 2D only: ellipse, circle, rectangle)
• bidimensional
• plane
• slice intersection (show slice intersections and allow shift/rotate them), crosshair
• linear transforms (2D: similar to vtkAffineWidget; 3D: current transform widget)
• ROI-based image window width/level

User features:

• Highlight/display info when hovering over: DONE
• Select entire markup or just a point: DONE
• Restricting placement and motion to other nodes (model surface, volume rendered image surface, etc.): partially done - point picking and movement in 3D view is restricted to visible surfaces (work well for volume rendered surfaces, too)
• One click 3D angle, length measurements from landmarks (latter is already available)
• Allow adding fiducials with undefined position to allow creating a template with only fiducial labels
• Fine control for fiducial placement/modifications by keyboard strokes
• Improved scaling of 3D glyphs and 2D points. For example, fixed in screen size, fixed relative to voxel size of the data set, etc.
• Projection display in slice views (similar to model projections, markup point projection)
• Right-click menu (delete, jump to slice, fit slice views to main axes - see also operations in the table widget in the Markups module GUI)
• Compute metrics (length, surface, diameter, etc.), allow them to be exported into table or other standard annotation file formats
• Allow undoing modifications
• Sensible behavior when non-linear transform is applied
• Keyboard shortcuts:
  • Control L to make a new fiducial list
  • Control I for persistent place toggle
  • Control M to make a ruler between last two placed fiducials
  • back tick and control/shift back tick to jump slices to fiducials
  • delete or backspace (or 'd') to delete fiducial under mouse: DONE
  • 'p' to place fiducial without having to click in window to grab focus
  • modifier key to move fiducials between slices in 2D
• scaling issues (2D widget is scaled down by 300x compared to 3D)
• Rubber band placement for rulers: DONE (after placing first point, line is displayed to the current mouse position)

User features:

• A start/end and moving event with seed number
• A start/end and hover event
• consistent widget interfaces for click to place as well as programmatic placement (vs them being added with zero size and having to be manipulated): DONE
• consistent distribution of settings from the widget to all handles (process events on/off, visibility, display properties).

Status

Since there are many small issues and ideas, we do not yet track the items using the Slicer bugtracker but use this spreadsheet: https://1drv.ms/x/s!Arm_AFxB9yqHtvxAgdAfdLS_GHsOxw

References

• https://discourse.slicer.org/t/a-users-proposal-for-markups-module-enhancements/3362
• https://discourse.slicer.org/t/markups-module-enhancements/4335
• https://issues.slicer.org/view.php?id=3849
• https://issues.slicer.org/view.php?id=3713
• https://na-mic.org/wiki/Projects:ARRA:SlicerAM:vtkWidgets
• https://github.com/Kitware/vtk-js/issues/682
• https://en.wikibooks.org/wiki/Online_OsiriX_Documentation/Navigational_shortcuts
• https://github.com/lassoan/Slicer/commit/cbd43fd2b8eb552e99f5dd29c0ed29b0d1af7e26
• https://www.vtk.org/Wiki/VTK_Widget_Examples

• Interactive Contour Editing thread on slicer-dev (2015)
• ARRA grant widget notes (2010)
• Mouse and Keyboard Shortcuts (nightly)
• Slicer3 Tcl SWidget, SeedSWidget, FiducialsSWidget