Slicer3:UIDesign

From Slicer Wiki
Jump to: navigation, search
Home < Slicer3:UIDesign

Return to Slicer3 Interface Design and Usability


Embodiment

  • Current schematic of the main application GUI: ( higher resolution image: Media:GUIMap.png )

SmallEmbodiment.png

  • Current schematic of a Slice Viewer and Controller: (higher resolution image: Media:SliceGUI.png )

SmallSliceGUI.png

Slicer users

Core user communities:

  • (NAC) Longitudinal and multi-channel dataset analysis
  • (BIRN/NA-MIC) Individual and group analysis
  • (IGT/AMIGO) Real-time control and tracking in the operating theater
  • (IGT) Neurosurgical planning and guidance

Types of users:

  • Algorithm researchers (who work within 3DSlicer's development environment and with associated toolkits)
  • Biomedical engineers (who rely on 3DSlicer's interactive enironment and scripting capabilities)
  • Application scientists (who use 3DSlicer as a desktop application and turnkey system)


Current status and TO-DO

The existing base classes form a basic framework for the UI. For the Alpha/Beta releases, we provided a main application window with the general layout described in our current design. For the Release deadline, we plan to complete the overall application GUI. Things we are focusing on:

  • choose radiobutton/checkbutton with/without icon convention
  • finish content for slicer human interface and style guidelines
  • theme font, match to screen resolution, set in application settings
  • finish slicer mouse modes
  • think through color palette to support semantic applications
  • update module GUI how-to
  • refine ROI toolbox design, work with developer to implement
  • refine lightbox viewer
  • implement mechanism to specify oblique view
  • build classes for pop-up information (help and context info)
  • copyrights

Working problem: spot GUI fixing

Scene snapshot panel

This GUI panel allows the scene to be recorded at any point and restored later; it's particularly useful for sharing relevant views of a scene with collaborators or for instruction. The layout can be made more compact and improved in general. Also, colors on the 'restore' button are overly bold, and the red/green imply more than they intend.

Current layout:

OldSceneSnapshot1.png

OldSceneSnapshot0.png

New design:

The restore button needs to convey three states:

  • no scene snapshots exist to restore

SceneSnapshot2.png

  • a scene snapshot is selected to be restored, but is not yet visible.

SceneSnapshot0.png

  • a selected scene snapshot has been restored, and is displayed.

SceneSnapshot1.png

Working problem: Application-wide support for ROI definition

GOALS:

  • determine relationships between label maps, implicit regions (spheres, circles, ellipses), and vector-format region definitions.
  • determine what kinds of ROIs we will allow users to create/edit at application level
  • determine how to expose ROI functionality application-wide (without "wormholing" to an ROI module)
  • provide tools for defining, editing ROIs
  • provide tools for converting ROIs into label maps (?)
  • provide good mouse+hotkey design
  • generalize implementation to eventually accommodate DICOM-format ROIs
  • determine relationship between fiducial lists and ROIs (from implementation pov).

Draft design:

Sketch of the draft interface for the "ROI toolbox" is below:

ROIsketch.png

Collecting feedback on the complete Interface & interaction design drafted at June NA-MIC Project Week meeting / MIT:

Working problem: Consistent checkbuttons and radiobuttons across platforms (Issue Resolved)

GOALS

Currently, radiobutton and checkbutton indicators display their state differently on different platforms Slicer supports. For instance, visually similar indicators on linux and windows actually represent different states. This can be confusing for people who work on both platforms. So instead of using the default Tk bitmaps for the indicators, we are proposing to use our own image data which will display the same on all platforms.

  • develop a custom slicer widget as an extension of vtkKWCheckButton
  • disambiguate current "deslected" radiobutton icon state -- it can be read as "disabled"
  • override method for turning the old indicator visible/invisible (always keep it off)
  • develop visually parseable convention for icons to represent:
    • selected
    • deselected
    • disabled
  • use methods SetImageToIcon and SetSelectImageToIcon to display selected/deselected icons, selected/deselected icons+indicators
  • provide blank icons with and without indicators for people to download.

Various experiments below for quickly readable icon state: either "selected" "deselected" or "disabled". Varying color, contrast, texture, etc. to find a design solution that reads out state quickly, at a glance. Drop shadow makes icons pop, but requires more pixels in each icon and we might have to re-do all existing icons to ensure good packing behavior. Also, we might get tired of this effect. Contrast differences add more visual "pop" than color differences. The greyed out dotted lines provide a good solution for "deselected" state, with good visual contrast to the black outlined "selected" state. Both are clearly distinguished from the "disabled" state. Options P&Q are preferred below.

RadioIcons.png

How do various icon configurations appear with and without icons, text, and indicators? The preferred convention would be to use either indicators + text or icons alone -- rather than indicators + icons together, or indicators, icons + text together. Sometimes this extra information may be necessary, so the last two embodiments are not ruled out.

RadioCheckButtons.png

Best without indicator, but ok with too.

Applying new design to Mouse Mode Toolbar

Easier to comprehend at a glance, and no ambiguity between "disabled" and "deselected"

MouseModeToolbar.png

Working problem: Navigating with fiducials (Issue resolved: implemented in Editor module and Edit Box)

GOALS:

  • use fiducials as markers for features of interest
  • allow navigation to fiducial points within a slice window and
  • allow navigation to same fiducial point in all slice windows (a feature we're calling 'jumpslice').
  • want to read-out which fiducial is currently being viewed while navigating
  • want to expose these features in top level in each slice controller widget
  • want also to keep the slice scrollbar from getting too small to use.
  • want to indicate which slice scrollbar direction is R,L, S,I, A,P, depending on orientation


Proposed modification of vtkSliceControllerWidget:

The widget below shows a read-out/entry for current fiducial; buttons for navigate-to-next-fiducial and navigate-to-previous-fiducial. Slice scrollbar is longer, and contains labels at each endpoint.

icons

  • user can navigate forward, backward using fiducial jump buttons
  • user can enter an ordinal number (or name?) to jump to a fiducial (above, third fiducial of 12 total in the list)
  • if viewers are unlinked, only the active slice viewer is updated
  • if viewers are linked, all viewers are updated to show the slice containing the same fiducial.
  • if a viewer's slice scrollbar is adjusted, its fiducial read-out is reset to ""
  • any time a new fiducial is placed in a viewer, the fiducial read-out is updated to that (current) fiducial.

Questions about placing and moving new fiducials:

  • should placing a new fiducial in the viewer cause a jumpslice event if all viewers are linked?

Seems we may not always want to trigger this by default. A reasonable solution for this is to unlink viewers when placing fiducials if you don't want the jumpslice behavior...

  • should moving a newly placed fiducial in the viewer cause a jumpslice event if all viewers are linked? Again, seems we may not want this default behavior. And again, it seems easy enough to unlink viewers if you don't want the jumpslice behavior.
  • can we undo the creation and placement of a fiducal easily, so that while exploring the data, user can click, look, delete, and click again -- as an alternative to grabbing and moving an existing fiducial?

Questions about navigation:

  • first time navigating, we start on fiducial#1. maybe we go forward to

fiducial 10 of 20. Next time navigating, do we continue on to 11, or restart at 1?

  • Will there ever be too many fiducials to navigate?
  • should the fiducial navigation entry (read-out) also take a fiducial NAME instead of number?
  • Do we want to include other navigation options?
  • With numerous fiducials, it may be hard to tell which fiducial is the corresponding one, when navigating them. Can we color the jump-fiducial temporarily to help integrate all displays?


Working problem: Cine display for viewing and analyzing multi-volume data in Slicer3

Design for a basic cine controller:

CineBasicController.png

Design studies for a cine display that allows viewing either corresponding slices in multiple volumes over time, or consecutive slices in a single volume over time. Version 4 is the most recent iteration, containing elements for specifiying the following animation parameters:

  • volumes (select a volume collection to animate)
  • volume (select a single volume to animate)
  • display (select Green, Yellow or Red slice viewer)
  • step (set slice increment within a volume, or volume increment within a collection)
  • speed (set animation speed from slow to fast)
  • range ( choose range of slices within a volume, or range of volumes within a group. In the former case, we assume the animation displays consecutive slices in the specified range, incrementing according to the step size. In this latter case, we assume you select the slice of interest via the usual slice viewer gui, and animation displays the corresponding slice in each volume within the collection, incrementing according to the step size. )
  • current ( show or choose current slice or current volume ).


And controller icons:

  • go to first frame
  • play in reverse
  • pause
  • stop
  • record a movie
  • play forward
  • go to last frame
  • loop
  • ping-pong


CineDisplayDesigns1-4.png

Slicer 2.6's Interval Browser controller for viewing and organizing timeseries data:

Ibrowser.png

Working problem: Mouse Modes in Slicer3

Currently, a left-mouse-button click in Slicer's 3D or Slice Viewers can mean different things at different times; this can be confusing to users, and can lead to mistakes. We are currently deciding a good paradigm for defining and controlling mouse modes in Slicer's 3D viewer and 2D slice windows. We rely on mouse interaction to transform the view, and to perform explicit actions like placing a fiducial, selecting and moving a fiducial, collecting voxel-value samples, and so on. The challenge is to make all actions possible with just the left mouse button (some mice have only one button), and to disambiguate what a left-mouse-button click in the 3D and Slice viewers means.

Goals: We'd like the interaction paradigm to have the following features:

  • Consistent in all slice windows and 3D viewer
  • Intuitive and easy for beginners
  • Fast for intermediate and expert users
  • Rely on only one mouse button (but accommodate multiple mouse buttons)
  • Support direct manipulation as much as possible
  • Provide clear visual feedback about current mouse mode
  • Support both view transformation (rotate, pan, zoom) as the primary mode, and mouse actions (pick, manipulate, place, etc...) as additional modes.

---

Current issues we want to address in existing Slicer3 implementation:

  • Mouse motion should default to "transform view" mode
  • Interaction relies on three mouse buttons (for rotate, pan, zoom, etc.) which aren't always present on all computers
  • It's currently easy to forget about persisting non-transform mouse modes (like pick or place) and assume that the view can be transformed -- leading to annoying errors.
  • Would like to free up right mouse button to invoke the display of a 'context menu', but well-established convention uses right mouse button for zooming in all viewers.
  • Would like BOTH intuitive one-handed mouse-only interaction to be possible AND hot-key assisted shortcuts to be available.
  • We need to combine the concepts of active (volume or model, etc.), selected (volume or model...) and picked (volume or model...)
  • We need to design a consistent rendering treatment for all selected objects (model, volume, fiducial, vertex, label).

Proposed mouse mode design

The following four mouse modes are proposed: Pick (or select), Pick+Manipulate (or select+manipulate), Place ( or create new object), and Transform View. Transform View is the default mode. All other modes can be invoked in a non-persistent manner from this mode with hot-keys. (which keys to choose?)


MouseModes.png

Current bindings:

  • 3D viewer: left mouse button >> rotate
  • 3D viewer: middle mouse button >> pan
  • 3D viewer: mouse wheel button >> zoom in/out
  • 3D viewer: right mouse button, pull drag >> zoom in
  • 3D viewer: right mouse button, push drag >> zoom out
  • Slice viewer: left mouse button >> select/move a fiducial point
  • Slice viewer: middle mouse button >> pan
  • Slice viewer: mouse wheel button >> scroll slices
  • Slice viewer: right mouse button, pull drag >> zoom in
  • Slice viewer: right mouse button, push drag >> zoom out
  • special key-bindings enable placing of fiducials

Proposed bindings:

Selection behavior

Currently 'active' models and volumes etc are persistent. However, it will probably be annoying for people to see the dataset persistently rendered as 'selected' -- and that will be a change to the status quo. Deselecting all, will have the effect of unloading all node selector widgets.


Developers and users: What do we need to pick?

At the June 2007 project week at MIT, we determined that we'll need to be able to pick several distinct things in Slicer, and the single "pick" mouse mode currently available was not going to provide the unambiguous picking functionality we need. To address this problem, we'll probably need to design a set of "pick" mouse modes. Please add your picking needs to the list developing below. Thanks!

  • single/multiple vertices on models (for plotting, we need to be able to get the closest vertex ID)
  • single/multiple cells on models ( need to calcuate the vertex usually)
  • single/multiple points on surfaces (for placing fiducials, get RAS)
  • single/multiple models
  • model groups
  • single/multiple "nodes" in general
  • "node" groups
  • single/multiple volumes
  • single/multiple fiducials
  • single/multiple ROIs
  • tensors
  • voxels in volumes
  • slices in volumes
  • subvolumes
  • general polygons?
  • annotations?
  • two points to make a line?
  • for tractography, we need to pick:
    • cell in a model (to get one fiber trajectory)
    • entire models (to pick a bundle)
    • ideally, a model group (to pick an entire structure like the corpus callosum that includes several bundles)
  • It would also be very cool to have a plane widget for 3D intersection tract selection.
  • for tractography and other things with an intermediate processing step before visualization (generation of glyphs or whatever)
    • it may be useful for picking to indicate what was picked, if several models are displayed per node
    • like: model #2 was picked, then my code can know that is a glyph model, rather than a line model, if that is ever needed.
    • this depends on the final implementation of Alex's new cool display API, whether there is ever >1 model allowed

User-suggestions for selection patterns

  • various lassos would be nice to select multiple of the thing current mode indicates
  • single click to select one of the thing the current mode indicates
  • control-click to select multiple of the thing the current mode indicates
  • a shift-click to select ALL of the thing the current mode indicates

Tests: visually rendering selected objects

State in MRML

Currently, Slicer uses the concept of active datasets (the active volume, an active model, and active fiducial list, etc.) Application-wide, only a single datatype can be active at one time. This set of active datatypes is represented in the vtkMRMLSelectionNode.

The fiducials module also uses the concept of selected fiducials within any fiducial list. Many command line modules use only selected fiducials in their processing.

We propose to collapse these into one state variable and moving that concept into the codebase, which includes the following work:

  • defining Selected=0 or 1 on a vtkMRMLTransformableNode.
  • modifying the vtkMRMLSelectionNode to include multiple selected objects within any datatype
  • extending the vtkSlicerNodeSelectorWidget to include checkbuttons rather than radiobuttons for nodes that can have multiple selections
  • Changing active to selected in the codebase, (Fiducials module, ROI module, SlicesControl, ViewControl, ApplicationLogic and MRML XML parsing). MRML XML parsing should continue to support the "active" tag to remain backward compatible.
  • Building the selection logic that appropriately marks objects as selected or deselected.
  • Enforce the habit of setting and using the Pick mode for selection in addition to just overloading the Transform View mode with hot-keys.

Other software conventions...

How Actions and View Transformations are done in Maya

Maya is a very heavily used professional 3D animation package. Basic mouse bindings are:

  • Rotate: alt-left mouse button
  • Pan: alt-middle mouse button
  • Zoom: alt-right or alt-right-and-middle mouse button
  • Actions: always left mouse button, specific action depends on current mode
  • Context menu: space bar

The advantage of this approach is that the left mouse button is freed up for many possible tasks such as selection, grabbing widget handles, etc. Disadvantage is that it requires two hands to operate. (A free copy of an almost-fully functional version of Maya is available for testing ).


How Actions and View Transformations are done in Blender

Blender is an Open Source modeling and animation package similar in many ways to Maya (almost like GIMP is to Photoshop). Basic mouse bindings are:

  • Rotate: alt-left mouse button OR middle mouse button
  • Pan: shift-middle mouse button
  • Zoom: control-middle mouse button
  • Actions: mostly left mouse button
  • Context menu: space bar


Note: Mac bindings

On a one-button mac, the following translations work by default in Mac OSX:

  • single button -> left mouse button
  • alt (option) + button -> middle mouse button
  • apple (command) + button -> right mouse button

---




GUI screenshots from other packages:

Below are snippets of SketchUp's interface, and Maya's Interface. Maya's mouse modes switch between object transforms (translate, rotate and scale), object or region select, and an extrude function. SketchUp's mouse-mode icons switch between view transforms (translate, rotate, zoom) and an object extrude function.

SketchUp Example

Maya Example

Return to Slicer3 Interface Design and Usability