Difference between revisions of "Documentation/Nightly/Developers/QtTesting"

From Slicer Wiki
Jump to: navigation, search
Tag: 2017 source edit
 
(41 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 +
<noinclude>{{documentation/versioncheck}}</noinclude>
 +
{|width="100%"
 +
|
 +
__TOC__
 +
|align="right"|
 +
[[Image: 3DSlicerQtTesting.png]]
 +
|}
 
=Overview=
 
=Overview=
 +
 +
{{documentation/banner
 +
| text  = Recording of macro using QtTesting should be considered experimental. <br> We suggest to implement workflow automation through Python scripts operating at a lower level by changing properties of MRML nodes and calling module logic functions. See https://discourse.slicer.org/t/macro-xml-to-python/15803 }}
  
 
QtTesting provides us a testing framework to test our application. It complements unit tests, by testing the application in its globality.
 
QtTesting provides us a testing framework to test our application. It complements unit tests, by testing the application in its globality.
Line 5: Line 15:
 
Following is a step by step guide to create slicer unit test, or module unit test, using this framework, which can be run as part of a nightly build automatic testing.<br>
 
Following is a step by step guide to create slicer unit test, or module unit test, using this framework, which can be run as part of a nightly build automatic testing.<br>
  
=Write a 3DSlicer UnitTest with QtTesting=
+
=Rules for Developers=
 +
 
 +
QtTesting records high level events, and try to not record mouse events as mouse press, release or move.<br>
 +
Furthermore, QtTesting is still under heavy development and some low level events won't be properly recorded.<br>
 +
 
 +
'''Please be aware of those following warning :'''
 +
 
 +
* Use '''''"activated"''''' signal instead of '''''"clicked"''''', when connection Qt views
 +
 
 +
=How QtTesting and CTK work=
 +
 
 +
{|width="100%"
 +
|align="left"|
 +
[[Image: ctkQtTestingSchemaPlay.png|550px|]]
 +
|align="left"|
 +
[[Image: ctkQtTestingSchemaRecord.png|550px]]
 +
|-
 +
|'''''How it works :'''''
 +
When pqTestUtility starts the playback, it calls pqEvenDispatcher to play a specific script. Then
 +
pqEventDispatcher finds an eventSource able to read the script. Then pqEventDispatcher forwards each
 +
event to the pqEventPlayer. Finally, this last class tries to find the best player to playback the action into the
 +
application. If the action has been done, we start over and over the same process until the end. If there is
 +
one error, or an action couldn’t be handled, the player fails with an error.
 +
|'''''How it works :'''''
 +
When pqTestUtility starts to record events, it finds the observer which will write the file, creates the file,
 +
and starts the pqEventRecorder.
 +
Then every time there is a user action in the application, a specific translator will catch it and transform it
 +
into an high level event. This high level event is sent to the pqEventTranslator, and then to the
 +
pqEventObserver which writes the action recorded into a specific language, XML for example.
 +
|}
 +
 
 +
=Write a 3DSlicer test using QtTesting=
 +
 
 +
QtTesting framework can also be use to create module test.<br> Use the description below, but instead of <code>Slicer/Applications/SlicerApp/Testing</code> all the path start by <code>Slicer/Path/To/The/Modules/Testing</code>
 +
 
 +
Following is the different step to create a test using QtTesting framework.
  
 
== Create your XML script ==
 
== Create your XML script ==
See [http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/SlicerApplication/QtTesting QtTesting user Wiki] to know how to create this xml script.<br>
+
See [[Documentation/Nightly/SlicerApplication/QtTesting|QtTesting user Wiki]] to know how to create this xml script.<br>
'''Save the xml script to''' <code>Slicer4/Application/SlicerApp/Testing/QtTesting/Slicer4.X.X_tutorial_MY_TUTORIAL_NAME.xml''</code><br>
+
'''Save the xml script to''' <code>Slicer/Applications/SlicerApp/Data/Input/MYTUTORIALNAME.xml''</code><br>
 
{{ambox  
 
{{ambox  
 
| type = content
 
| type = content
|text = If you needs data, please first add the data to the Sample Data module, and use this module during the test. See instruction [http://www.slicer.org/slicerWiki/index.php/SampleData here]<br>[[Documentation/{{documentation/version}}/Modules/SampleData Sample Data]] module according to the current Slicer version
+
|text = If you need data, please first add the data to the Sample Data module, and use this module during the test. See instruction [[SampleData|here]]<br>
 +
[[Documentation/{{documentation/version}}/Modules/SampleData Sample Data]] module according to the current Slicer version
 
}}
 
}}
  
Line 20: Line 66:
 
'''''Create a python file with the following lines'''''
 
'''''Create a python file with the following lines'''''
  
 +
    import os
 
     import slicer
 
     import slicer
     import slicer.testing
+
     import ApplicationsSlicerAppData as data
    import slicer.util
+
     filepath = data.input + '/MYTUTORIALNAME.xml')    ex : NeurosurgicalPlanningTutorial.xml
     filepath = os.environ['SLICER_HOME'] + 'path/to/your/script.xml'
 
 
     testUtility = slicer.app.testingUtility()
 
     testUtility = slicer.app.testingUtility()
 
     success = testUtility.playTests(filepath)
 
     success = testUtility.playTests(filepath)
     if success :
+
     if not success :
        slicer.util.exit(EXIT_SUCCESS)
+
      raise Exception('Failed to finished properly the play back !')
    else:
 
        slicer.util.exit(EXIT_FAILURE)
 
  
'''''Save to the directory ''''' <code>${Slicer_SOURCE_DIR}/Applications/SlicerApp/Testing/Python/ </code>
+
'''''Save to the directory ''''' <code>Slicer/Applications/SlicerApp/Testing/Python/MYTUTORIALNAME.py </code>
  
 
== Add the test in CMake ==
 
== Add the test in CMake ==
 
Adding the test in CMake will allow the test to be run with CTest and will also be run by the dashboard machines every nights.
 
Adding the test in CMake will allow the test to be run with CTest and will also be run by the dashboard machines every nights.
  
Edit the file <code>Slicer4/Applications/SlicerApp/Testing/Python/CMakeLists.txt</code><br>
+
Edit the file <code>Slicer/Applications/SlicerApp/Testing/Python/CMakeLists.txt</code><br>
Add your test at the end of the Slicer_USE_QTTESTING condition:
+
Add your test at the end of the Slicer_USE_QTTESTING condition and add the name in the <code>set_tests_properties</code> function :
 
   
 
   
 
     if(Slicer_USE_QTTESTING)
 
     if(Slicer_USE_QTTESTING)
Line 43: Line 87:
 
       ...
 
       ...
 
       slicer_add_python_test(
 
       slicer_add_python_test(
         SCRIPT pythonScriptName.py
+
         SCRIPT MYTUTORIALNAME.py
 
         SLICER_ARGS --launcher-no-splash --qt-testing)
 
         SLICER_ARGS --launcher-no-splash --qt-testing)
 +
      set_tests_properties(
 +
        py_NeurosurgicalPlanningTutorial
 +
        py_DiffusionTensorImagingTutorial
 +
        ....
 +
        py_MYTUTORIALNAME
 +
        ...
 +
        PROPERTIES RUN_SERIAL ON
 +
        )
 
       }
 
       }
  
Line 56: Line 108:
 
===Using CTest===
 
===Using CTest===
 
{|width = "100%"
 
{|width = "100%"
! style="border-bottom: 1px solid darkgrey;"| <small>Linux</small>
+
! style="border-bottom: 1px solid darkgrey;font-size: 75%;"|Linux
! style="border-bottom: 1px solid darkgrey;"| <small>Mac</small>
+
! style="border-bottom: 1px solid darkgrey;font-size: 75%;"|Mac
! style="border-bottom: 1px solid darkgrey;"| <small>Windows</small>
+
! style="border-bottom: 1px solid darkgrey;font-size: 75%;"|Windows
 
|-
 
|-
 
| colspan=2; valign="top" |
 
| colspan=2; valign="top" |
 
You need to open a terminal.<br>Then type in the terminal:<br><br>
 
You need to open a terminal.<br>Then type in the terminal:<br><br>
  $ cd path/to/Slicer4-Superbuild/Slicer-build
+
  $ cd path/to/Slicer-Superbuild/Slicer-build
  $ ctest -R MY_TEST_NAME -VV
+
  $ ctest -R MYTUTORIALNAME -VV
 
| valign="top" |
 
| valign="top" |
 
You need to open a terminal.<br>
 
You need to open a terminal.<br>
 
Start -> Microsoft Visual Studio 2008 -> Visual Studio Tools -> Visual Studio 2008 Command Prompt<br>
 
Start -> Microsoft Visual Studio 2008 -> Visual Studio Tools -> Visual Studio 2008 Command Prompt<br>
 
Then type in the terminal:<br>
 
Then type in the terminal:<br>
  $ cd path\to\Slicer4-Superbuild\Slicer-build
+
  $ cd path\to\Slicer-Superbuild\Slicer-build
  $ path\to\cmake\ctest.exe -R MY_TEST_NAME -VV
+
  $ path\to\cmake\ctest.exe -R MYTUTORIALNAME -VV
 
|}
 
|}
  
 
===Using the python script===
 
===Using the python script===
 
{|width="100%"
 
{|width="100%"
! style="border-bottom: 1px solid darkgrey;"| <small>Linux</small>
+
! style="border-bottom: 1px solid darkgrey;font-size: 75%;"|Linux
! style="border-bottom: 1px solid darkgrey;"| <small>Mac</small>
+
! style="border-bottom: 1px solid darkgrey;font-size: 75%;"|Mac
! style="border-bottom: 1px solid darkgrey;"| <small>Windows</small>
+
! style="border-bottom: 1px solid darkgrey;font-size: 75%;"|Windows
 
|-
 
|-
 
| colspan=2; valign="top" |
 
| colspan=2; valign="top" |
 
You need to open a terminal.<br>Then type in the terminal:<br><br>
 
You need to open a terminal.<br>Then type in the terminal:<br><br>
  $ cd path/to/Slicer4/Application/SlicerApp/Testing/Python
+
  $ cd path/to/Slicer/Application/SlicerApp/Testing/Python
  $ python MY_TEST_NAME.py
+
  $ python MYTUTORIALNAME.py
 
| valign="top" |
 
| valign="top" |
 
You need to open a terminal.<br>
 
You need to open a terminal.<br>
 
Start -> Microsoft Visual Studio 2008 -> Visual Studio Tools -> Visual Studio 2008 Command Prompt<br>
 
Start -> Microsoft Visual Studio 2008 -> Visual Studio Tools -> Visual Studio 2008 Command Prompt<br>
 
Then type in the terminal:<br>
 
Then type in the terminal:<br>
  $ cd path\to\Slicer4\Applicaiton\SlicerApp\Testing\Python
+
  $ cd path\to\Slicer\Applicaiton\SlicerApp\Testing\Python
  $ path\to\Slicer4-Superbuild\python-build\PCBuild\python.exe MY_TEST_NAME.py
+
  $ path\to\Slicer-Superbuild\python-build\PCBuild\python.exe MYTUTORIALNAME.py
 
|}
 
|}
  
Line 98: Line 150:
  
 
{|width="100%"
 
{|width="100%"
! style="border-bottom: 1px solid darkgrey;"| <small>Linux</small>
+
! style="border-bottom: 1px solid darkgrey;font-size: 75%;"|Linux
! style="border-bottom: 1px solid darkgrey;"| <small>Mac</small>
+
! style="border-bottom: 1px solid darkgrey;font-size: 75%;"|Mac
! style="border-bottom: 1px solid darkgrey;"| <small>Windows</small>
+
! style="border-bottom: 1px solid darkgrey;font-size: 75%;"|Windows
 
|-
 
|-
 
| colspan=2; valign="top" |
 
| colspan=2; valign="top" |
 
You need to open a terminal.<br>Then type in the terminal:<br><br>
 
You need to open a terminal.<br>Then type in the terminal:<br><br>
  $ cd path/to/Slicer4-Superbuild/CTK-build/QtTesting/Utilities/Scripts
+
  $ cd path/to/Slicer-Superbuild/CTK-build/QtTesting/Utilities/Scripts
 
  $ python TranslatorPlayerWizard.py MY_WIDGET_NAME
 
  $ python TranslatorPlayerWizard.py MY_WIDGET_NAME
 
  $ cd ../MY_WIDGET_NAME
 
  $ cd ../MY_WIDGET_NAME
Line 111: Line 163:
 
Start -> Microsoft Visual Studio 2008 -> Visual Studio Tools -> Visual Studio 2008 Command Prompt<br>
 
Start -> Microsoft Visual Studio 2008 -> Visual Studio Tools -> Visual Studio 2008 Command Prompt<br>
 
Then type in the terminal:<br>
 
Then type in the terminal:<br>
  $ cd path\to\Slicer4-Superbuild
+
  $ cd path\to\Slicer-Superbuild
 
  $ python-build\PCBuild\python.exe CTK-build\QtTesting\Utilities\Scripts\TranslatorPlayerWizard.py MY_WIDGET_NAME
 
  $ python-build\PCBuild\python.exe CTK-build\QtTesting\Utilities\Scripts\TranslatorPlayerWizard.py MY_WIDGET_NAME
 
  $ cd ../MY_WIDGET_NAME
 
  $ cd ../MY_WIDGET_NAME
Line 117: Line 169:
  
 
{{ambox
 
{{ambox
| type = speedy
+
| type =
 
| text = 4 files should have been created ! <br>  
 
| text = 4 files should have been created ! <br>  
 
<code>MY_WIDGET_NAMETranslator.cpp <br>  
 
<code>MY_WIDGET_NAMETranslator.cpp <br>  
Line 124: Line 176:
 
MY_WIDGET_NAMEPlayer.h</code><br>Those files are only the basic code. You now have to implement the functions, according to your widget, to record the events<br>Then add those files in the same folder as your widget.
 
MY_WIDGET_NAMEPlayer.h</code><br>Those files are only the basic code. You now have to implement the functions, according to your widget, to record the events<br>Then add those files in the same folder as your widget.
 
}}
 
}}
 +
 +
Finally move the 4 files to the same directory as your widget
 +
 +
$ cd path/to/Slicer-Superbuild/CTK-build/QtTesting/Utilities/MY_WIDGET_NAME
 +
$ mv MY_WIDGET_NAMETranslator.cpp MY_WIDGET_NAMETranslator.h MY_WIDGET_NAMEPlayer.cpp MY_WIDGET_NAMEPlayer.h Path/To/My/Widget/Directory
 +
 +
or just a simple copy to the same directory as your widget
 +
 +
$ cp MY_WIDGET_NAMETranslator.cpp MY_WIDGET_NAMETranslator.h MY_WIDGET_NAMEPlayer.cpp MY_WIDGET_NAMEPlayer.h Path/To/My/Widget/Directory
  
 
=Open bugs=
 
=Open bugs=

Latest revision as of 19:34, 26 April 2022

Home < Documentation < Nightly < Developers < QtTesting


For the latest Slicer documentation, visit the read-the-docs.


3DSlicerQtTesting.png

Overview

Recording of macro using QtTesting should be considered experimental.
We suggest to implement workflow automation through Python scripts operating at a lower level by changing properties of MRML nodes and calling module logic functions. See https://discourse.slicer.org/t/macro-xml-to-python/15803

QtTesting provides us a testing framework to test our application. It complements unit tests, by testing the application in its globality.

Following is a step by step guide to create slicer unit test, or module unit test, using this framework, which can be run as part of a nightly build automatic testing.

Rules for Developers

QtTesting records high level events, and try to not record mouse events as mouse press, release or move.
Furthermore, QtTesting is still under heavy development and some low level events won't be properly recorded.

Please be aware of those following warning :

  • Use "activated" signal instead of "clicked", when connection Qt views

How QtTesting and CTK work

CtkQtTestingSchemaPlay.png

CtkQtTestingSchemaRecord.png

How it works :

When pqTestUtility starts the playback, it calls pqEvenDispatcher to play a specific script. Then pqEventDispatcher finds an eventSource able to read the script. Then pqEventDispatcher forwards each event to the pqEventPlayer. Finally, this last class tries to find the best player to playback the action into the application. If the action has been done, we start over and over the same process until the end. If there is one error, or an action couldn’t be handled, the player fails with an error.

How it works :

When pqTestUtility starts to record events, it finds the observer which will write the file, creates the file, and starts the pqEventRecorder. Then every time there is a user action in the application, a specific translator will catch it and transform it into an high level event. This high level event is sent to the pqEventTranslator, and then to the pqEventObserver which writes the action recorded into a specific language, XML for example.

Write a 3DSlicer test using QtTesting

QtTesting framework can also be use to create module test.
Use the description below, but instead of Slicer/Applications/SlicerApp/Testing all the path start by Slicer/Path/To/The/Modules/Testing

Following is the different step to create a test using QtTesting framework.

Create your XML script

See QtTesting user Wiki to know how to create this xml script.
Save the xml script to Slicer/Applications/SlicerApp/Data/Input/MYTUTORIALNAME.xml

Create a Python script

This Python test will start 3DSlicer and run the previously recorded xml script

Create a python file with the following lines

   import os
   import slicer
   import ApplicationsSlicerAppData as data
   filepath = data.input + '/MYTUTORIALNAME.xml')     ex : NeurosurgicalPlanningTutorial.xml
   testUtility = slicer.app.testingUtility()
   success = testUtility.playTests(filepath)
   if not success :
     raise Exception('Failed to finished properly the play back !')

Save to the directory Slicer/Applications/SlicerApp/Testing/Python/MYTUTORIALNAME.py

Add the test in CMake

Adding the test in CMake will allow the test to be run with CTest and will also be run by the dashboard machines every nights.

Edit the file Slicer/Applications/SlicerApp/Testing/Python/CMakeLists.txt
Add your test at the end of the Slicer_USE_QTTESTING condition and add the name in the set_tests_properties function :

   if(Slicer_USE_QTTESTING)
     {   
     ...
     slicer_add_python_test(
       SCRIPT MYTUTORIALNAME.py
       SLICER_ARGS --launcher-no-splash --qt-testing)
     set_tests_properties(
       py_NeurosurgicalPlanningTutorial
       py_DiffusionTensorImagingTutorial
       ....
       py_MYTUTORIALNAME
       ...
       PROPERTIES RUN_SERIAL ON
       )
     }


You just wrote a Slicer Unit Test using QtTesting framework !

Run your test

You can either use directly CTest or the pyhton script created.

Using CTest

Linux Mac Windows

You need to open a terminal.
Then type in the terminal:

$ cd path/to/Slicer-Superbuild/Slicer-build
$ ctest -R MYTUTORIALNAME -VV

You need to open a terminal.
Start -> Microsoft Visual Studio 2008 -> Visual Studio Tools -> Visual Studio 2008 Command Prompt
Then type in the terminal:

$ cd path\to\Slicer-Superbuild\Slicer-build
$ path\to\cmake\ctest.exe -R MYTUTORIALNAME -VV

Using the python script

Linux Mac Windows

You need to open a terminal.
Then type in the terminal:

$ cd path/to/Slicer/Application/SlicerApp/Testing/Python
$ python MYTUTORIALNAME.py

You need to open a terminal.
Start -> Microsoft Visual Studio 2008 -> Visual Studio Tools -> Visual Studio 2008 Command Prompt
Then type in the terminal:

$ cd path\to\Slicer\Applicaiton\SlicerApp\Testing\Python
$ path\to\Slicer-Superbuild\python-build\PCBuild\python.exe MYTUTORIALNAME.py

Create custom translator/player

After created a custom widget, it might be possible that all its actions are not properly recorded, or played.
QtTesting provides us basic translator and player template, to create our custom translator and player according to the widget.

You can use the following command to create those basic files.

Linux Mac Windows

You need to open a terminal.
Then type in the terminal:

$ cd path/to/Slicer-Superbuild/CTK-build/QtTesting/Utilities/Scripts
$ python TranslatorPlayerWizard.py MY_WIDGET_NAME
$ cd ../MY_WIDGET_NAME

You need to open a terminal.
Start -> Microsoft Visual Studio 2008 -> Visual Studio Tools -> Visual Studio 2008 Command Prompt
Then type in the terminal:

$ cd path\to\Slicer-Superbuild
$ python-build\PCBuild\python.exe CTK-build\QtTesting\Utilities\Scripts\TranslatorPlayerWizard.py MY_WIDGET_NAME
$ cd ../MY_WIDGET_NAME

Finally move the 4 files to the same directory as your widget

$ cd path/to/Slicer-Superbuild/CTK-build/QtTesting/Utilities/MY_WIDGET_NAME
$ mv MY_WIDGET_NAMETranslator.cpp MY_WIDGET_NAMETranslator.h MY_WIDGET_NAMEPlayer.cpp MY_WIDGET_NAMEPlayer.h Path/To/My/Widget/Directory

or just a simple copy to the same directory as your widget

$ cp MY_WIDGET_NAMETranslator.cpp MY_WIDGET_NAMETranslator.h MY_WIDGET_NAMEPlayer.cpp MY_WIDGET_NAMEPlayer.h Path/To/My/Widget/Directory

Open bugs

Work in progress, following the two bug trackers :

Slicer issues

QtTesting issues