Documentation/Labs/VTK-String - Revision history

JChris.FillionR: /* Design */

2019-05-07T22:06:54Z

Design

Lasso: /* Design */

2019-05-03T16:10:43Z

Design

Lasso: /* Problem */

2019-05-03T16:08:04Z

Problem

Lasso: /* Design */

2019-05-03T16:01:58Z

Design

Lasso: /* Problem */

2019-05-03T12:44:30Z

Problem

Lasso: /* Design */

2019-05-03T12:38:50Z

Design

Lasso: /* Plan */

2019-04-25T02:54:24Z

Plan

Lasso: /* Problem */

2019-04-25T02:53:40Z

Problem

Lasso: Created page with "== Problem == Currently, encoding in VTK strings is not explicitly specified. When receiving a string from external libraries or using the string in operating system calls (e..."

2019-04-25T02:44:30Z

Created page with "== Problem == Currently, encoding in VTK strings is not explicitly specified. When receiving a string from external libraries or using the string in operating system calls (e..."

New page

== Problem ==

Currently, encoding in VTK strings is not explicitly specified. When receiving a string from external libraries or using the string in operating system calls (e.g., reading/writing files) then the behavior is often incorrect.

For example:
* files that have non-ASCII characters in their name cannot be opened
* when changing the application locale (so that some necessary special characters can be stored in a single byte), then generated files may become invalid (e.g., because decimal point is replaced by decimal comma)
* Python and Qt stores strings with known encoding, but there is no way to convert them to/from strings in VTK without loss of information

There is a [https://vtk.org/doc/nightly/html/classvtkUnicodeString.html vtkUnicodeString] class in VTK that you store string with a known encoding. It can store and provide string in utf8 and utf16 encoding. It is already used extensively in text rendering, arrays, tables, certain file export/import, but majority of VTK still uses const char* and get/set macros.

Using const char* for string storage, managing memory with vtkSetStringMacro/vtkGetStringMacro, and process strings with C string functions are all very outdated programming practices. VTK-based applications must choose between following VTK's approach and stuck with outdated practices or break away from it and live with inconsistencies in the code base and be cautious with managing strings (additional conversions, null-pointer checks are needed) - none of these options are good.

== Proposal ==

=== Design ===

Use an encoding-aware string class in VTK to store all strings.

[https://vtk.org/doc/nightly/html/classvtkUnicodeString.html vtkUnicodeString] is a good starting point, as it can store any string with known encoding and it is already in VTK, used in a number of VTK classes.

It could be renamed to ''vtkString'' to make the name shorter. It is also more clear if we don't include the name of a particular encoding in the class name (as in the future we might support multiple encodings inside the string class, not necessarily just Unicode). This would also consistent with how other libraries manage strings (see for example Qt's [https://doc.qt.io/qt-5/qstring.html QString]).

=== Plan ===

* Rename vtkUnicodeString by vtkString. Maybe improve the API with adding get as/set from Latin1.
* Replace all string attributes in VTK classes by vtkString (add new get/set macros, create object instance in the constructor)
* Update Python wrapping
* Maybe add automatic converters to ''const char*'' (can be disabled by CMake flags) to make update of application code easier
* Review all operating system calls and make sure strings are properly converted

@@ Line 21: / Line 21: @@
 Two design options may be considered:
-- A. Use an encoding-aware string class in VTK to store all strings. In the future, if most libraries and operating system APIs use utf8 encoding for std::string, then that special string class can be replaced by std::string.
+* A. Use an encoding-aware string class in VTK to store all strings. In the future, if most libraries and operating system APIs use utf8 encoding for std::string, then that special string class can be replaced by std::string.
-- B. Assume that all strings in VTK are utf8 encoded and store them in std::string.
+* B. Assume that all strings in VTK are utf8 encoded and store them in std::string.
 Option A:

@@ Line 20: / Line 20: @@
 === Design ===
-Use an encoding-aware string class in VTK to store all strings.
+Two design options may be considered:
 [https://vtk.org/doc/nightly/html/classvtkUnicodeString.html vtkUnicodeString] is a good starting point, as it can store any string with known encoding and it is already in VTK, used in a number of VTK classes. [https://github.com/dgobbi/vtk-dicom/blob/master/Source/vtkDICOMFilePath.cxx vtkDICOMFilePath] contains useful conversion code, too.

@@ Line 12: / Line 12: @@
 Using const char* for string storage, managing memory with vtkSetStringMacro/vtkGetStringMacro, and process strings with C string functions are all very outdated programming practices. VTK-based applications must choose between following VTK's approach and stuck with outdated practices or break away from it and live with inconsistencies in the code base and be cautious with managing strings (additional conversions, null-pointer checks are needed) - none of these options are good.
-It could be possible to state that "all strings in VTK are utf8 encoded", but it would be hard to enforce that this requirement is fulfilled in all VTK and all classes of VTK-based applications. It would be especially difficult to ensure that strings are transcoded when getting/setting data in VTK to/from other libraries and when VTK calls system APIs. It could have performance impact if we required all strings to be converted to utf8 to be stored in VTK (if we want to store a non-utf8-encoded string in VTK, we would need to always need to do an extra encoding and decoding operation).
+It could be possible to state that "all strings in VTK are utf8 encoded", see manifesto of advocates of this approach [http://utf8everywhere.org/ here]. It is certainly compelling that applications can be made to support Unicode without many changes, but it would be hard to enforce that this requirement is fulfilled in all VTK and all classes of VTK-based applications. It would be especially difficult to ensure that strings are transcoded when getting/setting data in VTK to/from other libraries and when VTK calls system APIs. In the long term, when all software libraries assume std::string is utf8-encoded and Windows has utf8 code page, utf8 everywhere is definitely a good solution, but it is questionable if we should try to jump there in one step, or via an intermediate step of using a dedicated encoding-aware string class.
 See some more discussion here: https://discourse.vtk.org/t/proposal-should-we-replace-vtkstdstring-with-std-string/796/14

@@ Line 24: / Line 24: @@
 [https://vtk.org/doc/nightly/html/classvtkUnicodeString.html vtkUnicodeString] is a good starting point, as it can store any string with known encoding and it is already in VTK, used in a number of VTK classes. [https://github.com/dgobbi/vtk-dicom/blob/master/Source/vtkDICOMFilePath.cxx vtkDICOMFilePath] contains useful conversion code, too.
-It could be renamed to ''vtkString'' to make the name shorter. It is also more clear if we don't include the name of a particular encoding in the class name (as in the future we might support multiple encodings inside the string class, not necessarily just Unicode). This would also consistent with how other libraries manage strings (see for example Qt's [https://doc.qt.io/qt-5/qstring.html QString]).
+It could be renamed to ''vtkString'' to make the name shorter. It is also more clear if we don't include the name of a particular encoding in the class name (as in the future we might support multiple encodings inside the string class, not necessarily just Unicode). This would also consistent with how other libraries manage strings (see for example [https://doc.qt.io/qt-5/qstring.html Qt's QString], [https://developer.gnome.org/glibmm/unstable/classGlib_1_1ustring.html Gnome's ustring]). There are a number of other implementations, such as [https://github.com/DuffsDevice/tiny-utf8 tiny-utf8]
 Maybe vtkDICOMFilePath should be added to VTK (as vtkFilePath, maybe parent class could be vtkString) for storing file paths. File paths need some extra features, such as conversion of slashes and handling of extended paths (\\?\) on Windows.

← Older revision		Revision as of 12:44, 3 May 2019
Line 11:		Line 11:

	Using const char* for string storage, managing memory with vtkSetStringMacro/vtkGetStringMacro, and process strings with C string functions are all very outdated programming practices. VTK-based applications must choose between following VTK's approach and stuck with outdated practices or break away from it and live with inconsistencies in the code base and be cautious with managing strings (additional conversions, null-pointer checks are needed) - none of these options are good.		Using const char* for string storage, managing memory with vtkSetStringMacro/vtkGetStringMacro, and process strings with C string functions are all very outdated programming practices. VTK-based applications must choose between following VTK's approach and stuck with outdated practices or break away from it and live with inconsistencies in the code base and be cautious with managing strings (additional conversions, null-pointer checks are needed) - none of these options are good.
		+
		+	It could be possible to state that "all strings in VTK are utf8 encoded", but it would be hard to enforce that this requirement is fulfilled in all VTK and all classes of VTK-based applications. It would be especially difficult to ensure that strings are transcoded when getting/setting data in VTK to/from other libraries and when VTK calls system APIs. It could have performance impact if we required all strings to be converted to utf8 to be stored in VTK (if we want to store a non-utf8-encoded string in VTK, we would need to always need to do an extra encoding and decoding operation).

	See some more discussion here: https://discourse.vtk.org/t/proposal-should-we-replace-vtkstdstring-with-std-string/796/14		See some more discussion here: https://discourse.vtk.org/t/proposal-should-we-replace-vtkstdstring-with-std-string/796/14

@@ Line 29: / Line 29: @@
 * Replace all string attributes in VTK classes by vtkString (add new get/set macros, create object instance in the constructor)
 * Update Python wrapping
 * Maybe add automatic converters to ''const char*'' (can be disabled by CMake flags) to make update of application code easier
 * Review all operating system calls and make sure strings are properly converted

← Older revision		Revision as of 02:53, 25 April 2019
Line 11:		Line 11:

	Using const char* for string storage, managing memory with vtkSetStringMacro/vtkGetStringMacro, and process strings with C string functions are all very outdated programming practices. VTK-based applications must choose between following VTK's approach and stuck with outdated practices or break away from it and live with inconsistencies in the code base and be cautious with managing strings (additional conversions, null-pointer checks are needed) - none of these options are good.		Using const char* for string storage, managing memory with vtkSetStringMacro/vtkGetStringMacro, and process strings with C string functions are all very outdated programming practices. VTK-based applications must choose between following VTK's approach and stuck with outdated practices or break away from it and live with inconsistencies in the code base and be cautious with managing strings (additional conversions, null-pointer checks are needed) - none of these options are good.
		+
		+	See some more discussion here: https://discourse.vtk.org/t/proposal-should-we-replace-vtkstdstring-with-std-string/796/14

	== Proposal ==		== Proposal ==