Page 1 of 1

LuxCore API and Unicode UTF-8 support

Posted: Tue May 19, 2020 1:38 pm
by Dade
Introduction

I have added the support for UTF-8 encoded strings to LuxCore API. This should allow the usage of not ASCII chars in object/material/texture names, file names, directories, etc. This is supposed to now allow the use of not latin languages (Chinese, Japanese, Russian, etc.) in LuxCore strings. The general philosophy LuxCore API now follow is this: http://utf8everywhere.org
Read the above link for a general introduction to the absolute mess that Unicode support is in general and across different OS.

Garbage in, garbage out

Even if LuxCore now supports UTF-8, the general rule "Garbage in, garbage out" still apply: you must pass UTF-8 encode strings to LuxCore API or the support for not latin languages will not work.

Linux and MacOS

Linux, MacOS and all Unix in general uses the same "utf8everywhere" philosophy so they should not require any special care.

Windows

Indeed, Windows is a mess as usually. It uses UTF-16 encoding ... mostly. So conversion from UTF-16 to UTF-8 should be applied to any string produced by any Windows API.

Python

As far as I know, Python uses the same UTF-8 encoding as LuxCore so it should work out of the box.

Blender

@B.Y.O.B.: I don't know how Blender works but I assume it will be UTF-8 too so BlendLuxCore may just work out of the box (i.e. passing the strings received from Blender to LuxCure API as they are).

External Libraries

There is also a problem with external libraries used by LuxCore. For instance checked OpenImageIO and it should work with UTF-8 encoded string too. There may be some other library to check (OpenVDB ?).

Testing

At the moment, I have done some test only on Linux and it seems to works fine with not-latin languages. I assume MacOS will work too. We need to check Windows.

Re: LuxCore API and Unicode UTF-8 support

Posted: Tue May 19, 2020 2:57 pm
by B.Y.O.B.
Dade wrote: Tue May 19, 2020 1:38 pm @B.Y.O.B.: I don't know how Blender works but I assume it will be UTF-8 too so BlendLuxCore may just work out of the box (i.e. passing the strings received from Blender to LuxCure API as they are).
Yes, seems to work fine.

Re: LuxCore API and Unicode UTF-8 support

Posted: Tue May 19, 2020 5:38 pm
by Rivental1
Thank you! Many of my files have russian names so it's very useful for me!

Re: LuxCore API and Unicode UTF-8 support

Posted: Thu May 21, 2020 1:58 pm
by acasta69
Dade wrote: Tue May 19, 2020 1:38 pm At the moment, I have done some test only on Linux and it seems to works fine with not-latin languages. I assume MacOS will work too. We need to check Windows.
I tried some very simple test on Windows using a filename with non-latin characters:
  • LuxCoreUI opens and renders correctly a file called 'שלום.cfg'
  • Blender saves and renders something called 'שלום.blend', and trying to export it saves '00001.bcf' in a folder named 'שלום_LuxCore'
  • pyluxcoretool.exe loads and renders that bcf from that folder
  • also renaming to 'שלום.bcf' works and 'שלום.png' is output correctly.
The scene used was the simple cube, so no OpenVDB involved.
Also it's the very first time I deal with something like this and have no idea if these tests are enough to say it's working. Do you have any suggestions about possible test cases?

P.S. Those characters are not displayed correctly in Windows console, even if I set it to UTF-8 with 'chcp 65001', nor in the Blender dialogs. I might have chosen an "unfortunate" language from the internet, it also seems to be written right-to-left. Using Cyrillic characters has no visualization problems.

Re: LuxCore API and Unicode UTF-8 support

Posted: Thu May 21, 2020 2:38 pm
by Dade
It should be enough, there may be some file read/write somewhere that may not works (opening .ies files or loading a background image, etc. some not spotted corner case): it is easy to miss something across half million lines of code. However it is already a quite good news that the general usage works.