LuxCore API and Unicode UTF-8 support

Discussion related to the LuxCore functionality, implementations and API.
Post Reply
User avatar
Dade
Developer
Posts: 4817
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

LuxCore API and Unicode UTF-8 support

Post by Dade » Tue May 19, 2020 1:38 pm

Introduction

I have added the support for UTF-8 encoded strings to LuxCore API. This should allow the usage of not ASCII chars in object/material/texture names, file names, directories, etc. This is supposed to now allow the use of not latin languages (Chinese, Japanese, Russian, etc.) in LuxCore strings. The general philosophy LuxCore API now follow is this: http://utf8everywhere.org
Read the above link for a general introduction to the absolute mess that Unicode support is in general and across different OS.

Garbage in, garbage out

Even if LuxCore now supports UTF-8, the general rule "Garbage in, garbage out" still apply: you must pass UTF-8 encode strings to LuxCore API or the support for not latin languages will not work.

Linux and MacOS

Linux, MacOS and all Unix in general uses the same "utf8everywhere" philosophy so they should not require any special care.

Windows

Indeed, Windows is a mess as usually. It uses UTF-16 encoding ... mostly. So conversion from UTF-16 to UTF-8 should be applied to any string produced by any Windows API.

Python

As far as I know, Python uses the same UTF-8 encoding as LuxCore so it should work out of the box.

Blender

@B.Y.O.B.: I don't know how Blender works but I assume it will be UTF-8 too so BlendLuxCore may just work out of the box (i.e. passing the strings received from Blender to LuxCure API as they are).

External Libraries

There is also a problem with external libraries used by LuxCore. For instance checked OpenImageIO and it should work with UTF-8 encoded string too. There may be some other library to check (OpenVDB ?).

Testing

At the moment, I have done some test only on Linux and it seems to works fine with not-latin languages. I assume MacOS will work too. We need to check Windows.
Support LuxCoreRender project with salts and bounties

User avatar
B.Y.O.B.
Developer
Posts: 3872
Joined: Mon Dec 04, 2017 10:08 pm
Location: Germany
Contact:

Re: LuxCore API and Unicode UTF-8 support

Post by B.Y.O.B. » Tue May 19, 2020 2:57 pm

Dade wrote:
Tue May 19, 2020 1:38 pm
@B.Y.O.B.: I don't know how Blender works but I assume it will be UTF-8 too so BlendLuxCore may just work out of the box (i.e. passing the strings received from Blender to LuxCure API as they are).
Yes, seems to work fine.

Rivental1
Posts: 28
Joined: Tue Mar 05, 2019 9:15 pm

Re: LuxCore API and Unicode UTF-8 support

Post by Rivental1 » Tue May 19, 2020 5:38 pm

Thank you! Many of my files have russian names so it's very useful for me!

acasta69
Developer
Posts: 361
Joined: Tue Jan 09, 2018 3:45 pm

Re: LuxCore API and Unicode UTF-8 support

Post by acasta69 » Thu May 21, 2020 1:58 pm

Dade wrote:
Tue May 19, 2020 1:38 pm
At the moment, I have done some test only on Linux and it seems to works fine with not-latin languages. I assume MacOS will work too. We need to check Windows.
I tried some very simple test on Windows using a filename with non-latin characters:
  • LuxCoreUI opens and renders correctly a file called 'שלום.cfg'
  • Blender saves and renders something called 'שלום.blend', and trying to export it saves '00001.bcf' in a folder named 'שלום_LuxCore'
  • pyluxcoretool.exe loads and renders that bcf from that folder
  • also renaming to 'שלום.bcf' works and 'שלום.png' is output correctly.
The scene used was the simple cube, so no OpenVDB involved.
Also it's the very first time I deal with something like this and have no idea if these tests are enough to say it's working. Do you have any suggestions about possible test cases?

P.S. Those characters are not displayed correctly in Windows console, even if I set it to UTF-8 with 'chcp 65001', nor in the Blender dialogs. I might have chosen an "unfortunate" language from the internet, it also seems to be written right-to-left. Using Cyrillic characters has no visualization problems.
Support LuxCoreRender project with salts and bounties

Windows 10 64 bits, i7-4770 3.4 GHz, RAM 16 GB, GTX 970 4GB v445.87

User avatar
Dade
Developer
Posts: 4817
Joined: Mon Dec 04, 2017 8:36 pm
Location: Italy

Re: LuxCore API and Unicode UTF-8 support

Post by Dade » Thu May 21, 2020 2:38 pm

It should be enough, there may be some file read/write somewhere that may not works (opening .ies files or loading a background image, etc. some not spotted corner case): it is easy to miss something across half million lines of code. However it is already a quite good news that the general usage works.
Support LuxCoreRender project with salts and bounties

Post Reply