Discussion on how to remove TkImg or improve platform compatibility

[Akuli]

This is a "continuation" of #17. I just installed from github...

$ python3.4 -m pip install --user git+https://github.com/redfantom/ttkthemes

...and I still get the same error:

>>> import ttkthemes
>>> root=ttkthemes.ThemedTk()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/akuli/.local/lib/python3.4/site-packages/ttkthemes/themed_tk.py", line 29, in __init__
    ThemedWidget.__init__(self, self.tk)
  File "/home/akuli/.local/lib/python3.4/site-packages/ttkthemes/_widget.py", line 46, in __init__
    self._img_support = self._load_tkimg()
  File "/home/akuli/.local/lib/python3.4/site-packages/ttkthemes/_widget.py", line 209, in _load_tkimg
    raise RuntimeError("Unsupported platform and/or architecture")
RuntimeError: Unsupported platform and/or architecture
>>> 

[RedFantom]

When installing directly from git, with the same command as you have used, except for using Python 3.6 instead of Python 3.4 on Ubuntu 17.04 64-bit, I cannot reproduce the issue.

Could you provide me with more details about your platform? Then I can spin up a Virtual Machine and try to reproduce the issue in the exact same environment.

[Akuli]

Python 3.4 and Python 3.7 on a 64-bit Ubuntu 14.04.
This bug is also annoying because some code is not using try-finally around os.chdir() correctly:

>>> import os
>>> os.getcwd()
'/home/akuli/porcu'
>>> import ttkthemes
>>> ttkthemes.ThemedStyle()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/akuli/.local/lib/python3.7/site-packages/ttkthemes/themed_style.py", line 29, in __init__
    ThemedWidget.__init__(self, self.tk)
  File "/home/akuli/.local/lib/python3.7/site-packages/ttkthemes/_widget.py", line 46, in __init__
    self._img_support = self._load_tkimg()
  File "/home/akuli/.local/lib/python3.7/site-packages/ttkthemes/_widget.py", line 209, in _load_tkimg
    raise RuntimeError("Unsupported platform and/or architecture")
RuntimeError: Unsupported platform and/or architecture
>>> os.getcwd()
'/home/akuli/.local/lib/python3.7/site-packages/ttkthemes'
>>> 

[Akuli]

Oops, I clicked the close button accidentally.

[Akuli]

I think I found the problem...

>>> ttkthemes._utils.get_tkimg_directory()
'/home/akuli/.local/lib/python3.7/site-packages/ttkthemes/tkimg/linux64'
>>> os.listdir('/home/akuli/.local/lib/python3.7/site-packages/ttkthemes/')   # no tkimg!
['_imgops.py', 'themes', '_widget.py', '__pycache__', 'themed_tk.py', '_utils.py', 'advanced', '__init__.py', 'themed_style.py']
>>> 

Maybe something's wrong with setup.py?

[Akuli]

Are you sure that this code is ok? https://github.com/RedFantom/ttkthemes/blob/master/setup.py#L23-L35

[Akuli]

Umm... can you just delete the weird if "pip" not in __file__ check? I don't know what it's supposed to do, but in any case, it's an ugly hack and it's causing this issue. When installing with git+, __file__ is something like '/tmp/pip-req-build-srr21gk7/setup.py' and the tkimg stuff doesn't get copied.

I liked this library when this was a "pip install and use and it works" thing instead of an ugly "pip" in __file__ hack :(

[RedFantom]

The real issue here is that I need to include a certain set of library files, tkimg in order to be able to work with PNG-images on platforms that do not support Tk 8.6. However, compiling tkimg from source is not possible on most modern Linux systems due to dependency conflicts and doing so on Windows is pretty unworkable too because it depends on Tcl header files and whatnot.

Thus, I have to include the compiled library files for the platforms they are available for. However, I do not want to include all library files for all platforms in all wheel packages, I only want to include a single set. But because it is not an extension that can be compiled upon building the wheel, I have to include the files in the package manually.

What files to select depends on the platform, and then the files can be copied over to the right folder and packaged in the wheel. When using a source distribution, all library files are included still and the code uses path finding to select the correct ones out of the full set during runtime.

I don't know why the files don't get copied in your particular case, and I would certainly like to know so I can fix this issue, but I can't reproduce it: It works fine for me when either using sdist or bdist. For now, I cannot delete the weird hack, because I do not know of a better way to implement this. If I could build the tkimg package, I would, but I have tried and failed (landing in apt-get dependency hell).

If you do know a better way of fixing this instead, I would love to hear it, but for now this is the only solution that I know of that works and allows non-Tk 8.6 platforms to work with PNG-images.

[Akuli]

Maybe it would be best to always include all binaries on all platforms for now? Nobody would notice the bigger download size anyway.

I can figure out a better alternative to the "pip" in __file__ hack if you tell me what it does and why.

Edit: I'm interested in the if "pip" not in __file__: line, not the code that follows it. I still have no idea why a file named ~/blahpip/setup.py should be treated differently than ~/blah/setup.py.

[Akuli]

Ummm, did I understand correctly? I think you're saying that tkimg is only needed for Tcl/Tk 8.5 and older. Would it be possible to instead use the built-in png support in Tcl/Tk 8.6 when that's available? I always use Tk 8.6 anyway.

[Akuli]

Sorry for the walls of text :( It's just annoying not to be able to install a library I really want to use.

[RedFantom]

Yes, I understand that that is frustrating. I really want to help and fix this, I just don't know how yet.

I want to offer everyone on every platform the same experience. Building just an sdist is something I will consider as a last option.

The pip check is performed to see if the package is being installed with pip, or whether a 'normal' execution of setup.py is performed.

For now, could you run setup.py as:
python setup.py bdist_wheel install?

I will spin up a fresh VM with no history of ttkthemes when I am back at my workstation.

[Akuli]

Why would pip need to be treated differently?

[Akuli]

Well, that command was very useful...

akuli@Akuli-Desktop:~/sourcestuff/ttkthemes$ python3.7 setup.py bdist_wheel install
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'bdist_wheel'

[Akuli]

Well, at least I have a working installation now. I wish pip installing wasn't so broken though. I ran python3.7 setup.py bdist and then moved a bunch of files from dist/home/akuli/.local/lib/python3.7/site-packages to ~/.local.

[RedFantom]

If your system does not recognize the bdist_wheel command, then what version of setuptools are you using? As said in the wiki, you should update pip and setuptools to the latest versions before installation.

It's not necessarily pip that needs to be run differently. The point is that when pip is being run, I would assume that either a wheel or an sdist package is being installed, and thus the libraries for the right platform are already in the package being installed and thus don't have to be copied over again. It would perhaps be possible to change this into an existence check instead. I did not account for the usage through a git-link (I didn't even know that existed).

Edit: For now, the packages on PyPI and python setup.py bdist_wheel install with the latest version of setuptools are guaranteed to work.

[Akuli]

when pip is being run, I would assume that either a wheel or an sdist package is being installed

In that case, why not check for their existance with e.g. os.path.exists or os.path.isdir? That would be much better than 'pip' in __file__.

what version of setuptools are you using?

Something too old. I just updated it.

[Akuli]

I can create a pull request that replaces 'pip' in __file__ with something not so shit, i.e. os.path.exists or something like that. What do you think?

[RedFantom]

Yes, that is okay with me. I will try to get to it as soon as I can, but that can take a few more days.

My concern when initialiy building this code, was that when using an existence check, I was not sure how compatible that would be with installing (and thus execution from) wheel packages and source distributions alike.

The real underlying issue is that the libraries are already compiled, and that is in no way officially supported by Python packaging. So it will always be a work-around-like situation.

[RedFantom]

Automatically closed after merging #23.

[maicol07]

Hi,
this issue happens any time I run my script. The solution included in update 2.0.5 seems doesn't work anymore...
Here is the shell/console log:

Python 3.6.4 (Windows 10 x64 1804)

Traceback (most recent call last):
  File "main.py", line 241, in <module>
    style.init()
  File "style.py", line 6, in init
    s=ThemedStyle()
  File "C:\Users\Maicol\AppData\Local\Programs\Python\Python36\lib\site-packages\ttkthemes\themed_style.py", line 29, in __init__
    ThemedWidget.__init__(self, self.tk)
  File "C:\Users\Maicol\AppData\Local\Programs\Python\Python36\lib\site-packages\ttkthemes\_widget.py", line 46, in __init__
    self._img_support = self._load_tkimg()
  File "C:\Users\Maicol\AppData\Local\Programs\Python\Python36\lib\site-packages\ttkthemes\_widget.py", line 205, in _load_tkimg
    raise RuntimeError("Unsupported platform and/or architecture")
RuntimeError: Unsupported platform and/or architecture

Process finished with exit code 1

Thanks

P.S.: Should I open a new issue or will you reopen this one?

[maicol07]

I found out that tkimg isn't installed. The folder C:\Users\Maicol\AppData\Local\Programs\Python\Python36\lib\site-packages\tkimg\win64 doesn't exists on my PC. How can I install it?

[RedFantom]

This really is the gift that keeps on giving.

You cannot install TkImg yourself. Well, you can, but it's complicated. It's a really old library that you cannot really feasibly compile anymore, so I have included it with ttkthemes. Do you know whether you are running a wheel or sdist installation? Could you try running pip install --upgrade ttkthemes so it is for certain you have 2.0.6 installed?

Update: I am as of yet unable to reproduce the issue for 2.0.6 with the following steps:

• Install Python 3.6 on Windows 10 64-bit with Tkinter enabled
• Use the most recent version of pip and setuptools to install the most recent version of ttkthemes
• Run an interactive shell, import ttkthemes and setup a ThemedStyle with a new Tk instance

I do not think the issue is related to the location where Python was installed (I had it installed in C:\Python, but for some reason the code path of _utils.py, line 100 is entered with line 103 containing the culprit for moving out of the right directory.

[maicol07]

Here is the result:

Requirement already up-to-date: ttkthemes in c:\users\maicol\appdata\local\programs\python\python36\lib\site-packages (2.0.6)
Requirement not upgraded as not directly required: pillow in c:\users\maicol\appdata\local\programs\python\python36\lib\site-packages (from ttkthemes) (5.1.0)

[RedFantom]

If check the provided wheel file for Windows 64-bit and Python 3.6, it does include tkimg. Could you check if the folder c:\users\maicol\appdata\local\programs\python\python36\lib\site-packages\ttkthemes\tkimg exists? And if so, does it contain a lot of folders, or a bunch of files?

Update: Oh, and could you make sure that you're not running inside a venv with an older version? It happened to me earlier today that I had an older version in ~/.local (but as far as I know, that particular folder only works on Linux) and my Python automatically loaded that over the system-wide one. The same applies to virtual environments.

[maicol07]

It doesn't exists...

[RedFantom]

That is very strange indeed. Did you install ttkthemes from PyPI using pip or by cloning the repository and running python setup.py bdist_wheel install?

Could you install the Windows 64-bit Python 3.6 wheel by downloading the file and then running pip install ttkthemes-2.0.6-cp36-cp36m-win_amd64.whl to make sure that you have a wheel installed?

[maicol07]

I installed ttkthemes from PyPi using pip.

With the wheel it works. I found out, when I'm installing it, that I had two versions of ttkthemes installed, the 2.0.6 and the 2.0.5. First, I uninstalled 2.0.6 and then installed the wheel, that uninstalled 2.0.5.

[RedFantom]

So it might have been loading 2.0.5 instead of 2.0.6, or it did not select the correct wheel for installing, somehow...? If you find out which of the two, that would be great. If not, then I guess I'll throw it on the heap of weird irreproducible things.

This is clearly a complicated issue. Before I released 2.0.0 I knew including tkimg would be a bit difficult, but I didn't quite expect this. I will leave this open, at least for a while, to see if other people still encounter issues with 2.0.6.

For future posters: If you have any issues, please post! I want to really make sure that it doesn't just work for me, but for everyone!

[Akuli]

How about just converting all images to gifs and getting rid of tkimg? Then we wouldn't have more "it doesn't work for me" issues, and if anyone happens to be using an OS that is actually unsupported, it would also work for them.

Edit: I use gifs with my own tkinter projects just because other image files don't work before Tk 8.6, and not everyone have 8.6 yet.

[RedFantom]

@Akuli Converting to GIF can have unintended consequences for some themes because the color space is more limited. It does support transparency though (at least with Tk >8.5, I'm not sure about 8.4).

Conversion would be a pretty big undertaking though. Conversion with PIL is not sufficient, because transparency conversion does not work quite correctly:

Conversion with imagemagick can work though:

Though in at least a few images conversion artifacts occur because those images do not only make uses of alpha pixels, but also the alpha channel in order to make for soft edges. They would have to be fixed by hand where possible, and the result would not be nearly as good as it is right now. Also, that would mean dynamic themeing would only be possible if tkimg is present, because that would require using PNG images still.

It would also mean that introducing new themes would be more complicated as well. I was hoping to port over a few more Gtk-themes like I did with equilux(#29 ), even though it's not completely done yet.

Considering everything, I will not convert the PNG-based themes to GIF-only images just yet. I first want to see if more people continue having issues. In the meantime, I will look into the possibility of having the package support an option for installing tkimg support at runtime, possibly by building a separate package to include tkimg.

Edit: And as a side-note: I have been using PNG-images and other formats in other projects for a long time using the PIL.ImageTk.PhotoImage class. Usage is simple:

from PIL.ImageTk import PhotoImage
from PIL import Image

image = Image.open("some_image.png")
photo = PhotoImage(image)  # Pass this to the image kwarg of Tkinter widgets

[Akuli]

Have you checked PIL's source to see how it passes the image data to tk?

[RedFantom]

PIL is an extensive C-library for Python, and checking how it passes its image data to Tk would involve basically writing something similar to the TkImg library. I can't use Python to create the theme images though.

I've been looking into the possibility of building a second library that can load tkimg into a Tk interpreter on a supported platform so I can remove it from here. Then this library can have GIF copies of all themes, and then only if the library is installed like pip install ttkthemes[tkimg] or something would it use the PNG-based themes. I personally think that that would be the cleanest solution. The GIF-color depth is not too limiting for even theme arc.

[Akuli]

Maybe it wouldn't be too bad to write a working png to gif converter in plain Tcl? If you're lazy, just take some conversion code written for another language and convert that to Tcl.

[RedFantom]

Tcl is an interpreted language, like Python, and I won't pretend to be good at it. Conversion during run-time would cost an enormous amount of time (relatively speaking) at library object creation, and that would most definitely be more than what removing numpy gained.

Plain Tcl would be too slow to do run-time conversion. There is no PIL library in Tcl, so conversion would have to be done by reading in the binary data of the files, then somehow manipulating that (which means studying how PNG and GIF-formats work in the first place) and then saving the manipulated file. That does not make sense. Once you know how to read the PNG image, it would be more efficient to simply pass that on.

Now the worst part is that passing on that data has to be done in C. Tcl supports custom image formats with its C-library, and surprise: That's exactly what TkImg is. I don't mind C, but I do mind reinventing the wheel.

Even conversion before start-up using Python would be too slow. The set_theme_advanced is already slow, but that's only when the user asks the library to do that. On top of that PIL is not capable of converting the alpha channel of the PNG images to the GIF images, creating the artifacts I have shown above in the screenshot.

From all this I conclude that there are three options:

• Only supporting GIF - I'd rather not because of the limited color space and the inevitable artifacts that it causes for themes that are originally PNG
• Supporting PNG only in Tk 8.6 - I'd rather not because I want the library to work the same across Python versions
• Supporting PNG with TkImg - Exactly where we are now, though the way the library is loaded could be improved by building it into its own package and making the PNG-support optional

So, I really appreciate the input, I do, but currently I just don't see any other feasible option. At least not one that is both relatively fast and will not take months with the amount of time I currently spend on this library.

[Akuli]

Plain Tcl would be too slow to do run-time conversion.

Theme images are very small. For example, if you have a 1500x1500 pixel photo, it has 10000 times more pixels than a 15x15 icon typically found in a theme. I don't think that speed would be a big issue with small pictures, even in plain Tcl. A 15x15 icon has 225 pixels, and a loop that runs 225 iterations per image doesn't sound too bad to me.

... then somehow manipulating that (which means studying how PNG and GIF-formats work in the first place) ...

I can look into this if you decide that this might be useful. I also know a few friends who might find this a nice challenge. Also, like you said, someone has done this before, so we could find someone else's code that does this and make a Tcl version of it.

...and then saving the manipulated file.

Tk images can take raw gif data directly, and there's no need to save it to a temporary file. It looks roughly like PhotoImage(format='gif', data=base64.b64encode(...)) in tkinter, so it's doable in Tcl.

... that would most definitely be more than what removing numpy gained.

Maybe we could convert the images when they are actaully needed (i.e. when changing a theme) instead of on startup?

What do you think?

[Akuli]

Hmmm, I think I'm really overcomplicating things...

If you look at e.g. tkimg/linux64, you can see that there are lots and lots of files for dealing with formats like ppm, ico, jpeg and so on. Does tkimg work for png files if you delete those other files? All of the remaining bits of tkimg for all platforms altogether would probably be smaller than what is currently in each platform-specific folder, so always including the files for all platforms could also be an option.

[RedFantom]

Yes, Tkinter can indeed take base64 encoded images to build PhotoImages from that. One theme I encountered that isn't published here because of licensing issues does that. It's pretty neat, because everything fits in a single file. But, the key is that you are still limited to the GIF-color space. That means that reading from a file compared to creating it that way makes zero difference...

Converting the images when they are actually needed would require dynamically loading the themes when they are needed, and that complicates things in the current Tcl file-structure. It might also increase Tcl overhead, but actually knowing for sure requires profiling. If the work is already done to convert the PNG images to GIF format (of which I doubt whether it's a good idea, see my first paragraph) then doing that would be relatively easy.

When I integrated TkImg I attempted to separate the files from each other, but then I got weird crashes of Tkinter (the kind that I had when the equilux theme was incomplete, full Tcl interpreter breakdowns while holding Python's GIL. Really nice.). I can try again at some later time, though.

[Akuli]

Ok, one more idea... how about just always including all tkimg files for all platforms, not just the target platform? Nobody would notice the bigger size of the package anyway, and we would stop getting "this didn't install correctly to me" problems.

[RedFantom]

Everything considered I think that providing an option upon installation for using the PNG-based themes and using the GIF-themes by default would not require a huge amount of work while offering the most flexibility in terms of supporting platforms, porting over themes and providing a simple interface.

I just really dislike the idea of copying Windows- and macOS-library files onto Linux machines (or any other combination) just because setting up the already compiled library files is difficult. They would literally be a waste of space. Also, when packaging with PyInstaller, that additional size is most definitely noticeable.

[Akuli]

How about checking if Tk 8.6 or newer is installed in setup.py? Then pyinstaller users could just use Tk 8.6 and have no size problems. Also, if tkimg doesn't get included at all on 8.6, ttkthemes can run on platforms whose tkimg binaries are not included.

I don't really like the idea of copying macOS files to Linux machines, but I really don't like random "it doesn't work for me" bug reports.

#32 seems like a lot of work. Are you sure there's no better option?

[RedFantom]

#32 might be a lot of work, but it not the most work out of the alternatives, and it does satisfy everything on what I would want from a solution:

• Correct loading of PNG-based themes (through GIF-images) in case TkImg cannot be loaded and Tk is version 8.5 or lower.
• Not including unnecessary files upon installation
• (Nice-to-have): Support for platforms that TkImg does not support in the first place, like the Raspberry Pi

Currently, platforms that do not support PNG only have the option of installing ttkthemes==1.5.2, but because I also have to include a source distribution for Linux platforms that do not use the wheel format I produce, it has to be selected manually and that doesn't sit right with me either.

So far, no one has indicated a reproducable problem that exists for the latest version, neither here nor on StackOverflow, so I am treating the current version as good until I have seen otherwise.

Executing checks in setup.py might be a possibility, but it might impede the possibility of producing wheel files. After all, wheel files are (pretty much) simply ZIP-files that are extracted, and I can't build wheel files specifically for Tk 8.5 and 8.6 separately...

Building a single source distribution that can perform the checks would be okay, but then the files to be copied would have to be dynamically set. That might be a neat solution, but depending on the implementation it might be more fragile than the current set-up...

[maicol07]

Does the issue with Pyinstaller happen also with cx_freeze? It raises me the same error.

UPDATE: I think I've found a fix. See PR below!

[RedFantom]

@maicol07 , could you please elaborate on the issue you are experiencing? Because this thread is not specifically about issues with PyInstaller. In fact, I use this library with PyInstaller and it works quite fine... If you have an issue with PyInstaller specifically, could you open a new issue so it can get the attention it deserves? This issue has transformed into a discussion instead.

[maicol07]

Well, the issue is that I build my app with cx_freeze and I get this error:

Is the thread related to Platform issues, right?

[RedFantom]

@maicol07 While this thread is related to platform issues, I would like to keep the actual, concrete issues in the current version with the path-finding and the discussion on how to improve the path-finding in future versions separate. This looks like a new issue indeed. Could you open a new issue, particularly so you can elaborate on why PR #33 implements a fix for the issue you are experiencing?

[RedFantom]

With the merging of #38, this discussion has come to an end. Most Python installations will now use the PNG-based themes by default, and those that do not always have GIF-counterparts available to them. Using PNG-based themes is still possible for those platforms by manually installing python-tkimg.

Thank you @Akuli for providing your input and pushing this issue and @maicol07 for reporting your issues.