@Confidant Reporting back as requested (I tried to keep it short–honest):
The kiosk, with version 1 of the application with almost full-sized images, was up and running at its first event last week. Thank you for all the help in the Haxe and OpenFl forums and some one-on-one help.
Zooming in and out of the huge images is fast and smooth, and panning works well, but boy do I wish Haxe applications targeting Windows could play video (would Away3D VideoTexture work or would the same licensing issues apply)! Anyone else want to pitch in for a bounty?
I had to use Electron. There is a big performance hit in one area when using Electron. But the kiosk needs to play video and that was the solution I was able to get working on time.
The goal was to be able to smoothly zoom in and out of huge images. And, this is for a kiosk, so other than OS stuff, my application is the only application running and I don’t care how long it takes the application to load up when launched.
Implementation:
My direct customer shot the photos in HDR, but did not maintain HDR through the image processing, so for now I don’t need to support that.
The touch display is a 50” 4K monitor in portrait orientation. The PC is an Intel NUC 11 Enthusiast with 64GB of memory and a decent M.2 drive. It has the mobile version of the 6GB RTX 2060 GPU and the CPU is an i7-1165G7. It is running Windows 11.
I had to go with 2 sizes of each of the 45 images. For the main menu, where you need to be able to quickly run through images, I used smaller 1080 x 1920 images. I preload those as OpenFl Bitmap’s.
The bigger images are PNGs with maximum compression. To work in Electron, the huge zoomable images had to be scaled down so neither dimension was greater than 16,384 pixels. They are all pre-loaded as ByteArray’s. When the viewer opts to look at a large image, it takes up to 5 seconds to covert it from a ByteArray to BitmapData so it can be displayed. That’s the big performance hit under Electron. Also, that conversion is a processing hog, so tweens (Actuate) of other things pretty much grind to a halt during the conversion.
Once the big image is converted and displayed, the viewer can zoom with two finger pinch/spread and pan by dragging with a single finger. The viewer can also zoom in/out and pan by holding down zoom or pan buttons. They all work fully as fast as desired. And fortunately, I did not need to use a multi-image format for the zooms!
I am super happy with the speed of the zoom using 2-finger pinch and spreads. You can go from a scale of around 0.1 up to 4 in a couple spreads with no lag. For finger zooming and panning I used openfl.ui.Multitouch (no gesture libraries).
Tradeoffs/Laments:
So, why do I wish I could target Windows directly? When targeting Windows rather than Electron, I can preload all the large images as Bitmaps (which of course are mostly BitmapData). I have an additional Bitmap for actually displaying the large images. When I want to change the image being displayed, I merely point the BitmapData property of that additional Bitmap to the BitmapData of the image I want to display. With an exe, I estimate it takes well under a quarter second to switch images. That is compared to around 5 seconds for converting a preloaded ByteArray image to BitmapData under Electron.
Under Windows, I’m guessing that when you load a Bitmap image, Haxe/OpenFl immediately expands it to the internal format needed for display. This is unlike Adobe AIR (and presumably Harmon AIR), which waits to expand the image until you actually display it—and then frees up the extra memory a little while after you quit displaying it.
I say this because the large images take up about 5GB on the hard drive, and my test application expands from about 98MB to about 28GB once all of the images are loaded. It further expands up to about 31GB when switching the images being displayed. I’m further guessing that with an exe, when you actually display the image, that is when it is transferred to GPU memory—and it is removed when you quit displaying it or perhaps on dispose() or disposeImage(). Just a guess because the GPU memory is only 6GB.
Under Electron, I’m wondering if Bitmap’s are by loaded into the GPU by default. Because under Electron, I can only preload about half the large Bitmap’s before the application chokes. If that is the case, and if there is a way to store a Bitmap (or at least BitmapData) outside of the GPU, then doing that might eliminate my 5 second delay when switching images.
The hit happens again, indirectly, during the attract loop where the large images are tweened on and off the screen. I need to have two images converted so one image tweens on while the other is tweening off. But I can’t convert the next image from ByteArray to BitmapData during those tweens, because the conversion processing would grind the tweens to halt. So once in position, an image has to stay on the screen for about 5 seconds while the next image is converted. My direct customer wants to speed that up. And I can do that by using smaller preloaded Bitmap images for the attract loop rather than the large images, but then my customer also gives up the possibility of being able to have the application zoom into the image detail during the attract loop. We’re not doing that yet, but we have talked about it.
So, with an exe, not only would I have vastly faster performance when switching images, but I would also have been able to go without creating a second smaller version of each image. The second version is undesirable because down the road the end customer wants to be able to add images on their own—as easily as possible. Extra image production is an extra step for something to go wrong, and I don’t know if they are savvy enough to be able to scale images to the maximum size that will fit the entire image within 1080 x 1920 pixels for the main menu (i.e., will they scale for the correct dimension—width vs height).
I might be able to speed up the Electron version by loading the images as Bitmap’s when they are needed (and then dumping them when done with them) rather than preloading them. On average the large images take about 2.4 seconds to load as Bitmaps on my development computer with a slower SSD (and under a Windows executable). It would certainly be worth trying.
I would also like to log kiosk usage by storing data in a CSV text file. I’ve done that many times with AIR kiosks and it should be easy under a Haxe/OpenFl exe. I think there is a way to do that under Electron, but I probably won’t go through the learning curve.
Caveat
As the end customer will want to add more images down the road, if I could use a Windows exe I would eventually need to switch computers to something that can have more than 64GB of memory. But I really like that Intel NUC we got, so if that happens soon, I could find another use for that, minimizing the cost of upgrading the computer.
Cheers.