COMMUNITY

Wanted: advice on fast bitmap scaling

Hello Haxe community,

I have to bid a project in a couple days and I am trying to figure out an approach and how much effort I will have to spend on image processing performance. I’m not sure how to target the GPU or even if I should.

I have tried to figure out where Haxe and OpenFL or additional libraries use hardware acceleration, but it is taking me too long. I’ve found a few image libraries, but again haven’t figured out if they use the GPU.

Thanks in advance for advice and pointing me towards articles/documentation.

The shortish version:
What is a good approach using Haxe and probably OpenFl to have very fast bitmap scaling for viewing and zooming in and out on individual very large (2GB) HDR images? This is for a kiosk application and I can specify the GPU, CPU, SSD, and amount of RAM as well as the monitor resolution—probably 4K maybe 8K.

Should I target the GPU or a CPU with many cores/threads (initiate multiple transform threads at once knowing that I will likely need one or more of them to zoom smoothly)? Would a game engine be overkill—that’s okay if it is fastest development path.

I’m quite new to Haxe and I haven’t yet figured out when the GPU is or isn’t targeted with Haxe and OpenFl. So far I have never directly used OpenGL or DirectX, so I am guessing that bindings such as OpenFl’s OpenGLRenderer might involve a greater learning curve than I have time for.

The touchscreen interface will need to be as minimal as possible, so I don’t need a lot of controls/components—at least not visible components.

The longer version—in case someone is curious and has plenty of excess time:
I need to figure out an approach for a potential touch screen kiosk application. I have done several kiosk projects using AS3/AIR to generate 64-bit Windows captive runtime applications. As a potential replacement for AS3/AIR/FlashDevelop, I started dabbling with Haxe, HaxeDevelop, and OpenFl back in March and then dropped it as other things took priority. Before dropping it I converted a small piece of an AS3/AIR kiosk application to Haxe/OpenFL and after much frustration was able to create a working EXE that utilized most of the common elements I tend use in kiosk applications.

The key aspect of this kiosk application will be to view large gorgeous HDR images on a 4K monitor (potentially 8K monitor down the road). A sample image I have is close to 2GB in size and has a pixel resolution of 12,752 x 19,136 (monitor will be in portrait orientation).

The kiosk needs to allow visitors to zoom from a scaled version of the entire image, down to having one image pixel to one monitor pixel, and any level in between. Visitors will also need to able to pan at any resolution. This needs to be done as elegantly as possible.

I have done something similar with aerial images of a city using AIR. I started with a few hundred large GIS aerial images shot at 0.5 ft per pixel and created around 20,000 images (2,000 x 2,000 pixels each) at 9 different levels of resolution (ft per pixel that is, ending with 128 feet per pixel), for zooming in and panning around the city. However, that zoom isn’t particularly elegant. From whatever resolution (feet per pixel) the viewer is on, I merely scale the current view of the bitmap(s), not the bitmap data, according to the amount of zoom, and then cut to bitmaps most closely matching the final resolution (ft per pixel) of the zoom. Zooming from the entire city view, down to street level view gives you a very ugly scaled up image while you are zooming in (spreading two fingers) until you let go and the new bitmaps appear. Zooming out leaves you with a lot of blank space as the area you are zooming out of gets smaller and smaller. Pulling images in for continual panning was plenty fast.

Note that in that project I did compress the aerial images, finding a tradeoff between speed of execution and image quality. This time I want to maximize image quality.

Also, this time I’m trying to avoid the pre-production of generating images at varying resolutions.

I can specify the computer hardware for the kiosk. It will be a Windows machine in part because I don’t have time to learn something else.

Repeating from above: I have never worked directly with DirectX or OpenGL and at this point I am hoping not to have to. I have tried to figure out where Haxe and OpenFL or additional libraries use hardware acceleration, but it is taking me too long. I’ve found a few image libraries, but again haven’t figured out if they use GPU. Also, as scaling will likely be my only transformation, I’m not sure if the libraries provide any benefit.

A few things I’ve glanced at along the way:

  • Core Architecture
  • bitmapData.draw() (this would be the multiple thread approach)
  • Haxe “image” library with GraphicsMagick installed
  • Haxe “bitmap” library
  • Haxe “magic” library with ImageMagick installed
  • Haxe “heaps.io” game engine
  • Feathers UI
  • starling
  • Haxeui.org

Closing:
Again, thanks in advance for advice and pointing me towards articles and documentation.

I can’t answer about which haxe library to use, but I can tell you that you do need GPU acceleration for this task, so you must pick a library that does that. Even with the fastest CPU you’d never be able to get smooth framerate at those resolutions if you tried to render it in software mode.

I think the dimensions of your image (even if it is HDR) will allow you to have it completely uploaded into video memory if you get a GPU with enough ram. If that’s the case your job shouldn’t be too hard.
If it doesn’t fully fit in VRAM you’ll have to create multiple versions of different quality and only load the high quality versions of the image when you zoom in. That can get quite complicated.

Since your image is HDR, do you intend to use an HDR display? If not then you can convert your image to truecolor instead to save in VRAM. If you do intend to use an HDR display it you will need to make sure your engine supports HDR rendering.

1 Like

@basro Thank you for the reply.

I intend to have an HDR display but have not yet started looking at what I need to do in software to support HDR images. I appreciate your note on this.

I did find some information on GPU usage under OpenFl–not sure how I missed it previously.

Forum info on OpenFl and GPU

It appears as if OpenFl uses the GPU most of the time for my target, but I still have some digging to do, because the bitmapData.draw() method apparently does not use the GPU and I need to figure out how to best utilize the GPU.

OpenFl seems to be a good direction to pursue, which is nice because I am quite familiar with AS3 and have tried out OpenFl. Also, there are libraries that sit on top of OpenFl that might be of use.

I hadn’t thought about maybe having to have multiple resolution images even with the use of the GPU. I will make sure we have plenty of VRAM. I already have a system for creating and utilizing such images, but for long term maintenance simpler is better.

Regards

Hi,

Loading 2Gb of image (in an already uncompressed format ?) seems overkill, the tile-pyramid approach that you seem to have already used, and that is commonly used for such applications (web cartography, ultra-high-res images), is probably much simpler, scalable and performant for a minimal overhead. It could be generated and cached by an existing script or via Haxe.

See for example OpenSeeDragon, an old library which allows smooth scrolling of high res images on common hardware. It supports pinch-zooming, so if using a browser in kiosk mode is compatible with your other needs, it might be a simple solution.

If a native binary or more control is needed, a prototype in OpenFL and/or in Kha (lower level graphic library) to test smooth zooming would be informative and maybe not to long to produce. I guess both make use of GPU as much as possible nowadays. You could take care of preloading and displaying the lower tile while zooming to prevent scale-up, and perfect the display, once the bid is accepted.

@tokiop Thank you for the good input. Sorry my slow reply, I didn’t have email notification set.

I like OpenSeeDragon and have seen sites that use it–without knowing what was underneath. I didn’t even realize that there is such a thing as a zooming image format–much less many of them. The photographer is starting a website and I will pass this on to him.

Viewing these I realized that a pre-determined zoom is more aesthetically pleasing than zooming to match finger gestures. Hmmm, I tested a button interface for zooming on my GIS project, but people are very accustomed to finger pinching and I had to go that route.

I had noticed Kha, but didn’t look very closely at it. I will take another peek.

We are targeting a museum quality experience, or at least being able to get there in two versions. So we are entertaining overkill for now–but we have not been assigned a budget yet.

I started looking for a single image solution after zooming in and out on a huge .ARQ file under Affinity Photo on a computer with an RTX3080 graphics card (not at my desk unfortunately). It is pretty smooth. It does show a little temporary blocking when you zoom fast, but it is not bad. I figured there has to be a way for me to do this.

For that GIS project, to create the thousands of images with multiple levels of resolution I wrote code that spit out data that was then used as input for Javascript routines I wrote for Photoshop batch processing. That production would be much less complex for this project, but the customer wouldn’t be able to just drop in additional images on their own in the future if we use multiple resolutions (unless of course the application has a mode where it creates them when the customer drops them in).

What I didn’t do in that project, was switch imagery (resolution) while zooming. That would have improved the aesthetic appeal.

I might try a test with OpenFl tiles, and see where that gets me.

Thanks again.

Hi. I think this could be solved quite easily using Kha. It is quite simple to use and is more low level with less overhead than OpenFL and gives you excellent access to handling drawing bitmaps, scaling etc on the GPU - at native speeds.

@d0oo0p Thank you for the insight on Kha. I had looked at Kha briefly, but haven’t done any of that type of low level graphics programming. The Kha API documentation (top level) has very little description of how to use the library. Many classes merely have their methods listed, with no explanation.

Is there a good documentation resource somewhere for a noob in low level graphics programming getting started with Kha? Something with an overview?

Here’s some nice starting points:

And Lewis Leptons Kha tutorial series is perfect for starters:

1 Like

@d0oo0p Those are good starting points. I haven’t installed Kha yet, but I’ve watched a handful of Lewis Lepton’s tutorial videos. I like the pace and level of detail. The samples link appears to have some samples that will be great for me (e.g., HDR).

Thanks!

This reminded me of a product called Zoomify. I used that a bit along time ago but I don’t know what it would take to integrate with your project.

@Confidant I checked out Zoomify. Zoomify uses multiple images (tens or tens of thousands of images). It has the same issues I’m trying to avoid as the GIS implementation I created has, but Zoomify does do it more elegantly. I could at least implement a bit of the nicer technique if I end up implementing a multiple image method. (e.g., when zooming out a long way, they use a static blurred version of the image as the background rather than a “blank” background as I had done).

I see Zoomify has “unconverted image viewing” in a couple of paid versions and drag and drop conversion too.

If we land the project, I’m looking forward to trying to use image processing on the fly with OpenFl and/or Kha, but Zoomify might be a good backup plan.

Thanks!

I might show some ignorance here so bear with me as I ramble since I find this interesting. Doing something “on the fly” will require maximum performance and I would think you would need a lot of RAM and would benefit greatly from a GPU. I am wondering if it’s better to use something that supports cacheing, i.e. a server solution (but you’d run it locally) which would save you some electricity in the long run. If you compiled Haxe to Python you could perhaps integrate a library like Dirpy. There’s also the old standards ImageMagick and GD with whatever server that has a cache.

Be sure to report back when you arrive at a solution!

@Confidant I appreciate the additional feedback–good stuff to consider.

The beauty of the situation is that this is a stand-alone kiosk project and I get to specify the computer hardware. I am thinking roughly the equivalent of: RTX-3080, i9 processor, as much RAM as a I need, and M.2 SSD.

It’s first deployment would be international event in New York, the second at a conference at a museum. We are hoping the museum will be interested in a permanent version, so we are putting budget towards elegant performance.

As it is a single kiosk, the hardware expense isn’t as large a factor as it would be for a multiple kiosk project. Of course we might get the project with less budget than asked for, and then I go back to the drawing board (and Zoomify is open source).

1 Like