An interactive northernlights experience created with the help of OpenFL, Haxe and Unreal Engine


(Datee) #1

So this is a bit of a different usecase for Haxe and OpenFL!

I was a technical director and lead developer for an interactive experience center about northern lights.
Located in the north of Norway, in the town of Alta, in a beautiful cathedral.

The interactive exhibition brings together different stories about the northern lights and its magical qualities, focusing on interactive experiences based on Sami myths. Visitors can create and play with the northern lights as they wave to the starlit sky, or invoke the aurora through song.

Technical description:
We developed several installations using the power of Haxe, OpenFL and Unreal Engine.
In most cases the front UI and user interface / logic was developed using Haxe and OpenFL utilizing a custom UI plugin for Adobe Animate targeting HxCpp / C++.

The effects layers was created with Unreal Engine with the additional use of 3rd party middleware particle system PopcornFX. The effects layer is blended in the displaylist in OpenFL using a custom c++ integration of the Spout SDK - , making it possible to launch Unreal Engine as a subprocess and streaming the graphics layer back into the OpenFL application.

Video about the project

Wave to the light and create your own northen lights - interactive particle system

Sing to the northern light and battle an opponent with Sami Joik - A “singstar” themed installation using realtime voice analyzing and particle systems running on two transparent OLED displays

(Philippe) #2

Brutal :smiley:

There seem to also be a tablet app - are there different apps?
Could you share a sample project with this setup?

(Nicolas Cannasse) #3

That’s very nice, would you be interested in talking about it at next Haxe Summit?

(Datee) #4

Thanks! It was a fun project to work on!
Actually, the tablet app in the video was developed separately by another developer and is done using Xamarin.

About a sample app, do you mean my custom Haxe/OpenFL/C++/Spout/Unreal graphics pipeline?
I think that would be possible to do!

(Datee) #5

Thanks Nicolas! Much appreciated coming from you! :smiley:

Actually i think the use-case and some of the technical solutions me and the team used for some of these installations are quite unique and they actually worked out pretty good! Maybe it could be interesting to talk about it on the Haxe Summit… hm :slight_smile:

(Rezmason) #6

Perhaps the Northern Lights will be visible from Seattle this May. :milky_way:


Amazing. That spout implementation is simply next level stuff.

(Pete Shand) #8

Hey @d0oo0p, awesome stuff! I remember seeing your posts about this on twitter. Are you able to explain a little more how you’re launching Unreal Engine as a subprocess and streaming the graphics layer back into the OpenFL application. Are they running as two different applications? or within the same app but two different processes. I’m pretty keen to start using Unreal a bit more, but I’m not totally convinced their 2d framework is up to scratch, so being able to merge the two is an ideal outcome.

(Datee) #9

Hey, thanks guys! :smiley:
Sure, i can explain a bit more about the pipeline :

First off there is a c++ application compiled with Haxe/hxcpp thats using OpenFL, and is responsible for the 2D gui / animations and the application logic. All GUI layouts and elements are created in Adobe Animate and is using a custom Animate plugin and framework developed by me, that allows code injection and components to be constructed from the timeline without the need for compilation during export, giving great flexibility and control.

Since the app is targeting win/c++ , we can write externs for external c++ libraries and windows functions. So what we did was to embed and extern the Spout SDK, to create a “spoutbitmap” that can be added to the displaylist, showing the spout data that is connected. Since Spout is based around sharing the graphics buffer, the cost for showing a 4K bitmap @ 60fps is pretty low.

Also we have done some functions to run a windows process from the application directly, so you have full control over that subprocess and you can start, hide and terminate it as you please. This means that when the application starts it runs the Unreal based part, which is separate application and hides the display window. Now the unreal “backend” is running but is not drawing anything onscreen or showing a window.

The UE4 application has several capture targets that feed the output to Spout, using the UE4 spout plugin, and this blends in very nicely with the displaylist in the OpenFL app. There is a custom local UDP connection set up between the Haxe app and the UE4 app that is used to do realtime communication and transfer data / values back and forth. On the Haxe side the data is processed and sent to the UE4 side where using blueprints and c++ that data drives values in the PopcornFX particle system, combined with UE4 Postprocessing to create the specific effects that is requred.

The “singstar” installation has a custom voice analyzer written with Openframeworks, that analyzes the song inputs and feeds this data into the Haxe application for processing, while the “wave” installation gets a datafeed constructed from several Kinects reading human position information and hand motions, processed via an Openframeworks application that unifies the data and feeds it into the Haxe application.

Hm, i think i will write a more detailed article about this somewhere, with code examples and a bit more in-depth information :slight_smile:

(david mouton) #10

I would love see you talking about it at the haxe summit.

(Pete Shand) #11

What version of UE4 are you running?
This video ( is pretty good at giving an overview of how to rebuild UE4 to support Spout, is this the same work flow you used? If so does it mean you have to use UE4.8 or does it work with the latest version?

(Datee) #12

Hi Pete. I belive i used 4.14, it was the latest version when development started. There is no need to rebuild UE4, you can just use this plugin, should work with all versions :