An interactive northernlights experience created with the help of OpenFL, Haxe and Unreal Engine

Showcase
1

So this is a bit of a different usecase for Haxe and OpenFL!

I was a technical director and lead developer for an interactive experience center about northern lights.
Located in the north of Norway, in the town of Alta, in a beautiful cathedral.
cathedral

The interactive exhibition brings together different stories about the northern lights and its magical qualities, focusing on interactive experiences based on Sami myths. Visitors can create and play with the northern lights as they wave to the starlit sky, or invoke the aurora through song.

Technical description:
We developed several installations using the power of Haxe, OpenFL and Unreal Engine.
In most cases the front UI and user interface / logic was developed using Haxe and OpenFL utilizing a custom UI plugin for Adobe Animate targeting HxCpp / C++.

The effects layers was created with Unreal Engine with the additional use of 3rd party middleware particle system PopcornFX. The effects layer is blended in the displaylist in OpenFL using a custom c++ integration of the Spout SDK - http://spout.zeal.co , making it possible to launch Unreal Engine as a subprocess and streaming the graphics layer back into the OpenFL application.

Video about the project


Wave to the light and create your own northen lights - interactive particle system

9_borealisalta
9_borealisalta1000Ă—562 72.3 KB

IMG_20170428_124938_rs
IMG_20170428_124938_rs1152Ă—648 340 KB

wave2
wave21200Ă—640 404 KB



Sing to the northern light and battle an opponent with Sami Joik - A “singstar” themed installation using realtime voice analyzing and particle systems running on two transparent OLED displays

8_borealisalta
8_borealisalta1000Ă—562 106 KB

6_borealisalta
6_borealisalta1000Ă—667 107 KB

sing1
sing11200Ă—640 464 KB

12 Likes
2

Brutal :smiley:

There seem to also be a tablet app - are there different apps?
Could you share a sample project with this setup?

3

That’s very nice, would you be interested in talking about it at next Haxe Summit?

3 Likes
4

Thanks! It was a fun project to work on!
Actually, the tablet app in the video was developed separately by another developer and is done using Xamarin.

About a sample app, do you mean my custom Haxe/OpenFL/C++/Spout/Unreal graphics pipeline?
I think that would be possible to do!

5

Thanks Nicolas! Much appreciated coming from you! :smiley:

Actually i think the use-case and some of the technical solutions me and the team used for some of these installations are quite unique and they actually worked out pretty good! Maybe it could be interesting to talk about it on the Haxe Summit… hm :slight_smile:

3 Likes
6

Perhaps the Northern Lights will be visible from Seattle this May. :milky_way:

1 Like
7

Amazing. That spout implementation is simply next level stuff.
Congrats.

1 Like
8

Hey @d0oo0p, awesome stuff! I remember seeing your posts about this on twitter. Are you able to explain a little more how you’re launching Unreal Engine as a subprocess and streaming the graphics layer back into the OpenFL application. Are they running as two different applications? or within the same app but two different processes. I’m pretty keen to start using Unreal a bit more, but I’m not totally convinced their 2d framework is up to scratch, so being able to merge the two is an ideal outcome.

9

Hey, thanks guys! :smiley:
Sure, i can explain a bit more about the pipeline :

First off there is a c++ application compiled with Haxe/hxcpp thats using OpenFL, and is responsible for the 2D gui / animations and the application logic. All GUI layouts and elements are created in Adobe Animate and is using a custom Animate plugin and framework developed by me, that allows code injection and components to be constructed from the timeline without the need for compilation during export, giving great flexibility and control.

Since the app is targeting win/c++ , we can write externs for external c++ libraries and windows functions. So what we did was to embed and extern the Spout SDK, to create a “spoutbitmap” that can be added to the displaylist, showing the spout data that is connected. Since Spout is based around sharing the graphics buffer, the cost for showing a 4K bitmap @ 60fps is pretty low.

Also we have done some functions to run a windows process from the application directly, so you have full control over that subprocess and you can start, hide and terminate it as you please. This means that when the application starts it runs the Unreal based part, which is separate application and hides the display window. Now the unreal “backend” is running but is not drawing anything onscreen or showing a window.

The UE4 application has several capture targets that feed the output to Spout, using the UE4 spout plugin, and this blends in very nicely with the displaylist in the OpenFL app. There is a custom local UDP connection set up between the Haxe app and the UE4 app that is used to do realtime communication and transfer data / values back and forth. On the Haxe side the data is processed and sent to the UE4 side where using blueprints and c++ that data drives values in the PopcornFX particle system, combined with UE4 Postprocessing to create the specific effects that is requred.

The “singstar” installation has a custom voice analyzer written with Openframeworks, that analyzes the song inputs and feeds this data into the Haxe application for processing, while the “wave” installation gets a datafeed constructed from several Kinects reading human position information and hand motions, processed via an Openframeworks application that unifies the data and feeds it into the Haxe application.

Hm, i think i will write a more detailed article about this somewhere, with code examples and a bit more in-depth information :slight_smile:

6 Likes
10

I would love see you talking about it at the haxe summit.

4 Likes
11

What version of UE4 are you running?
This video (UE4 To Spout LiveStream Tutorial Edited by Lynn Jarvis (Part 1) - YouTube) is pretty good at giving an overview of how to rebuild UE4 to support Spout, is this the same work flow you used? If so does it mean you have to use UE4.8 or does it work with the latest version?

1 Like
12

Hi Pete. I belive i used 4.14, it was the latest version when development started. There is no need to rebuild UE4, you can just use this plugin, should work with all versions : GitHub - AleDel/Spout-UE4: Spout Plugin for Unreal Engine