a couple of days ago I released a smol game using HaxeFlixel (sources here: GitHub - AlexHaxe/haxe-same) . it’s a very simple game and probably could use some better graphics (PRs welcome).
what probably only very few people noticed: the game comes with unittests and fully automated UI tests using the latest haxeium release version.
anyone familiar with Selenium test framework will find a few similarities when looking at haxeium lib. both try to solve the same problem: ``does the button in my app actually do the thing when users press it?´´.
without an automated UI testing framework you will have to manually test your app (or game) every time you make any change to it or at least every time you are preparing a release. depending on the size of your app that can take a long time where one or multiple testers will have to click through your app again and again and again, filling out test protocol after test protocol (at least in the corporate world) - and most likely they still won’t find everything.
frameworks for automated UI tests like haxeium can help reduce manual testing by having your computer or your continuous integration systems perform some parts of it. so you can run UI tests on every commit making sure your app still works as expected.
obviously even automated UI tests are not a cure-all - they only test what you tell them to. so they will be as blind as you make them.
Selenium works by opening a browser window to visit your webapp’s URL and then it sends mouse and keyboard events to individual HTML elements and reads out state and / or attributes of those elements. to anyone who’s ever looked at the HTML sources of a HaxeFlixel project using html5 target it should be very clear that Selenium cannot really work in such a project. the reason is simple: there is a big canvas element that shows your app / game, but that canvas element is just a bunch of pixels, there is no way for Selenium to dig into it and access your FlxG.game or any of your FlxSprite objects. all Selenium can do is click on some coordinates and then read out pixels to see if some have changed colours. that’s probably not very stable and not a very good basis for writing UI tests.
that’s where haxeium comes in. unlike Selenium which basically puppeteers a browser window and interacts with your webapp from the outside, a part haxeium lib gets compiled into a version of your app. from there it can access your components and sprites or even internal data structures. of course having to compile a version of your app that includes haxeium lib is not ideal since you are not running UI tests against a full release version of your app, but it’s as close as you can get.
how does it work?
first you install haxeium lib via haxelib install haxeium and haxelib install hxWebSockets. then you add it to your project.xml
<haxelib name="haxeium" if="uitests" />
<haxelib name="hxWebSockets" if="uitests" />
and then you initialise it in your main class
#if uitests
new haxeium.drivers.FlixelHaxeUIDriver("ws://127.0.0.1:9999");
#end
so now everytime you compile your app with -D uitests it will include haxeium lib which will try to open a websocket connection to 127.0.0.1 on port 9999. unless there is a test runner application listening on that port, haxeium lib will not do much else and it will not hinder any normal function of your app. as soon as a test runner accepts a connection it will start running tests.
currently haxeium can run on HTML5, Hashlink and C++ targets. for HTML5 you have to bring your own webserver (using lime test html5 -D uitests is sufficient).
the test runner side currently only supports C++ target. it is usually a very simple utest project and it doesn’t require any flixel, openfl, lime or other libraries your main project uses. just utest, haxeium and hxWebSockets is all you need.
your UI test runner’s main class initialises and runs test classes just like any regular utest project. before it can run UI tests it needs to set up an AppDriver instance and an AppRestarter, e.g. (assuming your app was compiled using lime build hl -D uitests.)
new AppDriver("127.0.0.1", 9999, new AppRestarter("./hxSame", [], "export/hl/bin"));
AppDriver opens a websocket server and manages communication between test runner and your app. it provides an API to find and interact with components or elements of your app.
AppRestarter is responsible for starting and killing instances of your app. in almost all cases you want to run your UI tests on a fresh instance of your app simply to avoid issues where a failed UI test leaves your app in a state it can’t recover from. such a case would result in all subsequent UI tests also failing. which is a pretty unenjoyable scenario to find yourself in. with AppRestarter tests get a fresh instance every time. and they can assume that your app always starts in the same state.
obviously there are things like savegames and other persistent data. if your app uses that you will need to add code to wipe those storage locations to make sure your testcases find your app in a state they expect.
haxeium lib comes with two (three) base classes for your testclasses: TestBaseAllRestarts restarts your app for every test. TestBaseOneInstance only starts one instance for the whole testclass, so all tests within it use the same instance. usually you want your tests to have a fresh instance so TestBaseAllRestarts is what you should start with. of course you can always come up with your own variant and extensions.
both classes extend from a common base class called TestBaseScreenshotAssert. which provides all the assertion functions you would find in utest.Assert with one helpful addition: every assert in that base class will record a screenshot of your app when an assertion fails. you can still use pure utest.Assert if you want, but having a screenshot for every failure or error during UI testing is very helpful when trying to figure out what went wrong.
writing tests is obviously a bit more involved compared to simple unittests. you always have to factor in some communication overhead and also have to make sure your app has had enough time to process your input (e.g. a mouse click) and has updated it’s state and UI.
lets look at PlayStateTest.hx to see how testSolve_12345 testcase works:
Wait.untilElementBecomesVisible(ById("seed"));
waits for a HaxeUI component with id="seed" to become visible.
var seed = driver.findElement(ById("seed"));
finds HaxeUI component with id="seed" and returns an instance of Element which represents the element on test runner side.
new Actions().doubleClick(seed).pause(0.2).sendKeys("12345").perform();
creates a new Actions object, adds a doubleClick action on seed, a pause of 200ms and then sends “12345” to seed component which happens to be a textfield. doubleclicking results in all existing text to be selected, sending “12345” simulates key presses that input that sequence into the textfield.
Wait.untilPropertyEqualsValue(seed.locator, "text", "12345");
waits until seed component’s text property equals “12345”.
var button = driver.findElement(ById("setseed"));
button.click();
Sys.sleep(0.02);
finds button component with id="setseed", clicks it and waits for 20ms to make sure app had time to process that event.
clickField(col, row);
a custom function (see below) to click on any field by specifying column and row. I’ve recorded a number of clicks that will solve the game for seed “12345”. and I use clickField calls to make UI test replay those clicks, which should result in a field that is completely cleared and the game ends in a victory.
var winLossLabel = driver.findElement(ById("winloss"));
equals("VICTORY!", winLossLabel.text);
finds the HaxeUI label with id="winloss" and checks its text property to see if it displays “VICTORY!”, indicating and verifying the game has ended in a win.
function clickField(col:Int, row:Int) {
var flx = driver.findElement(ByPath(["flixel.group.FlxTypedGroup-1", 'game.FieldSprite-${col + row * 15}']));
flx.click();
Sys.sleep(0.02);
}
function clickField tries to locate a FieldSprite according to its col and row coordinates and then clicks on it. it then uses a sleep of 20ms to make sure the game had time to process the click.
ByPath is how haxeium allows you to find HaxeFlixel sprites, since there is no easy way to reference any element of flixel by e.g. ID. unlike HaxeUI you cannot easily search for a specific sprite. so haxeium has to work around that limitation.
in flixel-land ByPath(["flixel.group.FlxTypedGroup-1", 'game.FieldSprite-${col + row * 15}']) basically translates to:
var flxObj = FlxG.state.members.filter(m -> m.className == "flixel.group.FlxTypedGroup")[1].members.filter(m -> m.className == "game.FieldSprite")[col + row * 15];
it looks complicated but it allows you to address individual sprites in flixel’s object hierarchy. and if you use custom classes for your sprites to can easily select the right one even after someone added, removed or rearranged parts of your game’s UI.
alternatively the above could have been written as ByPath(["1", '${col + row * 15}']) which is much shorter, but only works as long as your add calls always stay in the same order, because it would translate to var flxObj = FlxG.state.members[1].members[col + row * 15];.
you can probably see where this might become a problem and cause UI tests to fail for every little change you make in your app.
if you made it this far why not download hxSame and run the tests for yourself and play around with it to see what happens when tests fail, etc.
as demonstrated haxeium in it’s current state can run UI tests in HaxeFlixel projects. whether it will work for you and your project remains to be seen. there’s things that haxeium currently can’t do (e.g. all those sleep calls in my UI tests ideally shouldn’t be there) and there’s probably a few use cases that I haven’t seen or planned for, but the foundation is there and I’m pretty sure it can be expanded to make it work.
P.S. the animated gif you see at the top is not a recording of a user pressing all the things. it’s a recording of a UI test run. of course you know that already because you downloaded and ran the tests for yourself!