Library for acceptance tests

1

Dear Community,

I’m decided to test Haxe 4 in my working project. I work with agility and BDD (Behaviour Driven Development) mostly in acceptance testing level.
I haven’t found a library for Specification By Example (like behave in python, JBehave in Java, SpecFlow in C#, Cucumber,…). I found buddy library, it’s a first step better than nothing, but it isn’t enough for me for many reasons that I won’t explain here. It will perhaps be one of my later stuff.

Firstly, I’m looking for a testing library enabling to do BDD. I mean coding the behaviour test, and run it, before coding the tested code. I found hxtf, it looks very nice for BDD in unit test context, but not for BDD in acceptance test context. I explain my point of vue in following examples.

Consider I write a CLI in Haxe, available in many target languages (ex: C++, Python, Java, C#), with some input arguments and output result in a file. In an acceptance point of view, it’s interesting to write black box acceptance tests. Consider that the tests are in a separate project (the “testing project”), that tests the CLI projet (the SUT, System Under Test), so that the testing project can be compiled and run in a target language, and SUT can be in another target language. Tests will run SUT in a separate process, with different arguments, and then verify the result file content.

Another example could be a GUI application, tested by UI (ex: a web application, tested with Selenium+others).

In all cases, the “testing project” can be written for only one target, whereas SUT is really written for all target languages. This is a topic that concerns any Haxe project, when considering acceptance tests, for BDD or simply validation.

Do you know any Haxe library that helps to do this kind of work ?
Probably you know Haxe libraries that do such stuff, but haven’t externalized it in a separate library ?

2

I use selenium to test a few web projects. I use Haxe’s Java target to build a small selenium test application. Technically you simply need to put -java-lib libs/selenium-standalone.jar in your .hxml file and you should have access to all types and methods from Haxe without writing externs. When I started doing selenium tests in Haxe I had to add some externs manually, because it wouldn’t work otherwise, but it’s not a complete implementation of all externs for the entire lib.
I haven’t revisited my externs, I can probably drop most (if not all) of them. I guess I will revisit it next time things break.

Unfortunately my selenium tests (and externs) are not available online.
I am not aware of other libraries to help with black box testing (but I might not be up to date).

I guess your best course of action is to select one existing test framework (utest, munit, buddy, etc.). Create a common base class that helps with setup and result validation. And then write your tests as if they were unittests, you can even use assert or fail() calls in your validation base class to signal a failed test to the framework so it can create proper reports.
It’s what I did for my selenium tests.
I chose munit for my tests, simply because it is compatible with mcover which allows collecting of coverage data (when running unittests or module tests of course).

3

Thanks ablum for your experience and suggests.

munit is interesting for supporting all Haxe targets. Do you know if it works with Haxe 4 ? It’s not clear on munit page.
Unfortunately, a lot of libraries in haxelib supports only some targets, it’s a problem when you want to use Haxe for almost all Haxe targets.
What I’m looking for is outside of the scope of unit testing frameworks. I’m afraid I will have to use one of them to begin with.

There’s a part I’m not sure to well understand:

If I have well understood, you write your tests with Haxe and munit, and compile & run it under Java target with external dependency to “selenium-standalone.jar”. I suppose that library “selenium-standalone.jar” was created with java without Haxe, by including selenium-java library (and others) and an empty main. What is not clear is how you can use Selenium java types of “selenium-standalone.jar” in your tests written in Haxe language.

4

Munit runs fine with Haxe 4

Most unit testing frameworks don’t really care what you test or how your tests come up with results, so using an existing unittest framework will give you testrunners and reporting for free (some even have IDE integration and/or give you CI compatible reports)

Correct, usually targeting Chrome browser, but Firefox or Edge should also work.

selenium-standalone-x.x.x.jar is what you download from Selenium’s homepage, it’s what you would normally use if you wrote Selenium tests in a plain Java project.
By using -java-lib <lib.jar> Haxe will try to read that jar file during compilation and technically all types inside should automatically be available to your Haxe code, which means you don’t have to write externs.

Externs are comparably to typescript’s .d.ts files, where you tell Haxe that there is a type (e.g. a class or interface) with certain methods or fields that will be available at runtime. Since your externs contain names, parameters, types and return types Haxe can fully type your code, and trusts you that an external library containing all those types will be present when you run your project.

When writing externs you can concentrate on things that you actually use in your project so you don’t have to write full externs, you just need the parts that you use in your projects.

I think Java, C# and Flash are the only Haxe targets that support such automatic externs.

5

Thanks ablum for all these usefull precisions.
All is clear for me now.
If I have time, I will perhaps write the library I’m looking for :wink:

6

Hi @gassho, maybe you find this library interesting: https://github.com/haxeui/haxe-bdd
It is not actively maintained, but was used in conjunction with: https://github.com/haxeui/haxeui-bdd
Hope it helps :slight_smile:

7

I would say to do BDD you should make the tests as close to end to end as possible. For sure you should have unit and integration tests, and you can write them ‘bdd’ style with for example buddy, but that doesn’t mean they actually are BDD.
For those you can use for example travix to run your tests on different targets.

But to do actual BDD, at the top of unit/integration tests you have your “user journeys”, “specification by example” or “acceptance tests” where you perform user actions and make sure the whole system works as intended for your most important user journeys.

That really has to be done in the target runtime that the user will use.
I think a lib to handle bdd for all haxe targets is probably more hassle than it is worth. Just write your cucumber tests, and then whatever script you run in your CI system can easily run the tests with the appropriate tools on an appropriate environment for various targets. So for a web application test spin up an image with cucumber-js and selenium and run the tests on that, or use a could service like browserstack. For native versions perhaps appium will work.

If you are testing cli applications it would mean having ability to read Cucumber and in your step definitions execute shell commands. That is something that could be written in haxe, but has nothing with haxe to do really. It is not something have experience with, but a quick google turned up this:

8

Hi @Leo, sounds great. Did you check the haxe-bdd project linked to above? It seems to implement Gherkin in pure Haxe. An OS-level input driver is here. It need some extension for non-Windows, but otherwise maybe this is already an all-target Haxe BDD system? Not sure, @ianharrigan ? :wink:

9

Ah, yeah, forgot about that set of libs - right, so haxe-bdd, as you say, is a haxe port of gherkin. I cant remember fully, but it supports all the “important” stuff. Scenarios, scenario outlines, backrounds, tags, substs, data tables, etc. It was ported from gherkin in java, but i have no idea what version now… an older one presumably, though, to be fair, the gherkin ast isnt that complex, so i doubt it wont do as prescribed.

The more important part, ofc, is using that ast to actually do something (ie, “cucumber”)… for haxe-bdd it was a “special” hscript that would match the gherkin steps. It all worked fine last time i played with it - which was a LONG time ago.

haxeui-bdd was an extension of all of these things to specifically functionally test haxeui apps. Again, this was all working last time i used it - however, this wont work now as it relied on a thing called “haxeui remoting”, which was a way to query details about a haxeui via sockets - its not a bad idea, but it fell behind and so i removed it, and thus haxeui-bdd with it.

That all said and done, haxe-bdd should be a pretty solid base for something a little more interesting in this avenue, and now, knowing alot more about macros in general i think it would be pretty “simple” (in a macro kinda way) to take a .feature file and generate something alot more interesting based on it (ie, haxe code rather than pattern matching at runtime).

The input simulator is, as it sounds, a way of simulating input (mouse & keyboard) on windows - one annoying thing about this method i remember though is that the window has to be “in focus” which means you cant run your features and just “let them run”, you have to sit there and watch it, which gets old fast, so not like selenium in that way (although, ofc, selenium is web only, the simulator isnt, it clicks anywhere on your desktop as if you had yourself)

So… … … the gherkin parser is:

The “runner” (which is what you would think of as cucumber is here):

Some examples of step def implementations can be found here:

And an example .feature of what the gherkin can parse and the runner can, well, run, is here:

So, long post / response. But, if anyone is willing to breathe some life into this project, at least the runner part then im more than willing to help - ill take a look at the haxeui part again separately as im not sure how best to reimplement haxeui-remoting again, probably using GitHub - ianharrigan/hxWebSockets: hxWebSockets - websockets for all haxe platforms where the UI starts a server, not sure).

I use cucumber at work all the time and its pretty great when you have a solid .feature set - it catches things unit tests never could all the time.

Hope that helps a little - sorry for the read! :smiley:

Cheers,
Ian

10

Indeed it seems nice with a haxe BDD testing lib when it is tied to a ui lib, nice work Ian!

My reply was more considering a general solution for testing various haxe projects, which would be tricky.
Testing end to end it doesn’t really matter what stack you used to create the application, in the end you have your canvas, dom, native or cli application which would be no different to test than if it was written in js, c++ or bash. So when possible the quickest way is just to use the testing tools available for that target.

For something like a canvas application it would only be possible to test in a sane way if you use a framework that offers a way to automate, doing things like checking state and interacting with UI objects. Reading pixels and specifying coordinates to click on quickly becomes unmanageable.
So for part of a framework like haxeui it makes a lot of sense, however it also means it will be very hard to make your testing scale.

In our case we get the most value from our acceptance tests from the fact that we don’t have to check all our changes manually on a bunch of browsers and devices all the time. On each push it runs through features on a bunch of selected devices and browsers on browserstack.
So to make that possible our games have the ui rendered as dom so it can be automated with selenium. However that of course means we have to put more effort to make native versions of our games if needed in the future.

A solution like haxeui and haxe-bdd with a middleware that would make it possible to run the tests using standard solutions like webdriverio and selenium-grid would be very very nice.