Headless testing in JS

Hello there,
I’m looking for a way to test JS code without opening a browser.

Currently I’m using a combination of munit with mocatoo for testing some JS code. It’s running on CircleCI and it seems like some kind of browser is opening in there. Unfortunately I have to abandon CircleCI and run tests on something that cannot open a browser, so everything pretty much stopped.
I thin solution for that might be using a headless browser, but I couldn’t find any information of how to set it up. I was also considering checking out some other testing frameworks (based on the latest Pool results), but I couldn’t find much info either. Now I’m not sure if there’s actually not that much info regarding this topic or my Google-Fu is getting weaker.

Is anyone here using one of the test framework with headless browser? Or maybe have any tips or suggestions about how to test it?

Thanks in advance :slight_smile:

I guess it depends how complete you need the browser mocking to be.

I’m using jsdom (npm package) for that, with a quick extern I added to haxe-enzyme.

I then call JsdomSetup.init() in my main class __init__ like here.

tink_unittest doesn’t rely on a browser.

You can run it through travix, which is a helper script that helps you setup the js runtime environment. Currently it uses the phantomjs headless browser. But there are plans to migrate to headless chrome

But you will also need to use tools like Selenium, which automate the operation of an actual browser.


This is very-simply because the operation of your script does depend on which browser [version …] is running it, and “headless mocking” cannot completely replace this factor. Two different browser-development teams will always read the same standards-document in slightly different ways … :roll_eyes:

Munit should now support node as a target. Just add -lib hxnodejs in your test hxml.

Thanks for all the ideas. I couldn’t try them today unfortunately but will carry on tomorrow.
tink_unittest does look very interesting, but I’ll have to check few specific cases.

OH! Even better if I could keep the Munit tests. I’ve added -lib hxnodejs to test.hxml but I’m getting an error :frowning:

MUnit Results
Error running 'node'
    Module /Users/slawomir/test-project/build/js_test.js requires node.js version 4.0.0 or higher
            window.document.getElementById("munit").innerHTML = "";

ReferenceError: window is not defined
    at delayStartup (/Users/slawomir/test-project/build/js_test.js:362:4)
    at new TestMain (/Users/slawomir/test-project/build/js_test.js:367:2)
    at Function.TestMain.main (/Users/slawomir/test-project/build/js_test.js:372:2)
    at /Users/slawomir/test-project/build/js_test.js:10909:10
    at Object.<anonymous> (/Users/slawomir/test-project/build/js_test.js:10910:3)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)

Which seems a bit weird because:

  • I have node v10.15.1
  • Autogenerated TestMain.hx is trying to access js.Browser.document (I think that’s the window issue)

After revisiting munit Github readme I did notice a note about -lib hxnodejs and TestMain.hx update. After updating it node. JS version error was gone and munit no longer wanted to access window element. That means I can update some libraries to be tested in node, but few other ones do require DOM elements. Definitely a progress tho :slight_smile:

That error means you need a hxnodejs from GitHub. There hasn’t been a Haxelib release in quite a while.

ReferenceError: window is not defined

That’s the kind of thing you need jsdom (or an alternative) for, to allow your nodejs tests to use [basic?] DOM API.

you can try to use puppeteer or webdriver to test in headless browsers, have used both

@3vilguy what is your source for mocatoo? From what I remember, the GIT repo you indicated didn’t work with Haxe 4.

@Kyliathy After double checking I’m actually using a mockatoo_DoclerLabs fork. Also I’m still using Haxe 3.4.7

1 Like

Yes Docler maintains an up to date version of Mockatoo

Headless testing is an excellent part of “continuous quality-control” practices which aim to detect problems caused by new commits as early as possible. The test suites should grow with each new feature and the entire suite should be run and re-run continuously to detect regression.

Nevertheless, before release “you need to put your head on.” You need to test against a broad range of actual browsers and browser versions.

I very recently encountered an obvious bug in Red Hat’s(!) employment portal in which the code expects two fields to be entered (“user-name,” “password”), but only one shows up on the Mac Safari browser. You therefore can’t proceed. You can see the error-message plainly on the JS console. You don’t see it on Chrome, but of course that’s not the point. They deployed a change but did not thoroughly test it on actual browsers first. This happens far more often than you think.

A “headless” engine is always supposed to do the right thing to see if your code does the right thing. But, when the software is deployed into the field, it may well encounter [recent(!) …] browsers that do not do “the right thing.”

You should also do “monkey testing” – as in, “a million monkeys.” Here, Selenium basically pokes buttons at random, whether they’re the “right” one or the “wrong” one, then analyzes each response to look for errors. You’d also be surprised how many web-sites screw up when you go backward in their intended sequence, then forward again. Because the “testers” (sic …) never actually did that. (Or, you “reload the page” and find that some of the internal state is now gone.)

Summary for the Impatient: If you are not interested in improving your Testing approach Please Disregard This Post.

Strongly Suggested:
Attitude When Doing Testing Should Be There Are Defects To Be Found.
The problem IS How To Find Defects Effectively ?

Practically REQUIRED
NUMBER 1 Way To Help Avoid Defects Before Testing ( Giant Process DEFECT ! )
Output of ALL compiles need to be captured before Tests. AND Reviewed By Testers!
IF There Are Warnings / Errors THEN Developers Need To Clean Up The Application!
And Just Disabling Warnings Is Indication Of A LAZY Developer and Should Not Be Allowed Unless After Careful Review ( and not allow original Developer to have a vote ! ).

The Compiler and Interpreter providers go to a lot of Time and Effort to let anyone using those tools get good feedback about Warnings (Problems that happen to your Users but you cant reproduce locally) or Errors (Problems that are so bad that Lazy Developers like me are forced to take action to fix).

I have literally lost count of the number of times some Developer (including my earlier Self) would think a Warning is “HARMLESS” only to find after more Debugging that the Warning directly indicated the starting point (Root Cause) of a serious DEFECT.
Alternate Name: Keep Those Lazy Developers Honest !

Important Safety Tip: Even Testing Frameworks and UI / GUI Unit Tests may have built in Blind Spots or Weaknesses. If you use the Palm Gremlins point of view you may help yourself to avoid deploying Defects.

Palm had a great built in approach called Gremlins. Automatic pseudo random Keyboard AND Gesture AND GUI Buttons, Sliders, etc activation. Anything in the UI that could change or any UI control was tested. There was a pretty strong guidance that Palm would check up on Your App by running Gremlins at Palm before you would be approved to publish App. And the Gremlins ran As Fast As Possible (way faster than Monkeys) and would find subtle timing window Defects that no other testing approach could find. Gremlins could use a random start value or you could give it a value if you needed to reproduce a defect that would only happen after 900,000 Gremlin entries. Our little startup would let Gremlins run overnight and If count got to 1M then Maybe would trust Our Own Code. Typically would start a few more runs with different start numbers to get that “Warm and Fuzzy”.

Another thing to look for in Testing Frameworks is excellent support for Debugging.
Best approach might be:
Just always run a Debug Build of your app under the Debugger for automatic tests.
Set Breakpoints where the Code is handling unusual Exceptions or unexpected Errors.
I think your Headless approach should work but I am really Ignorant of Web dev until a few months ago.
What I expect any Developer would want:
maybe a Callback into the App from the Browser when the Browser (or fake Browser) thinks it found a Warning or Error. What would be awesome for Debugging is to have a related Call Stack with parameters at the time of the Warning or Error.
Definitely a captured Log file saved that includes enough actions and other stuff before the Warning or Error to give some meaningful context to help isolate the Defect.
What would be awesome is if the Logging of the Test tool(s) could provide a Call Stack also or at least the most recently called part of your Application including all the parameters.

Deterministic way to reproduce the Defect.
If you use all the same initial settings for the Browser, Debugger and anything else will the Error always happen? If not, see below.

Sometimes it is Easier to try testing with a Release build to also help with finding a Defect that a Debug build finds. Optimization changes Timing of your App and related support Tools and all that.

After the Debug build seems to do OK then test with a Release build.
Hopefully you will not find any tricky Timing Window Defects in your App or the Libraries your App uses or other places.
IF you do AND you do not have Source code to change (3rd party Library, Framework, Tool or even OS): THEN add some code in your Application that is DEFENSIVE. The added code explicitly checks for the Warning or Error conditions Before OR Immediately After and Then does a Retry approach. I had to deal with an API that returned absolutely No Error but a requested Network setup just quietly FAILED. So I put in code that even after a so called Successful return would check the Network to be sure it actually worked. Very subtle as would only happen on certain deployments of the OS with certain background processes running. And in this case the silent No Error when there really IS An ERROR would happen originally less than 1 in 1000 tests (the DREADED Timing Window Defect class).

Suggest you try running some background processes on same platform as Browser or Headless Browser. After all Users don’t just Run your App only but rather run several processes and yours is just 1 app. You can try with both Debug and Release builds.
Again another way that changes the Timing of your App and especially the Resource Availability that your App will See. Load testing, Stress testing, low RAM tests, low Disk space tests, Network Busy / Not Available, etc.

Another Weakness of Testing:
There is a fairly large burden from using some Testing approaches.
It requires creating tests that really test Limits, Corner Cases, Bad Data input, etc.
Relying on Automated Tests may completely miss some really important Defects because no tests really hit the logical limits or assumptions of the Application. Bad Data showing various Security Defects or Math values being used incorrectly for example.

What might give you some help is what I call Testing by Contract.
The idea can be used for White Box and Black Box Tests along with practically any other type of Test.

You apply the Ideas from Design by Contract of Pre Conditions, Invariants and Post Conditions when you look at the various Application interfaces, UI, public API of the Application: settings, preferences, Haxe code interfaces, etc.

Think about if the calling (Test code or script) gives questionable or bad input that only fails to honor 1 Pre Condition out of 4. What Error returns or Exceptions are reasonable to expect?

In a similar way what is reasonable if after return from the Application an Invariant is broken.

In a similar way see if any Post Conditions were not supported.

AND If you really want to cut back on Defects.
Inspection(s) of source(s) is a process that I have seen work very impressively.
In one Inspection meeting over 50 Defects were identified in 1 hour by 5 people in a source code file. Yes there is significant overhead to Inspection meetings. Another reason to schedule them as early as possible in your process. Because Agile does not change any fundamentals about the COST of Defects. Early Correction is a Huge Saving.
And the Inspections approach detects types of Defects that literally NO amount of even very dedicated and otherwise thoughtful dynamic Tests will FIND. Pair programming in Agile will help find SOME of what Inspections find but I would think not as completely. And I am guessing that trying to convince any Agile team to actually use the Inspections approach would be difficult to do.

Spoken as former Palm and Intel employee :slight_smile:
p.s. really incomplete summary for other Platforms:
Mobile platforms: Same as above (use the Emulator / Simulator, Luke!)
Desktop / Laptop: Test on both Slowest and Fastest reasonable HW / OS.
Server(s): Minimum Performance metrics are required to be met. Meet them always!
Network(s) Like server(s) with more Protocol combinations / connections to FAIL.