OK I installed munit, made it work, but now I’m stuck.
ExampleTest.hx runs fine, I’m only using the nodejs target in test.hxml.
But, when I add imports (like import utils.StringUtils;) it can’t find the class.
I did add class paths to the test.hxml, like
-cp …/…/shared/haxecode/
So this is then compiled into a haxe … command with all the parameters I added in test.hxml.
I think the error is caused by calling into -main TestMain, which sets up a TestRunner, and running all tests.
So, I wonder, what am I missing here?
I checked the relative paths and installed munit in such a way that those class paths should resolve fine.
Another weird thing I noticed was that even though a haxe … command is being run, it does not accept all parameters that it should be accepting; for example, --macro … causes problems when running the test. (so far, -lib works, -cp, -D, …/, and -js)
Any help on going forward would be greatly appreciated! (as otherwise this munit thing is pretty useless) TIA!
I don’t think I fully understand your question.
I assume you are using haxelib run munit to run your tests. I usually don’t do that, I always create my own TestMain and set everything up myself, including my own test.hxml. that way you can control what runs and what command line options are used.
Thanks Alexander for your reply.
Sorry if I’ve been unclear - using testing libraries other than haxe.unit.* has been difficult, because of lack of knowing how the test.xhml should pass on haxe parameters like -lib, -cp, -D, and --macro.
I’m testing munit and buddy - so far I’ve come really far with munit, but when compiling using
haxelib run munit test
the process reports an error when adding --macro parameters.
I also see problems when doing import .xxx (referring to packages as subdirectories) - at the ExampleTest.hx level, that is.
Could you please explain a little bit more on how you create your own TestMain and test.hxml?
BTW: I’d be willing to post the code I have written so far - maybe I’m just doing things the wrong way. Let me know if that would be helpful, and TIA for any help you can offer - we really want to leave haxe.unit.* behind and move on to some better option.
you can see a small sample here: haxe-test-adapter/samples/munit at master · vshaxe/haxe-test-adapter · GitHub (along with samples for buddy, utest and even old haxeunit (still partially supported through hx3compat lib)).
of course the sample is very compressed and doesn’t do much since it’s supposed to showcase test-adapter lib. you probably have better and more complex testcases, but it shows how to set up TestRunner, TestSuites and TestCases. whether you need multiple TestSuites is up to you. usually you have one TestCase class per tested class with testcases matching their package structure.
I would suggest you also take a look at utest simply because it’s more actively maintained compared to munit. the only reason why I would use munit over utest would be if you had testcases that need to be generated by a macro. last time I checked utest didn’t like that.
OK, I took a look at your haxe-test-adapter and it looks great.
I tried all samples, and from running them in VS Code, I’d like to go ahead with Buddy.
Why Buddy? Well, it has a BDD approach, where live documentation would be automatically created by our developers. The other testing frameworks do not support it. The only con I see for now is that the Empty Test case passes, which should return an error since there’s no minimum of one assertion in that test. But that would be a minor issue I think as it’s simply something we need to remember.
So, going back, we have a couple of tests implemented which were developed by a developer some years ago. The effort then stopped. Back then the haxe.unit.* classes were used. The process was as follows: in a directory named unittests, a unittests.hxml lives which has a set of haxe parameters: -lib, -D, -cp, --macro, -main, -dce, -js, and -cmd.
So, using -main UnitTests, this command was used to point to the main method in UnitTests.hx, which lives in the same directory.
We use the following command to run these tests (both manually and in our build and CI/CD pipelines): haxe unittests.hxml
This sets up the haxe.unit.TestRunner in UnitTests.hx, add a series of tests, then run them; output is very simple, just some text in the command prompt.
So, now I’d like to do the same in for example Buddy (most of the others do have similar structures).
If I understand well the way it all works, to avoid having to use all those haxe parameters in test.hxml, I would expect them to be able to pass them in each TestCase so that we can be specific about what is being needed as each test case runs. Would that be correct, or would it be better to just pass them all along? If so, where would I need to do that? Because we actually have test.hxml and main and test cases living on different directory levels, so the relative paths would work for only one file structure. I hope I made my point clear here… any helpful suggestions?
I do also have a question on the current test.hxml content, you use both -main Main and -x Main - what is the reason for that?
Unless your test classes need completely different command line parameters I would not have multiple different hxml test files. those would make sense e.g. when you want to run tests for different targets.
apart from that you can have as many test classes in whatever package structure you like, they can all run in one single haxe unittest.hxml call.
in our test-adapter samples there is at least one test class that is in a subfolder for each testing framework, so that’s not only possible but rather expected for any larger project. as I said a test class usually sits in the same package structure as the class being tested - so unless your project is rather flat, your test classes won’t all sit in root package (not that they couldn’t).
as for output you should take a look at different reporters that your testing framework offers, especially in regards to what your CI software supports, because you might be able to get it to generate graphs and trends of tests and failures. like identifying and automatically marking flaky tests, etc.
another topic you might want to look into is adding coverage to the mix. where you can see which lines of code your tests actually touch, helping you to improve your tests.
as to the -x Main / -main Main it’s probably some leftover. use whatever options you need to run your unittests.
Thanks for your patience and trying to help me out, I really appreciate that!
I have to say thank you because you pointed me to your haxe-test-adapter - I tried it and it looks awesome - something we’d really like to adopt and use in our project!
So, over the weekend, I saw the 6 samples and seeing them appear in the testing pane on the left working very nicely.
So I decided to give it a go on our codebase which is pretty extensive - not really flat, but it does not look very organized either.
First thing I did was adding a test.hxml at the project root, and added the following commands (it’s the folder where our current tests live):
–cwd productname/unittests
unittests.hxml
I then edited the settings.json in .vscode to have the following:
{
“haxeTestExplorer.testCommand”: [
“${haxe}”,
“test.hxml”,
“-lib”,
“test-adapter”
]
}
and then it did appear in the TEST EXPLORER pane.
The tests ran fine and I could even run a specific test.
So far really good!
So, then I decided to add another target - buddy.
This time, the test.hxml had this content:
–cwd productname/buddytests
unittests.hxml
I setup the file structure for the Main class and test cases, and voilá the tests ran fine.
One thing to note the earlier tests (ours) did still appear and I think that maybe is something to look at - a cleanup of older (and not relevant) tests should be done whenever a change occurs in test.hxml.
Then, I did a stash in git of my latest changes including the untracked code, and this morning I applied it to continue my work. To my surprise, the TEST EXPLORER did not show up anymore. I’m sure I had not changed anything - so what can be the cause of that?
I tried it again on the samples and those still work fine.
I also started again almost from scratch, but the TEST EXPLORER does not appear.
Would you have any idea why this happened and how can I get it to appear again?
TIA!
OK, after having played a bit (well, almost half a day) with the various samples I finally noticed that some of the TEST EXPLORERs did disappear also (even while they were nicely appearing yesterday!).
But, when I opened at a minmum the text.hxml and Main.hx then the TEST EXPLORER with the test suddenly appeared again.
So I tried that on our project and now it appeared again too (so far only our current haxeunit tests).
While this is good, of course, I’d expect a bit more feedback from the process that is checking whether a test is available or not, and if not, what is exactly the problem (it may be a build or a parse error, for instance). Currently nothing shows up, which is frustrating.
As a user we can then more easily fix the issue at hand, and continue.
So, for now I’m good to go ahead - I have been asked to showcase this after tomorrow’s daily scrum meeting, so hopefully it won’t break again…!
when running unittests test-adapter lib will generate a folder called .unittest in your project root where it will store results after running your tests. test explorer extension watches that folder for changes to refresh results in test explorer view.
your --cwd … most likely messes with that, or rather puts .unittest folder in a subfolder, so test explorer won’t find it, because it only looks in your workspace root. since you cleared everything it doesn’t even find old results from your haxeunit tests. so I guess that explains some of your observations.
also there should be a buddy Tests category in OUTPUT view which might give some info, also Extension Host should be stack trace free (since it shows entries for all extensions, you might have a broken one without knowing).
as for buddy based tests, they tend to generate a rather flat hierarchy. because each class starts at root and there is no support for using package names to build additional hierarchy levels like with other testing frameworks (buddy doesn’t keep package info in test results). the only way to build a hierarchy is to nest describe testcases.
OK thanks for your comments!
I know this haxe.unit thing is only temporal and I used it to demo the capabilities of your product this very morning, and pretty much the whole team was impressed from what they saw (old test run vs. the new testrun inside VS Code; I also demoed Buddy and some debugging tools that we can use to build our test cases).
I do see the Buddy problem at the Main level but we do see the TestCase3 being added through a directory / package. this last approach should be the one we’d be using.
I’d like to add MCover to this mix (that’s yours too right?) - I’m going to try this today and see how things evolve.
Important Q: What is the minimum amount of libraries and extensions (and maybe other software) we’d need to add your haxe-test-adapter, buddy, and mcover? (I ask because during my research on munit I even had to install VS2017 Pro because it uses the MsDev command)
With this information we can ask several team members to testdrive this initiative.
TIA and keep up the good work!
good news!
mcover is from the same people that wrote munit (with which it has strong ties), my only involvement were a few patches here and there. unfortunately mcover has no active maintainer.
most of my projects now use my own coverage library called instrument. so chances of me providing more patches to mcover are not zero, but close to.
as to your dependencies: I don’t know where Visual Studio comes in, and why you would need it for unittesting or coverage, but then again I don’t use Windows, so I wouldn’t know.
test-adapter lib has one dependency to json2object, which might come with additional sub-dependencies. take a look at files in buddy/haxe_libraries/ (or any other sample folder) to see which libs and what versions get pulled in for each sample.
these samples and test-adapter itself uses lix for dependency and version management (npm i lix), making sure that you get the exact same Haxe and library versions every time you build your project on your own pc or on a CI/CD server or anywhere else.
Nice comments, thanks!
We’d use VS Code to develop and run our test cases.
And to see code coverage we’d be using VS Code also, this is then helpful for Developers that are about to commit some new stuff into the CI/CD pipeline, and they can then run a command which then reports on code coverage - and will tell them if the commit would be approved - approved means the coverage percentage should not drop but should be equal or greater than the last one reported. This encourages adding testing (which should be useful at all times).
I like the addition of lix, too, it’s a very important part of running a smooth CI/CD pipeline.
Q, is your library instrument available for download through github? Id like to take a closer look. I’ll still have a closer look at that Massive mcover, to see how that integrates, and how that one would compare with yours.
Another Q: your samples are based on multi-root workspace in VS Code. How can we create multiple targets (for example, haxe.unit, munit, and buddy) in our single codebase? I’ve tried creating different testxxx.hxml in our root but the settings only define one single hxml - and if I switch the old tests remain in the TEST EXPLORER pane - something I already mentioned before, and you told me that is actually not correct. So, our Q is then, how should we handle this correctly? (we’d need to switch between targets with the TEST EXPLORER updating as we switch)
(note that the --cwd parameter was needed to change the working directory and then the relative paths in the executed xhml all work fine - for now there’s no intent from the team to change that - the idea is just to be able to see and run the tests, and if possible (and easy) to do add some basic debugging capabilities to it)
TIA!
GitHub - AlexHaxe/haxe-instrument: a profiling and coverage library for Haxe or haxelib install instrument
test-adapter supports multiroot workspaces, vshaxe does not. so working with multiroot workspaces in Haxe lacks almost all Haxe supporting features.
furthermore test-adapter supports only one set of test results for any given project, so only in the case of a multiroot workspace can you mix results from multiple different projects in test explorer view.
if you have one testXXX.hxml file per test framework then you can still run each of them manually and have their results shown in test explorer, if they contain a line with -lib test-adapter and run from project root (so no --cwd). of course in test explorer UI the button to run tests will always run whatever haxeTestExplorer.testCommand specifies.
if you absolutely must run unittests from a subfolder, you need to open that folder as a new root folder in vscode (or a second window), then you will see test results of those tests.
you might be able to get away opening your main folder and then adding the subfolder with your buddy tests to your workspace (creating a multiroot workspace), but I haven’t tested that, so you will have to experiment a bit.
Thanks for sharing this instrument library - will definitely look into that when we’re getting to that part!
For now, we’re concentrating on setting up a usable directory structure so we can actually add any target testing framework while we’re still evaluating.
I’ve looked at creating multi-root workspace before, so yeah I tried that and this seems to work pretty well so far.
The only thing that I cannot do is have specific names for the test.hxml file - it always need to be the same name: test.xhml. I tried to use a different name, but then this error appears (even though both settings.json and launch.json have that different name - in this case unittests.hxml):
Executing task in folder unittests: C:\HaxeToolkit\haxe\haxe.exe test.hxml -lib test-adapter
Error: Could not process argument test.hxml (file not found)
Invalid character:
- The terminal process “C:\HaxeToolkit\haxe\haxe.exe ‘test.hxml’, ‘-lib’, ‘test-adapter’” terminated with exit code: 1.
So not sure if I’m doing something wrong, if not, well that is not a big problem actually, as the TEST ADAPTER pane shows the tests by workspace in a tree like manner.
So right now we have added two folders in this new multi-root workspace: buddy and the current haxe.unit tests that we want to replace.
I think using multi-root workspaces is a nice feature and the best option to go forward in this case. And, as such, there is no need to use params like --cwd anymore.
it sems currently you cannot have a per folder setting for haxeTestExplorer.testCommand, it seems to be one configuration for the whole workspace. I believe it can be made to run individual commands for each folder, but I have to look into it.
so for now you need to have identical test.hxml files for each folder in your multiroot workspace, at least until next potential release.
also make sure your multiroot workspace doesn’t prevent you from using languageserver / vshaxe, because it doesn’t support multiroot yet. I guess for the first folder in a multiroot workspace it should be fine. and since your buddy tests are contained within your main project, you can access them though your main project. and you can easily use a dummy hxml containing class paths to all src folders as your selected Haxe configuration, so you can get completion, etc. for all files. but I suggest you do some research if there are any downsides to that approach and everything works for you.
OK, as the TEST EXPLORER shows the actual test names, I have no problem using just test.hxml.
So far I haven’t seen problems working with multi-root workspaces, except for some tests that popup in the TEST EXPLORER. This is because I added a third folder to the workspace, it added a Go test (of which I’m not sure where that came from), and of course the contained tests show up also.
I can do a Test hide there, and now that looks good again.
Q is, where can I find the configuration of Hiding a test, can you tell me? (that would be useful to know in case we want to unhide them)
Currently I’m implementing my first tests using TDD and so far it goes pretty well - so thanks again for all your hard work!
I don’t see an option to auto hide tests. the only way seems to be through UI. I don’t know how Go finds test cases, so maybe there is a way to make it not find them.
have fun writing tests!