Support Haxe on NPM

13

It might make sense to make 3.4.4 the default, now.

So based on our discussion so far, two things come up in regards to improving compatibility with NPM:

1.) Make it possible to use “neko” from an NPM environment
2.) Make it easier to access haxelibs from NPM

There are @types/<lib name> packages on NPM, that represent typescript types for libraries. What if the Haxelib server automatically published Haxe libraries under something such as @haxelib/<lib name>, using the same version tag? It would have to create a package.json for each library, and (ideally) haxelib or at least -lib would be able to find these libraries (though -cp node_modules/<lib name>/ works in a pinch)

EDIT: Please no one use this name prefix until we’ve discussed it more, since NPM does not allow you to re-release the same version tag, it’s important that we have things right and settled before we start using up repo names and tags

14

@damoebius Haxe npm surely needs to download neko as well, and possibly it should also have a cache mechanism to not re-download the same binaries all the time.

@singmajesty I think David has looked into publishing to npm; in fact it could very nicely line up with @back2dos’s haxeshim to transform the -lib automagically.

15

haxeshim and switchx are available through npm (only).

The haxe and neko binaries are reused between projects and it works on Mac, x64 Linux and x86 Windows - I don’t know if 32 bit is still a thing, but since Windows will run the binaries just fine, it seemed like the more defensive choice … if there’s a need to improve on this, feel free to open an issue.

That said, npm is just used as a distribution channel, because it’s practically ubiquitous. Beyond that, there’s literally no integration.

Using @haxelib/<lib name> in the manner described will be hard if not impossible. It’s certainly not an issue of just adding one class path. You want the extraParams.hxml to work correctly and you also need to deal with the fact that npm will install dependencies in a nested manner, so you will have to deal with that part too (because we don’t use require to load dependencies). Better yet, if you depend on @haxelib/foo and @haxelib/bar and both depend on @haxelib/boink, it’s installed twice and you have to pick one that works for both - if that’s possible at all. Simply put: the way node (and therefore npm) handles dependencies is not compatible with haxe, except in the most trivial cases.

Alternative: The registry accepts additional fields, so having a haxeDependencies field that simply contains the dependencies from the haxelib.json will work just fine. You can act on that configuration in a postinstall hook and also you won’t have to set up mirroring between haxelib and npm (although you can still add that later to increase availability).

16

I think it’s all solvable (that’s not like haxelib does a good job at handling conflicting transitive dependencies anyway) but the benefit isn’t clear and it’s generally a bit off topic.
Let’s keep it about having a solid way to install haxe+neko from npm :slight_smile:

17

Of course it is solvable somehow. Just not by relying on npm to install the dependencies. I don’t see what haxelib’s inadequacies have to do with that :wink:

It’s my understanding that this is all about reducing friction for npm users. I don’t see this target audience accepting a solution that has no simple way to use 3rd party code. It makes sense to configure these dependencies in the package.json, because that meets general expectation - going against it will cause friction.

Sure, these are two separate problems. But a solution for just one of them doesn’t really solve anything at all at the bottom line.

18

Then let’s have a discussion (here?) on how to use npm to manage haxelibs, directly or indirectly. I think it could work - what problem do you see in npm dependencies resolution? Beside having to potentially choose one of several versions.

19

Hi,
We have to talk about features, and then find a technical solution. Do you want me to organize a conf call with skype ?

20

So I see a number of possible improvements that are coming up about haxelib for an NPM-based project:

Idea 1

Using haxelib install within the NPM-based project causes haxelib to set up the dependency in “node_modules”, with a fake package.json

Idea 2

Allow haxelib to be sensitive to an additional “package.json” field, specific to haxelib. Ideally, make it simple to integrate for npm install to cause haxelib to install its dependencies as well. Additionally (for bonus points) haxelib install should write to “package.json” with a hard-coded version, to simplify recreating the same environment

Idea 3

Not exclusive from the above two ideas, but improve haxelib so we can install in an NPM environment, and have it not prompt for the haxelib path, but presume that it will be saved under “node_modules”, or “haxelib_modules”, or some other dedicated path when working in this workflow

Idea 4

As mentioned previously, find ways to improve Neko to work as an installed tool within this environment

There are probably more?

21

I think the suggestion is to allow managing haxelibs through npm.

It would mean:

  • making the haxelibs compatible with npm (e.g. generate a package.json), including haxelib dependencies resolution, so you can use npm to manage them,
  • and finding a way for the haxe compiler to resolve haxelibs inside node_modules.

And yes we also need to install neko along with haxe from npm.

22

It should be easy to download neko along with haxe npm.
In the first version of this module i had a wrapper to the haxe bin to convert -lib arguments to -cp and support node_modules. It could be a solution again and maybe for the haxelib bin too.

I think we should be able to do all that things without updating haxelib.

1 Like
23

On that note, it should be possible to detect (and use) extraParams.hxml inline when calling the Haxe build process, if present.

There are complicated cases that could be somewhat hard to support, but the simple cases would support most projects anyway, and help make this more comfortable.

24

What do you think about this change on package.json management ?

  • Remove the config section
  • Use a custom property haxeDependencies
{
	"name": "application-name",
	"version": "0.0.1",
	"private": true,
	"dependencies": {
		"express": "2.4.7",
		"jade": ">= 0.0.1"
	},
	"haxeDependencies": {
		"haxe": "3.2.1",
		"haxelib": "3.2.0",
		"neko": "2.0.0",
		"pixijs": "4.5.5",
		"perfjs": "1.1.18"
	}
}

Like that, we can manage haxe, haxelib, and neko versions, and also haxelib dependencies

25

You’d still need haxe in the main dependencies to install your tool.

And this means that it still uses haxelib to install dependencies and not using npm or storing the libs under node_modules, right? That would be a bit a missed opportunity, isn’t it?

If we were to try managing haxelibs using npm we could try hacking a npm repository proxy that would return haxelib packages maybe? GitHub - verdaccio/verdaccio: 📦🔐A lightweight private proxy registry build in Node.js

26

Right, i forgot it.
We need haxe in the npm dependencies.
My exemple means i can manage haxelib dependencies but it still possible to use npm if the lib is available. It’s because there is too much relevant libs only available on haxelib.

Is it possible to set a new npm registery from package.json or from the haxe module for exemple ?
Not sure.
I was thinking about using directly haxelib cli to download dependencies into an haxe_modules folder.

27

I’ve been using switchx, haxeshim and lix since they were released and it just works so well. It fetches the correct haxe version (and neko if missing) while all you need to start is npm, no other requirements. If you want to setup your package.json like the typescript example above all you’d need is something like this:

// package.json
{
  "scripts": {
    "postinstall": "lix download",
    "build": "haxe build.hxml"
  },
  "devDependencies": {
    "haxeshim": "^0.9.2",
    "lix.pm": "^0.11.8"
  }
}
// .haxerc (can be created through switchx)
{
  "version": "3.4.3",
  "resolveLibs": "scoped"
}

This is will install the correct haxe version (you can even pick nightlies) and all (pinned down) haxelibs with only npm install. The .haxerc file gets created via switchx and libs managed via lix. I get that the discussion here is about managing haxelibs through npm, but if you’re actually looking for a way of managing reproducible builds right now, this is the best solution.

28

Mmm, what do you think if we use it as the official haxe module ? Could it be the solution ?

29

I like this proposal, unless our haxelibs are packaged to be compatible with NPM, it makes sense to maintain a “haxe_modules” directory, and to automatically point the local haxelib to use this directory. The main missing point is to wire up support for npm install for doing the install, but that could easily be handled with a script

30

Perhaps there are features from “haxeshim” that could be used in the official “haxe” module, or merged. However, I think it is important that it does work locally (and does not replace your global “haxe” script) which may already be the current behavior, I don’t like the requirement for “lix.pm” (strange name) though if someone doesn’t want to use it.

31

It’s all local if you need it to be. The example I presented uses npm and its scripts. Haxeshim provides the haxe bin and lix.pm the lix bin. It doesn’t mess with your existing haxe install. It would if you install switchx or haxeshim globally, but that’s true for any global npm package which provides binaries.

Most of the features in haxeshim/lix were mentioned here: [stage2] Haxelib replacement · Issue #34 · HaxeFoundation/haxe-evolution · GitHub
Lix (however strange the name) is a package manager for haxe, like haxelib, but with a few much needed features to lock down builds and is probably a lot saner to recommend than using haxelib.

I just thought I’d mention these again, because it seems people like exploring a lot of options (haxelib rewrite, libs on npm, …) while not giving what’s there already a try (and it’s pretty good!). Jason did a nice writeup here: Lix: A step forward in dependency management for Haxe projects – Jason O'Neil

32

Does switchx support project-local copies of Haxe, or is it designed just to switch a global context of Haxe between versions?

This is important, because Project A might use Haxe 3.4.4, while Project B is on bleeding edge, and they may be building at the same time on the same system.

npm-haxe has worked well for me, except the Haxelib it includes is not compatible with Haxe 3.4, and I find that the versioning is a bit off-putting. I would like to be able to npm install haxe@3.4.4 and know that’s the version used, but I understand why this is so.

TypeScript has a good workflow, where you npm install --save-dev typescript and forget it. I think npm install --save-dev haxe has potential to be that easy… if it gets updated with a newer Haxelib and a newer default Haxe version.

I like "haxeDependencies", but I agree with Philippe that distributing through NPM would make sense. That’s one reason I suggested the idea of @haxe/<lib> like TypeScript’s @type/ namespace, but we would need to get it sorted out right before starting, since NPM won’t allow a re-release of the same version number.