Proposal to facilitate creation of new targets

I have a proposal.

I’m putting this out here, before going any other route, because… well, maybe it’s a silly proposal; maybe it cannot be done because x, y and z; maybe it has been tried before but failed because reasons…
Anyway, here it is:

Proposal: Allow Haxe targets to be implemented as plugins, in the form of executables / servers.

I propose a system whereby it should be possible to implement a new Haxe target backend, not (necessarily) in OCaml, but as an executable that can be implemented in any language capable of creating standalone executables that work on the platform where the haxe executable works (e.g., in Haxe, or in the target language itself). The haxe executable can then run the plugin executable, request any information is needs from it, send back any information the plugin needs, and finally instruct the plugin to generate the output.

Such plugins would have to follow a well-defined pattern:

E.g., on Windows, suppose our haxe.exe is located in C:\HaxeToolkit\haxe\, and we want to implement a (completely hypothetical, of course) COBOL target.

For this purpose, there could be a “targets.json” in the same folder where the haxe executable is located, and it could look something like this:

{
  "targets" : {
    "COBOL" : {
      "enabled"   : true,   // Haxe compiler will only consider this a valid target if this is set to true.
      "switch"    : "cbl",  // The command line switch (in this case "-cbl" or "--cbl") that tells the
                            // Haxe compiler to use this target.
                            // This should also then be a defined compiler directive in haxe code:
                            // 
                            //     #if cbl
                            //       // Only compile this if target is cbl
                            //       doCobolThing(1,2,3);
                            //     #end
                            //
      "generator" : "./targets/cobol/cobol.exe", // The actual target plugin executable / server
    },
    "Ada" : {
      // etc
    },
    "Fortran" : {
      // etc
    }
  }
}

The Haxe compiler, upon receiving the –cbl switch, should check whether a targets.json exists, then whether the target exists in targets.json, then check whether enabled == true, and then run ./targets/cobol/cobol.exe, as a server.

The plugin should continue running, as a server, until the Haxe compiler terminates it, or the Haxe compiler itself is terminated.

Communication between Haxe compiler and plugin executable would have to follow a strict protocol, much like the Language Server Protocol, in the form of JSON-RPC strings.

I imagine compiler and plugin versions would have to be checked whether mutually compatible (either side can decide it can or cannot handle the other side’s version, and only if they both say yes, then proceed; a bit like Brexit, only smarter, and more productive…), then the Haxe compiler would have to query what kind of target it is:

  • Are you a sys target?
  • Is your type system static or dynamic?
  • Do you support overloaded functions?
  • Do you support threading?

Etc.

Finally, when all the negotiating is done, the compiler should send the AST, instruct the plugin to generate the output, and the plugin should report a success or failure result back to the Haxe compiler.

===

Is this something that sounds interesting, feasible, doable, silly…?

:grinning::haxe::slightly_smiling_face::haxe::grinning:

Simple answer: If it was that easy, we would already have it :slight_smile:

1 Like

What would the stumbling blocks be?

I’d imagine the main issue would be having an API between generators and the rest of the compiler that’s guaranteed to remain stable. Right now, it’s fairly easy to make big internal changes because all the generators can easily be adjusted in one go, once you have to care about stability of the API and compatibility with third-party generators it becomes a fair bit trickier.

1 Like

Yes, I see that, that’s a definite downside…
The main targets should, of course, remain fully integrated in Haxe, as they are now.
My proposal most certainly doesn’t mean: “Oh let’s re-implement JavaScript as an external executable now!”

I see it more like a possible infrastructure, where people can more easily write back-ends, in their favourite language, and submit them as independent side-projects, not really bound to the main Haxe project.

The dependency should be strictly one-way: Target depends on Haxe: yes! Haxe depends on target? What target?!?

And if such a side project becomes popular (say, e.g., a Rust target, a Go target, a Dart target, etc), then it might be interesting to implement them in OCaml and make them become part of the main Haxe project.

Yes, I see.
I guess “it” (my proposal) would have to be seen as one target: All external targets would have to follow Haxe’s rules, and not vice-versa.

Out-of-date plugins (i.e., the ones not maintained and thus falling behind) would eventually be invalidated automatically, by failing the version comparison test, or probably better: by failing to answer a specific question, e.g.:
Haxe compiler asks: Hey, you are sys , but can you do threads?
Plugin: What are threads?!?
Haxe: Bye!

I foresee an issue with a proliferation of languages being used to implement these external parts. Most people probably wouldn’t implement anything in OCaml unless they absolutely had to. :wink: Which means that they wouldn’t, and soon enough we would have things wandering this way and that.

1 Like

Then how about… something that would facilitate writing new targets in Haxe?

(I know this has been discussed before; maybe it should be revisited, because we can probably all agree that we love Haxe!)

Okay, alternate proposal…

Alternate Proposal: Create a parallel project, an external generator, ONE external generator, written in pure Haxe, targeting Hashlink (or Neko, or any sys target that is trivially easy to get running on all platforms), that enables the creation of new targets in Haxe .

This way, any new target can be written in either OCaml, or Haxe.

From the developer’s perspective (the dev writing Haxe code and targeting language X), there should be NO difference whatsoever.

It would have to be a project that gets developed in 100% tandem with the OCaml Haxe compiler project; they’re simply 2 code bases that stay 100% in sync all of the time. Rigorous unit testing would have to make sure of this.

There’s no need for any “targets.json” file anymore, nor any need for any “./targets/*” folders. There would simply be:

  • The Haxe compiler executable (as we all know and very much love it), compiled from OCaml sources, as it is now; and:

  • A new haxe-ext executable (e.g. on Windows: haxe-ext.exe, on *nix: haxe-ext).

Instead of having e.g. a “genrust.ml”, “gengo.ml”, etc, it would have a “genrust.hx”, “gengo.hx”, …

The main Haxe compiler would only need to run the haxe-ext executable if and when it encounters a command line option it doesn’t automatically recognise (e.g., “haxe --rust”, “haxe --go”, etc), or when it shows its options (e.g. “haxe --help”).

If the main Haxe compiler encounters a command line switch it doesn’t know, it runs haxe-ext as a server, and interrogates it for its implemented targets, via JSON-RPC.

If the target exists in haxe-ext (and apparently not in haxe itself), then the Haxe compiler interrogates haxe-ext for any target-specific info it needs:

  • Is it a sys target?

  • Is the type system static or dynamic ?

  • Does it support threading ?

Etc.

Finally, when all the negotiating is done, the main Haxe compiler should send the AST, instruct haxe-ext to generate the output, and haxe-ext should report a success or failure result back to the main Haxe compiler.

It’s already possible to write generators in Haxe by using haxe.macro.Context.onGenerate().
Though, the performance won’t be so good as with Ocaml.

1 Like

Still better than sending all of the typed ast to another process :slight_smile: which is a big limitation for external targets.

Nice examples of creating generators using just Haxe macros:

GitHub - kevinresol/hxgenjs: Extensible JS generator for Haxe , Haxe to ES6
GitHub - darmie/wrenegade: Wrenegade is a toolkit for developing backends for embedded Wren based projects. , Haxe to Wren
GitHub - RapidFingers/Craxe: Haxe to nim transpiler , Haxe to Nim
https://github.com/nadako/hxtsdgen , Haxe to TypeScript externs

4 Likes

One more:

https://github.com/YellowAfterlife/hxpico8, Haxe to Lua / pico-8

1 Like

Time to learn me some OCaml then!
(I want top performance)