Some questions about the ammer library (FFI, haxelib, externs)

I’d like to understand more about the purpose and rationale of
the ammer library.

I understand that if you want to use an external library (native to the platform you’re targetting) that you need to find or write externs for it.

My understanding is that a number of libraries at the haxelib repo
are exactly that — externs to native libraries (for a given target platform). (Is it common practice to tag these with the extern tag?)

Is Haxe’s FFI mechanism built into its externs functionality? (That is, are externs how Haxe does FFI?)

Does ammer auto-generate externs?

The ammer page says, “The goal is for end-users to be able to install a C library and its corresponding ammer library, then be able to use the C library in Haxe code on any ammer-supported platform.” I don’t understand this — the C library is a native library, so it’s only on one platform (native), and so you can only use it via hxcpp or hashlink targets, correct?

1 Like

First of all, thanks for the interest in ammer!

I talked a bit about the motivation for this project here.

I admit the language surrounding externs, libraries, native libraries, and so on can be a bit confusing or ambiguous. For the purposes of ammer, a “native library” is a library distributed to end users as a .dll, .so, or .dylib file (on Windows, Linux, and Mac OS X, respectively). Such a library is typically written in C or C++, although it really doesn’t matter – in the end it just contains machine code suitable for the target architecture.

Examples of native libraries: libpng, libuv, SDL2, zlib, and many, many more.

This is a bit different from the way things can be “native” in the context of Haxe. There are many “target native” libraries in Haxe. These libraries only work on a particular target, and maybe need Haxe externs. On Java, for example, libraries are distributed as .jar files. In the JavaScript ecosystem, npm hosts packages, many of which have Haxe externs.

Examples: JUnit (Java), socket.io + hxsocketio (JavaScript).

If you didn’t consider FFI mechanisms, then it would seem that native libraries are a subset of target native libraries, since something like libpng could only work on the hxcpp target.

But, as it turns out, all of Haxe system targets have some FFI mechanisms, allowing them to use native libraries if things are set up correctly. It is important to note that FFI allows using native libraries without recompiling the runtime itself (without recompiling the hl executable, or the php interpreter, or node, etc). Unfortunately, “setting things up correctly” is very different from target to target, and none of our targets really support it out of the box, except hxcpp (which can only do this because it can link against native libraries at compile time like any C++ program can).


An example C library

As an example, consider a simple C library that has this function:

int add_numbers(int a, int b) {
  return a + b;
}

We can compile it to a native library with a command that depends on the OS, e.g. gcc -dynamiclib -o libnative.dylib native.c on OS X. So at this point, we have a .dll, .so, or .dylib, in addition to the C headers of the library (just says int add_numbers(int, int); in this case).

Without ammer

Now we want to use this library in HashLink. We set up the glue code:

#include <hl.h>
#include <native.h>
HL_PRIM int w_add_numbers(int a, int b) {
  return add_numbers(a, b);
}
DEFINE_PRIM(_I32, w_add_numbers, _I32 _I32);

And compile with something like gcc -fPIC -o glue_native.o -c glue_native.c -I /path/to/native/include && gcc -I /path/to/native/include -D LIBHL_EXPORTS -m64 -shared -o glue_native.hdll glue_native.o -lhl -L/path/to/native/lib -lnative. Then we make the HashLink externs:

@:hlNative("glue_native")
class Native {
	public static function w_add_numbers(a:Int, b:Int):Int {
		return 0; // function gets replaced after loading
	}
}

And now we can call the function on HashLink with the Haxe code Native.w_add_numbers(1, 2).

But the above process is very different for all of our platforms, because each target has different method signature requirements, specifies FFI primitives differently, needs to compile against different libraries, etc etc. I believe this should not be the responsibility of a library maintainer, because it is error-prone, and leads to a lot of code duplication.

With ammer

With ammer, we only need the externs in the form of a “library definition”:

class Native extends ammer.Library<"native"> {
  public static function hello(a:Int, b:Int):Int;
}

We specify the paths to the library and its headers when compiling:

--library ammer
-D ammer.lib.native.include=/path/to/native/include
-D ammer.lib.native.library=/path/to/native/library

And we’re done! ammer then takes care of creating the target-specific glue code and invoking the C compiler with the correct parameters to produce the FFI-containing dynamic library. Most importantly, any target that ammer supports (including future additions) should work with the library definition.


TL;DR: yes, ammer dynamically generates externs for native libraries (as in, .dll, .so, .dylib) and makes such libraries available by means of FFI on any ammer-supported target – currently HashLink, hxcpp, very soon Lua and JavaScript, then more. :slight_smile:

11 Likes

Thanks for the detailed reply, and for the link to your talk, Aurel! Much appreciated. I understand a little more now (for example, the distinction between native and “target native” (wrt Haxe)).

When HashLink first came out, and I saw that you could access native C libraries with it, I figured that you could now access native C libraries in two ways:

  • find/write externs (an externs lib) that specifically works with hxcpp, or
  • find/write externs (an externs lib) that specifically works with HL/C.

Like you said in your talk, regarding the pre-ammer situation: “you need to write slightly different versions of the same code for each target”.

(See also the referenced common native API github issue.)

I’d assumed that, with Haxe, from a user’s perspective (just using Haxe to create an app) there’s no “FFI” required per se — you just choose your target, find or write externs, and your code gets translated/transpiled to the target language which then just calls that target’s libraries like any other target-lang code would.

But, as it turns out, all of Haxe system targets have some FFI mechanisms, allowing them to use native libraries if things are set up correctly. It is important to note that FFI allows using native libraries without recompiling the runtime itself (without recompiling the hl executable, or the php interpreter, or node, etc).

Ok, so that’s happening under Haxe’s hood for each target, to make it so users can just go ahead and write those externs. But, looking at your C code example above, it looks like HashLink requires the user to create that C glue code as well. (Thanks so much for that C code, HL example above.)

At 17:37 in your talk (where you talk about binding to libuv), it looks like you have to write two parts: one is C code (the FFI — just declarations?), and the other is the Haxe extern code (using things for HL like @:hlNative). So, is that C glue code only required for HashLink, and not hxcpp?

Anyhow, it looks like ammer unifies those two ways (hxcpp and HL/C) of accessing native code, going with the simplest way to write that interface/glue/externs code. That is, ammer generates the externs/glue-code for both targets — hxcpp and HL/C.

Since, say, Lua is a Sys target, what does it mean for ammer to provide access to a native (ex. .so) lib? Does that mean, that by whatever means Lua accesses native libs, that ammer would abstract over that for me? Or are you talking about accessing the lua.so?..

As another example, I think Python uses ctypes to access native libs (though, possibly via other means as well). If ammer were to support Python, would it then use something like ctypes under its hood, such that I could use native libs (C libs, .so libs) from my Haxe code via Python? (Note though, I think Python already has bindings to SDL, so would ammer side-step/recreate these?)

Ah. I like your diagram at 20:39. Would be nice to include that somewhere in your ammer docs.

TL;DR …

Not too long! Thanks for all those details! :smiley: