Haxe cpp/hl extern generator

aka larsbounty#2.

I’ve just seen it got bumped again, so I kinda forced to report. I didn’t really want to do that because I wasn’t sure if anything will come out of this (I’m still not sure). But here it is.

Idea is to convert this

@:hlNative( 'lib' )
class A {
    function b( x:String, y:Int ):String;
}

to this (for hl):

class A {
    static inline public function b( x:String, y:Int ):String {
        return @:privateAccess String.fromUtf8( b_wrap( x.toUft8(), y ) );
    }

    @:hlNative( 'hxlib', 'b_wrap' )
    static public function b_wrap( x:hlBytes, y:Int ):hl.Bytes {
        return null;
    }
}

or this (for hxcpp):

class A {
    static public var b:(String,Int)->String = cpp.Lib.load( 'hxlib', 'b_wrap', 2 );
}

Or something like that. Also, appropriate cpp file gets generated and compiled into ndll or hdll.

What works:

  • it compiles and runs on hxcpp
  • you can pass Ints, Floats, and Strings and have them returned back
  • examples that are required in original bounty do (kind of) work (maybe)

What does not work (aka strings are tricky):

  • if you have a c api of type void concat( const char *x, const char * y, char * result ); then you’ll have to write the code that explicitly allocates new string, gets the pointer, calls this function and returns the result.
  • api that allocates memory and returns it will leak (trying to figure out how to allocate finalizers without copying everything).
  • if api requires you to pass allocated memory then you have to convert haxe.io.Bytes to memory pointer on the c side explicitly (it can be made automatic but it is as it is for now).
  • all this fiddling on hl target with to/from utf8. It is (should be) slow, but more than that it does not work. For example, on windows everything that is not ascii is (I believe) in ucs2. So if wrapped library uses unicode then it will most likely crash. And even for non-unicode appfunctions it will only work for lower part of ascii table.
  • c++ compilation is tricky. For now there is no way to add libraries, header file paths, flags etc to build.xml, because I’ve no idea how to do that (more metadata, most likely). Regarding hl I don’t even know how I should compile.
  • the whole api is ad-hoc and I’m not sure if it will be usable for something bigger than this example.
  • hl build should still crash at runtime.

Right now api looks like this: https://pastebin.com/dYk59e3b
And generated hxcpp code looks like this: https://pastebin.com/zVVN86Xm
And its output (lines do not match because I’ve cleaned some comments): kinda_works

If someone has any suggestions (especially about how to convert does not work into works) or sees what that I do does not make sense then please tell.

Also, as you may see, stringConcat2 should return “test123” but it returns true. This is because I return const char*, then hxcpp calls ToValue on result and there is no overload for ToValue( const char* ), so msvc decides that closest type to const char* is bool and here we go. I’m not fixing it because fix will leak memory anyway and if I fix memory leak this will work automatically.

Also, I think I may have broken something because even though add_nums_wrap receives ints and returns int, and it’s defined with DEFINE_PRIME, val_int is still called on both arguments and return value is passed to alloc_int. Pretty sure this is not intended because iirc performance was one promised feature of CFFIPrime over “standard” neko-inspired cffi.

FAQ:
Q: Code where?
A: I won’t share it at least before example starts to work out of the box, and I don’t have to compile libraries manually.

Q: Code when?
A: I’m really not sure. Not any time soon.

Q: Why not use @:native/@:nativeGen thing for hxcpp?
A: I would like to but 1. it does not look like it is easy to generate, and 2. I have no idea how it works.

I think linc does add stuff to build.xml

Yeah, they do it by adding :buildXml meta which does not work in my case because I need to compile a separate dll. Maybe there is a way to actually compile a dll and then let hxcpp continue building with same xml file, but I’m not aware of that. Right now I just make another build.xml and invoke “haxelib run hxcpp” on that.
I think I’ll move away from hxcpp build system because there are several problems with it (apart from me still not understanding how it works):

  1. I think it’s a bug. If you add <libpath name="."/> then dot will be stripped, and output will be just -L nothing, which will mostly screw you up depending on where it will end in the command-line (I’m pretty sure gcc does not allow for -L be an alias for -L"."). And the current directory is not in the default linker search path.
  2. There is a <lib/> node which has two attributes: “name” and “base” (there is third, but it’s not relevant in my case). “base” is a “cross-platform” way of adding libraries, because it requires just a lib name and appends all neccessary information automatically. But there is a problem: by default, msvc linker expects libs to have .lib extension, while mingw expects them to have .a. When you use <lib base=“xx”/> and specify mingw toolchain, hxcpp thinks that xx will have .a extension. The thing is mingw actually can link against .lib and there are many libraries that only have .lib’s shipped with them, so lots of the time you’ll get a linker error, because it searches for the wrong lib.
    You can use the “name” attribute, but it requires you to specify actual raw command-line (<lib name="-lxx"/> for mingw), which will result in “ignored unknown directive” for other compilers.
    Also, there is no way to know in macro what toolchain hxcpp will use to compile the project (which makes sense, because decision happens much later). So all that means is I have to setup/detect c++ compiler manually, which kinda defeats the point of using build.xml system. And I have to do that for hl build anyway so why bother.

Or maybe I should forget about compiling stuff and just spit the generated file and let the user deal with it (through hxml’s --cmd for example). This will make thing much simpler for me, and right now compilation probably would not work for anything but gcc anyway.

Maybe there should be a hxhl (just like hxcpp/hxjava) repo that contains a build toolchain.
So that hl-extensions can utilize that and let user build the extension on their machine, pretty much like linc.

I guess.
Compilation still does not work properly. Use g++ for better experience or -D generate-only and compile manually.
Can pass String, Bool, Int, Single, Float, UInt, UInt32 and haxe.io.Bytes to native functions. Bytes are autoconverted to raw memory pointer. You cannot return Bytes. UInt and Int32 should probably work on 32bit machines but I would not bet on that. If you need Int32 then mask higher bits manually. These should not be used yet. Abstracts and typedefs should work properly.
While it’s technically allowed (I didn’t check) to compile with “haxe --cpp path/to/folder/on/different/drive”, you should not do that, because it will break the macro.
There is a small doc file inside.
Things to do (in no particular order):

  1. Strings on hl. There is still all this to/fromUtf8 stuff going on
  2. Ability to return constant strings from :w_code blocks
  3. Make U/Ints(32) actually work
  4. Compilation
  5. Investigate about all this Dynamic stuff on hxcpp
  6. Maybe specific types, like Uint8, Float32 etc
  7. Maybe Int64
  8. Try to wrap something a little bit more complex (like SDL, for example) and see how it goes
  9. Pass haxe objects to C side (very low priority).