Hxcpp: Ways to retain objects outside of Haxe?

Let’s say I want to interop with some external C/C++ libraries which has their own memory management and they expose an allocation function:

// allocates memory of the specified size
// and returns the pointer to the memory
void* externalAlloc(size_t size)

then later on, just before the library wants to free the memory, it will invoke a callback with the pointer of the allocated memory, so we can do our cleanup:

void onFinalize(void* data)

so here are some example code in plain C for such external library (modified from lua embedding examples):

int main() {
  // allocate memory
  DIR **d = (DIR **) externalAlloc(sizeof(DIR *));

  // set the value into the memory
  *d = opendir("some/path");

  // ...(omitted) also somehow set up the finalize callback ...
}

void* onFinalize(void* data) {
  // the library gives us back the data pointer
  // we cast it back to proper types and do cleanup
  DIR **d = (DIR **)data;
  closedir(*d);
}

And my question is how to do the equivalent in Haxe, I tried the following (long code, may need to scroll):

@:native('externalAlloc')
// Q: not quite sure if the return type should be
// `cpp.Star` or `cpp.RawPointer` or `cpp.Pointer`
extern function externalAlloc<T>(size:cpp.SizeT):cpp.Pointer<T>;

static function main() {
  allocate();
}

static function allocate() {
  // prepare a haxe value
  var foo = Foo();

  // allocate memory externally
  var ptr:cpp.Pointer<Foo> = externalAlloc();

  // store the haxe value in the externally allocated memory
  // Q:  not sure if this is the correct way
  cpp.Native.set((cast ptr:cpp.Star<Foo>), foo);
  
  // normally the foo value will be ready for GC
  // after this function scope because it is a local var
  // so we somehow need to let the GC know that we want to keep it
  // Q: not sure if this is the correct way
  untyped __cpp__('GCAddRoot( (hx::Object **)&{0}.mPtr )', foo);

  // alternatively, I think we can also store the value in some
  // global/static Haxe data structure, such as a Map, to retain it
  // globalMap.set(foo, true)

  // ...(omitted) also somehow set up the finalize callback ...
}

// Q: again, not sure about the type of the argument
static function onFinalize(data:cpp.RawPointer<Void>) {
  // Q: not sure if this the correct way to get the haxe value back
  final foo = cpp.Native.get((cast data:cpp.Star<Foo>));

  // reverse whatever done to retain the value:
  untyped __cpp__('GCRemoveRoot( (hx::Object **)&{0}.mPtr )', foo);

  // alternatively:
  // globalMap.remove(foo);

}
class Foo {}

Observation:
if the allocation is called multiple times like below, it will either get stuck, or seg fault.

static function main() {
  for(i in 0...100) allocate();
}

I also noticed that the following code will always show the same address for every invocation of allocate(), not sure if that makes sense.

// inside `allocate()`
untyped __cpp__('printf("&{0}.mPtr = 0x%p\\n", &{0}.mPtr)', foo);

There are virtually no documentation for the API used here (cpp.Pointer, cpp.Native, GCAddRoot, etc). I’m struggling to wrap my head around all these. Could someone shed some light on it for me please?

References:

Are you wanting to store a haxe object in a GC root allocated by the externalAlloc function? If so that should be easy to do with a little bit of C++.

void* root_object(::Dynamic obj) {
	auto alloc = externalAlloc(sizeof(::hx::Object*));
	auto root  = static_cast<::hx::Object**>(alloc);

	*root = obj.mPtr;

	::hx::GCAddRoot(root);

	return root;
}

void onFinalize(void* ptr) {
	::hx::GCRemoveRoot(static_cast<::hx::Object**>(ptr));
}

That seems to be similar to some of the sample code you posted. You could then extern it in haxe with something like this.

extern class Rooter {
  @:native('::root_object')
  function root(in:Any):cpp.Star<cpp.Void>;
}

function main() {
  final foo  = new Foo();
  final root = Rooter.root(foo);
}

The pointer returned by root can be safely passed around any native code and it will be kept up to date if the GC moves the object held within. I’m not sure how the finalise function is invoked in your example, but I’m guessing thats part of whatever library you’re using.

Hope that helps.

1 Like

@Aidan63 Thanks very much it seems to work!

Apparently the main mistake I made is storing the obj itself in the foreign pointer instead of obj.mPtr. It also seem to be able to cast the pointer back a haxe object in haxe code as follow, though I am not sure if this is correct because of the mPtr indirection:

// `ptr` is again the result of `externalAlloc`
function retrieveHaxeValue(ptr:cpp.RawPointer<Void>) {
  final foo = cpp.Native.get((cast ptr:cpp.Star<Foo>));
  trace(foo);
}

I guess my immediate problem is solved but I think it is nice to have documentation somewhere for the following:

  • relationships/differences between cpp.Pointer, cpp.RawPointer and cpp.Star (perhaps also cpp.Reference)
  • meaning of .mPtr and how to GC Add/Remove Root

I don’t think I’ve used Native.get before, but I have some memory of using Pointer.reinterpret to go from a void* to haxe object in the past (I try to keep this sort of pointer wrangling in C++ now so can’t remember the exact details).

function retrieveHaxeValue(ptr:cpp.RawPointer<Void>) {
	final foo : Foo = cpp.Pointer.fromRaw(ptr).reinterpret().value;
	trace(foo);
}

There has been some recent pointer type documentation, someone asked a similar question on Discord and I put my answer on the cookbook site. Deployment of that site doesn’t work at the moment so you have to go directly to the markdown file.

Rooting isn’t documented anywhere but is roughly as follows.

First its important to know a bit about the C++ the compiler generates, for a given haxe class

class Foo {}

The compiler generates two C++ classes.

// internal gubbins ommitted

class Foo_obj : public hx::Object {}

class Foo : public hx::ObjectPtr {
    Foo_obj* mPtr;
}

Foo_obj which extends hx::Object is whats allocated and managed by the GC. ObjectPtr subclasses can live on the stack and hold a pointer to a GC object. These objects also contain functions for casting, null checking, and various other things.

That’s why we access the mPtr field when rooting, ::Dynamic is a subclass of hx::ObjectPtr which holds a pointer to a GC hx::Object which is what we want to root. Not the container class holding that pointer.

The root functions operate on two levels of pointers (::hx::Object**) as it then allows the GC to update the hx::Object as and when it gets moved around GC memory.

I can look into getting this info into the cookbook as well.

1 Like

Awesome note on the pointer types, thanks!

One more follow up question: how to retain a haxe String object?

Example:

::String str = // ... obtain a haxe string somehow
passCharStarToNativeCode(str.utf8_str());
// do I need to retain the haxe string object so the char pointer
// remains valid? if so, how to? ::String is not ::hx::Object so
// the previous GCAddRoot code wouldn't work?

That utf8_str function will allocate GC memory and store the utf8 string in it, the root API only allows operating on ::hx::Objects, not raw GC pointers, so there is no way to ensure its retained.

The solution to this is to use the utf8_str overload which accepts a hx::IStringAlloc pointer, this interface allows you to allocate the buffer the utf8 string is placed into instead of it going into GC memory.

Hxcpp has a built in implementation of that interface called hx::strbuf which uses non GC heap memory and you can see it used in various places in hxcpp where it calls into native code.

::String str = //

hx::strbuf buffer;
passCharStarToNativeCode(str.utf8_str(&buffer));

The buffer owned by strbuf will be released when its destructor is called, so you might not want it to live on the stack depending on your use case.

1 Like

Yes in that case I need to allocate the buffer in the heap so I suppose something like this would work:

::String str = //

hx::strbuf *buffer = new hx::strbuf();
passCharStarToNativeCode(str.utf8_str(buffer));

// later
delete buffer;