Reconstituting a cpp.RawPointer of a Haxe Dynamic object passed into C++ and then later returned out of context

Some Background

I’m working on an integration of the straight-C Duktape embedded JavaScript engine into a hxcpp project. I want to use Haxe reflection and/or RTTI to do run-time binding of Haxe objects into JavaScript.

To bind a Haxe object into a JS object, first you have to create a JS object, and then populate it with property getters and functions that will call into the backing Haxe object. To correlate the JS object to the Haxe object, I need to store some way of getting back to the Haxe object. If I were in C/C++, the most obvious way to do this would be to just stick a pointer to the C++ object as a hidden property of the JS object. Duktape provides such a facility (and even a primitive “pointer” JS Value type for the purpose).

The Problem

The problem is when I pass the pointer into Duktape, and pull it out later in the getter, it crashes when I try to reference it.

I am extracting the pointer with code like this

function pushNativeObject(value:Dynamic):Void {
  var raw = cpp.RawPointer.addressOf(value);
  untyped __cpp__("duk_push_pointer(context, static_cast<void*>(raw))");
  referencedHaxe.set(value, true);
}

In this case I’ve already determined through Type that value is in fact an instance of a Haxe class.

Of course, the static_cast looks scary, but I should be able to go to and from void* without any damage.

I add the object to a referencedHaxe:ObjectMap<Dynamic, Bool>, because otherwise there may be nothing to prevent Haxe’s GC from reaping it. That said, I know in this case it’s still referenced on the stack.

I’m reconstituting the pointer with code like this

// ... code to get the property onto the DukTape stack ..
var p:cpp.RawPointer<Void> = untyped __cpp__("duk_require_pointer(context, -1)");
var raw:RawPointer<Dynamic> = untyped __cpp__("static_cast<Dynamic*>(p)");
var ptr = cpp.Pointer.fromRaw(raw);
var ref:Dynamic = ptr.ref;
ref.toString();  // This crashes.

Definitely pulling more funny stuff here. I couldn’t figure out how to cast the cpp.RawPointer<Void> to cpp.RawPointer<Dynamic> in Haxe, so I do it not-so-slyly in C++. Then I work my way back to a Dynamic from there using the given APIs.

The traces indicate that the pointer is the same numeric value at both ends.

The generated code looks reasonable to me

(massaged for brevity)

::Dynamic* raw = &(value);
duk_push_pointer(context, raw);

and

void* p = duk_to_pointer(context, index);
::Dynamic* raw = static_cast<Dynamic*>(p);
::cpp::Pointer<  ::Dynamic > ptr = ::cpp::Pointer_obj::fromRaw(raw);
::Dynamic ref = ptr->get_ref();
ref->toString();

I’ve been muddling through to get this far, but I don’t have a great concept of how Dynamic works in this case. It seems possible Dynamic is a container that contains the type metadata, and getting a pointer to it is really just referring to the Dynamic container on the stack. But, if so, I don’t know how to accomplish getting the pointer to the underlying object instance, whatever it is (or how to reconstitute it on the other side). What am I doing wrong?

Note: I could just generate a random number and use it as a Map key to get the object back on the Haxe side, since I have to retain the objects in Haxe anyway. Given the overhead of having any kind of scripting engine, it’s probably not too terrible. But it seems like I should be able to do it this more direct way.

At first glance it does look ok, so I guess the garbage collector might be moving things.

I’m not 100% sure you can use them like that but you could try adding
-D HXCPP_GC_MOVING=0 -D HXCPP_GC_GENERATIONAL=0
to your hxml.

Also did you try printing the pointers to be sure you are getting the correct value?

Thanks, Valentin! I did print the pointers, and so I know that the address of the pointer is “correct” in that I’m getting back what I put in.

I didn’t know the GC could move objects in memory. That makes using actual pointers non-viable. But, in that case, there must be a handle that I could use instead that WOULD be consistent for the life of the object. I’ll try disabling those features and see if it changes anything. If it was a GC moving issue, I would expect the results to be inconsistent. I could print the pointer before and after the test and see if the numbers change.

Digging into some of the hxcpp C++ internals, this cpp::Pointer constructor seemed interesting, but I haven’t decoded it enough to make full sense of it.

inline Pointer( const Dynamic &inRHS) { ptr = inRHS==null()?0: (T*)inRHS->__GetHandle(); }

::Dynamic inherits from hx::ObjectPtr<hx::Object>, so it is implemented as a pointer-wrapper. hx::Object is the base class for all Haxe-generated C++ classes, so that all makes sense.

The C++ code generated for RawPointer<Dynamic>.addressOf(value) is &value - I think would only be correct if operator& is overloaded on ::Dynamic. Which I can’t seem to find evidence of. But, I tried getting the raw pointer in different functions, and they all returned the same result.

Additionally, I found hx::HxObject::GetPtr(), which I was able to try to use instead, and it returns a very different number, but it didn’t “just work,” and might need a different method of reconstitution.

I did some experiments, since I’m new to Haxe and hxcpp in general:

var obj = new TestObject();
var objDynamic:Dynamic = obj;
trace(cpp.Pointer.addressOf(obj));
trace(cpp.Pointer.fromRaw(cpp.RawPointer.addressOf(obj)));
trace(cpp.Pointer.addressOf(objDynamic));
trace(cpp.Pointer.fromRaw(cpp.RawPointer.addressOf(objDynamic)));
trace(hex(untyped __cpp__("(intptr_t)&obj")));
trace(hex(untyped __cpp__("(intptr_t)objDynamic.GetPtr()")));
trace(hex(untyped __cpp__("(intptr_t)&objDynamic")));

Produces this output:

Pointer(00000024196FEB58)
Pointer(00000024196FEB58)
Pointer(00000024196FEB60)
Pointer(00000024196FEB60)
00000024196FEB58
0000013FC5DFDFCC
00000024196FEB60

The pointer value of the object is not the same as the pointer value of the Dynamic! But, it’s very close - just 2 bytes different. Since the Dynamic is a box for the object, I wouldn’t expect them to be SO close in memory. Unless there’s some allocation trick where some metadata is stored with the allocation in the two bytes before the pointer I haven’t looked into that, yet. [Edit: The pointers are 8 bytes apart, because hex, which is more plausible.]

RawPointer.addressOf seems to consistently translate into just using operator&, and thus gives the same results as it with the same input. Great!

Just getting a Pointer, and printing it gives the same results as getting a RawPointer and converting it to a Pointer and printing that, so that object → RawPointerPointer seems like it should work.

Also, I notice that the pointers are the same before and after the test, so the GC doesn’t seem to be moving anything yet.

I was able to get something to work, but only by diving into the internals of hxcpp that probably are not guaranteed to remain stable between versions. An official solution would be better!

But, here’s what I learned:

  • All Haxe classes Foo are translated into a C++ class Foo_obj that extends hx::Object, and then the C++ Foo type is typedef’d to be hx::ObjectPtr<Foo_obj>. Haxe functions taking Foo are taking the C++ Foo class, and are essentially passing pointer wrappers by value. Everyone probably already knew this, but TIL.
  • When getting Pointer.addressOf() or RawPointer.addressOf(), they get the address of the ObjectPtr box, not the object itself. This means they are pointing to the stack, and will not be stable without extreme care.
  • hx::ObjectPtr has a method GetPtr() which gets the underlying pointer (or handle?) to the referenced object. This seems like a heap pointer and does not depend on which box on the stack is wrapping it.
  • I couldn’t find a hxcpp API to get this value nor could I find one to put it back inside a Dynamic, so I shamefully dropped into untyped C++ for this.

With these converters, my test works. I don’t know if I will be subject to heap compaction, so this may still have such problems.

static function fromVoidPointer(p:cpp.RawPointer<Void>):Dynamic {
    return untyped __cpp__("Dynamic(static_cast<hx::Object*>(p))");
}

static function toVoidPointer(o:Dynamic):cpp.RawPointer<Void> {
    return untyped __cpp__("o.GetPtr()");
}

This is less work than a hashtable lookup, but feels fragile. Is there an officially sanctioned way to do this?

The thing to note about addressOf and fromRaw is that they seem to only be useful for externs, and should never be used on any Haxe-defined object, because they’ll always be grabbing stack addresses of pointer wrappers, not pointers (or handles) to the heap objects themselves.

Sorry about bringing you all along with me through my learning experiences. I hope that this is useful to someone else someday.

3 Likes