I see two problems here. First, Haxe is interpreting the angle brackets as “greater than” and “less than” signs. So it thinks you wrote Array is less than Vector3 is greater than [blank]. It expects an expression after >, and when it doesn’t find one it throws an error. If this was the only problem, you could work around it by converting to string and examining the string.
But there’s also the second problem, which is that the information you’re looking for does not exist at runtime. At runtime, all arrays are Array<Dynamic>. Maybe points started out as an Array<Vector3>, maybe it was Array<Vector2>, or maybe even Array<Array<String>> (though I hope not). Type.typeof() can’t tell the difference.
I see a couple options:
Test the contents of the array. Type.typeof() can tell the difference between a Vector3 and a Vector2 no problem.
Write one function to handle Array<Vector3> and another to handle Array<Vector2>. Then you won’t have to deal with Type.typeof() at all.
Only use Vector3, but sometimes set z=0 so that it’s equivalent to Vector2.
Advanced option: use macros. Since they run at compile time, they can tell between the types of array.