COMMUNITY

Tink_web FormFile error


(Tom Rhodes) #1

Hi,

I’m tinkering with tink_web and trying to upload a file, the response I’m getting back from the node server is…

“422 (missing field “content” at characters 5 - 92 in {“f”: ----> {“fileName”:“Fattura … n/pdf”,“size”:20430} <---- })”

…and sure enough when I go and look at what got sent, there is no field “content”. The horror!

“{fileName: “Fattura_6_MDM_Latina_Marzo.pdf”, mimeType: “application/pdf”, size: 20430, read: ƒ, saveTo: ƒ}”

Now, I think I’m doing things right in my request :slight_smile:

var formFile = FormFile.ofBlob(file.name, file.type, fileBytes);
		var client = new JsClient();
		client.request
		(
			new OutgoingRequest
			(
				new OutgoingRequestHeader
				(
					Method.POST,
					new Host("incoming.mdm"),'/upload'
				)
				,Json.stringify({f:formFile})
			)
		)
		.next(function(res) return res.body.all())
		.handle(function(o) switch(o)
		{
			case Success(body): trace(body);
			case Failure(e): trace(e);
		});

When I went poking around in FormFile and UploadedFile, I can see FormFile doing something with data.content, but I don’t understand where that is being generated from the ofBlob call. There is some magic funk in UploadedFile which fries my tiny mind and I can’t follow it :smiley:

Anyone know what happened to my content?

Cheers,

Tom.


(Juraj Kirchheim) #2

Hey Tom,

you’re not really supposed to make HTTP requests yourself with tink_web (that is not to say you can’t, but it’s sort of missing the point).

Rather, you would do something like:

var p = new tink.web.proxy.Remote<Routes>( //Routes being the type over which you route on the server ... it's best to make it an interface
  new JsClient(), 
  { host: 'localhost:2000' } //or something
);
p.theFunctionThatExpectsTheFile({ f: FormFile.ofBlob(file.name, file.type, fileBytes) });

Best,
Juraj


(Tom Rhodes) #3

Hey Juraj,

I’m a world leader in missing the point so no surprise there :wink:

Ok, I’ve got this on the nodeJS container (your term I think) for the backend…

class Server
{
	static function main()
	{
		var container = new NodeContainer(1982);
		var router = new Router<Root>(new Root());
		container.run(function(req)
						{
							return router.route(Context.ofRequest(req))
							.recover(OutgoingResponse.reportError);
						});
		trace("server started");
	}
}

class Root
{
	public function new() {}

	@:post 
  	public function upload(body: { f: FormFile } )
  	{
    	trace(body.f);
    	return body.f;
    }
}

…not trying to do anything with it yet just wanted to see it arrive ok. The first lot of code was all client side. So do you mean that I shouldn’t be using the FormFile type at all on the client? Just send some JSON with the fields that the function on the server side expects (name, mime, data)?

Thanks,

Tom.


#4

I’m not good in abstractions but should not be FormFile.ofBlob declared as returning FormFile type?
… in Json.stringify formFile is treated as ForrmFile (as declared) while being in fact UploadedFile (no content field, read and saveTo existing)

issue?


(Juraj Kirchheim) #5

The problem is not with the server side, but with the client side: you should not perform the request yourself. Instead, you should use a Remote. The mechanism is quite similar to haxe remoting, except that it doesn’t use haxe serialization, but rather reflection free encoders/decoders for standard formats like application/json, application/x-url-form-encoded or multipart/form-data (the latter two being if you wish to submit the data directly from an html form).

You could compose the request yourself if you wanted. It would have to be something like this:

Json.stringify({
  f:{ 
    mimeType: file.type, 
    fileName: file.name, 
    content: haxe.crypto.Base64.encode(fileBytes)
  }
});

But I advise against it :wink:


#6

Juraj, correct me if I’m wrong…
leaving Remote for a while, suppose we’re testing this separately … all client side

Usually (standard) uploading files we ‘emit’ multipart/form-data to be parsed/join parts on server.
FormFile shouldn’t be an abstract for form encoded data?
It has JsonFileRep but it contains binary data field, not directly JSON-able - ofBlob shouldn’t encode it for us? This way Json.stringify( f: fileForm ) should work as expected.

In another scenario Json.stringify is aware of binary data and can/could/should handle it w/o earlier encoding?

What about another scenario: uploading multiple files in one request and different formats, f.e. binary beside base64? FormFile needs format (enum) field?

Going to server side - UploadedFile represents already decoded data - where is file-data parsing realized from multipart data - we have acces to raw data? F.e. for saving very big files uploaded in chunks or any custom handling?


(Tom Rhodes) #7

Ok, I’ll look at the Remote way of doing it, in that case I’m still not passing a FormFile right? As far as I’ve understood what you’re saying FormFile is server side only?

Thanks, will play tomorrow :slight_smile:


(Juraj Kirchheim) #8

Going to server side - UploadedFile represents already decoded data - where is file-data parsing realized from multipart data - we have acces to raw data? F.e. for saving very big files uploaded in chunks or any custom handling?

So, first of all, tink_web allows posting data with different content types and will use the content-type to pick the right one. I’m surprised that the request even works without content-type header, but maybe I put in some automatic fallback that I forgot about :smiley:

You can use multipart/form-data on the server, assuming you compile with -lib tink_multipart or you’re on php/neko or some other environment that parses multipart anyway. If this is the case, you will get a FormFile which has a read and a saveTo method, to either stream the file or save it to a location of your choice. Looking at the code it seems like the parsers in tink_multipart put the files into memory, rather than into tmp files - something to be aware of when it’s tink_web handles multipart processing (as opposed to Apache/nginx). On the plus side, body parsing starts after routing and after authentication. And if you really need to put through enormous amounts of data my suggestion would be to define body:RealSource and then you can stream the body wherever you want to.

On the client side you can of course construct these multipart request in any imaginable way. You can have a plain html form. You can use whatever native APIs are around (e.g. use XHR with FormData). You can use tink_http directly as Tom did, in which case you’ll want to use new tink.Multipart() (again requires -lib tink_multipart ) to construct the request body correctly.

Alternatively, you can let tink_web handle all of the client side request construction for you (regardless of whether the server actually uses tink_web or not). It keeps all the tedious parts out of the way.

You can post files to tink_web endpoints while using JSON for your request body. Whether or not that is appropriate is for the caller to choose. The Remote (i.e. the client end of tink_web) will always prefer JSON if available - you can disable JSON for a route by setting @:accepts('multipart/form-data').

FormFile shouldn’t be an abstract for form encoded data?
It has JsonFileRep but it contains binary data field, not directly JSON-able - ofBlob shouldn’t encode it for us? This way Json.stringify( f: fileForm ) should work as expected.

For JSON, tink_web relies on tink_json which handles binary data transparently (through base64). The JsonFileRep is used to tell tink_json how to represent a FormFile: a name, a type and the binary content (which again it knows to represent as its base64 encoded counterpart). FormFile is there to create a common type to represent files that works for both multipart and json. That’s also why always base64 encoding the payload is not an option.


(Juraj Kirchheim) #9

As far as I’ve understood what you’re saying FormFile is server side only?

Not at all: FormFile can be used on both the client and server. The thing is, you want to do this instead:

var r = new Remote<Root>(new JsClient(), { host: 'localhost:1982' } );
r.upload({ f: FormFile.ofBlob(file.name, file.type, fileBytes) })
.handle(function(o) switch(o)
		{
			case Success(body): trace(body);
			case Failure(e): trace(e);
		});

Based on the metadata and type information provided, the Remote knows to JSON encode the file and to POST it to /upload. If you rename the route to /upload-file or change the method to PUT, the code will continue working. You avoid making your client fragile by manually repeating information about the server. Rather, the client and server share a common understanding of the API and tink_web does the boring plumbing. It’s not only significantly shorter, it’s also entirely type safe.


(Tom Rhodes) #10

I can’t get it going, will start another thread about the remoting though with a stripped back example.

I’ll go for the XHR and FormData now, been tinkering for a while and need to get something useful done today :wink:

tink_web looks great though, I love how the routing works it’s a thing of beauty, The @:get, @:post etc methods make the code super easy to read and having type safe remoting both ends is awesome if I can get it going…


#11

…OK… I looked at sources and tests …
it looks somewhat strange, unusual … but nice [magic] … you can define function / route as expecting one file and it’ll work even ‘being feed’ with multipart request containing a few files encoded - testMultipart()

I didn’t found test (or docs) for creating requests with binary data …
it looks like it could be possible to send it directly as json encoded object/structure (FormFile) or [more standard] as multipart (hard way, no docs)

IMHO the first method should work in exactly way as Tom tried in the first post - if not fully, at least Json should contain content field [base64 encoded binary]

Are you 1000% sure that FormFile.ofBlob works as expected? toJson result (used by Json.stringify ?) souldn’t contain size/read/saveTo as in post above?


(Tom Rhodes) #12

I still couldn’t get FormFile working, in the end I did it like this on the client as Juraj suggested using the native APIs…

                var XHR = new XMLHttpRequest();
		XHR.addEventListener("error", fileUploadRequestError);
		XHR.addEventListener("load", fileUploadRequestLoaded);
		
		var fileUploadFormData = new FormData();
		fileUploadFormData.append("name",file.name);
		fileUploadFormData.append("type",file.type);
		fileUploadFormData.append("content", Base64.encode(fileBytes));

		XHR.open("POST", "/upload", true);
		XHR.send(fileUploadFormData);

…interestingly there explicity setting the Content-type header to multipart/form-data broke it, though it does itself in this case and works. Go figure.

Server side is this…

    @:post 
  	public function upload(body: { name:String, type:String, content:String } )
  	{
    	        var fileBytes = Base64.decode(body.content);
    	        var fileName = body.name;	
    	        sys.io.File.saveBytes(fileName,fileBytes);
    	        return body.name;
    }

…whilst trying with FormFile, I got type errors about UploadedFile not being a FormFile etc. This way works but as you can see I’m not using FromFile anymore!


(Juraj Kirchheim) #13

I didn’t found test (or docs) for creating requests with binary data …

You can specify the body to be Bytes so you’ll get it all in one chunk, or as tink.io.RealSource which allows you to stream it.

it looks like it could be possible to send it directly as json encoded object/structure (FormFile) or [more standard] as multipart (hard way, no docs)

You can send it using multipart. This is primarily supported, because it’s the browser’s default way of sending files. Outside the browser, people tend to prefer just using JSON, which is why there is a corresponding representation for files. That’s all tink.web.FormFile does: tell tink_json how to encode tink.http.UploadedFile, by specifying implicit cast to and from tink.json.Representation<T>.

IMHO the first method should work in exactly way as Tom tried in the first post - if not fully, at least Json should contain content field [base64 encoded binary]

It does exactly that if you use tink_json. The standard JSON stringifier has no way of knowing how to represent anything that has no direct equivalent in JSON. As said before, tink.web.FormFile exists solely for one reason: to tell tink_web how to json encode tink.http.UploadedFile. It doesn’t seem reasonable for me to expect the standard JSON parser to deal with any of this.

The structure expected by the server side JSON parser is { f: { mimeType:String, fileName:String, content:Base64String }}. How you produce such a document is for you to choose. This is one way:

haxe.Json.stringify({
  f:{ 
    mimeType: file.type, 
    fileName: file.name, 
    content: haxe.crypto.Base64.encode(fileBytes)
  }
});

Here’s another:

tink.Json.stringify({ f: FormFile.ofBlob(file.type, file.name, fileBytes) })

Or you use the Remote and let it abstract away all the noise. Will post a full example on the other thread later today :wink:


#14

this is clue of a problem,
I’m affraid there is a bad declaration of returned type in FormFile.hx
try changing

  static inline public function ofBlob(name:String, type:String, data:Bytes):UploadedFile 

into

  static inline public function ofBlob(name:String, type:String, data:Bytes):FormFile 

#15

as expected … using multipart/form-data is a way harder to use, needs content-length header and different body structure (see DispatchTest.hx or source of any e-mail with attachement)

tink should help in creating these on client-side … but I don’t know how to create sth similiar … StructuredBody with FormFile[-s] and mixed with text part (e-mail message with encoded txt/html) ?