The Wayback Machine - https://web.archive.org/web/20170718183359/https://github.com/w3c/FileAPI/issues/73
Skip to content

Add readAsSharedArrayBuffer to FileReader #73

Open
topicus opened this Issue Mar 3, 2017 · 3 comments

Comments

Projects
None yet
2 participants

topicus commented Mar 3, 2017

To allow workers play with file contents currently you have three options (maybe more):

  • Post a message to the worker with the File (structured cloning is costly).
  • Read the file as an array buffer and transfer it to the worker context (it's no longer accesible from other contexts).
  • Read the file as an array buffer and copy it's contents to a shared array buffer (still a copy is needed).

If you need to parallelize tasks over file contents (e.g. parsing a big csv file) there is no way to do it without perform a copy which is costly operation when you deal with large files.

It would be great to have a readAsSharedArrayBuffer to read the file straight in a shared array buffer and thus avoid the copy. It would be cool to allow file reading in parallel but that's another issue.

Is there any other way to perform this with the current specs that I'm missing?

Contributor

domenic commented Mar 3, 2017

This is the wrong level at which to solve this problem. We should go around duplicating every API on the platform that produces ArrayBuffers to create a second parallel version that creates a SharedArrayBuffer. If nothing else, the cost of keeping all these APIs in sync is very high.

Instead, what you need is a feature (probably an ECMAScript-level feature) for creating a SharedArrayBuffer from a non-shared ArrayBuffer, without memory copies. (Presumably this would detach the original ArrayBuffer.)

Contributor

domenic commented Mar 3, 2017

Also,

Post a message to the worker with the File (structured cloning is costly).

[citation needed]. It should be extremely cheap, especially for Files (which behind the scenes are basically a filename).

topicus commented Mar 3, 2017

@domenic About File structured cloning I misinterpreted a benchmark, it's very cheap as you mentioned.

Regarding readAsSharedArrayBuffer maybe adding a param to readAsArrayBuffer can be an option:

fileReader.readAsArrayBuffer([blob[, shared]])

It would be backward compatible and just change the underlying class to be used.

Besides that I agree with you it would be great to allow create a SharedArrayBuffer from an ArrayBuffer. Maybe something like this https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer/transfer for SharedArrayBuffers.

SharedArrayBuffer.transfer([oldBuffer[, byteLength]]);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can't perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Morty Proxy This is a proxified and sanitized view of the page, visit original site.