The Wayback Machine - https://web.archive.org/web/20200914065510/http://github.com/microsoft/Win2D/issues/477
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request for sample of a IBasicVideoEffect that uses a custom PixelShaderEffect #477

Open
rolandsmeenk opened this issue Dec 29, 2016 · 5 comments
Labels

Comments

@rolandsmeenk
Copy link

@rolandsmeenk rolandsmeenk commented Dec 29, 2016

All samples I found either use built-in Win2D effects or perform their processing on CPU.

@shawnhar shawnhar added the suggestion label Dec 30, 2016
@shawnhar
Copy link
Member

@shawnhar shawnhar commented Dec 30, 2016

We don't have a sample that shows exactly this, but all the necessary pieces are covered in different samples. Once you have an implementation of IBasicVideoEffect with the ProcessFrame method set up to use Win2D drawing APIs, the fact that the input and output bitmaps are in fact frame of a video doesn't many any difference to which Win2D APIs you can use, so at this point you can look at any of the Win2D drawing samples (not just video ones) to learn how to use these Win2D APIs.

Specifically for this I would recommend looking at these parts of Example Gallery:

  • The DisplacementEffect, DynamicBlurVideoEffect, or ExampleVideoEffect classes to learn how to implement an IBasicVideoEffect that will apply a Win2D image effect to frames of a video
  • BasicVideoEffectExample to learn how to apply an IBasicVideoEffect to a video file
  • Or CameraEffectExample to learn how to apply an IBasicVideoEffect to a realtime camera feed
  • CustomEffects to learn how to implement your own image effects (using PixelShaderEffect)
@rolandsmeenk
Copy link
Author

@rolandsmeenk rolandsmeenk commented Dec 30, 2016

Ok, thanks. I already looked at those examples and I now have a custom PixelShaderEffect working with a webcam feed. But I still do not know the proper way to do asynchronous resource loading for a PixelShaderEffect inside a IBasicVideoEffect implementation.

@shawnhar
Copy link
Member

@shawnhar shawnhar commented Dec 30, 2016

You probably don't want to be waiting for anything asynchronous from inside the IBasicVideoEffect implementation. Once video processing is started, you need to process frames promptly without waiting for async I/O.

Instead, do whatever resource loading you need before starting up the video/camera/etc, so you have any needed data already in memory and ready to go.

@rolandsmeenk
Copy link
Author

@rolandsmeenk rolandsmeenk commented Jan 16, 2017

I keep banging my head against loading resources (CanvasBitmap) for a VideoEffect and cannot find a way that works. Since I only have a CaptureElement combined with a MediaCapture object there are no Win2D objects to use for resource loading. I guess I need access to an ICanvasResourceCreator thas is attached to the Direct3DDevice that is used by the MediaCapture object, but I don't know how to do that. Loading inside the VideoEffect is unwanted (and fails). Loading while using an invisible CanvasControl I run into an "Objects used together must be created from the same factory instance" error.
A little help would be appreciated.

@shawnhar
Copy link
Member

@shawnhar shawnhar commented Jan 18, 2017

When using IBasicVideoEffect, the media runtime will pass in whatever IDirect3DDevice is to be used for video processing when it calls your SetEncodingProperties method. Typically (such as in the sample IBasicVideoEffect implementations in the ExampleGallery\Effects folder) you will implement SetEncodingProperties by calling CanvasDevice.CreateFromDirect3D11Device and hanging onto the resulting Win2D device.

Once you have a CanvasDevice, you can use that for any Win2D resource creation. For something like loading a texture which is an asynchronous operation, you will have to decide what do about frames that are requested while the asynchronous I/O is still in progress. Unlike the Win2D controls, IBasicVideoEffect doesn't have any concept of it being in an initializing state that is not yet ready to render, so as soon as you return from SetEncodingProperties, it will start requesting frames.

One option would be to kick off your async I/O from inside SetEncodingProperties, then early out from any frame processing requests that may arrive before the bitmap has finished loading.

Another would be to pre-load your bitmap data from the filesystem into memory before you activate the IBasicVideoEffect at all, then point the Win2D bitmap load functions at a memory stream which you know will not actually trigger any I/O, so could block on rather than letting it run asynchronously.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.
Morty Proxy This is a proxified and sanitized view of the page, visit original site.