VideoFrameProcessor

Kotlin |Java

@UnstableApi
public interface VideoFrameProcessor

Known direct subclasses

DefaultVideoFrameProcessor

DefaultVideoFrameProcessor

A VideoFrameProcessor implementation that applies GlEffect instances using OpenGL on a background thread.

Interface for a video frame processor that applies changes to individual video frames.

The changes are specified by Effect instances passed to registerInputStream.

Manages its input Surface, which can be accessed via getInputSurface. The output Surface must be set by the caller using setOutputSurfaceInfo.

VideoFrameProcessor instances can be created from any thread, but instance methods for each stream must be called from the same thread.

Summary

Nested types
`public interface VideoFrameProcessor.Factory` A factory for `VideoFrameProcessor` instances.
`@Documented @Retention(value = RetentionPolicy.SOURCE) @Target(value = ElementType.TYPE_USE) @IntDef(value = ) public annotation VideoFrameProcessor.InputType` Specifies how the input frames are made available to the `VideoFrameProcessor`.
`public interface VideoFrameProcessor.Listener` Listener for asynchronous frame processing events.

Constants
`default static final long`	`DROP_OUTPUT_FRAME = -2` Indicates the frame should be dropped after `renderOutputFrame` is invoked.
`default static final int`	`INPUT_TYPE_BITMAP = 2` Input frames come from a `Bitmap`.
`default static final int`	`INPUT_TYPE_SURFACE = 1` Input frames come from a `surface`.
`default static final int`	`INPUT_TYPE_SURFACE_AUTOMATIC_FRAME_REGISTRATION = 4` Input frames come from the `input surface` and don't need to be `registered` (unlike with `INPUT_TYPE_SURFACE`).
`default static final int`	`INPUT_TYPE_TEXTURE_ID = 3` Input frames come from a `traditional GLES texture`.
`default static final ImmutableList<Effect>`	`REDRAW` A maker for passing to `registerInputStream` to signal a redraw.
`default static final long`	`RENDER_OUTPUT_FRAME_IMMEDIATELY = -1` This field is deprecated. Pass `nanoTime` to `renderOutputFrame` to render an output frame immediately.
`default static final long`	`RENDER_OUTPUT_FRAME_WITH_PRESENTATION_TIME = -3` Indicates the frame should preserve the input presentation time when `renderOutputFrame` is invoked.

Public methods
`abstract void`	`flush()` Flushes the `VideoFrameProcessor`.
`abstract Surface`	`getInputSurface()` Returns the input `Surface`, where `VideoFrameProcessor` consumes input frames from.
`abstract int`	`getPendingInputFrameCount()` Returns the number of input frames that have been made available to the `VideoFrameProcessor` but have not been processed yet.
`abstract boolean`	`queueInputBitmap( Bitmap inputBitmap, TimestampIterator timestampIterator )` Provides an input `Bitmap` to the `VideoFrameProcessor`.
`abstract boolean`	`queueInputTexture(int textureId, long presentationTimeUs)` Provides an input texture ID to the `VideoFrameProcessor`.
`abstract void`	`redraw()` Updates an `available frame` with the modified effects.
`abstract boolean`	`registerInputFrame()` Informs the `VideoFrameProcessor` that a frame will be queued to its `input surface`.
`abstract void`	`registerInputStream( @VideoFrameProcessor.InputType int inputType, Format format, List<Effect> effects, long offsetToAddUs )` Informs the `VideoFrameProcessor` that a new input stream will be queued with the list of `Effects` to apply to the new input stream.
`abstract void`	`release()` Releases all resources.
`abstract void`	`renderOutputFrame(long renderTimeNs)` Renders the oldest unrendered output frame that has become `available for rendering` at the given `renderTimeNs`.
`abstract void`	`setOnInputFrameProcessedListener( OnInputFrameProcessedListener listener )` Sets the `OnInputFrameProcessedListener`.
`abstract void`	`setOnInputSurfaceReadyListener(Runnable listener)` Sets a listener that's called when the `input surface` is ready to use.
`abstract void`	`setOutputSurfaceInfo(@Nullable SurfaceInfo outputSurfaceInfo)` Sets the output surface and supporting information.
`abstract void`	`signalEndOfInput()` Informs the `VideoFrameProcessor` that no further input frames should be accepted.

Constants

DROP_OUTPUT_FRAME

default static final long DROP_OUTPUT_FRAME = -2

Indicates the frame should be dropped after renderOutputFrame is invoked.

INPUT_TYPE_BITMAP

default static final int INPUT_TYPE_BITMAP = 2

Input frames come from a Bitmap.

INPUT_TYPE_SURFACE

default static final int INPUT_TYPE_SURFACE = 1

Input frames come from a surface.

When receiving input from a Surface, the caller must register input frames before rendering them to the input Surface.

INPUT_TYPE_SURFACE_AUTOMATIC_FRAME_REGISTRATION

default static final int INPUT_TYPE_SURFACE_AUTOMATIC_FRAME_REGISTRATION = 4

Input frames come from the input surface and don't need to be registered (unlike with INPUT_TYPE_SURFACE).

Every frame must use the input stream's registered frame format. Also sets the surface's default buffer size.

INPUT_TYPE_TEXTURE_ID

default static final int INPUT_TYPE_TEXTURE_ID = 3

Input frames come from a traditional GLES texture.

REDRAW

default static final ImmutableList<Effect> REDRAW

A maker for passing to registerInputStream to signal a redraw.

RENDER_OUTPUT_FRAME_IMMEDIATELY

default static final long RENDER_OUTPUT_FRAME_IMMEDIATELY = -1

RENDER_OUTPUT_FRAME_WITH_PRESENTATION_TIME

default static final long RENDER_OUTPUT_FRAME_WITH_PRESENTATION_TIME = -3

Indicates the frame should preserve the input presentation time when renderOutputFrame is invoked.

Public methods

flush

abstract void flush()

Flushes the VideoFrameProcessor.

All the frames that are registered prior to calling this method are no longer considered to be registered when this method returns.

Listener methods invoked prior to calling this method should be ignored.

Throws
`java.lang.UnsupportedOperationException`	If the `VideoFrameProcessor` does not accept `surface input`.

getInputSurface

abstract Surface getInputSurface()

Returns the input Surface, where VideoFrameProcessor consumes input frames from.

The frames arriving on the Surface will not be consumed by the VideoFrameProcessor until registerInputStream is called with INPUT_TYPE_SURFACE.

For streams with INPUT_TYPE_SURFACE, the returned surface is ready to use immediately and will not have a default buffer size set on it. This is suitable for configuring a android.media.MediaCodec decoder.

For streams with INPUT_TYPE_SURFACE_AUTOMATIC_FRAME_REGISTRATION, set a listener for the surface becoming ready via setOnInputSurfaceReadyListener and wait for the event before using the returned surface. This is suitable for use with non-decoder producers like media projection.

Throws
`java.lang.UnsupportedOperationException`	If the `VideoFrameProcessor` does not accept `surface input`.

getPendingInputFrameCount

abstract int getPendingInputFrameCount()

Returns the number of input frames that have been made available to the VideoFrameProcessor but have not been processed yet.

queueInputBitmap

abstract boolean queueInputBitmap(
    Bitmap inputBitmap,
    TimestampIterator timestampIterator
)

Provides an input Bitmap to the VideoFrameProcessor.

Can be called many times after registering the input stream to put multiple frames in the same input stream.

An implementation could recycle the passed in Bitmap, so it may not be suitable for reuse.

Parameters
`Bitmap inputBitmap`	The `Bitmap` queued to the `VideoFrameProcessor`.
`TimestampIterator timestampIterator`	A `TimestampIterator` generating the exact timestamps that the bitmap should be shown at.

Returns
`boolean`	Whether the `Bitmap` was successfully queued. A return value of `false` indicates the `VideoFrameProcessor` is not ready to accept input.

Throws
`java.lang.UnsupportedOperationException`	If the `VideoFrameProcessor` does not accept `bitmap input`.

queueInputTexture

abstract boolean queueInputTexture(int textureId, long presentationTimeUs)

Provides an input texture ID to the VideoFrameProcessor.

It must be only called after setOnInputFrameProcessedListener and registerInputStream have been called.

Parameters
`int textureId`	The ID of the texture queued to the `VideoFrameProcessor`.
`long presentationTimeUs`	The presentation time of the queued texture, in microseconds.

Returns
`boolean`	Whether the texture was successfully queued. A return value of `false` indicates the `VideoFrameProcessor` is not ready to accept input.

redraw

abstract void redraw()

Updates an available frame with the modified effects.

This method can be called from any thread.

registerInputFrame

abstract boolean registerInputFrame()

Informs the VideoFrameProcessor that a frame will be queued to its input surface.

Must be called before rendering a frame to the input surface. The caller must not render frames to the input surface when false is returned.

Returns
`boolean`	Whether the input frame was successfully registered. If `registerInputStream` is called, this method returns `false` until `onInputStreamRegistered` is called. Otherwise, a return value of `false` indicates the `VideoFrameProcessor` is not ready to accept input.

Throws
`java.lang.UnsupportedOperationException`	If the `VideoFrameProcessor` does not accept `surface input`.
`java.lang.IllegalStateException`	If called after `signalEndOfInput` or before `registerInputStream`.

registerInputStream

abstract void registerInputStream(
    @VideoFrameProcessor.InputType int inputType,
    Format format,
    List<Effect> effects,
    long offsetToAddUs
)

Informs the VideoFrameProcessor that a new input stream will be queued with the list of Effects to apply to the new input stream.

After registering the first input stream, this method must only be called after the last frame of the already-registered input stream has been registered, last bitmap queued or last texture id queued.

This method blocks the calling thread until the previous calls to this method finish, that is when onInputStreamRegistered is called after the underlying processing pipeline has been adapted to the registered input stream.

Parameters
`@VideoFrameProcessor.InputType int inputType`	The `InputType` of the new input stream.
`Format format`	The `Format` of the new input stream. The `colorInfo`, the `width`, the `height` and the `pixelWidthHeightRatio` must be set.
`List<Effect> effects`	The list of `effects` to apply to the new input stream.
`long offsetToAddUs`	The offset that must be added to the frame presentation timestamps, in microseconds. This offset is not part of the input timestamps. It is added to the frame timestamps before processing, and is retained in the output timestamps.

release

abstract void release()

Releases all resources.

If the VideoFrameProcessor is released before it has ended, it will attempt to cancel processing any input frames that have already become available. Input frames that become available after release are ignored.

This method blocks until all resources are released or releasing times out.

This VideoFrameProcessor instance must not be used after this method is called.

renderOutputFrame

abstract void renderOutputFrame(long renderTimeNs)

Renders the oldest unrendered output frame that has become available for rendering at the given renderTimeNs.

This will either render the output frame to the output surface, or drop the frame, per renderTimeNs.

This method must only be called if renderFramesAutomatically was set to false using the Factory and should be called exactly once for each frame that becomes available for rendering.

The renderTimeNs may be passed to eglPresentationTimeANDROID depending on the implementation.

Parameters
`long renderTimeNs`	The render time to use for the frame, in nanoseconds. The render time can be before or after the current system time. Use `DROP_OUTPUT_FRAME` to drop the frame or `RENDER_OUTPUT_FRAME_WITH_PRESENTATION_TIME` to render the frame to the `output surface` with the presentation timestamp seen in `onOutputFrameAvailableForRendering`. If the frame should be rendered immediately, pass in `nanoTime`.

setOnInputFrameProcessedListener

abstract void setOnInputFrameProcessedListener(
    OnInputFrameProcessedListener listener
)

Sets the OnInputFrameProcessedListener.

Parameters
`OnInputFrameProcessedListener listener`	The `OnInputFrameProcessedListener`.

setOnInputSurfaceReadyListener

abstract void setOnInputSurfaceReadyListener(Runnable listener)

Sets a listener that's called when the input surface is ready to use.

setOutputSurfaceInfo

abstract void setOutputSurfaceInfo(@Nullable SurfaceInfo outputSurfaceInfo)

Sets the output surface and supporting information. When output frames are rendered and not dropped, they will be rendered to this output SurfaceInfo.

The new output SurfaceInfo is applied from the next output frame rendered onwards. If the output SurfaceInfo is null, the VideoFrameProcessor will stop rendering pending frames and resume rendering once a non-null SurfaceInfo is set.

If the dimensions given in SurfaceInfo do not match the output size after applying the final effect the frames are resized before rendering to the surface and letter/pillar-boxing is applied.

The caller is responsible for tracking the lifecycle of the surface including calling this method with a new surface if it is destroyed. When this method returns, the previous output surface is no longer being used and can safely be released by the caller.

signalEndOfInput

abstract void signalEndOfInput()

Informs the VideoFrameProcessor that no further input frames should be accepted.

Throws
`java.lang.IllegalStateException`	If called more than once.

VideoFrameProcessor

Summary

Nested types

Constants

Public methods

Constants

DROP_OUTPUT_FRAME

INPUT_TYPE_BITMAP

INPUT_TYPE_SURFACE

INPUT_TYPE_SURFACE_AUTOMATIC_FRAME_REGISTRATION

INPUT_TYPE_TEXTURE_ID

REDRAW

RENDER_OUTPUT_FRAME_IMMEDIATELY

RENDER_OUTPUT_FRAME_WITH_PRESENTATION_TIME

Public methods

flush

getInputSurface

getPendingInputFrameCount

queueInputBitmap

queueInputTexture

redraw

registerInputFrame

registerInputStream

release

renderOutputFrame

setOnInputFrameProcessedListener

setOnInputSurfaceReadyListener

setOutputSurfaceInfo

signalEndOfInput