TL;DR: we should check if there are new frames available to queue to the
ExternalTextureProcessor before actually queueing a frame.
The overall flow on the external texture processor:
- `SurfaceTexture.onFrameAvailable` is called on `ExtTexMgr`, and
- it calls `updateTexImage()`, and sets `frame`
- it calls `maybeQueueFrameToExtTexProc()`
- the frame is queued to `ExtTexProc` if `frame` is set
- From `ExtTexProc.queueInputFrame()`:
- notifies the `frameProcessorListener` of available frame
- notifies the `inputListener` of `onReadyToAcceptInputFrame`
- (`ExtTexMgr` is the listener), it calls `maybeQueueFrameToExtTexProc()`
again
-- Parallelly --
- `ExtTexProc` calls `inputListener.onInputFrameProcessed`, when the frame is
released
- (`ExtTexMgr` is the listener), sets `frame` to `null`
*Problem*
This logic relies on `frame` to be cleared at the right time.
In transformer, it's OK b/c `ExtTexProc` release the frame immediately in
`queueInputFrame()` and calls `onInputFrameProcessed` which also reset `frame`
But in previewing, the frame is not released for a while, up to 10 ms.
In this case, `frame` will not reset in this 10 ms, and
`maybeQueueFrameToExtTexProc()` is repeatedly queueing the same input frame.
PiperOrigin-RevId: 470211620
Use the PQ OETF and EOTF to ensure that intermediate fragment shader operations
using PQ are in linear BT.2020 rather than PQ and HLG-1 BT.2020.
Also, swap the OETF and EOTF in shaders, as they were used incorrectly before
Manually tested by verifying transformer demo HLG and PQ videos look the same with and without this CL, including with a BitmapOverlayProcessor enabled to test flows both with one MatrixTransformationProcessor that skips HDR TFs, and with one that doesn't.
PiperOrigin-RevId: 469736067
Adds a method to FrameProcessor.Listener to be called when an
output frame is available and a method releaseOutputFrame in
FrameProcessor allowing the caller to trigger release of the
oldest available output frame at a given timestamp. Late frames
or frames with unset release times are dropped in the
FinalMatrixTransformationProcessorWrapper.
More than one output frame can become available before they are
released if the penultimate GlTextureProcessor is capable of producing
multiple output frames. Processing continues while waiting for
releaseOutputFrame to be called. Frame release tasks are prioritized
over other tasks.
PiperOrigin-RevId: 468473072
Manually tested using transformer demo HLG videos. Before this CL, RGB values after the YUV to RGB conversion reached up to 1.025. After this CL, RGB values correctly clamp at 1.0.
PiperOrigin-RevId: 468426092
FrameProcessingTaskExecutor should be released on error.
There can be a delay until this happens, so
FrameProcessingTaskExecutor will cancel any pending tasks
and drop new tasks until it is released.
PiperOrigin-RevId: 468171820
This is needed as a pre-requisite for allowing MCVR to control
FrameProcessor frame release for previewing.
Submitting a high-priority task is conceptually different from
posting at the front of a single queue of tasks, as the high-priority
tasks are executed in FIFO order among themselves. This will ensure
that frame release tasks submitted in close succession are executed
in the order they are submitted but before any lower priority tasks.
PiperOrigin-RevId: 467675137
* Rename all Rgba instances to Rgb.
* Remove alpha value from the RGBA Matrices and apply the 4x4 matrix
only to the R, G, B channels.
* Restore the alpha from the input unchanged.
PiperOrigin-RevId: 467208888
This change adds ExternalTextureManager which implements
InputListener to only queue input frames to the
ExternalTextureProcessor when it is ready to accept an input
frame. This replaces the old retry-logic in GlEffectsFrameProcessor.
Before this change, the retrying in GlEffectFrameProcessor wasted
CPU time if input becomes available faster than the
ExternalTextureProcessor can process it.
PiperOrigin-RevId: 467177659
This change adds a new method onReadyToAcceptInputFrame to
GlTextureProcesssor.InputListener and changes maybeQueueInputFrame
to queueInputFrame, removing the boolean return value.
This avoids the re-trying in ChainingGlTextureProcessorListener
by allowing it to only feed frames from the producing to the consuming
GlTextureProcessor when there is capacity.
MediaPipeProcessor still needs re-trying when processing isn't 1:1.
PiperOrigin-RevId: 466626369
While HDR is most closely tied to the color transfer (ex.
COLOR_TRANSFER_SDR is the only one explicitly mentioning dynamic
range), technically color spaces may be associated with HDR as well,
like BT.2020 commonly being used for HDR rather than BT.709 for SDR.
Therefore, it's more specific to mention just that the transfer is HDR.
PiperOrigin-RevId: 466316960
This simplifies ChainingGlTextureProcessor as it now only connects a
consuming and a producing GlTextureProcessor rather than a previous,
current, and next GlTextureProcessor.
Also use default no-op implementations of the listeners in
SingleFrameGlTextureProcessor and MediaPipeProcessor to avoid
null-checks.
PiperOrigin-RevId: 466301642
This allows the GlEffectsFrameProcessor to later handle HLG and PQ
differently, or limited and full color range differently.
No functional change intended in this CL.
PiperOrigin-RevId: 466070764
* Add RgbaMatrix interface implementation.
* Add Builder class for easy adjustments.
* Adjust existing RgbaMatrixPixelTests to use new RgbAdjustment class.
PiperOrigin-RevId: 465545429
