It can be confusing how to use RgbMatrix for app devs not comfortable with image
manipulation, so add a helper class that simply translates rgb values the same
constant brightness value, to update the brightness values.
PiperOrigin-RevId: 509287229
GLEffectsFrameProcessor, MatrixShaderProgram and FinalMatrixShaderProgramWrapper are currently setup to handle the input frames coming from an external input (i.e. a video decoder). Image input is loaded into Bitmap objects at the start of the pipeline, so they are not produced externally. The changes provide a way for the frame processing pipeline to handle this "internal" (i.e. non-external) input.
PiperOrigin-RevId: 508645244
Based on [this conversation thread](https://chat.google.com/room/AAAA--f88ao/76Rem_cRCK8), I've opted to update the existing FrameProcessor.create() rather than deprecate it, as it is unlikely to be in use by apps outside google3.
PiperOrigin-RevId: 506920930
Set the correct output color on the debug SurfaceViewWrapper, so that SDR contents
can have an output transfer of either GAMMA_2_2 (Gamma 2.2) or SDR (SMPTE 170M).
This fixes an issue where in-app tone-mapping would output gamma 2.2, and the
SDR value incorrectly hardcoded here would lead to an error in the OpenGL, which
does not support SMPTE 170M.
PiperOrigin-RevId: 506684602
With the current ExtTexMgr,
it can happen that
- `x` frames are registered, but haven't arrived yet
- flush
- need to drop `x` frames when they arrive on SurfaceTexture
- status is reset to 0 pending, 0 available, drop `x` when frames arrive
- register one frame
- status is set to 1 pending, 0 available, drop `x` when frames arrive
- flush
- now the number of frame to drop is reset to `pending - available = 1`
- but it should be `x+1`
This CL solves the issue by reporting (by running the afterFlushTask) flush completes only after all the pending frames before calling flush are accounted for.
PiperOrigin-RevId: 506310671
Flushing resets all the texture processors within the `FrameProcessor`. This
includes:
- At the back, the FinalMatrixTextureProcessorWrapper, and its MatrixTextureProcessor
- At the front, the ExternalTextureManager
- All the texture processors in between
- All the ChainingGlTextureProcessorListeners in between texture processors
- All the internal states in the aforementioned components
The flush process follows the order, from `GlEffectsFrameProcessor.flush()`
1. Flush the `FrameProcessingTaskExecutor`, so that after it returns, all tasks queued before calling `flush()` completes
2. Post to `FrameProcessingTaskExecutor`, to flush the `FinalMatrixTextureProcessorWrapper`
3. Flushing the `FinalMatrixTextureProcessorWrapper` will propagate flushing through, via the `ChainingGlTextureProcessorListener`
Startblock:
has LGTM from christosts
and then
add reviewer andrewlewis
PiperOrigin-RevId: 506296469
The eotf is needed so that overlay (image) colors are correctly interpreted and mixed the linear video colors.
Also replaces the 100winners.png with "homemade" image file.
Added GlEffectsFrameProcessor test to justify that the color looks correct at the end of frame processing.
PiperOrigin-RevId: 506290309
For HLG input in transformer, FinalWrapper is configured to only output HLG to
encoder. But since DebugPreview is configured to take PQ for HDR content, the
color will not look correct.
This CL allows overriding the MatrixTP output transfer function, so
that FinalWrapper can output
- HLG to encoder
- PQ to debug preview
PiperOrigin-RevId: 506022840
Implementations outside media3 should be able to throw FrameProcessingException if they come across an error during configure().
PiperOrigin-RevId: 506020149
Add checks to GL tone-mapping pixel tests, to ensure the device's decoder, API
version, and OpenGL implementation support GL tone-mapping before attempting it.
These tests should be run on mobile harness, to detect per-device failures, and
so are moved to transforemr/mh. Per b/263395272, these tests should ultimately
be in an effect/mh directory.
PiperOrigin-RevId: 505749974
Tested adding 3, 15,16 different overlays in a single OverlayTextureProcessor. The program errored out when 16 overlays were added.
PiperOrigin-RevId: 504810157
This is necessary in order to move HDR to SDR tone-mapping tests to transformer/mh,
and to move the test runner to androidx.media3.test-utils, which should not have to
include androidx.media3.effects.
PiperOrigin-RevId: 504607929
This information may be needed for applications to create their own OpenGL contexts
and textures passed into or interacting with Effects.
PiperOrigin-RevId: 504342258
Based on experimentation, when CameraX is producing input for frame processor the buffer size is not being set to match the camera capture resolution, so the output has lower resolution than expected. Expose the default buffer size setter on `SurfaceTexture` to allow apps to process frames at full resolution for use cases like this one where the producer doesn't override the default buffer size.
PiperOrigin-RevId: 504022107
Also remove usages of TransformationRequest convenience methods
(setScale, setRotationDegrees and setResolution).
Some usages of setResolution can't be removed yet because they are used
for fallback.
PiperOrigin-RevId: 502415748
This is needed for constrained multi-asset to shift the timestamps of
the media items that are not the first in the sequence.
PiperOrigin-RevId: 502409923
Also, omit the "actual" label from output files, as this boilerplate isn't necessary
(it doesn't disambiguate between any other saved filename like "expected").
PiperOrigin-RevId: 502378188
Makes GlEffectsFrameProcessorPixelTest slightly more modular in preparation for
copying this into transformer/mh.
Refactoring change. No functional change intended.
PiperOrigin-RevId: 501902223
Frame cache compensates for the fluctuation in frame processing times.
Imagine a frame takes 10ms to process, and the interval between two frames is
33ms. The third frame took 40ms to process.
If we don't have frame cache:
- Process frame 1, ready after 10ms, starts playback, now t=0 ms
- Start processing frame 2, ready at t=10ms,
- Release frame 2 at t=33ms
- We start processing the third frame at t=33ms
- The third frame is due presentation at t=66ms
- But frame 3 is available at t=73ms, late
If we have a frame cache of say 3 frams,
- Process frame 1, ready after 10ms, starts playback, now t=0 ms
- Start processing frame 2, ready at t=10ms
- Start processing frame 3, ready at t=50ms
- Release frame 2 at t=33ms
- Start frame 4, ready at t=60ms
- Frame 3 is due presentation at t=66ms
- Frame 3 isn't late
PiperOrigin-RevId: 501869948
Move BitmapTestUtil from media3.effect to media3.test.utils.
This allows this class to be accessed from both transformer and effect tests.
Refactoring change. No functional change intended.
PiperOrigin-RevId: 501837130
The migration strategy is to deprecate `androidx.media3.session.BitmapLoader` and copy the file into common since BitmapLoader is a public interface that apps could be relying on.
PiperOrigin-RevId: 501266521
Based on experimentation it seems that buffers can occasionally (roughly 1% of test runs) be dropped when rendering off-screen from EGL on the emulator. Specifically, in this test, sometimes after rendering three buffers with distinct timestamps only the first and third buffers' timestamps are handled in the `ImageReader`'s image available callback causing the assertion checking all frames rendered to fail. This behavior seems to be independent of the nanosecond presentation time attached to the buffers (as expected for off-screen rendering).
Introducing a pause of 1 second between rendering each frame reduces the flake rate to around 1/2000. This increases the run time of some of the tests, so this change also removes the 5 second `FRAME_PROCESSING_WAIT_MS` (it seems to be unnecessary when rendering off-screen) and instead uses a latch to wait until the frame processor has handled 'end of stream'.
PiperOrigin-RevId: 499440591
This CL replaces concurrent collections and atomic primitives with a single
lock, this way the code is easier to reason about.
PiperOrigin-RevId: 496718057
FrameProcessor already support using different transfer function for input and
output color. This CL has two major changes:
- Create an eglSurface that recognizes BT.2020 PQ
- This requires a separate extension that works only after 33
- So we current throw, if input is HDR, and this extension doesn't work
- Create FrameProcessor with PQ output transfer function
PiperOrigin-RevId: 496023758
