364 Commits

Author SHA1 Message Date
huangdarwin
05782a7e99 Compositor: Add support for mismatched timestamps.
This means we now require 2+ input frames per input, and compare the primary
stream timestamp with secondary stream timestamps in order to select the
correct output timestamp. We also must release frames and back-pressure as
soon as possible to avoid blocking upstream VFPs.

Also, improve signalling of VFP onReadyToAcceptInputFrame

PiperOrigin-RevId: 553448965
2023-08-07 11:29:44 +00:00
sheenachhabra
876b767356 Fix nit issues in muxer tests
PiperOrigin-RevId: 553145104
2023-08-07 11:25:20 +00:00
huangdarwin
4302102cf0 Test: Use TestName in media3.effects tests.
Instead of duplicating strings

PiperOrigin-RevId: 552865414
2023-08-07 11:22:02 +00:00
huangdarwin
f389a85eea Compositor: Extract interface.
This allows for custom implementations of this interface, like
a TestVideoCompositor or partner-implemented implementation

PiperOrigin-RevId: 552541631
2023-08-01 14:35:29 +01:00
claincly
c221958889 Remove setInputFrameInfo
After this change, every queued bitmap is treated as an individual input stream
(like a new MediaItems).

This change merges the FrameDropTest and FrameDropPixelTest into one (while maintaining all the test cases)

- This is accomplished by generating bitmaps with timestamps on it in FrameDropTest and compare them with goldens (one may call this a pixel test, please lmk if you want this to be renamed)
- The most part of the change comes from DefaultVideoFrameProcessorVideoFrameRenderingTest. The overall working is
  - We bypass the input manager
  - The TestFrameGenerator generates frames based on timestamps. In this case, we generate frames with timestamps on it
  - The generated frame is sent to texture output and in turn saved to bitmaps
  - We then compare the generated bitmap with the goldens

PiperOrigin-RevId: 551795770
2023-08-01 14:17:59 +01:00
huangdarwin
bd6a36033e Compositor: Move shared test logic to TestRunner.
Move shared test logic to the test runner.

This does increase indirection, which isn't usually preferable in tests.
However, we will have many different tests that would use logic
like this, so this allows us to reduce repetition.

PiperOrigin-RevId: 551536438
2023-08-01 14:13:25 +01:00
samrobinson
54093a152e Integrate AudioMixer for audio export.
Adds support for compositions with multiple audio sequences.

PiperOrigin-RevId: 550880626
2023-08-01 14:03:22 +01:00
huangdarwin
35bee3299c Remove VisibleForTesting visibility in 8k workaround.
PiperOrigin-RevId: 550843914
2023-08-01 14:01:02 +01:00
Googler
465e7c1277 Rewrite tracing to be more flexible and provide more info.
For each event, the timestamp and presentation time is logged. The trace can
then be dumped to a tsv file and easily imported in a spreadsheet.

PiperOrigin-RevId: 550839156
2023-08-01 13:59:51 +01:00
huangdarwin
e5e3db954f Remove setting the interrupted flag when unneeded.
When rethrowing InterruptedException, this
Thread.currentThread().interrupted() is not needed.

PiperOrigin-RevId: 550586853
2023-08-01 13:56:35 +01:00
claincly
22d30b3d74 Fix some mal formatted TODOs
PiperOrigin-RevId: 550507208
2023-08-01 13:51:58 +01:00
huangdarwin
d7b5ab9f23 Add Thread.currentThread().interrupt() for InterruptedExceptions.
PiperOrigin-RevId: 549975878
2023-08-01 13:49:48 +01:00
huangdarwin
5858723a06 Effect: Add Compositor signalEndOfInputStream and onEnded.
signalEndOfInputStream is needed for when streams have different amounts of
frames, so that if the primary stream finishes after a secondary stream, it
can end without waiting indefinitely for the secondary stream's matching
timestamps.

onEnded mirrors this API on the output side, which will be necessary to
know when to call signalEndOfInput on downstream components (ex. on downstream)
VideoFrameProcessors

PiperOrigin-RevId: 549969933
2023-08-01 13:48:36 +01:00
sheenachhabra
c2615a679f Skip tests if device needs disable 8k workaround
PiperOrigin-RevId: 549967326
2023-08-01 13:47:26 +01:00
huangdarwin
9128293236 Test: Use correct test skipping in muxer androidTest.
PiperOrigin-RevId: 549601882
2023-08-01 13:37:31 +01:00
huangdarwin
747b31b3c5 Test: Add non-mh androidTest tests to MH_TESTS / triage.
PiperOrigin-RevId: 549005874
2023-07-20 10:25:30 +01:00
huangdarwin
0c29dacde3 Compositor: Use timestamps to release frames.
Also, implement back-pressure to avoid requesting more than all of the
compositor's texturepool textures

PiperOrigin-RevId: 548179800
2023-07-20 10:04:55 +01:00
huangdarwin
8c4aa6b75d Compositor: Add 10 frame test.
This serves as a sort of test with more frames for the compositor for now
(before more varied video system tests come later when integrating with
Transformer), showing it doesn't error out and outputs the right
amount of frames.

Due to the VFPTestRunner having a 5s timeout, and mostly due to presubmit
emulators being very slow with OpenGL, there is a sort of limitation
to how many frames this type of test can have, depending on the test target

PiperOrigin-RevId: 548159762
2023-07-20 10:03:55 +01:00
huangdarwin
c122c0ca05 Compositor: Add support for multiple bitmaps, and add testrunner
* Allow more than one input bitmap at a time.
* Allow Compositor to take in and set up an Executor. Otherwise,
Compositor resources may be created on one thread and accessed on another.
* Add a Compositor TestRunner to reuse test code more.
* Update VideoFrameProcessingTaskExecutor to use a new onError listener, so
that it's more reusable in non-DVFP contexts, like for Compositor.

PiperOrigin-RevId: 547206053
2023-07-13 15:45:30 +01:00
samrobinson
9b06da8174 Remove redundant assertion.
TransformerAndroidTestRunner#run will throw any exception that occurs,
so there is no need to assert the result exception is null.

PiperOrigin-RevId: 546923823
2023-07-13 15:35:50 +01:00
huangdarwin
7c9e6ad2ea Effect: Only allow setExecutorService with @NonNull values.
PiperOrigin-RevId: 546828814
2023-07-13 15:30:07 +01:00
claincly
9c74e78f33 Revert DefaultCodec to final
The class is made non-final for a test in 12584fbb20.

PiperOrigin-RevId: 546718462
2023-07-13 15:23:07 +01:00
huangdarwin
cd0b45092e Test: Rename bitmap test method from Current to Focused GlFrameBuffer
Frame buffers are focused, not current. EGL Contexts may be current but this
is a prerequisite for focusing a framebuffer, and this alone isn't
sufficient to make a bitmap.

Clear this up as it can make some complex tests harder to understand, if the
methods needed don't match up in naming.

PiperOrigin-RevId: 546305118
2023-07-13 15:15:34 +01:00
huangdarwin
7a368b9a11 Effect: Remove GlTextureInfo accessor methods.
The extra check was a bit excessive, especially as it's called multiple times
per frame.

PiperOrigin-RevId: 545657102
2023-07-13 14:51:38 +01:00
huangdarwin
fa781782da Effect: Share ExecutorService and thread between VFP instances.
This allows us to simplify and reduce thread initialization, hops and the
synchronization cost associated with them, especially when more and more VFPs
are used.

PiperOrigin-RevId: 545622169
2023-07-05 13:58:43 +00:00
kimvde
81c83e4213 Remove HDR modes from TransformationRequest
PiperOrigin-RevId: 545352306
2023-07-05 09:16:52 +00:00
huangdarwin
9945033867 Effect: Create basic VideoCompositor.
Allow two DefaultVideoFrameProcessor instances to output to one
VideoCompositor, which then outputs a frame.

PiperOrigin-RevId: 544705821
2023-07-05 09:02:45 +00:00
tofunmi
d14122b029 Create queue of sync objects in FinalShaderProgramWrapper
PiperOrigin-RevId: 544683524
2023-07-05 09:00:20 +00:00
tofunmi
6f9731ce7e TextureInput: Advise client to delete syncObject after use
PiperOrigin-RevId: 544670927
2023-07-05 08:57:55 +00:00
huangdarwin
a875fa72a8 GL: Move fallback to glFinish to GlUtil.
This may happen, for example, on older OpenGL versions.

This may be used in the compositor too.

PiperOrigin-RevId: 544613566
2023-07-05 08:54:24 +00:00
tofunmi
a98b33b03c Replace glFinish in texture output with Gl sync token
Providing the sync token in the api allows the client to decide which waiting method they would like to use depending on the use case, allowing them to optimise if possible.

PiperOrigin-RevId: 543997311
2023-06-29 23:07:56 +00:00
tofunmi
8cecb93570 Texture Input: Add GLSyncToken to the OnInputFrameProcessedListener
PiperOrigin-RevId: 543867944
2023-06-29 23:02:20 +00:00
claincly
f98a10f3f2 Move GlShaderProgram creation away from VFP creation
PiperOrigin-RevId: 543773418
2023-06-29 23:00:11 +00:00
kimvde
b358d075a1 Replace usages of deprecated TransformationRequest setter
PiperOrigin-RevId: 543765376
2023-06-29 22:59:06 +00:00
sheenachhabra
8735382590 Move SSIM related files to test_util
PiperOrigin-RevId: 543444043
2023-06-29 22:51:20 +00:00
tonihei
2322462404 Do not trim audio samples by changing their timestamp
MP4 edit lists sometimes ask to start playback between two samples.
If this happens, we currently change the timestamp of the first
sample to zero to trim it (e.g. to display the first frame for a
slightly shorter period of time). However, we can't do this to audio
samples are they have an inherent duration and trimming them this
way is not possible.

#minor-release

PiperOrigin-RevId: 543420218
2023-06-29 22:50:04 +00:00
claincly
12584fbb20 Add test for force EOS when a decoder drops frames
This is for the timer added in abd96598d9

PiperOrigin-RevId: 542571387
2023-06-23 16:42:46 +00:00
tofunmi
09fe0d7390 Correct the DefaultFrameDroppingShaderProgram logic
copies the previous texture to a temp texture to ensure the correct frame is queued.

#minor-release

PiperOrigin-RevId: 541972349
2023-06-22 15:29:41 +00:00
claincly
4cb5f38007 Increase pixel difference
PiperOrigin-RevId: 541860433
2023-06-20 14:05:07 +01:00
huangdarwin
949e9cbd96 GL: Rename GL methods.
To more accurately describe what they do, especially as Compositor will
starts to use more contexts or threads, and it's important to know what
needs to be reset/recreated/focused before what methods.

PiperOrigin-RevId: 541010135
2023-06-19 16:29:03 +01:00
huangdarwin
daa42322d7 GL: Remove configAttributes from createFocusedPlaceholderEglSurface.
PiperOrigin-RevId: 540901189
2023-06-19 16:27:02 +01:00
claincly
0d3082e6ad Reduce the pixel differnce in TransformerSequenceEffectTest
As the test images are re-gened on the emulators.

PiperOrigin-RevId: 540851288
2023-06-19 16:23:13 +01:00
tonihei
51fb72b00d Replace deprecated NullableType from checkerframework with our own one
The existing NullableType has been deprecated 5 years ago and causes
crashes in Kotlin apps because Kotlin doesn't recognize this annotation
as a nullable type annotation.

While we can't align on a single @Nullable annotation yet, we can at
least replace this one by JSR305's @Nonnull(MAYBE) as it fulfils all
requirements, including full Kotlin compatiblity. To avoid the
cumbersome name, we can redefine it as our own @NullableType
annotation. (We can't use @Nullable to avoid name clashes with the main
@Nullable annotation from AndroidX)

Issue: google/ExoPlayer#6792
PiperOrigin-RevId: 540497469
2023-06-19 16:08:20 +01:00
claincly
5961637c0a Add test for per-MediaItem effect
The test transcodes four media items with distinct effects, keeping one frame
each; extracts the four frames in the produced video, and compares them against
the expected results.

PiperOrigin-RevId: 539697344
2023-06-14 20:27:37 +01:00
claincly
4b1ac2f172 Factor out video decoding and fix two minor issues
1. Not treating 0 as valid buffer index
2. Not handling the case the last frame is a comparison frame

PiperOrigin-RevId: 539607482
2023-06-12 14:09:48 +00:00
tofunmi
219212d54f rename videoFrameProcessorTestRunner.queueInputBitmap
This methods currently registers and queues the bitmaps Will need to have a method that just queues bitmaps in future CLs.

#minor-release

PiperOrigin-RevId: 539605628
2023-06-12 14:06:22 +00:00
huangdarwin
0c924fcb40 Test: Add no-op effect test for GL tone mapping.
To ensure no regressions for the potentially confusing pipeline of:
* HDR electrical -> SDR linear EOTF+OOTF, and
* SDR linear -> SDR electrical OETF

PiperOrigin-RevId: 538741079
2023-06-08 12:46:34 +00:00
tofunmi
ce203ccfed DefaultVideoFrameProcessorMultipleTextureOutputPixelTest fixes & cleanup
PiperOrigin-RevId: 538495675
2023-06-08 12:45:07 +00:00
huangdarwin
9dad207603 Test: Move duplicated GL tone mapping logic into helper methods.
PiperOrigin-RevId: 538491957
2023-06-08 12:43:40 +00:00
huangdarwin
adf53b4d50 Effect: Add multiple texture output test.
Confirms that multiple textures can be output, and that timestamps and pixels
are as expected.

PiperOrigin-RevId: 538459296
2023-06-08 12:37:00 +00:00