19010 Commits

Author SHA1 Message Date
tofunmi
499dd5b84e remove redundant identity setting in the OverlayShaderProgram
PiperOrigin-RevId: 552416606
2023-08-01 14:26:07 +01:00
tofunmi
bb699e41c9 Add png extractor
Implementing a basic extractor that reads the whole png file into the trackOutput as one sample.

PiperOrigin-RevId: 551897619
2023-08-01 14:24:47 +01:00
ibaker
a700974d8c Mark x-media3-cues MIME type as text type
PiperOrigin-RevId: 551853336
2023-08-01 14:23:44 +01:00
andrewlewis
ddb0f86604 Fix parsing of H.265 sequence parameter sets
Fix short term reference picture list parsing. Before this change, `deltaPocS0`
was derived by adding one to the value of the syntax element
`delta_poc_s0_minus1`, but (maybe surprising) the specification actually says
that `DeltaPocS0[stRpsIdx][i]` should be assigned the negation
`-(delta_poc_s0_minus1[i] + 1)` on the first iteration, then that value added
to the previous value on previous iterations. See equations (7-67) to (7-70) in
the 2021-08 version of the H.265/HEVC specification.

Also read the number of long term reference pictures once rather than on every
loop iteration (subsection 7.3.2.2.1).

PiperOrigin-RevId: 551852999
2023-08-01 14:22:29 +01:00
simakova
415b17c1e1 Allow an empty video effects array
PiperOrigin-RevId: 551829203
2023-08-01 14:21:20 +01:00
christosts
444f0023a4 Define the VideoSink interface
PiperOrigin-RevId: 551825940
2023-08-01 14:20:18 +01:00
samrobinson
b15e7553e5 Add the hasSource method to the AudioMixer interface.
PiperOrigin-RevId: 551804976
2023-08-01 14:19:04 +01:00
claincly
c221958889 Remove setInputFrameInfo
After this change, every queued bitmap is treated as an individual input stream
(like a new MediaItems).

This change merges the FrameDropTest and FrameDropPixelTest into one (while maintaining all the test cases)

- This is accomplished by generating bitmaps with timestamps on it in FrameDropTest and compare them with goldens (one may call this a pixel test, please lmk if you want this to be renamed)
- The most part of the change comes from DefaultVideoFrameProcessorVideoFrameRenderingTest. The overall working is
  - We bypass the input manager
  - The TestFrameGenerator generates frames based on timestamps. In this case, we generate frames with timestamps on it
  - The generated frame is sent to texture output and in turn saved to bitmaps
  - We then compare the generated bitmap with the goldens

PiperOrigin-RevId: 551795770
2023-08-01 14:17:59 +01:00
tofunmi
ebbcec9a20 Add imageDurationMs to MediaItem
PiperOrigin-RevId: 551776350
2023-08-01 14:16:48 +01:00
samrobinson
ccdde8806e Account for endTimeUs when outputting with no sources.
When endTimeUs is set, restrict output to that point.

PiperOrigin-RevId: 551555392
2023-08-01 14:15:34 +01:00
claincly
5ba0ec9b40 Remove stale TODO
It says RTSP doesn't support this in the javadoc.

PiperOrigin-RevId: 551541942
2023-08-01 14:14:34 +01:00
huangdarwin
bd6a36033e Compositor: Move shared test logic to TestRunner.
Move shared test logic to the test runner.

This does increase indirection, which isn't usually preferable in tests.
However, we will have many different tests that would use logic
like this, so this allows us to reduce repetition.

PiperOrigin-RevId: 551536438
2023-08-01 14:13:25 +01:00
claincly
e77c318d96 Create Util method for SparseArray.contains()
And fixed usages in Transformer and Effect.

PiperOrigin-RevId: 551484860
2023-08-01 14:12:10 +01:00
bachinger
17ee5277d5 Only use result.sendError where supported by legacy media library
`MediaLibraryServiceLegacyStub` handles various edge cases by calling
`result.sendError(null)` with the intention to send back an error to
the legacy browser [1].

`MediaBrowserServiceCompat` of the legacy media1 Compat library has an
inner base class `Result` that has a default implementation of
`onErrorSent` that throws an `UnsupportedOperationException` [2].
However, most anonymous inner classes for `Result` created in
`MediaBrowserServiceCompat` do not override `onErrorSent` [3].

Hence Media3 must not call `sendError` in these cases. Instead we call
`sendResult(null)` according to what the default implementation of
the callbacks in `MediaBrowserServiceCompat` do ([4] as an example).

Issue: androidx/media#78
Issue: androidx/media#334

[1] https://github.com/androidx/media/blob/release/libraries/session/src/main/java/androidx/media3/session/MediaLibraryServiceLegacyStub.java#L200
[2] https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:media/media/src/main/java/androidx/media/MediaBrowserServiceCompat.java;l=872
[3] https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:media/media/src/main/java/androidx/media/MediaBrowserServiceCompat.java;l=578-604?q=MediaBrowserServiceCompat&ss=androidx
[4] https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:media/media/src/main/java/androidx/media/MediaBrowserServiceCompat.java;l=1395

PiperOrigin-RevId: 551210137
2023-08-01 14:11:08 +01:00
tonihei
f766936140 Fix ms to us conversion bug in DecoderVideoRenderer
The current code multiplies the value by 1000 twice,
effectively converting to nanoseconds.

#minor-release

PiperOrigin-RevId: 551129750
2023-08-01 14:09:56 +01:00
samrobinson
b7d7027424 Update Composition javadoc for expanded audio support.
PiperOrigin-RevId: 550922961
2023-08-01 14:08:46 +01:00
tonihei
782f5b2a0e Fix last sample detection issues
1. For codecs that change timestamps, we need to use greater
   or equal instead of just equal for comparison.
2. We should only make this comparison once a value has been
   set.
3. The largest queue timestamp isn't updated before using it
   for this detection, meaning that we also mark the
   second-to-last sample as the last one.

PiperOrigin-RevId: 550901978
2023-08-01 14:06:55 +01:00
andrewlewis
9a2bfc4042 Prioritize H.265 and H.264 over other formats
This should make no functional difference because `SampleExporter` always
checks for H.265 and H.264 first. However, in case we ever change that code,
these are used in priority order so it's better to order them accordingly.

PiperOrigin-RevId: 550894935
2023-08-01 14:05:42 +01:00
tonihei
79c8b0e0f8 Set correct start time for newly enabled renderers during playback
We currently pass in the time at which the stream originally started,
but newly enabled renderers should get the current playback position
instead.

PiperOrigin-RevId: 550894630
2023-08-01 14:04:35 +01:00
samrobinson
54093a152e Integrate AudioMixer for audio export.
Adds support for compositions with multiple audio sequences.

PiperOrigin-RevId: 550880626
2023-08-01 14:03:22 +01:00
samrobinson
357c458028 Ensure EOS is queued after processing generated silence with effects.
When generating silence for AudioProcessingPipeline, audio never
queued EOS downstream.

Linked to this, when silence followed an item with audio, the silence
was added to SilentAudioGenerator before the mediaItem reconfiguration
occurred. If the silence had effects, the APP would be flushed after
silence queued EOS, resetting APP.isEnded back to false, so AudioGraph
never ended.

Regression tests reproduce failure without fix, but pass with it.

PiperOrigin-RevId: 550853714
2023-08-01 14:02:09 +01:00
huangdarwin
35bee3299c Remove VisibleForTesting visibility in 8k workaround.
PiperOrigin-RevId: 550843914
2023-08-01 14:01:02 +01:00
Googler
465e7c1277 Rewrite tracing to be more flexible and provide more info.
For each event, the timestamp and presentation time is logged. The trace can
then be dumped to a tsv file and easily imported in a spreadsheet.

PiperOrigin-RevId: 550839156
2023-08-01 13:59:51 +01:00
bachinger
71af48429f Fix flaky test in MediaControllerListenerTest
PiperOrigin-RevId: 550625956
2023-08-01 13:58:49 +01:00
ibaker
449cf55523 Add comment about out-of-order timestamps in AtomParsers.parseStbl
PiperOrigin-RevId: 550596173
2023-08-01 13:57:36 +01:00
huangdarwin
e5e3db954f Remove setting the interrupted flag when unneeded.
When rethrowing InterruptedException, this
Thread.currentThread().interrupted() is not needed.

PiperOrigin-RevId: 550586853
2023-08-01 13:56:35 +01:00
huangdarwin
ef39d317d5 Compositor: Signal onEnded when frames finish compositing.
PiperOrigin-RevId: 550576032
2023-08-01 13:55:21 +01:00
bachinger
a90ec96425 Use MediaUtils.intersect in MediaControllerImplBase
PiperOrigin-RevId: 550566322
2023-08-01 13:54:11 +01:00
bachinger
2c19399d3e Move command button related helper methods to CommandButton
PiperOrigin-RevId: 550558261
2023-08-01 13:53:09 +01:00
claincly
22d30b3d74 Fix some mal formatted TODOs
PiperOrigin-RevId: 550507208
2023-08-01 13:51:58 +01:00
jbibik
f4ad26451f Deprecate MimeTypes.TEXT_EXOPLAYER_CUES for APPLICATION_MEDIA3_CUES
Instead of `text/x-exoplayer-cues`, we will use `application/x-media3-cues`.

The prefix should be `application` not `text` since the encoded form is arbitrary bytes, not necessarily text. The name should not reference `exoplayer`, since the Media3 Extractors (which are not part of `exoplayer`) produce this format.

PiperOrigin-RevId: 550181852
2023-08-01 13:50:50 +01:00
huangdarwin
d7b5ab9f23 Add Thread.currentThread().interrupt() for InterruptedExceptions.
PiperOrigin-RevId: 549975878
2023-08-01 13:49:48 +01:00
huangdarwin
5858723a06 Effect: Add Compositor signalEndOfInputStream and onEnded.
signalEndOfInputStream is needed for when streams have different amounts of
frames, so that if the primary stream finishes after a secondary stream, it
can end without waiting indefinitely for the secondary stream's matching
timestamps.

onEnded mirrors this API on the output side, which will be necessary to
know when to call signalEndOfInput on downstream components (ex. on downstream)
VideoFrameProcessors

PiperOrigin-RevId: 549969933
2023-08-01 13:48:36 +01:00
sheenachhabra
c2615a679f Skip tests if device needs disable 8k workaround
PiperOrigin-RevId: 549967326
2023-08-01 13:47:26 +01:00
huangdarwin
b1cf5fb320 Effect: Reduce Compositor exception try/catch redirection.
The VideoFrameProcessingTaskExecutor already handles wrapping
GlUtil.GlException errors into VideoFrameProcessingExceptions.
Therefore, stop wrapping this ourselves, so that errors are
attributed to the correct line number with less indirection.

PiperOrigin-RevId: 549959258
2023-08-01 13:46:28 +01:00
tofunmi
7eee15ecb4 Update OverlaySettings to explicitly state overlay transformations
Alters the OverlayShaderProgram implementation to support rotations, however we need to apply the transformations separately in order for them to work as expected so the matrix is removed from the interface in favour of explicit methods.

Adds a rotation test to ensure this ability doesn't regress

PiperOrigin-RevId: 549890847
2023-08-01 13:45:22 +01:00
bachinger
d658de5944 Mark media notification controller and filter command buttons
The `MediaNotificationManager` registers an internal controller
to each session. This change marks this controller through its
connection hints and provides an API for apps to hide
implementation details of the marking.

Issue: androidx/media#389
PiperOrigin-RevId: 549712768
2023-08-01 13:44:12 +01:00
jbibik
25253698bc TtmlParser implementation - moved from TtmlDecoder
`TtmlDecoder` which used to be `SimpleSubtitleDecoder` will now be called `TtmlParser` and implement SubtitleParser interface. For backwards compatibility, we will have the same functionality provided by `DelegatingSubtitleDecoder` backed-up by a new `TtmlParser` instance.

PiperOrigin-RevId: 549700490
2023-08-01 13:43:12 +01:00
jbibik
1c1b78e635 Test for Decoder with Mp4WebvttParser corrected
PiperOrigin-RevId: 549629289
2023-08-01 13:42:06 +01:00
jbibik
e11e484bc3 Do not emit empty cues in SsaParser
It is not part of the `SubtitleParser` interface's promise to give events for all changes to do with `Cue` objects. So while in the past, we would have `endTimeUs` of one Cue event being the `startTimeUs` of the next one, now we have that encoded in `durationUs` and can skip event with empty `Cues`.

PiperOrigin-RevId: 549629157
2023-08-01 13:41:11 +01:00
tonihei
85cde93e6c Remove C2Mp3TimestampTracker
This tracker aims to replicate the behavior of a specific codec to
ensure MediaCodecRenderer correctly detects stream and output
format transitions. The class was needed because MediaCodecRenderer
made assumptions about codec behavior this codec did not fulfil (in
particular, changing timestamps and number of samples).

Since then, MediaCodecRenderer was made more robust to this kind of
codec behavior in general and currently has no assumptions that
require any special handling of this codec. This means we can remove
the workaround completely.

PiperOrigin-RevId: 549610989
2023-08-01 13:39:53 +01:00
jbibik
27bda610aa Standardise the use of ParsableByteArray use among SubtitleParsers
PiperOrigin-RevId: 549609028
2023-08-01 13:38:38 +01:00
huangdarwin
9128293236 Test: Use correct test skipping in muxer androidTest.
PiperOrigin-RevId: 549601882
2023-08-01 13:37:31 +01:00
simakova
3d0d5f5c95 Update README file
Add a link to the media dev center and add Transformer to the list of libraries

PiperOrigin-RevId: 549598105
2023-08-01 13:36:21 +01:00
ibaker
1cfaae3450 Enable WebVTT subtitles in MkvPlaybackTest
This is no longer flaky because WebVTT subtitles are decoded on the
loading thread (enabled in this test with
`defaultExractorsFactory.setTextTrackTranscodingEnabled(true)` - and
supported for WebVTT since f0f24aa0d4).

PiperOrigin-RevId: 549594291
2023-08-01 13:35:11 +01:00
tonihei
b4860fada0 Document MediaCodecRenderer's stream and codec behavior assumptions
The class tries to be flexible to support as many different input
and codec behavior combinations as possible. But so far it didn't
spell out its remaining assumptions and explicit non-assumptions,
making it hard to know which behavior to rely on.

PiperOrigin-RevId: 549589347
2023-08-01 13:34:02 +01:00
ibaker
1ccedf8414 Tighten the demo app's handling of DrmInitData for downloads
This code is Widevine specific. `OfflineLicenseHelper.downloadLicense`
requires the passed `Format` to have a `DrmInitData.SchemeData` with
Widevine UUID and non-null `data` field. The demo app tries to check
this in advance (to avoid an exception later), but its checks are
looser than those made by `OfflineLicenseHelper`. This change tightens
the checks to match.

Issue: androidx/media#512
PiperOrigin-RevId: 549587506
2023-08-01 13:31:57 +01:00
jbibik
f4c66d82dd TtmlDecoderTest refactor not to use TtmlNode or TtmlStyle
PiperOrigin-RevId: 549584918
2023-08-01 13:30:39 +01:00
samrobinson
beceb996ac Create ExportException for audio from UnhandledAudioFormatException.
All audio processing export errors are related to a UAFE - passing
audioFormat separately is redundant.

PiperOrigin-RevId: 549427685
2023-07-20 10:35:45 +01:00
rohks
cdb174c91a Add field measured throughput (mtp)
Updated `ExoTrackSelection` to provide the most recent bitrate estimate, enabling the inclusion of measured throughput (mtp) as a CMCD-Request field in Common Media Client Data (CMCD) logging.

Additionally, made changes to the `checkArgument` methods in `CmcdLog` to prevent the use of default values in certain cases.

#minor-release

PiperOrigin-RevId: 549369529
2023-07-20 10:34:32 +01:00