18999 Commits

Author SHA1 Message Date
huangdarwin
bd6a36033e Compositor: Move shared test logic to TestRunner.
Move shared test logic to the test runner.

This does increase indirection, which isn't usually preferable in tests.
However, we will have many different tests that would use logic
like this, so this allows us to reduce repetition.

PiperOrigin-RevId: 551536438
2023-08-01 14:13:25 +01:00
claincly
e77c318d96 Create Util method for SparseArray.contains()
And fixed usages in Transformer and Effect.

PiperOrigin-RevId: 551484860
2023-08-01 14:12:10 +01:00
bachinger
17ee5277d5 Only use result.sendError where supported by legacy media library
`MediaLibraryServiceLegacyStub` handles various edge cases by calling
`result.sendError(null)` with the intention to send back an error to
the legacy browser [1].

`MediaBrowserServiceCompat` of the legacy media1 Compat library has an
inner base class `Result` that has a default implementation of
`onErrorSent` that throws an `UnsupportedOperationException` [2].
However, most anonymous inner classes for `Result` created in
`MediaBrowserServiceCompat` do not override `onErrorSent` [3].

Hence Media3 must not call `sendError` in these cases. Instead we call
`sendResult(null)` according to what the default implementation of
the callbacks in `MediaBrowserServiceCompat` do ([4] as an example).

Issue: androidx/media#78
Issue: androidx/media#334

[1] https://github.com/androidx/media/blob/release/libraries/session/src/main/java/androidx/media3/session/MediaLibraryServiceLegacyStub.java#L200
[2] https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:media/media/src/main/java/androidx/media/MediaBrowserServiceCompat.java;l=872
[3] https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:media/media/src/main/java/androidx/media/MediaBrowserServiceCompat.java;l=578-604?q=MediaBrowserServiceCompat&ss=androidx
[4] https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:media/media/src/main/java/androidx/media/MediaBrowserServiceCompat.java;l=1395

PiperOrigin-RevId: 551210137
2023-08-01 14:11:08 +01:00
tonihei
f766936140 Fix ms to us conversion bug in DecoderVideoRenderer
The current code multiplies the value by 1000 twice,
effectively converting to nanoseconds.

#minor-release

PiperOrigin-RevId: 551129750
2023-08-01 14:09:56 +01:00
samrobinson
b7d7027424 Update Composition javadoc for expanded audio support.
PiperOrigin-RevId: 550922961
2023-08-01 14:08:46 +01:00
tonihei
782f5b2a0e Fix last sample detection issues
1. For codecs that change timestamps, we need to use greater
   or equal instead of just equal for comparison.
2. We should only make this comparison once a value has been
   set.
3. The largest queue timestamp isn't updated before using it
   for this detection, meaning that we also mark the
   second-to-last sample as the last one.

PiperOrigin-RevId: 550901978
2023-08-01 14:06:55 +01:00
andrewlewis
9a2bfc4042 Prioritize H.265 and H.264 over other formats
This should make no functional difference because `SampleExporter` always
checks for H.265 and H.264 first. However, in case we ever change that code,
these are used in priority order so it's better to order them accordingly.

PiperOrigin-RevId: 550894935
2023-08-01 14:05:42 +01:00
tonihei
79c8b0e0f8 Set correct start time for newly enabled renderers during playback
We currently pass in the time at which the stream originally started,
but newly enabled renderers should get the current playback position
instead.

PiperOrigin-RevId: 550894630
2023-08-01 14:04:35 +01:00
samrobinson
54093a152e Integrate AudioMixer for audio export.
Adds support for compositions with multiple audio sequences.

PiperOrigin-RevId: 550880626
2023-08-01 14:03:22 +01:00
samrobinson
357c458028 Ensure EOS is queued after processing generated silence with effects.
When generating silence for AudioProcessingPipeline, audio never
queued EOS downstream.

Linked to this, when silence followed an item with audio, the silence
was added to SilentAudioGenerator before the mediaItem reconfiguration
occurred. If the silence had effects, the APP would be flushed after
silence queued EOS, resetting APP.isEnded back to false, so AudioGraph
never ended.

Regression tests reproduce failure without fix, but pass with it.

PiperOrigin-RevId: 550853714
2023-08-01 14:02:09 +01:00
huangdarwin
35bee3299c Remove VisibleForTesting visibility in 8k workaround.
PiperOrigin-RevId: 550843914
2023-08-01 14:01:02 +01:00
Googler
465e7c1277 Rewrite tracing to be more flexible and provide more info.
For each event, the timestamp and presentation time is logged. The trace can
then be dumped to a tsv file and easily imported in a spreadsheet.

PiperOrigin-RevId: 550839156
2023-08-01 13:59:51 +01:00
bachinger
71af48429f Fix flaky test in MediaControllerListenerTest
PiperOrigin-RevId: 550625956
2023-08-01 13:58:49 +01:00
ibaker
449cf55523 Add comment about out-of-order timestamps in AtomParsers.parseStbl
PiperOrigin-RevId: 550596173
2023-08-01 13:57:36 +01:00
huangdarwin
e5e3db954f Remove setting the interrupted flag when unneeded.
When rethrowing InterruptedException, this
Thread.currentThread().interrupted() is not needed.

PiperOrigin-RevId: 550586853
2023-08-01 13:56:35 +01:00
huangdarwin
ef39d317d5 Compositor: Signal onEnded when frames finish compositing.
PiperOrigin-RevId: 550576032
2023-08-01 13:55:21 +01:00
bachinger
a90ec96425 Use MediaUtils.intersect in MediaControllerImplBase
PiperOrigin-RevId: 550566322
2023-08-01 13:54:11 +01:00
bachinger
2c19399d3e Move command button related helper methods to CommandButton
PiperOrigin-RevId: 550558261
2023-08-01 13:53:09 +01:00
claincly
22d30b3d74 Fix some mal formatted TODOs
PiperOrigin-RevId: 550507208
2023-08-01 13:51:58 +01:00
jbibik
f4ad26451f Deprecate MimeTypes.TEXT_EXOPLAYER_CUES for APPLICATION_MEDIA3_CUES
Instead of `text/x-exoplayer-cues`, we will use `application/x-media3-cues`.

The prefix should be `application` not `text` since the encoded form is arbitrary bytes, not necessarily text. The name should not reference `exoplayer`, since the Media3 Extractors (which are not part of `exoplayer`) produce this format.

PiperOrigin-RevId: 550181852
2023-08-01 13:50:50 +01:00
huangdarwin
d7b5ab9f23 Add Thread.currentThread().interrupt() for InterruptedExceptions.
PiperOrigin-RevId: 549975878
2023-08-01 13:49:48 +01:00
huangdarwin
5858723a06 Effect: Add Compositor signalEndOfInputStream and onEnded.
signalEndOfInputStream is needed for when streams have different amounts of
frames, so that if the primary stream finishes after a secondary stream, it
can end without waiting indefinitely for the secondary stream's matching
timestamps.

onEnded mirrors this API on the output side, which will be necessary to
know when to call signalEndOfInput on downstream components (ex. on downstream)
VideoFrameProcessors

PiperOrigin-RevId: 549969933
2023-08-01 13:48:36 +01:00
sheenachhabra
c2615a679f Skip tests if device needs disable 8k workaround
PiperOrigin-RevId: 549967326
2023-08-01 13:47:26 +01:00
huangdarwin
b1cf5fb320 Effect: Reduce Compositor exception try/catch redirection.
The VideoFrameProcessingTaskExecutor already handles wrapping
GlUtil.GlException errors into VideoFrameProcessingExceptions.
Therefore, stop wrapping this ourselves, so that errors are
attributed to the correct line number with less indirection.

PiperOrigin-RevId: 549959258
2023-08-01 13:46:28 +01:00
tofunmi
7eee15ecb4 Update OverlaySettings to explicitly state overlay transformations
Alters the OverlayShaderProgram implementation to support rotations, however we need to apply the transformations separately in order for them to work as expected so the matrix is removed from the interface in favour of explicit methods.

Adds a rotation test to ensure this ability doesn't regress

PiperOrigin-RevId: 549890847
2023-08-01 13:45:22 +01:00
bachinger
d658de5944 Mark media notification controller and filter command buttons
The `MediaNotificationManager` registers an internal controller
to each session. This change marks this controller through its
connection hints and provides an API for apps to hide
implementation details of the marking.

Issue: androidx/media#389
PiperOrigin-RevId: 549712768
2023-08-01 13:44:12 +01:00
jbibik
25253698bc TtmlParser implementation - moved from TtmlDecoder
`TtmlDecoder` which used to be `SimpleSubtitleDecoder` will now be called `TtmlParser` and implement SubtitleParser interface. For backwards compatibility, we will have the same functionality provided by `DelegatingSubtitleDecoder` backed-up by a new `TtmlParser` instance.

PiperOrigin-RevId: 549700490
2023-08-01 13:43:12 +01:00
jbibik
1c1b78e635 Test for Decoder with Mp4WebvttParser corrected
PiperOrigin-RevId: 549629289
2023-08-01 13:42:06 +01:00
jbibik
e11e484bc3 Do not emit empty cues in SsaParser
It is not part of the `SubtitleParser` interface's promise to give events for all changes to do with `Cue` objects. So while in the past, we would have `endTimeUs` of one Cue event being the `startTimeUs` of the next one, now we have that encoded in `durationUs` and can skip event with empty `Cues`.

PiperOrigin-RevId: 549629157
2023-08-01 13:41:11 +01:00
tonihei
85cde93e6c Remove C2Mp3TimestampTracker
This tracker aims to replicate the behavior of a specific codec to
ensure MediaCodecRenderer correctly detects stream and output
format transitions. The class was needed because MediaCodecRenderer
made assumptions about codec behavior this codec did not fulfil (in
particular, changing timestamps and number of samples).

Since then, MediaCodecRenderer was made more robust to this kind of
codec behavior in general and currently has no assumptions that
require any special handling of this codec. This means we can remove
the workaround completely.

PiperOrigin-RevId: 549610989
2023-08-01 13:39:53 +01:00
jbibik
27bda610aa Standardise the use of ParsableByteArray use among SubtitleParsers
PiperOrigin-RevId: 549609028
2023-08-01 13:38:38 +01:00
huangdarwin
9128293236 Test: Use correct test skipping in muxer androidTest.
PiperOrigin-RevId: 549601882
2023-08-01 13:37:31 +01:00
simakova
3d0d5f5c95 Update README file
Add a link to the media dev center and add Transformer to the list of libraries

PiperOrigin-RevId: 549598105
2023-08-01 13:36:21 +01:00
ibaker
1cfaae3450 Enable WebVTT subtitles in MkvPlaybackTest
This is no longer flaky because WebVTT subtitles are decoded on the
loading thread (enabled in this test with
`defaultExractorsFactory.setTextTrackTranscodingEnabled(true)` - and
supported for WebVTT since f0f24aa0d4).

PiperOrigin-RevId: 549594291
2023-08-01 13:35:11 +01:00
tonihei
b4860fada0 Document MediaCodecRenderer's stream and codec behavior assumptions
The class tries to be flexible to support as many different input
and codec behavior combinations as possible. But so far it didn't
spell out its remaining assumptions and explicit non-assumptions,
making it hard to know which behavior to rely on.

PiperOrigin-RevId: 549589347
2023-08-01 13:34:02 +01:00
ibaker
1ccedf8414 Tighten the demo app's handling of DrmInitData for downloads
This code is Widevine specific. `OfflineLicenseHelper.downloadLicense`
requires the passed `Format` to have a `DrmInitData.SchemeData` with
Widevine UUID and non-null `data` field. The demo app tries to check
this in advance (to avoid an exception later), but its checks are
looser than those made by `OfflineLicenseHelper`. This change tightens
the checks to match.

Issue: androidx/media#512
PiperOrigin-RevId: 549587506
2023-08-01 13:31:57 +01:00
jbibik
f4c66d82dd TtmlDecoderTest refactor not to use TtmlNode or TtmlStyle
PiperOrigin-RevId: 549584918
2023-08-01 13:30:39 +01:00
samrobinson
beceb996ac Create ExportException for audio from UnhandledAudioFormatException.
All audio processing export errors are related to a UAFE - passing
audioFormat separately is redundant.

PiperOrigin-RevId: 549427685
2023-07-20 10:35:45 +01:00
rohks
cdb174c91a Add field measured throughput (mtp)
Updated `ExoTrackSelection` to provide the most recent bitrate estimate, enabling the inclusion of measured throughput (mtp) as a CMCD-Request field in Common Media Client Data (CMCD) logging.

Additionally, made changes to the `checkArgument` methods in `CmcdLog` to prevent the use of default values in certain cases.

#minor-release

PiperOrigin-RevId: 549369529
2023-07-20 10:34:32 +01:00
claincly
177187e237 Allow multiple inputs to the processing graphs
The new logic covers the old single-sequence use cases.

PiperOrigin-RevId: 549307918
2023-07-20 10:33:35 +01:00
jbibik
f0f24aa0d4 WebvttParser implementation - moved from WebvttDecoder
`WebvttDecoder` which used to be `SimpleSubtitleDecoder` will now be called `WebvttParser` and implement `SubtitleParser` interface.

For backwards compatibility, we will have the same functionality provided by `DelegatingSubtitleDecoder` backed-up by a new `WebvttParser` instance. `WebvttSubtitle` will still be used behind the scenes to handle overlapping `Cues`.

PiperOrigin-RevId: 549298733
2023-07-20 10:32:13 +01:00
ibaker
5d453fcf37 Add an Extractor to parse subtitles before SampleQueue
The end-to-end test output for the overlapping SRT and SSA subtitles
is currently incorrect. They will be fixed in a future change that
updates `TextRenderer` to support this overlap.

The 'extra' samples visible in the extractor test output files are
'empty cue list' samples produced by `SsaParser`. They will go away
when this implementation is updated to remove this behaviour and rely
on `CuesWithTiming.durationUs` instead (the 'empty list' behaviour is
not required by the `SubtitleParser` interface).

PiperOrigin-RevId: 549264593
2023-07-20 10:30:16 +01:00
tonihei
5c5e7cd257 Update storage access in main demo app
The main demo app was still targeting API 29 to avoid scoped storage
restrictions. It is now updated to 34 (like the rest of the demo apps)
and handles scoped storage as it should handle it.

More specifically:
 - We need to request READ_MEDIA_... permissions instead of
   READ_EXTERNAL_STORAGE from API33.
 - The legacy scoped storage opt-out can be removed
 - READ_MEDIA_... permissions don't allow arbitrary file access
   if the file doesn't end in a typical media file extension, so
   this change adds a remark on the guide page to place samples
   in the app-specific directory.
 - We also don't have to request permissions for the app-specific
   directories.
 - Custom json files can't be placed in arbitray local locations
   because they don't end in a media file extension, as there is
   no way we can request a permission to load them. This means we
   can remove the storage request logic in SampleChooserActivity.

Issue: google/ExoPlayer#6045
PiperOrigin-RevId: 549252474
2023-07-20 10:29:02 +01:00
jbibik
d9daa392d5 Util for Subtitle to CuesWithTiming conversion
PiperOrigin-RevId: 549027279
2023-07-20 10:27:47 +01:00
tonihei
da99f9937d Rollback of 7996766b22
*** Original commit ***

Rollback of b69b33206e

*** Original commit ***

Mark output sample as decode-only based on start time

We currently do the same check on the input timestamps and
expect the output timestamps to match. Some codecs produce
samples with modified timestamps and the logic is a lot safer
when the comparison with the start time is done on the output
side of the codec.

Issue: google/ExoPlayer#11000

***

***

PiperOrigin-RevId: 549019403
2023-07-20 10:26:25 +01:00
huangdarwin
747b31b3c5 Test: Add non-mh androidTest tests to MH_TESTS / triage.
PiperOrigin-RevId: 549005874
2023-07-20 10:25:30 +01:00
tonihei
8064c6df83 Declare foreground service type for DownloadService
This ensures the DownloadService stays functional on Android 14
where defining this type is required. On Android 14 and above,
the app also needs to define the DATA_SYNC permission, which is
added to the demo app as well. In the future, this service type
will no longer be supported and DownloadService needs to be
rewritten with another background scheduling framework.

Issue: google/ExoPlayer#11239
PiperOrigin-RevId: 548994842
2023-07-20 10:23:56 +01:00
samrobinson
035934c6d4 Combine audio mixing logic across Transformer.
AudioMixingUtil#mix handles input & output in float or Int16 PCM. Given
Float and Int16 use different sample ratnes, this util handles
conversion between the two, based on the encoding being mixed to.

Migrate AudioMixer to use the util, removing AudioMixingAlgorithm
interface and implementation. ChannelMixingAudioProcessor will be
migrated after additional performance checks.

PiperOrigin-RevId: 548994584
2023-07-20 10:22:58 +01:00
jbibik
9520180fbb DelegatingSubtitleDecoder tests refactor
PiperOrigin-RevId: 548991428
2023-07-20 10:21:43 +01:00
tonihei
b0ec2cc701 Remove timestamp clipping in MergingMediaPeriod.TimeOffsetSampleStream
This was done because it was deemed correct to only start at timestamp
zero when the code was originally written. However, in case of
prerolling from a keyframe, many samples will get the same timestamp,
which is not correct and interferes with downstream logic
that deals with timestamps.

PiperOrigin-RevId: 548986160
2023-07-20 10:20:22 +01:00