473 Commits

Author SHA1 Message Date
samrobinson
65d653c577 Move Encoder quality API to VideoEncoderSettings.
Some other minor nits and adjustments to the API logic.

PiperOrigin-RevId: 459490431
(cherry picked from commit 91f1777741149ecf67759da8eb68828e1d92624b)
2022-07-07 12:54:02 +00:00
huangdarwin
bde5f9e30d HDR: Remove ColorInfo.SDR constant
The SDR constant also specified a color space and range, in addition to
C.COLOR_TRANSFER_SDR. However, it turns out that SDR videos may use different color
space and range values, so following prior ExoPlayer conventions to have `null`
mean "generic SDR" is preferable here.

PiperOrigin-RevId: 459296746
(cherry picked from commit 7078ce312d3650ef70dcd8ed236af88dc07b5333)
2022-07-06 18:11:11 +00:00
huangdarwin
c199608589 HDR: Throw error if attempting HDR editing under API 31.
HDR editing is not supported under API 31

PiperOrigin-RevId: 459211106
(cherry picked from commit ab7747d9533c0ed1c5d096743741607b0e08c609)
2022-07-06 10:25:30 +00:00
huangdarwin
5ab53d3dc1 HDR: Remove unused EGL_GL_COLORSPACE_KHR attribute.
PiperOrigin-RevId: 459106221
(cherry picked from commit 1869855c90f67e8735f3a58361e36db2c421f838)
2022-07-05 18:47:29 +00:00
samrobinson
ebf1ae49b7 Implement device based encoder bitrate mapping.
This feature is disabled by default for now.

PiperOrigin-RevId: 458932471
(cherry picked from commit 7d2eb7601947570245dcd2ff86b7dcf1c8393d40)
2022-07-04 17:32:27 +00:00
hschlueter
3bdecf2ac9 Fallback to SDR if encoder doesn't support HDR (HLG only).
If the input is HDR (HLG), check encoder capabilities for HDR support
and request tone-mapping to SDR during decoder configuration otherwise.
Capabilities are only checked for API 31 and above, as HDR editing is
not supported before.

As the encoder capabilities check needs to happen before selecting the
encoder to use (as this may depend on the resolution output by the
effects chain), the EncoderWrapper checks all candidate encoders
for the MIME type for HDR capabilities and only requests fallback to
SDR if none of them support it.

When the actual encoder is selected, the wrapper checks that it matches
one of the encoders is checked capabilities for.

PiperOrigin-RevId: 458511599
(cherry picked from commit 9c8dcb402b45620174ae6be2d943ff8f32e4da14)
2022-07-01 18:34:02 +00:00
huangdarwin
28cf112e01 HDR: Configure GL shaders and encoder.
Configure the GL shaders and encoder to take in HDR metadata.

This mostly just consists of passing the Format.colorInfo through
the VideoTranscodingSamplePipeline down to the encoder, rather than passing
the PQ-ness down to the GL step.

Due to b/237674316, this will remove HDR10+ support temporarily to introduce
support for HLG10.

Manually tested to confirm that HLG10 operations that don't affect color display
correctly after this CL with "HDR editing" in the demo checked, and continue to display incorrectly (as before this CL) without the option unchecked.

PiperOrigin-RevId: 458490810
(cherry picked from commit a0870a42be5336b313f79f596e94411563e14720)
2022-07-01 16:57:43 +00:00
claincly
379ecd198f Fix error in frame rate handling.
The old getString() will throw because FRAME_RATE can only be float or int.

PiperOrigin-RevId: 458481251
(cherry picked from commit deea5c927a61da5a84c534b485ad8a249d10cbd6)
2022-07-01 16:05:39 +00:00
claincly
e0752a3a7a Find only REGULAR_CODECS in EncoderUtil.
We used "ALL_COOECS" previously, and it is not necessary because "ALL_CODECS"
additionally the codecs that support tunneling/secure decoding, which there
is no use case in Transformer.

PiperOrigin-RevId: 458470278
(cherry picked from commit 3df4f3eb19aaa3486d2b3f8153be9167bd31027d)
2022-07-01 14:58:47 +00:00
claincly
f2b4701e92 Round the frame rate in MediaFormat
Although MediaCodec claims supporting float frame rate, encoder init failed on
API21 Nexus 5. Since it's just a performance hint to the codec, it's OK to
generalize it to other API versions.

PiperOrigin-RevId: 458434650
(cherry picked from commit 1f47fa832c6228a5545d8a15c9c09fe76c17dfc5)
2022-07-01 11:06:12 +00:00
hschlueter
d48507a2d1 Move DebugViewProvider out of Transformer class.
The GlEffectsFrameProcessor that will be part of the effects module
uses the DebugViewProvider. So it does not make sense for it
to be an inner interface of Transformer.

PiperOrigin-RevId: 458014932
(cherry picked from commit cd0e5b99de9577ff1af0da15a9ea6c9041f38361)
2022-06-29 17:56:53 +00:00
hschlueter
1a1c6e94bb FrameProcessor: Replace SurfaceInfo.Provider with setter.
The FinalMatrixTransformationProcessorWrapper ensures that the
surface is only replaced when it is not being rendered to and vice
versa.

PiperOrigin-RevId: 458007639
(cherry picked from commit 234015cb952fbdaf274f1fa9005ddc0d2b4ba17a)
2022-06-29 17:27:05 +00:00
hschlueter
129cfb6eeb Fix outputHeight pass-through condition.
The outputHeight in the TransformationRequest is the height of
the frame as it would be displayed (i.e., after applying any
rotation specified in the format). So pass-through should only
be used if the requested outputHeight matches the input
format's height after applying the rotation.

PiperOrigin-RevId: 457934867
(cherry picked from commit 87beb273e464d291423ccb4129c83f36195343f1)
2022-06-29 10:28:54 +00:00
hschlueter
c250d2d791 Allow changing input pixelWidthHeightRatio for GlEffectsFrameProcessor.
pixelWidthHeightRatio is now passed to setInputFrameInfo instead of
the factory.

PiperOrigin-RevId: 457696703
(cherry picked from commit b7b5f20e592bee772772eb1b009c7f5a96077520)
2022-06-28 11:22:45 +00:00
hschlueter
864230fd11 Rename setVideoFrameEffects to setVideoEffects.
PiperOrigin-RevId: 457023382
(cherry picked from commit 4819b28587c41e75c716ba3fa84fa84a8934c78e)
2022-06-24 17:04:45 +01:00
hschlueter
928036523a Allow FrameProcessor input surface size changes.
This will be useful for downgrading to a lower resolution during
a slow preview and for processing slide-shows once sequential
multi-asset editing is supported.

PiperOrigin-RevId: 457017255
(cherry picked from commit 30c52c58f0772e43ca4e4639120d95f41ea8a9b5)
2022-06-24 16:31:31 +01:00
hschlueter
71f3ebb79c Extract FrameProcessor interface from GlEffectsFrameProcessor.
PiperOrigin-RevId: 456814150
(cherry picked from commit f3893c146d5e4521b84af45551ea5fda4d9c66bd)
2022-06-23 19:00:01 +01:00
hschlueter
4fc36bafdd Disable flaky transformer emulator test on API 31.
videoEncoderFormatUnsupported_completesWithError() has recently
been flaky on API 31 emulators on presubmit because a different
exception than the expected exception is thrown.
This disables it on those emulators to reduce testing noise
until the underlying problem is investigated and resolved.

PiperOrigin-RevId: 456765512
(cherry picked from commit 938d3c2e5b6f016f7075a3d2d162d37c1951d825)
2022-06-23 15:44:02 +01:00
hschlueter
498123da68 Rename FrameProcessorChain to GlEffectsFrameProcessor.
This change is just renaming. There is no functional change intended.
The FrameProcessor interface will be created in a follow-up.

PiperOrigin-RevId: 456741628
(cherry picked from commit 709224fb1ed3a1dab1433a5dc7fbb33a04eecaae)
2022-06-23 13:13:56 +01:00
samrobinson
90c904b089 Add a Builder for DefaultEncoderFactory.
PiperOrigin-RevId: 456728032
(cherry picked from commit 352967f65661296ff5551138fd6346fce0c52a3f)
2022-06-23 11:37:19 +01:00
hschlueter
26ee3d32b2 Use GlTextureProcessor to avoid redundant copy in MediaPipeProcessor.
After this change GlEffects can use any GlTextureProcessor not just
SingleFrameGlTextureProcessor.
MediaPipeProcessor now implements GlTextureProcessor directly which
allows it to reuse MediaPipe's output texture for its output texture
and avoids an extra copy shader step.

PiperOrigin-RevId: 456530718
(cherry picked from commit 69ab79418ef21033ca25cebdf5a5e80752818ab5)
2022-06-22 17:16:54 +01:00
hschlueter
0007a47365 Support chaining async GlTextureProcessors in FrameProcessorChain.
After this change, FrameProcessorChain chains any GlTextureProcessors
instead of only SingleFrameGlTextureProcessors.

The GlTextureProcessors are chained in a bidirectional manner using
ChainingGlTextureProcessorListener to feed input and output related
events forward and release events backwards.

PiperOrigin-RevId: 456478414
(cherry picked from commit 555ab97e34d6586d38a9979bc0595a4818b7ecb8)
2022-06-22 12:03:41 +01:00
huangdarwin
4773fc6873 FrameProcessor: Use factories instead of a builder for Presentation.
PiperOrigin-RevId: 456064021
(cherry picked from commit f5d8800d51def6304db9c1f5a8f5ea3f6e3c69a6)
2022-06-20 14:26:12 +01:00
hschlueter
73b57d3c77 Implement chaining GlTextureProcessor.Listener.
In follow-ups the FrameProcessorChain will set an instance of this
listener for each GlTextureProcessor to chain it with its previous
and next GlTextureProcesssor.

PiperOrigin-RevId: 455628942
(cherry picked from commit 981baae7095ef34c548df600bfbea1f7765dbb9e)
2022-06-17 16:57:46 +01:00
hschlueter
c16e5ac47c Wrap processor chain executor service for better exception handling.
The wrapper
* catches exceptions for each task and notifies the
  listener (this will be used more in follow-ups when processFrame
  is split into lots of listeners and callbacks),
* removes finished tasks from the queue and signals any exceptions
  that occurred to the listener each time a new task is executed.

PiperOrigin-RevId: 455345184
(cherry picked from commit ee847d92c50c8d2929da287e1103110149d78170)
2022-06-16 10:41:23 +00:00
hschlueter
8c465a7860 Replace FrameProcessorChain#isEnded with listener method.
PiperOrigin-RevId: 455114693
(cherry picked from commit fdfca880195da2ca5e5d0ef002feb8144009384c)
2022-06-15 13:27:35 +00:00
hschlueter
b7118486dc Use FrameProcessorChain#SurfaceProvider for encoder compat transform.
This change adds a SurfaceProvider interface which is necessary to
allow for texture processors whose output size becomes available
asynchronously in follow-ups.
VTSP's implementation of this interface wraps the encoder and provides
its input surface together with the output frame width, height, and
orientation as used for encoder configuration.
The FrameProcessorChain converts the output frames to the provided
orientation and resolution using a ScaleToFitTransformation and
Presentation replacing EncoderCompatibilityTransformation.

PiperOrigin-RevId: 455112598
(cherry picked from commit ea7f1ca1e32f19131cec690c6934d82c81a2d6d0)
2022-06-15 13:14:56 +00:00
hschlueter
56f234ea75 Check targetSdkVersion for frame dropping workaround.
Based on
https://developer.android.com/reference/android/media/MediaCodec#using-an-output-surface,
frame dropping behaviour depends on the target SDK version.
After this change transformer will only use
MediaFormat#KEY_ALLOW_FRAME_DROP if both the target and system SDK
version are at least 29 and default to its pre 29 behaviour where each
decoder output frame must be processed before a new one is rendered
to prevent frame dropping otherwise.

Also remove deprecated Transformer.Builder constructor without a
context and the context setter.

PiperOrigin-RevId: 453971097
(cherry picked from commit a105d033a7d44b8c3afb0e8134ad65f624caf256)
2022-06-09 18:22:39 +00:00
hschlueter
0b08396255 Make GlUtil.GlException checked and remove flag to disable it.
Transformer always enabled glAssertionsEnabled, so there should
be no functional change.

ExoPlayer previously disabled glAssertionsEnabled, so GlUtil logged
GlExceptions instead of throwing them. The GlExceptions are now
caught and logged by the callers so that there should also be no
functional change overall.

This change also replaces EGLSurfaceTexture#GlException with
GlUtil#GlException.

PiperOrigin-RevId: 453963741
(cherry picked from commit cc1f32d094cc4371d92b3ddab144d8965e1bc1c3)
2022-06-09 17:53:39 +00:00
claincly
780631c14b Ensure re-encode on performance tests
PiperOrigin-RevId: 453933854
(cherry picked from commit 12d775f055faac1b9de2c53fd296c5b1d0f89a32)
2022-06-09 15:41:09 +00:00
huangdarwin
5349312f50 Transformer GL: Split Presentation and Crop.
This removes the prior restriction of needing to remember not to crop and set aspect ratio in the same Presentation.Builder, and makes each class a bit more targeted.

This is partially made feasible by the past work to merge consecutive
MatrixTransformations into a single MatrixTransformationFrameProcessor, which
ensures that there's no loss in quality between successive MatrixTransformations.

PiperOrigin-RevId: 453660582
(cherry picked from commit c5b881e0899e713a7c632da48f1cd64b1394a221)
2022-06-08 12:51:28 +00:00
hschlueter
8973efd199 Mention alpha for matrix transformation background color.
PiperOrigin-RevId: 453633920
(cherry picked from commit cdb80387828c0087d9c9b96c55f303b978847910)
2022-06-08 09:50:18 +00:00
hschlueter
dac3878273 Implement default GlTextureProcessor in SingleFrameGlTextureProcessor.
SingleFrameGlTextureProcessor is now an abstract class containing a
default implementation of the more flexible GlTextureProcessor interface
while still exposing the same simple abstract methods for single frame
processing it previously did.

FrameProcessorChain and GlEffect will be changed to use
GlTextureProcessor in follow-ups.

PiperOrigin-RevId: 453633000
(cherry picked from commit 457f446114823831b940bf46fb256daa0ea00815)
2022-06-08 09:44:25 +00:00
hschlueter
f4dfbdbc37 Add interface for async texture processors.
Implementations of this interface will be able to drop or add frames,
change timestamps, accept multiple input frames before producing
output, and process frames on their own background thread.

A default implementation of this interface will be added to SingleFrameGlTextureProcessor in a follow-up.

PiperOrigin-RevId: 453159835
(cherry picked from commit 63436390de55741620ab3faccbbc504af24c2686)
2022-06-06 10:18:26 +00:00
hschlueter
27f23041cf Add async error listener to transformer to avoid exception wrapping.
This internal listener avoids wrapping the TransformationExceptions
in PlaybackExceptions that are handled via the Player.Listener and
is also used for FrameProcessingExceptions which already avoided
the PlaybackException layer previously.

This listener will also be useful in follow-ups for encoder-related
TransformationExceptions that are thrown in the SurfaceProvider that
will be called on the GL thread.

PiperOrigin-RevId: 452074575
(cherry picked from commit 35b5147eb1404698132d9b063c905202c22e31d8)
2022-05-31 17:01:02 +00:00
hschlueter
22ee071b53 Move program initialization to texture processor constructor.
Once the more advanced GlTextureProcessor interface exists,
it will be possible to change the output size of a GlTextureProcessor
between frames. To keep the re-configuration based on the frame sizes
minimal, things indepedent of the frame size, such as the GlProgram,
can be initialized in the constructor.

PiperOrigin-RevId: 451997584
(cherry picked from commit 87ab96d352c0ffbf83d514d93193638895b61e5e)
2022-05-31 09:36:18 +00:00
claincly
a629d09458 Skip BitrateAnalysisTest if device is incapable.
Add bitrate check to the "can encode" criteria.

PiperOrigin-RevId: 451868042
2022-05-30 16:51:51 +00:00
andrewlewis
b25d00a795 Retain stream time offsets through codecs
ExoPlayer applies a large time offset to buffers so that, if the input has negative timestamps, generally buffers seen by the decoders should have positive timestamps. Modify how the offset is handled in `Transformer` so that decoders and encoders generally see positive timestamps, by leaving the offset on samples when reading them in the base renderer (remove the code that removed the offset), and then removing the offset when muxing. Also update the frame processor chain and slow motion flattening code to retain the existing behavior after this change (these both need original media presentation timestamps)

Tested via existing end-to-end tests and manually verified that the overlay frame processor shows the expected original media timestamps.

Aside: we don't need the same logic as ExoPlayer to track stream offsets across the decoder yet, because we don't try to handle stream changes during playback in single asset editing. (There is an edge case of multi-period DASH that may not work but I doubt anyone will use that as input to `Transformer` before we change the code to handle multi-asset properly.) In future we should try to refactor interaction with the decoder to use the same code for Transformer and ExoPlayer.
PiperOrigin-RevId: 451846055
2022-05-30 16:45:15 +00:00
samrobinson
a1b89561c1 Treat pixels as unsigned and correct pixel count division.
PiperOrigin-RevId: 451428202
2022-05-30 16:42:38 +00:00
huangdarwin
67354cf2a7 Transformer: Add 8k24fps transform test.
Most devices won't support 8k decoding, so they'll skip this test entirely.

As the video is quite short, this test shouldn't be any longer than the nearby,
long-running 4k60 test.

PiperOrigin-RevId: 451423368
2022-05-30 16:41:41 +00:00
samrobinson
6c4f6ecf46 Clarify variables and improve documentation of SSIM.
PiperOrigin-RevId: 451392021
2022-05-30 16:39:50 +00:00
samrobinson
224761833f Use updated test name in BUILD and docs.
PiperOrigin-RevId: 451384408
2022-05-30 16:39:01 +00:00
hschlueter
9649411a4f Use video passthrough if clipping starts at key frame.
PiperOrigin-RevId: 451380267
2022-05-30 16:37:19 +00:00
hschlueter
3e8a2f104c Expand SSIM skipping to all Nexus API 21 devices.
PiperOrigin-RevId: 451371681
2022-05-30 16:36:28 +00:00
huangdarwin
099a542dac Transformer: Skip SSIM in tests when fallback is applied.
SSIM calculation requires the input and output dimensions to be identical.

For devices that can't encode the input dimensions, skip SSIM calculations and
log the cause. Only apply this on tests where the encoder may not support the
input file dimensions.

PiperOrigin-RevId: 451364904
2022-05-30 16:35:35 +00:00
hschlueter
fe3831c5b4 Fix handling clipping in transformer renderers.
Decode-only video frames (needed when the frame at / first frame after the
clipping start is not a key frame) need to be decoded but not passed to
the frame processor chain or encoder.

The clipping start offset needs to be removed from the frame timestamps
in the passthrough and video pipelines.
There are no changes needed for this in the audio pipeline, as it doesn't
use the input timestamps -- it uses its own timestamps derived from the
buffer sizes instead.

Also add demo option to try this out.

#minor-release

PiperOrigin-RevId: 451353609
2022-05-30 16:34:38 +00:00
hschlueter
0eaf3d30c8 Rename GlFrameProcessor to SingleFrameGlTextureProcessor.
Also update names of implementations to match design doc.
In follow-ups, SingleFrameGlTextureProcessor will become
an abstract implementation of a new GlTextureProcessor
interface.

Texture processor makes sense as it processes OpenGL textures.
The term frame processor will be used for something else in
follow-ups.

PiperOrigin-RevId: 451142085
2022-05-30 16:31:13 +00:00
hschlueter
de1dceedc1 Skip SSIM calculation on Nexus 5 API 21.
There is a problem with the ImageReader formats used by the
SSIM helper that only occurs for Nexus 5 API 21, so as a workaround
we can skip the SSIM calculation on Nexus 5 API 21.

This skips just the SSIM calculation (by setting the value to
1.0 instead and logging). The tests still run when SSIM is skipped
so that we can detect other failures.

PiperOrigin-RevId: 450903183
2022-05-26 11:56:20 +00:00
hschlueter
678b6c0438 Replace 640x360 H264 test video with 320x240 H264 video.
Decoding and encoding 320x240 H264 video should be supported
on all devices from Android 5.0 based on CDD requirements.
https://source.android.com/compatibility/5.0/android-5.0-cdd#5_2_video_encoding

640x360 encoding doesn't seem to be supported on Nexus 5.

PiperOrigin-RevId: 450901715
2022-05-26 11:55:21 +00:00
hschlueter
d840aa4dc1 Skip transform4K60 test if 4K decoding is not supported.
4K decoding is not supported (not required to be supported) on all
devices, e.g., Nexus 5 does not support it.

PiperOrigin-RevId: 450682519
2022-05-26 11:50:24 +00:00