311 Commits

Author SHA1 Message Date
hschlueter
63dcdf5803 Add listener for FrameProcessingExceptions.
This listener replaces
FrameProcessorChain#getAndRethrowBackgroundExceptions.
The listener uses a new exception type FrameProcessingException
separate from TransformationException as the frame processing
components will be made reusable outside of transformer soon.

PiperOrigin-RevId: 447455746
2022-05-10 17:47:22 +01:00
hschlueter
2e544224c2 Fix GlUtil vector size constant name.
This constant is used for https://docs.gl/es2/glVertexAttribPointer
which takes the number of components per generic vertex attribute
(meaning the size of the individual coordinate vectors here) not the
number of attributes (the number of vertices that the old constant
name referred to).

PiperOrigin-RevId: 447427241
2022-05-10 17:42:39 +01:00
samrobinson
ff7629aba5 Match the suggested bitrate to the actual Kush Gauge formula.
As defined in Kush Amerasinghe's paper 'H.264 For the rest of us'.

PiperOrigin-RevId: 446988272
2022-05-09 11:05:43 +01:00
huangdarwin
e60ba54ddf Transformer GL: Remove obsolete TODOs.
We won't try to provide/rethrow helpful error messages that are already
provided by GL, as this sort of task would expand into writing a GL verifier.
A Gl verifier is unnecessarily complex for minimal payoff, especially as Apps
expected to read GL error messages would mostly be those writing custom
GlFrameProcessors, who should be already be familiar with reading GL error
messages anyways.

PiperOrigin-RevId: 446950837
2022-05-09 11:04:05 +01:00
hschlueter
e9919f6da1 Remove redundant attribute from matrix transformation vertex shader.
The texture coordinates can be derived from the frame coordinates.

PiperOrigin-RevId: 446770538
2022-05-09 11:01:33 +01:00
hschlueter
4b8ff72d30 Use MatrixTransformation instead of wrapping its GlFrameProcssor.
ScaleToFitFrameProcessor, PresentationFrameProcessor,
and EncoderCompatibilityFrameProcessor now each implement
MatrixTransformation instead of wrapping
MatrixTransformationFrameProcessor.

PiperOrigin-RevId: 446480286
2022-05-09 10:58:56 +01:00
hschlueter
c94035278c Separate matrix effect specification and implementation.
This change splits AdvancedFrameProcessor into 4 files:
- MatrixTransformationFrameProcessor for the GlFrameProcessor
  implementation
- MatrixTransformation and GlMatrixTransformation for the GlEffect
  specification
- MatrixUtils for the static matrix helpers

PiperOrigin-RevId: 446236384
2022-05-09 10:51:14 +01:00
hschlueter
b410a922fe Introduce GlEffect interface for effect specification.
PiperOrigin-RevId: 446143537
2022-05-09 10:44:30 +01:00
hschlueter
3869c0a851 Return recommended output size for FrameProcessorChain.
Also add warning for what can happen if this is ignored.

Previously the return value was overridden by setOutputSurface, now
setOutputSurface only overrides the values using internally.

PiperOrigin-RevId: 445377036
2022-05-09 10:38:32 +01:00
huangdarwin
0de9955eb2 Transformer: Misc nits
* Add a /* paramName= */ comment
* Remove an unnecessary comma.
* Remove extra "internal " for a b/### link

PiperOrigin-RevId: 445169649
2022-05-09 10:30:18 +01:00
claincly
c82cbc094f Change video asset used in TranscodeQualityTest
The old video has a `irregular' resolution of 1080x720. Change to a 1920x1080
video.

PiperOrigin-RevId: 445124065
2022-05-09 10:28:43 +01:00
huangdarwin
0f3e63909c Transformer: Restrict frame buffer size for all omx decoders API >= 29.
PiperOrigin-RevId: 445119411
2022-05-09 10:27:58 +01:00
claincly
564dc11f28 Add device capability based test skipping.
Change test name prefix from `transform` to `transcode` for consistency.

PiperOrigin-RevId: 445106224
2022-05-09 10:25:07 +01:00
huangdarwin
f269963e89 Transformer GL: Add support for pixelWidthHeightRatio.
To ensure frame processor operations operate on square pixels,
make the frame taller or wider for non-square input pixels.

In addition to automated tests, this was tested by changing the
inputFormat.pixelWidthHeightRatio in the TransformerVideoRenderer.

PiperOrigin-RevId: 444553517
2022-05-09 10:11:18 +01:00
huangdarwin
22c9b8168d Transformer: remove unused bitmap test option.
Remove unused saveTestBitmapToCacheDirectory throwOnFailure option.

#cleanup

PiperOrigin-RevId: 444516857
2022-04-26 12:05:30 +01:00
huangdarwin
3cf7f58edf Transformer GL: save actual bitmap before checking pixel diffs.
This allows the actual bitmap to be saved, even if output
dimensions are different than expected. Otherwise, differing
output dimensions would throw an exception, preventing the bitmap
from being saved.

PiperOrigin-RevId: 444512210
2022-04-26 11:37:03 +01:00
samrobinson
c87d03e78e Add 30 second high motion clips to AndroidTestUtil.
30 second clips at higher resolutions are useful for testing encoder
behaviour compared to shorter clips.

PiperOrigin-RevId: 444505865
2022-04-26 10:59:07 +01:00
huangdarwin
f1cda5fe2a FrameProcessor: Avoid early rounding in Presentation output dimensions.
PiperOrigin-RevId: 444253425
2022-04-25 14:38:17 +01:00
hschlueter
02743456a0 Use last frame processor to draw on debug surface view.
The last frame processor could use a different drawing command
than a four vertex triangle strip. So we need call its drawing
method again instead of assuming the shape.

PiperOrigin-RevId: 444217274
2022-04-26 15:04:33 +01:00
huangdarwin
36d190bb66 Transformer GL: Avoid no-op ScaleToFit / Presentation FrameProcessors
Avoid no-op FrameProcessors in VideoTranscodingSamplePipeline, to avoid
creating resources for and executing GL for a no-op vertex+fragment shader.

EncoderCompatibilityFrameProcessor will still always be exercised.

PiperOrigin-RevId: 443675833
2022-04-26 15:03:22 +01:00
andrewlewis
c8744ee9bd Add an SSIM test with 360p input
This test should run on all devices from API 21 (the media uses Baseline
profile level 3.0 H.264) to give us coverage of the full pipeline (forcing
re-encoding) and SSIM calculation on all devices.

PiperOrigin-RevId: 443650002
2022-04-26 15:00:34 +01:00
andrewlewis
0a378a4555 Fix encoder configuration before API 25
Float I-frame intervals are only supported from API 25 onwards (see https://developer.android.com/reference/android/media/MediaFormat#KEY_I_FRAME_INTERVAL).

Setting a null value for profile/level can cause a native crash when configuring the decoder.

PiperOrigin-RevId: 443647924
2022-04-26 14:59:44 +01:00
claincly
f9bce2a6b6 Remove a completed TODO item
PiperOrigin-RevId: 443620931
2022-04-26 14:58:37 +01:00
huangdarwin
3cfdfb41c6 FrameProcessor: Create EncoderCompatibilityFrameProcessor.
Split rotationDegrees changes to EncoderCompatibilityFrameProcessor, a new
FrameProcessor.

This removes automatic rotationDegrees adjustments from Presentation, which
allows Presentation to be used for changes before the end of a
FrameProcessorChain pipeline.

PiperOrigin-RevId: 443387226
2022-04-26 14:57:40 +01:00
andrewlewis
27c1876c9f Expand Samsung/1+ workaround to API 31 + 32
The issue with decoding getting stuck is still reproducible on `samsung/beyond1lteeea/beyond1:12/SP1A.210812.016/G973FXXUEGULB:user/release-keys`.

PiperOrigin-RevId: 443368546
2022-04-26 14:55:39 +01:00
hschlueter
6939464f91 Don't use API 26 SurfaceTexture constructor in frame processor tests.
After this change the test will use EGL_NO_SURFACE or a pixel buffer surface if using no surface is not supported.

PiperOrigin-RevId: 443113794
2022-04-26 14:53:31 +01:00
claincly
f903e4a9f8 Apply resolution fix.
Some devices under report their resolution support, like 2144 for 2160 in
H265, 1072 for 1080 in H264. This CL only takes care of these two cases,

- reporting 1920x1080 is supported when the device reports 1920x1072, and
- reporting 3840x2160 is supported when the device reports 3840x2144

PiperOrigin-RevId: 443095042
2022-04-26 14:50:35 +01:00
samrobinson
d1f8e96d9c Add 4 different video resolutions for bitrate analysis.
PiperOrigin-RevId: 442844189
2022-04-26 14:48:20 +01:00
huangdarwin
7f7e84bee3 FrameProcessor: Improve PresentationFrameProcessor javadocs.
PiperOrigin-RevId: 442835160
2022-04-26 14:47:21 +01:00
hschlueter
5248bbb882 Add context param to GlFrameProcessor initialize().
PiperOrigin-RevId: 442826391
2022-04-26 14:46:24 +01:00
hschlueter
afa551ab8f Block until FrameProcessorChain is released.
This is safer because it will prevent any future problems with
creating a new FrameProcessorChain before the previous one has
completed its async release.

From [eglDestroyContext documentation](https://www.khronos.org/registry/EGL/sdk/docs/man/html/eglDestroyContext.xhtml):
"If the EGL rendering context is not current to any thread,
eglDestroyContext destroys it immediately."
The context isn't current to any thread here because GlUtil calls
eglMakeCurrent with EGL_NO_CONTEXT before calling eglDestroyContext.
So everthing should be released once the FrameProcessorChain's
release task terminates.

PiperOrigin-RevId: 442807484
2022-04-26 14:45:31 +01:00
claincly
7a67609561 Add checks for device capability in tests.
PiperOrigin-RevId: 442751310
2022-04-26 14:42:31 +01:00
huangdarwin
3f4d03e731 FrameProcessor: Add checks to ensure width and height are positive.
Negative and zero values should be disallowed.

PiperOrigin-RevId: 441757246
2022-04-26 14:39:29 +01:00
huangdarwin
29e32540de FrameProcessor: Move test bitmaps to a separate directory.
* Group what's now many related test PNGs by moving them to their own directory.
* Move bitmap references to files where they're used, as each bitmap is only
used once each, except the original bitmap.

PiperOrigin-RevId: 441485489
2022-04-26 14:36:31 +01:00
hschlueter
22e016e851 Clarify TransformationRequest setResolution javadoc.
Scaling and rotation using ScaleToFitFrameProcessor may change the
the resolution and aspect ratio, so defaulting to the same as the
input is no longer accurate.

PiperOrigin-RevId: 441463349
2022-04-26 14:35:12 +01:00
hschlueter
f25c912c72 Rename GlFrameProcessor updateProgramAndDraw to drawFrame.
What a minimal implementation should include is now explained in the
interface javadoc while the method name reflects what the method does.

PiperOrigin-RevId: 441432059
2022-04-26 14:34:11 +01:00
huangdarwin
34f014ce5c FrameProcessor: Add aspect ratio changes to Presentation.
PiperOrigin-RevId: 441250773
2022-04-26 14:31:14 +01:00
olly
83daa052cb Rename TracksInfo and TrackGroupInfo
1. TracksInfo is renamed to Tracks
2. TracksInfo.TrackGroupInfo is renamed to Tracks.Group

PiperOrigin-RevId: 441232373
2022-04-26 14:30:14 +01:00
hschlueter
af8435816a Make TransformerEndToEndTest constants private.
PiperOrigin-RevId: 441230941
2022-04-26 14:29:19 +01:00
samrobinson
36c5b9407c Remove transformer from inputValues json.
Transformer resolves to null when wrapped, this is a debug line
that was not removed.

PiperOrigin-RevId: 441200790
2022-04-26 14:28:13 +01:00
hschlueter
9b4aba33a3 Fix FrameProcessorChain method ordering.
signalEndOfInputStream makes more sense before isEnded and
getPendingInputFrameCount is related to registerInputFrame.

PiperOrigin-RevId: 441134418
2022-04-26 14:26:17 +01:00
samrobinson
0a16813f1c Allow suppression of AssertionError in TransformerAndroidTestRunner.
PiperOrigin-RevId: 440952335
2022-04-26 14:25:15 +01:00
samrobinson
8521d48f86 Fix incorrect test id.
PiperOrigin-RevId: 440906834
2022-04-26 14:24:21 +01:00
claincly
803cf64a06 Removing using the test asset that has Mp3 audio track.
The stream with the mp3 track is added because

- We only encode to AAC
- We only encode when the source track is not AAC

Now that we have a way to force encoding, we no longer need the mp3 track.

The test asset is kept for later parameterized testing.

PiperOrigin-RevId: 440876080
2022-04-26 14:23:26 +01:00
hschlueter
20547c3392 Add WorkerThread annotation to FrameProcessorChain helpers.
PiperOrigin-RevId: 440347550
2022-04-26 14:20:31 +01:00
huangdarwin
8a7fcb4eae FrameProcessor: Add setCrop to Presentation.
PiperOrigin-RevId: 440325693
2022-04-26 14:19:38 +01:00
claincly
1ca80741e5 Change SSIM transcode MIME type to H264.
The MIME type was set to H265 to force transcoding. Now that we have an encoder
factory that forces encoding, switching back to H264 ensures the quality test
is conducted on more devices (those don't support H265 can be tested now).

However, H265 should be part of the quality test after we have proper mechanism
to skip test based on device capability.

PiperOrigin-RevId: 440132471
2022-04-26 14:18:35 +01:00
claincly
7d0034748f Add trying 1/3 resolution in fallback.
1/4 is probably unnecessary, as 1/3 of 4k is 720, which should be supported on
all practical encoders.

PiperOrigin-RevId: 440055144
2022-04-26 14:14:33 +01:00
claincly
a2d92ec623 Change SSIM test threshold from .95 to .90.
We don't currently have enough understanding of the correlation between a
specific SSIM score and video quality. Dropping to .90 to make most tests pass.
Especially when there's no discernible difference from the videos with .9 and
.95 SSIM.

PiperOrigin-RevId: 440047551
2022-04-26 14:13:31 +01:00
claincly
1da7fd8956 Add test to query device capabilities.
PiperOrigin-RevId: 439861685
2022-04-26 14:11:27 +01:00