MediaCodec Class

MediaCodec class can be used to access low-level media codec, i.e. encoder/decoder components.

Syntax

[Android.Runtime.Register("android/media/MediaCodec", DoNotGenerateAcw=true)]
public sealed class MediaCodec : Java.Lang.Object

Remarks

MediaCodec class can be used to access low-level media codec, i.e. encoder/decoder components.

MediaCodec is generally used like this:

java Example

 MediaCodec codec = MediaCodec.createDecoderByType(type);
 codec.configure(format, ...);
 codec.start();

 // if API level <= 20, get input and output buffer arrays here
 ByteBuffer[] inputBuffers = codec.getInputBuffers();
 ByteBuffer[] outputBuffers = codec.getOutputBuffers();
 for (;;) {
   int inputBufferIndex = codec.dequeueInputBuffer(timeoutUs);
   if (inputBufferIndex >= 0) {
     // if API level >= 21, get input buffer here
     ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferIndex);
     // fill inputBuffers[inputBufferIndex] with valid data
     ...
     codec.queueInputBuffer(inputBufferIndex, ...);
   }

   int outputBufferIndex = codec.dequeueOutputBuffer(timeoutUs);
   if (outputBufferIndex >= 0) {
     // if API level >= 21, get output buffer here
     ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferIndex);
     // outputBuffer is ready to be processed or rendered.
     ...
     codec.releaseOutputBuffer(outputBufferIndex, ...);
   } else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
     // no needed to handle if API level >= 21 and using getOutputBuffer(int)
     outputBuffers = codec.getOutputBuffers();
   } else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
     // Subsequent data will conform to new format.
     // can ignore if API level >= 21 and using getOutputFormat(outputBufferIndex)
     MediaFormat format = codec.getOutputFormat();
     ...
   }
 }
 codec.stop();
 codec.release();
 codec = null;

Each codec maintains a number of input and output buffers that are referred to by index in API calls.

For API levels 20 and below: The contents of these buffers are represented by the ByteBuffer[] arrays accessible through MediaCodec.GetInputBuffers and MediaCodec.GetOutputBuffers.

After a successful call to MediaCodec.Start the client "owns" neither input nor output buffers, subsequent calls to MediaCodec.DequeueInputBuffer(long) and MediaCodec.DequeueOutputBuffer(.BufferInfo, System.Int64) then transfer ownership from the codec to the client.

The client is not required to resubmit/release buffers immediately to the codec, the sample code above simply does this for simplicity's sake. Nonetheless, it is possible that a codec may hold off on generating output buffers until all outstanding buffers have been released/resubmitted.

Once the client has an input buffer available it can fill it with data and submit it it to the codec via a call to MediaCodec.QueueInputBuffer(int, System.Int32, System.Int32, System.Int32, System.Int32). Do not submit multiple input buffers with the same timestamp (unless it is codec-specific data marked as such using the flag MediaCodec.BufferFlagCodecConfig).

The codec in turn will return an output buffer to the client in response to MediaCodec.DequeueOutputBuffer(.BufferInfo, System.Int64). After the output buffer has been processed a call to MediaCodec.ReleaseOutputBuffer(int, System.Boolean) will return it to the codec. If a video surface has been provided in the call to MediaCodec.Configure(MediaFormat, Android.Views.Surface, Android.Views.Surface, Android.Views.Surface), MediaCodec.ReleaseOutputBuffer(int, System.Boolean) optionally allows rendering of the buffer to the surface.

Input buffers (for decoders) and Output buffers (for encoders) contain encoded data according to the format's type. For video types this data is all the encoded data representing a single moment in time, for audio data this is slightly relaxed in that a buffer may contain multiple encoded frames of audio. In either case, buffers do not start and end on arbitrary byte boundaries, this is not a stream of bytes, it's a stream of access units.

Most formats also require the actual data to be prefixed by a number of buffers containing setup data, or codec specific data, i.e. the first few buffers submitted to the codec object after starting it must be codec specific data marked as such using the flag MediaCodec.BufferFlagCodecConfig in a call to MediaCodec.QueueInputBuffer(int, System.Int32, System.Int32, System.Int32, System.Int32).

Codec specific data included in the format passed to MediaCodec.Configure(MediaFormat, Android.Views.Surface, Android.Views.Surface, Android.Views.Surface) (in ByteBuffer entries with keys "csd-0", "csd-1", ...) is automatically submitted to the codec, this data MUST NOT be submitted explicitly by the client.

Once the client reaches the end of the input data it signals the end of the input stream by specifying a flag of MediaCodec.BufferFlagEndOfStream in the call to MediaCodec.QueueInputBuffer(int, System.Int32, System.Int32, System.Int32, System.Int32). The codec will continue to return output buffers until it eventually signals the end of the output stream by specifying the same flag (MediaCodec.BufferFlagEndOfStream) on the BufferInfo returned in MediaCodec.DequeueOutputBuffer(.BufferInfo, System.Int64). Do not submit additional input buffers after signaling the end of the input stream, unless the codec has been flushed, or stopped and restarted.

Seeking & Adaptive Playback Support

You can check if a decoder supports adaptive playback via NoType:android/media/MediaCodecInfo$CodecCapabilities;Href=../../../reference/android/media/MediaCodecInfo.CodecCapabilities.html#isFeatureSupported(java.lang.String). Adaptive playback is only supported if you configure the codec to decode onto a Android.Views.Surface.

For decoders that do not support adaptive playback (including when not decoding onto a Surface)

In order to start decoding data that's not adjacent to previously submitted data (i.e. after a seek) one mustMediaCodec.Flush the decoder. Any input or output buffers the client may own at the point of the flush are immediately revoked, i.e. after a call to MediaCodec.Flush the client does not own any buffers anymore.

It is important that the input data after a flush starts at a suitable stream boundary. The first frame must be able to be decoded completely on its own (for most codecs this means an I-frame), and that no frames should refer to frames before that first new frame. Note that the format of the data submitted after a flush must not change, flush does not support format discontinuities, for this a full MediaCodec.Stop, MediaCodec.Configure(MediaFormat, Android.Views.Surface, Android.Views.Surface, Android.Views.Surface), MediaCodec.Start cycle is necessary.

For decoders that support adaptive playback

In order to start decoding data that's not adjacent to previously submitted data (i.e. after a seek) it is not necessary to MediaCodec.Flush the decoder.

It is still important that the input data after the discontinuity starts at a suitable stream boundary (e.g. I-frame), and that no new frames refer to frames before the first frame of the new input data segment.

For some video formats it is also possible to change the picture size mid-stream. To do this for H.264, the new Sequence Parameter Set (SPS) and Picture Parameter Set (PPS) values must be packaged together with an Instantaneous Decoder Refresh (IDR) frame in a single buffer, which then can be enqueued as a regular input buffer. The client will receive an MediaCodec.InfoOutputFormatChanged return value from MediaCodec.DequeueOutputBuffer(.BufferInfo, System.Int64) or NoType:android/media/MediaCodec$Callback;Href=../../../reference/android/media/MediaCodec.Callback.html#onOutputBufferAvailable(android.media.MediaCodec, int, android.media.MediaCodec.BufferInfo) just after the picture-size change takes place and before any frames with the new size have been returned.

Be careful when calling MediaCodec.Flush shortly after you have changed the picture size. If you have not received confirmation of the picture size change, you will need to repeat the request for the new picture size. E.g. for H.264 you will need to prepend the PPS/SPS to the new IDR frame to ensure that the codec receives the picture size change request.

States and error handling

During its life, a codec conceptually exists in one of the following states: Initialized, Configured, Executing, Error, Uninitialized, (omitting transitory states between them). When created by one of the factory methods, the codec is in the Initialized state; MediaCodec.Configure(MediaFormat, Android.Views.Surface, Android.Views.Surface, Android.Views.Surface) brings it to the Configured state; MediaCodec.Start brings it to the Executing state. In the Executing state, decoding or encoding occurs through the buffer queue manipulation described above. The method MediaCodec.Stop returns the codec to the Initialized state, whereupon it may be configured again, and MediaCodec.Release brings the codec to the terminal Uninitialized state. When a codec error occurs, the codec moves to the Error state. Use MediaCodec.Reset to bring the codec back to the Initialized state, or MediaCodec.Release to move it to the Uninitialized state.

The factory methods MediaCodec.CreateByCodecName(string), MediaCodec.CreateDecoderByType(string), and MediaCodec.CreateEncoderByType(string) throw Java.IO.IOException on failure which the caller must catch or declare to pass up. MediaCodec methods throw Java.Lang.IllegalStateException when the method is called from a codec state that does not allow it; this is typically due to incorrect application API usage. Methods involving secure buffers may throw NoType:android/media/MediaCodec$CryptoException;Href=../../../reference/android/media/MediaCodec.CryptoException.html#MediaCodec.CryptoException(int, java.lang.String), which has further error information obtainable from NoType:android/media/MediaCodec$CryptoException;Href=../../../reference/android/media/MediaCodec.CryptoException.html#getErrorCode().

[Android Documentation]

Requirements

Namespace: Android.Media
Assembly: Mono.Android (in Mono.Android.dll)
Assembly Versions: 0.0.0.0
Since: Added in API level 16