Showing posts with label Android Multimedia. Show all posts
Showing posts with label Android Multimedia. Show all posts

Android Multimedia: CODECS

The word CODEC means Encoder + Decoder.


A codec is a way of compressing and decompressing digital files. Each codec uses a slightly different set of algorithms to accomplish this. 

Multimedia data is compressed or encoded by encoders and decompressed or decoded by decoders.

Traditionally a codec is a software algorithm/module runs on a computing device. Sometimes Digital Signal Processors (DSP) are used for realizing faster codec algorithms.

Software algorithms are in-adequate to handle today’s challenging system demands in terms of data rate, video resolution, sampling rate etc.

Today most of the multimedia systems use HW accelerated Codecs. HW accelerators are dedicated HW module inside a processor for faster multimedia data processing



Examples of speech codecs 
  • G723, G729 used for VOIP and video conferencing 
  • AMR-NB used in GSM & UMTS networks and systems those record voice 
  • AMR-WB, AMR-WB+ used in CDMA networks 
Examples of Audio codecs 

  • MP3 – Used in all systems those can play music 
  • AAC - Used in most of the systems those can play music & record 
  • WMA – Microsoft’s proprietary codec used in many systems 
  • RA – Real audio used by Real Networks. 
Examples of Video codecs 
  • MPEG1 – Used in Video CD
  • MPEG2 – Used in DVD 
  • MPEG4, H264 – Used in computer, handheld devices, digital TV transmission 
  • WMV - Microsoft’s proprietary codec used in many systems 
  • RV – Real video used by Real Networks 
Examples of image codecs 

  • JPEG, JPEG 2000, PNG, GIF

Audio Module Changes in Android Verions

Android Jelly Bean(4.2):

Low-latency audio:

Android 4.2 improves support for low-latency audio playback, starting from the improvements made in Android 4.1 release for audio output latency using OpenSL ES, Soundpool and tone generator APIs. These improvements depend on hardware support — devices that offer these low-latency audio features can advertise their support to apps through a hardware feature constant. New AudioManager APIs are provided to query the native audio sample rate and buffer size, for use on devices which claim this feature.

Android Jelly Bean(4.1):

Media codecs: The MediaCodec class provides access to low-level media codecs for encoding and decoding your media. You can instantiate a MediaCodec by calling createEncoderByType() to encode media or call createDecoderByType() to decode media. Each of these methods take a MIME type for the type of media you want to encode or decode, such as"video/3gpp" or "audio/vorbis".

With an instance of MediaCodec created, you can then call configure() to specify properties such as the media format or whether or not the content is encrypted.

Record audio on cue:
New method startRecording() allows you to begin audio recording based on a cue defined by a MediaSyncEvent. TheMediaSyncEvent specifies an audio session (such as one defined by MediaPlayer), which when complete, triggers the audio recorder to begin recording. For example, you can use this functionality to play an audio tone that indicates the beginning of a recording session and recording automatically begins so you don't have to manually synchronize the tone and the beginning of recording.

Timed text tracks:
The MediaPlayer now handles both in-band and out-of-band text tracks. In-band text tracks come as a text track within an MP4 or 3GPP media source. Out-of-band text tracks can be added as an external text source viaaddTimedTextSource() method. After all external text track sources are added, getTrackInfo() should be called to get the refreshed list of all available tracks in a data source.

To set the track to use with the MediaPlayer, you must call selectTrack(), using the index position for the track you want to use.


Audio effects:
The AudioEffect class now supports additional audio pre-processing types when capturing audio:
  • Acoustic Echo Canceler (AEC) with AcousticEchoCanceler removes the contribution of the signal received from the remote party from the captured audio signal.
  • Automatic Gain Control (AGC) with AutomaticGainControl automatically normalizes the output of the captured signal.
  • Noise Suppressor (NS) with NoiseSuppressor removes background noise from the captured signal.
You can apply these pre-processor effects on audio captured with an AudioRecord using one of the AudioEffect subclasses.

Gapless playback:
You can now perform gapless playback between two separate MediaPlayer objects. At any time before your first MediaPlayer finishes, call setNextMediaPlayer() and Android attempts to start the second player the moment that the first one stops.

Android Ice Cream Sandwich(4.0):

Low-level streaming multimedia:
Android 4.0 provides a direct, efficient path for low-level streaming multimedia. The new path is ideal for applications that need to maintain complete control over media data before passing it to the platform for presentation. For example, media applications can now retrieve data from any source, apply proprietary encryption/decryption, and then send the data to the platform for display.

To support this low-level streaming, the platform introduces a new native API based on Khronos OpenMAX AL 1.0.1.

Audio remote controls:Android 4.0 adds a new audio remote control API that lets media applications integrate with playback controls that are displayed in a remote view. Media applications can integrate with a remote music playback control that’s built into in the platform’s lock screen, allowing users to control song selection and playback without having to unlock and navigate to the music app.

Using the audio remote control API, any music or media app can register to receive media button events from the remote control and then manage play state accordingly. The application can also supply metadata to the remote control, such as album art or image, play state, track number and description, duration, genre, and more.

New media codecs and containers:
Android 4.0 adds support for additional media types and containers to give developers access to the formats they need. For high-quality compressed images, the media framework adds support for WebP content. For video, the framework now supports streaming VP8 content. For streaming multimedia, the framework supports HTTP Live streaming protocol version 3 and encoding of ADTS-contained AAC content. Additionally, developers can now use Matroska containers for Vorbis and VP8 content.

Android Multimedia: Media Player States





  • Playback control of audio/video files and streams is managed as a state machine. 
  • The following diagram shows the life cycle and the states of a Media Player object driven by the supported playback control operations. 
  • Media Player has several internal states and can be changed by different operations. 
  • Media Player is a controller class that has a lot of low level function calls which directly invoke the functions in the native media framework library; it can handle not only video but audio content as well;
Player States:
  • Idle: When the Media Player object is created using new or after reset () is called, it is in idle state. Call the Media Player constructor to create the object; the Media Player object is in idle state when created; 
                         a) MediaPlayer mp = new MediaPlayer (); 
                              It creates new MediaPlayer object, and we can interact with the player object. 
                             After we need to initialize the player with SetDataSource (...) method. 
                         b) public static MediaPlayer create (Context context, Uri uri): 
                              It creates a MediaPlayer for a given Uri. On success the prepare () also called 
                              no need to call prepare again. 
                              After successful of create we can give command play ().                                
                              Here the argument uri is the Uri from which to get the datasource. 
                         c) public static MediaPlayer create (Context context, int resid) 
                             It create a MediaPlayer for a given resource id, on success prepare () also 
                             called, no need to call prepare again. After successful of create we can give 
                             command play (). 
                             Here the resid is the resource to use as the datasource                         
                         d) public static MediaPlayer create (Context context, Uri uri, SurfaceHolder 
                             holder) 
                             It create a MediaPlayer for a given Uri, and uses the holder to render the 
                             video, onsuccess it calls the prepare() method. After successful of create we 
                             can give command play (). 
                    Note: After b,c,d calls the player in “Prepared State”, we can give play command 
                              after successful completion of these methods. 


  • End: After release () is called it is in End state.                                                                         After calling release on MediaPlayer it will be in End state. After release, the object is no longer available. We can call this (release) method in any state of the player. 
  • Initialized: calling setDatsource () method transfers a Media player objects in the idle state to initialized state. 
          Media Player class has several setDataSource () signatures to accept different content 
          type than URI. After this call, the Media Player object is in “Initialized” state; 

          After SetDataSource we can prepare the player in two ways 
                   1) Prepare(): If it is a local file or raw file with the project resource then we can give 
                        prepare command. We can call prepare on player if player in initialized or 
                        Stopped state only. 
                   2) PrepareAsync (): If it is a URL and we need to stream the data from network 
                        and play that data in that case we need to give prepareAsync () command for non 
                        blocking functionality. We can call prepare on player if player in initialized or 
                        Stopped state only. 


  • Preparing: If we call the prepareAsync (), now the player is in preparing state. After the required buffer gathered with it will go to prepared state and it will call the onPrepared listener attached to the player object. And the player currently in Prepared State. 
  • Prepared: If we call prepare (), player is in Prepared state. 
           Media Player object must first enter the Prepared state before playback can be started. 
           In this state, the Media Player object can be operated to play, pause and stop, etc. 

  • Playing: The playback starts after start () of the Media Player object is invoked inside of the onPrepared () callback function. The Media Player object is at the “Started” state now and can be paused and stopped. 
  • Stop: Call stop () of Media Player to stop the playback. This sets the object to the “Stopped” state. Once in the “Stopped” state, the Media Player must be prepared again in order to play. 
          We can give stop command to the player if the player current state is Prepared, Started, 
          Stopped, Paused, and Playback Completed. 
  • Paused: We can give the pause command on the player if its current state is Started or Paused. 
  • Playback Complete: If the Player completes the playing of the given file, the player current state is PlaybackCompleted. In this state we can restart the player using start () command. 
  • Error: If any errors occurred while preparing or playing the player state will be in Error state. It will call the OnError listener attached to the player.

OpenMax

OpenMAX (Open Media Acceleration) is a royalty-free, cross-platform set of C-language programming interfaces that provides abstractions for routines especially useful for audio, video, and still images. It's intended for devices that process large amounts of multimedia data in predictable ways.

OpenMAX provides three layers of interfaces: 

                 Application Layer (AL)
                 Integration Layer (IL) 
                 Development Layer (DL). 

OpenMAX is managed by the non-profit technology consortium Khronos Group.

1) OpenMax Al interface between Applications like Media Player and Media framework.
2) OpenMax IL interface between Media framework and Codec’s.
3) OpenMax DL interface between Codec’s and Hardware.

OpenMax IL:
The OpenMax IL (Integration Layer) API defines a standardized media component interface to enable developers and platform providers to integrate and communicate with multimedia codec’s implemented in hardware or software”.


The Open MAX IL API allows the user to load, control, connect, and unload the individual components.

OpenMax IL Features:

  • A flexible component-based API core 
  • Ability to easily plug in new codec’s 
  • Capable of being implemented as either static or dynamic libraries 
  • Ease of communication between the client and the codec’s and between codec’s themselves
OpenMax IL Use:
           Open MAX IL Components are used like any typical Multimedia features such as playing video, recording audio and capturing images. For example, two Open MAX IL Components, decoder and render, are loaded by Open MAX IL and communicate with each other for decoding and rendering.

In the OpenMax IL, components represent individual blocks of functionality. Components can be sources, sinks, codec’s, filters, splitters, mixers, or any other data operator.
  • An IL client always communicates with a component via the IL core.
  • The IL client uses the Open MAX core for loading and unloading components, setting up direct communication between two Open MAX components, and accessing the component’s method functions. 
  • IL Client: MDF (Multimedia Device Framework) plays the role of the Open MAX IL client within the Open MAX IL architecture. 
  • IL core: Open MAX IL Core is used to load and unload the Open MAX IL Components and to facilitate communication between them. Open MAX IL Core is a platform-specific entity which is used to load and unload Open MAX IL Components, and to facilitate communication between them.
OMX Components handle data communication through ports. A port is an interface that represents: the connection to other components, the type of streamed data and the buffers needed to maintain such connection. In Open MAX a buffer is an entity that holds the information and represents the minimum unit of data that can be exchanged between two OMX Components. One OMX Component sends/receives buffers to/from other OMX Components through output/input ports.

OpenMax IL Profiles:Base Profile: support non tunneled communication
Interop Profile: support non tunneled communication and tunneled communication.

Non tunneled communication: exchanging data buffers b/w IL client and component.
Tunneled communication: components to exchange data buffers directly with each other.
Proprietary Communication:Third party based

OpenMax IL Component States:
  • WAIT FOR RESOURCES 
  • IDLE 
  • EXECUTING 
  • PAUSED 
  • INVALID 
  • LOADED 
  • UNLOADED

Non Tunnel Call Sequences:
1) Non tunnel initialization:
  • IL client shall call OMX_getHandle function, which activates actual component creation by core. 
  • The core passes IL client callback functions to the component by means of the Set Callbacks. 
  • If successful, OMX will be in OMX_StateLoaded state. 
  • IL Client configures the component and its ports using OMX_SetParameters. 
  • After configuration, It can request the component to make the state transition to OMX_StateIdle. After that only client can set up the buffers to component to use all of its ports. 
  • IL Client use OMX_AllocateBuffer and OMX_UseBuffer to set up buffers. 
  • If OMX_UseBuffer is used, the IL client shall have allocated buffer and passed it to component. 
  • IL client may ask component to allocate buffer and buffer header using OMX_AllocateBuffer. 
2) Non tunnel Data Flow:
     IL Client entirely responsible for moving data buffers among components if data tunneling is not used.
  • IL Client delivers data buffers to component using OMX_EmptyThisBuffer call. 
  • As soon as one buffer is available from the component output port, component shall send OMX_FillBufferDone call back. 
3) Non tunneled De Initialization:
  • First switch the components to the OMX_StateIdle state so that all buffers are returned to their suppliers. 
  • The component to change its state to OMX_StateLoaded.                                                                    The IL client shall free all of the component’s buffers by calling OMX_FreeBuffer for each buffer.                                                                                                                            OMX_FreeBuffer : free port buffer headers 
  • When all of the buffers have been freed, the component shall complete the state transition. 
  • The IL client calls the OMX_FreeHandle function that disposes of the component.