Mpeg-4 streaming system with adaptive error concealment

This is a division of U.S. application Ser. No. 10/990,818, filed Nov. 16, 2004, which is incorporated herewith by reference.

The present invention generally relates to an MPEG-4 streaming system, and more specifically to an MPEG-4 streaming system with adaptive error concealment scheme to improve the overall quality of the transmitted video contents over error prone environment.

It has been a constant challenge for the research community and the industry to search for a better service quality for video streaming over the error-prone environment such as Internet, as the video bitstreams may be corrupted by random error or suffer packet loss in the channels.

To address the aforementioned problem, the MPEG-4 video coding standard is developed to provide users a new level of performance for various video communication services, such as video-on-demand (VOD) over the Internet or mobile multimedia applications. An MPEG-4 video system uses a robust encoded bitstream and a resilient decoding process. The robust encoded bitstream is used in the encoder to help, with some coding overhead, the recovery from error corruption. One of the methods for creating a robust bitstream is to insert additional intra blocks to stop error propagation in decoder. But the insertion of intra blocks will slightly decrease coding efficiency. Thus, the trade-off of the error propagation and coding efficiency must be built to achieve a good performance for MPEG-4 video encoders.

Cote, Shirani and Kossentini proposed an adaptive intra refreshment (IR) scheme for H.263 under the consideration of rate distortion optimization (IEEE Journal on Selected Areas in Communications, vol. 18, pp. 952-965, No. 6, 2002). The rate distortion optimization is to improve the timing of intra block insertion to achieve the optimized usage of IR based on the Internet conditions.

Another method is to use an error resilient decoding process, which can locate errors and then conceal the lost slices. The error location methods utilize useful header information available at the decoder for coding process resynchronization. For error resilience, MPEG-4 provides several tools, including the resynchronization marker (RM), the data partition (DP), and the reverse variable length coding (RVLC). The optimal usage of the error resilient tools is not specified in the video specification. To further enhance the error-resilient ability, the selection of the optimal parameters, intra refreshment, advanced error detection and concealment methods are required to improve the reconstructed video quality.

Several error concealment methods are developed for either spatial error concealment (SEC) or temporal error concealment (TEC). The SEC techniques exploit the spatial redundancy within a picture, while the TEC techniques exploit the temporal similarity of frames in a sequence. For spatial error concealment, various interpolation methods, such as multi-directional interpolation (Valente, et al., IEEE Transaction On Consumer Electronics, vol. 147, No. 3, 2001), and quadri-linear interpolation (Kwok, et. al., IEEE Transaction On Consumer Electronics, vol. 39, No. 3, 1993), are developed in addition to the widely used bi-linear interpolation (Kaiser, et. al., Signal Processing: Image Communication, vol. 14, No. 6-8, 1999). The multi-directional interpolation needs all neighboring macro blocks (MB) to correctly decide the edge direction in the lost MB and requires much more computational complexity. The quadri-linear interpolation is an area-based interpolation which takes the nearest four pixels to interpolate the recovered pixel. Two refinements are introduced by Kwok et. al. One is to increase the weight of nearer direction and the other is to take average of nearest pixels and their neighboring two pixels instead of nearest pixels only. The refinements will make the visual quality smoother.

For temporal error concealment, blind selection of motion vector such as mean, medium, nearest motion vector of surrounding motion vectors have been used. Boundary matching algorithm (BMA) is the most common method that uses the boundary properties to choose a best motion vector. There are two kinds of BMA. One is using boundary gradient to choose a result which makes the boundary match between lost MB and its neighbors. This method can be called a spatial BMA because it uses the spatial boundary correlation. The other BMA method is using boundary difference between the current frame and the previous frame. This method can be called a temporal BMA because it uses the temporal boundary correlation. Other temporal concealment method, such as decoder motion vector estimation (DMVE), uses search range and surrounding area to find a best motion vector according to temporal BMA or uses search range to refine the best motion vector of neighbors. It is obvious that the DMVE costs much more computational complexity due to testing more motion vectors and surrounding lines used for motion estimation.

As spatial concealment is suitable for the area in which spatial correlation is higher than temporal correlation, and temporal concealment is suitable for the area in which temporal correlation is higher than spatial correlation, several hybrid error concealment methods are developed to take advantages of their respective strength. A general hybrid scheme is that spatial concealment is used for I-VOP and temporal concealment is used for P-VOP. Further refinement strategies are also developed to improve the performance of the hybrid concealment methods. For example, the majority of I-VOPs excluding the first VOP have temporal correlation; thus, the temporal methods are used to conceal the VOP. For pictures having conditions, such as scene change, fad in, or fad out, and less temporal correlation, the spatial methods are used to conceal the VOP. The approach proposed by Kraiser et. al. uses spatial activity and temporal activity to decide the use of spatial concealment or temporal concealment. Spatial activity is calculated by computing the variance of nearest neighboring macro-block. Temporal activity is calculated by computing the mean square error between co-located macro-blocks. When the temporal activity is larger than spatial activity, spatial concealment is used, and vice versa. Other approaches use the boundary smoothness property. The ratio of boundary gradient of lost macro-block to boundary gradient of above and below macro-blocks is used to decide if the boundary gradient of lost macro-block is too large and requires the use of spatial concealment instead of temporal method.

However, as more and more applications and activities are brought to the Internet, the competition for bandwidth and the fluctuation of the bandwidth availability is more severe than before. It is, therefore, necessary to device an MPEG-4 streaming system with adaptive error concealment capability in order to deliver performance to the video services.

The present invention has been made to overcome the aforementioned drawback of conventional techniques used in MPEG-4 delivery in an error-prone environment. The primary object of the present invention is to provide an MPEG-4 system with error concealment for video service under the network with packet loss.

The second object of the present invention is to provide an encoder for use in an MPEG-4 video streaming system. The encoder uses an intra-refreshment technique is used to make coded bitstream more robust against noise in order to stop error propagation. The rate-distortion optimization criterion is also introduced to adaptively update in synchronization with intra-coded blocks adaptively based on the true network condition with minimal overhead. The Lagrange multiplier is modified to achieve the best rate distortion balance. In addition, a decoder loop is used in the encoder and is synchronized with the true decoder to achieve the best performance and avoid mismatch with the decoder used in the MPEG-4 system.

The third object of the present invention is to provide a decoder which is able to achieve resilient decoding from any kind of noise and enhance the reconstructed image quality with spatial and temporal hybrid concealment method. The result shows that a 3.65-9.71 dB further improvement on peak-signal-to-noise-ratio (PSNR) can be achieved in comparison with the existing methods that adopt spatial copy and zero motion concealment in decoding.

The fourth object of the present invention is to provide a rate distortion optimized intra-refresh (RDIR) method for improving the bit-stream structure according to the network condition to an encoder system with least overhead.

The fifth object of the present invention is to provide an error concealment method combining hybrid concealment scheme and block-based refinement.

The foregoing and other objects, features, aspects and advantages of the present invention will become better understood from a careful reading of a detailed description provided herein below with appropriate reference to the accompanying drawings.