Methods and apparatus for use in multi-view video coding   

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser. No. 60/830,206, filed 11 Jul. 2006, which is incorporated by reference herein in its entirety.

TECHNICAL FIELD

The present principles relate generally to video encoding and decoding and, more particularly, to methods and apparatus for use in Multi-view Video Coding (MVC).

In the current implementation of Multi-view Video Coding (MVC) compliant with the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group-4 (MPEG-4) Part 10 Advanced Video Coding (AVC) standard/International Telecommunication Union, Telecommunication Sector (ITU-T) H.264 recommendation (hereinafter the “MPEG-4 AVC standard”), there is no provision to identify a specific view and to signal the camera parameters. This view information is needed for several reasons. View scalability, view random access, parallel processing, view generation, and view synthesis are all Multi-view Video Coding requirements which utilize the view id information. Moreover, several of these requirements also utilize camera parameters which are currently not passed in a standardized way.

In a first prior art approach, a method is proposed to enable efficient random access in multi-view compressed bit streams. In the proposed method, a new V-picture type and a new View Dependency SEI message are defined. A feature required in the proposed V-picture type is that V-pictures shall have no temporal dependence on other pictures in the same camera and may only be predicted from pictures in other cameras at the same time. The proposed View Dependency SEI message will describe exactly which views a V-picture, as well as the preceding and following sequence of pictures, may depend on. The following are the details of the proposed changes.

With respect to V-Picture syntax and semantics, a particular syntax table relating to the MPEG-4 AVC standard is extended to include a Network Abstraction Layer (NAL) unit type of 14 corresponding to a V-picture. Also, the V-picture type is defined to have the following semantics:

V-picture: A coded picture in which all slices reference only slices with the same temporal index (i.e., only slices in other views and not slices in the current view). When a V-picture would be output or displayed, it also causes the decoding process to mark all pictures from the same view which are not IDR-pictures or V-pictures and which precede the V-picture in output order to be marked as “unused for reference”. Each V-picture shall be associated with a View Dependency SEI message occurring in the same NAL.

With respect to the view dependency Supplemental Enhancement Information message syntax and semantics, a View Dependency Supplemental Enhancement Information message is defined with the following syntax:

view_dependency ( payloadSize ) {  num_seq_reference_views ue(v)  seq_reference_view_0 ue(v)  seq_reference_view_1 ue(v)  ...  seq_reference_view_N ue(v)  num_pic_reference_views ue(v)  pic_reference_view_0 ue(v)  pic_reference_view_1 ue(v)  ...  pic_reference_view_N ue(v) } where num_seq_reference_views/num_pic_reference_views denotes the number of potential views that can be used as a reference for the current sequence/picture, and seq_reference_view_i/pic_reference_view_i denotes the view number for the ith reference view.

The picture associated with a View Dependency Supplemental Enhancement Information message shall only reference the specified views described by pic_reference_view_i. Similarly, all subsequent pictures in output order of that view until the next View Dependency Supplemental Enhancement Information message in that view shall only reference the specified views described by seq_reference_view_i.

A View Dependency Supplemental Enhancement Information message shall be associated with each Instantaneous Decoding Refresh (IDR) picture and V-picture.

The first prior art method has the advantage of handling cases where the base view can change over time, but it requires additional buffering of the pictures before deciding which pictures to discard. Moreover, the first prior art method has the disadvantage of having a recursive process to determine the dependency.

SUMMARY

These and other drawbacks and disadvantages of the prior art are addressed by the present principles, which are directed to methods and apparatus for use in Multi-view Video Coding (MVC).

According to an aspect of the present principles, there is provided an apparatus. The apparatus includes an encoder for encoding anchor and non-anchor pictures for at least two views corresponding to multi-view video content. A dependency structure of each non-anchor picture in a set of non-anchor pictures disposed between a previous anchor picture and a next anchor picture in display order in at least one of the at least two views is the same as the previous anchor picture or the next anchor picture in display order.

According to another aspect of the present principles, there is provided a method. The method includes encoding anchor and non-anchor pictures for at least two views corresponding to multi-view video content. A dependency structure of each non-anchor picture in a set of non-anchor pictures disposed between a previous anchor picture and a next anchor picture in display order in at least one of the at least two views is the same as the previous anchor picture or the next anchor picture in display order.

According to yet another aspect of the present principles, there is provided an apparatus. The apparatus includes a decoder for decoding anchor and non-anchor pictures for at least two views corresponding to multi-view video content. A dependency structure of each non-anchor picture in a set of non-anchor pictures disposed between a previous anchor picture and a next anchor picture in display order in at least one of the at least two views is the same as the previous anchor picture or the next anchor picture in display order.

According to still yet another aspect of the present principles, there is provided a method. The method includes decoding anchor and non-anchor pictures for at least two views corresponding to multi-view video content. A dependency structure of each non-anchor picture in a set of non-anchor pictures disposed between a previous anchor picture and a next anchor picture in display order in at least one of the at least two views is the same as the previous anchor picture or the next anchor picture in display order.

According to a further aspect of the present principles, there is provided an apparatus. The apparatus includes a decoder for decoding at least two views corresponding to multi-view video content from a bitstream. At least two Groups of Pictures corresponding to one or more of the at least two views have a different dependency structure. The decoder selects pictures in the at least two views that are required to be decoded for a random access of at least one of the at least two views based upon at least one dependency map.

According to a yet further aspect of the present principles, there is provided a method. The method includes decoding at least two views corresponding to multi-view video content from a bitstream. At least two Groups of Pictures corresponding to one or more of the at least two views have a different dependency structure. The decoding step selects pictures in the at least two views that are required to be decoded for a random access of at least one of the at least two views based upon at least one dependency map.

These and other aspects, features and advantages of the present principles will become apparent from the following detailed description of exemplary embodiments, which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present principles may be better understood in accordance with the following exemplary figures, in which:

FIG. 1 is a block diagram for an exemplary Multi-view Video Coding (MVC) encoder to which the present principles may be applied, in accordance with an embodiment of the present principles;

FIG. 2 is a block diagram for an exemplary Multi-view Video Coding (MVC) decoder to which the present principles may be applied, in accordance with an embodiment of the present principles;

FIG. 3 is a diagram for an inter-view-temporal prediction structure based on the MPEG-4 AVC standard, using hierarchical B pictures, in accordance with an embodiment of the present principles;

FIG. 4 is a flow diagram for an exemplary method for encoding multiple views of multi-view video content, in accordance with an embodiment of the present principles;

FIG. 5 is a flow diagram for an exemplary method for decoding multiple views of multi-view video content, in accordance with an embodiment of the present principles;

FIG. 6A is a diagram illustrating an exemplary dependency change in non-anchor frames that have the same dependency as a later anchor slot, to which the present principles may be applied, in accordance with an embodiment of the present principles;

FIG. 6B is a diagram illustrating an exemplary dependency change in non-anchor frame that have the same dependency as a previous anchor slot, to which the present principles may be applied, in accordance with an embodiment of the present principles;

FIG. 7 is a flow diagram for an exemplary method for decoding multi-view video content using a random access point, in accordance with an embodiment of the present principles;

FIG. 8 is a flow diagram for another exemplary method for decoding multi-view content using a random access point, in accordance with an embodiment of the present principles; and

FIG. 9 is a flow diagram for an exemplary method for encoding multi-view video content, in accordance with an embodiment of the present principles.

DETAILED DESCRIPTION

The present principles are directed to methods and apparatus for use in Multi-view Video Coding (MVC).

The present description illustrates the present principles. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the present principles and are included within its spirit and scope.

All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the present principles and the concepts contributed by the inventor(s) to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.

Moreover, all statements herein reciting principles, aspects, and embodiments of the present principles, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the present principles. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (“DSP”) hardware, read-only memory (“ROM”) for storing software, random access memory (“RAM”), and non-volatile storage.

Other hardware, conventional and/or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

In the claims hereof, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements that performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The present principles as defined by such claims reside in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.

Reference in the specification to “one embodiment” or “an embodiment” of the present principles means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

As used herein, “high level syntax” refers to syntax present in the bitstream that resides hierarchically above the macroblock layer. For example, high level syntax, as used herein, may refer to, but is not limited to, syntax at the slice header level, Supplemental Enhancement Information (SEI) level, picture parameter set level, sequence parameter set level and NAL unit header level.

Also, as used herein, “anchor slot” refers to the time at which a picture is sampled from each view and each of the sampled pictures from each view is an anchor picture.

Turning to FIG. 1, an exemplary Multi-view Video Coding (MVC) encoder is indicated generally by the reference numeral 100. The encoder 100 includes a combiner 105 having an output connected in signal communication with an input of a transformer 110. An output of the transformer 110 is connected in signal communication with an input of quantizer 115. An output of the quantizer 115 is connected in signal communication with an input of an entropy coder 120 and an input of an inverse quantizer 125. An output of the inverse quantizer 125 is connected in signal communication with an input of an inverse transformer 130. An output of the inverse transformer 130 is connected in signal communication with a first non-inverting input of a combiner 135. An output of the combiner 135 is connected in signal communication with an input of an intra predictor 145 and an input of a deblocking filter 150. An output of the deblocking filter 150 is connected in signal communication with an input of a reference picture store 155 (for view i). An output of the reference picture store 155 is connected in signal communication with a first input of a motion compensator 175 and a first input of a motion estimator 180. An output of the motion estimator 180 is connected in signal communication with a second input of the motion compensator 175

An output of a reference picture store 160 (for other views) is connected in signal communication with a first input of a disparity estimator 170 and a first input of a disparity compensator 165. An output of the disparity estimator 170 is connected in signal communication with a second input of the disparity compensator 165.

An output of the entropy decoder 120 is available as an output of the encoder 100. A non-inverting input of the combiner 105 is available as an input of the encoder 100, and is connected in signal communication with a second input of the disparity estimator 170, and a second input of the motion estimator 180. An output of a switch 185 is connected in signal communication with a second non-inverting input of the combiner 135 and with an inverting input of the combiner 105. The switch 185 includes a first input connected in signal communication with an output of the motion compensator 175, a second input connected in signal communication with an output of the disparity compensator 165, and a third input connected in signal communication with an output of the intra predictor 145.

Turning to FIG. 2, an exemplary Multi-view Video Coding (MVC) decoder is indicated generally by the reference numeral 200. The decoder 200 includes an entropy decoder 205 having an output connected in signal communication with an input of an inverse quantizer 210. An output of the inverse quantizer is, connected in signal communication with an input of an inverse transformer 215. An output of the inverse transformer 215 is connected in signal communication with a first non-inverting input of a combiner 220. An output of the combiner 220 is connected in signal communication with an input of a deblocking filter 225 and an input of an intra predictor 230. An output of the deblocking filter 225 is connected in signal communication with an input of a reference picture store 240 (for view i). An output of the reference picture store 240 is connected in signal communication with a first input of a motion compensator 235.

An output of a reference picture store 245 (for other views) is connected in signal communication with a first input of a disparity compensator 250.

An input of the entropy coder 205 is available as an input to the decoder 200, for receiving a residue bitstream. Moreover, a control input of the switch 255 is also available as an input to the decoder 200, for receiving control syntax to, control which input is selected by the switch 255. Further, a second input of the motion compensator 235 is available as an input of the decoder 200, for receiving motion vectors. Also, a second input of the disparity compensator 250 is available as an input to the decoder 200, for receiving disparity vectors.

An output of a switch 255 is connected in signal communication with a second non-inverting input of the combiner 220. A first input of the switch 255 is connected in signal communication with an output of the disparity compensator 250. A second input of the switch 255 is connected in signal communication with an output of the motion compensator 235. A third input of the switch 255 is connected in signal communication with an output of the intra predictor 230. An output of the mode module 260 is connected in signal communication with the switch 255 for controlling which input is selected by the switch 255. An output of the deblocking filter 225 is available as an output of the decoder.

In an embodiment of the present principles, a high level syntax is proposed for efficient processing of a multi-view sequence. In particular, we propose creating a new parameter set called View Parameter Set (VPS) with its own NAL unit type and two more new NAL unit types to support multi-view slices, with the NAL unit types including a view identifier (id) in the NAL header to identify to which view the slice belongs. For view scalability and backward compatibility With decoders compliant with the MPEG-4 AVC standard, we propose to maintain one MPEG-4 AVC compliant view which we call an “MPEG-4 AVC compliant Base View”.

As used herein, “high level syntax” refers to syntax present in the bitstream that resides hierarchically above the macroblock layer. For example, high level syntax, as used herein, may refer to, but is not limited to, syntax at the slice header level, Supplemental Enhancement Information (SEI) level, picture parameter set level, and sequence parameter set level.

In the current implementation of the Multi-view Video Coding system described above as having no provision to identify a specific view and to signal camera parameters different views are interleaved to form a single sequence instead of treating the different views as separate views. Since the syntax is compatible with the MPEG-4 AVC standard, as noted above, it is presently not possible to identify which view a given slice belongs to. This view information is needed for several reasons. View scalability, view random access, parallel processing, view generation, and view synthesis are all Multi-view Video Coding requirements which need to identify a view. For efficient support of view random access and view scalability, it is important for the decoder to know how different pictures depend on each other, so only pictures that are necessary are decoded. Camera parameters are needed for view synthesis. If view synthesis is eventually used in the decoding loop, a standardized way of signaling camera parameters needs to be specified. In accordance with an embodiment, a view parameter set is used.

In an embodiment, it is presumed that one view is needed that is fully backward compatible with the MPEG-4 AVC standard for the purpose of supporting non-MVC compatible but MPEG-4 AVC compatible decoders. In an embodiment, it is presumed that there will be views that are independently decodable to facilitate fast view random access. We refer to these views as “base views”. A base view may or may not be compatible with the MPEG-4 AVC standard, but an MPEG-4 AVG compatible view is always a base view.

Turning to FIG. 3, an inter-view-temporal prediction structure based on the MPEG-4 AVC standard, using hierarchical B pictures, is indicated generally by the reference numeral 300. In FIG. 3, the variable I denotes an intra coded picture, the variable P denotes a predictively coded picture, the variable B denotes a bi-predictively coded picture, the variable T denotes a location of a particular picture, and the variable S denotes a particular view to which corresponds a particular picture.

In accordance with an embodiment, the following terms are defined.

An “anchor picture” is defined as a picture the decoding of which does not involve any picture sampled at a different time instance. An anchor picture is signaled by setting the nal_ref_idc to 3. In FIG. 3, all pictures in locations T0, T8 . . . , T96, and T100 are examples of anchor pictures.

A “non-anchor picture” is defined as a picture which does not have the above constraint specified for an anchor picture. In FIG. 3, pictures B2, B3, and B4 are non-anchor pictures.

A “base view” is a view which does not depend on any other view and can be independently decoded. In FIG. 3, view S0 is an example of base view.

Also, in an embodiment, a new parameter set is proposed called the View Parameter Set with its own NAL unit type and two new NAL unit types to support Multi-view Video Coding slices. We also modify the slice header syntax to indicate the view_id and the view parameter set to be used.

The MPEG-4 AVC standard includes the following two parameter sets: (1) Sequence Parameter Set (SPS), which includes information that is not expected to change over an entire sequence; and (2) Picture Parameter Set (PPS), which includes information that is not expected to change for each picture.

Since Multi-view Video Coding has additional information which is specific to each view, we have created a separate View Parameter Set (VPS) in order to transmit this information. All the information that is needed to determine the dependency between the different views is indicated in the View Parameter Set. The syntax table for the proposed View Parameter Set is shown in TABLE 1 (View Parameter Set RBSP syntax). This View Parameter Set is included in a new NAL unit type, for example, type 14 as shown in TABLE 2 (NAL unit type codes).

In accordance with the description of the present invention, the following terms are defined:

view_parameter_set_id identifies the view parameter set that is referred to in the slice header. The value of the view_parameter_set_id shall be in the range of 0, to 255.

number_of_views_minus—1 plus 1 identifies the total number of views in the bitstream. The value of the number_of_view_minus—1 shall be in the range of 0 to 255.

avc_compatible_view_id indicates the view_id of the AVC compatible view. The value of avc_compatible_view_id shall be in the range of 0 to 255.

is_base_view_flag[i] equal to 1 indicates that the view i is a base view and is independently decodable. is_base_view_flag[i] equal to 0 indicates that the view i is not a base view. The value of is_base_view_flag[i] shall be equal to 1 for an AVC compatible view i.

dependency_update_flag equal to 1 indicates that dependency information for this view is updated in the VPS. dependency_update_flag equal to 0 indicates that the dependency information for this view is not updated and should not be changed.

anchor_picture_dependency_maps[i][j] equal to 1 indicates the anchor pictures with view_id equal to j will depend on the anchor pictures with view_id equal to i.

non_anchor_picture_dependency_maps[i][j] equal to 1 indicates the non-anchor pictures with view_id equal to j will depend on the non-anchor pictures with view_id equal to i. non_anchor_picture_dependency_maps[i][j] is present only when anchor picture_dependency maps[i][i] equals 1. If anchor_picture_dependency_maps[i][j] is present and is equal to zero non_anchor_picture_dependency_maps[i][j] shall be inferred as being equal to 0.

TABLE 1 view_parameter_set_rbsp( ) { C Descriptor  view_parameter_set_id 0 ue(v)  number_of_views_minus_1 0 ue(v)  avc_compatible_view_id 0 ue(v)  for( i = 0; i <= number_of_views_minus_1; i++ ) {  is_base_view_flag[i] 0 u(1)  dependency_update_flag 0 u(1)  if (dependency_update_flag == 1) {   for(j = 0; j <= number_of_views_minus_1; j++) {   anchor_picture_dependency_maps[i][j] 0 f(1)   if (anchor_picture_dependency_maps[i][j] == 1)    non_anchor_picture_dependency_maps[i][j] 0 f(1)   }  }   }   for( i = 0; i <= number_of_views_minus_1; i++ ) {  camera_parameters_present_flag 0 u(1)  if (camera_parameters_present_flag == 1) {   camera_parameters_1_1[i] 0 f(32)

Download full PDF for full patent description/claims.




You can also Monitor Keywords and Search for tracking patents relating to this Methods and apparatus for use in multi-view video coding patent application.

Patent Applications in related categories:

20130148743 - Method and system to improve the transport of compressed video data - A method comprises determining a plurality of time intervals Tp and Tn within a variable bit rate (VBR) representation of an image sequence. The time intervals Tp are those in which a number of blocks of information per unit time is greater than a baseline value. The time intervals Tn ...


###


Other recent patent applications listed under the agent :