Play back apparatus, playback method and program for playing back 3d video   

Abstract: The playback apparatus realizes stereoscopic viewing by overlaying planar or stereoscopic graphics over stereoscopic video in a way that reduces eye strain using following method in abstract: A graphics plane holds therein data composed of graphics data. A shift engine shifts, in a case when a composition unit composites the graphics data with a left-view video frame, coordinates of each of the pixels is shifted in a first horizontal direction, and in a case when the composition unit composites the graphics data with a right-view video frame, coordinates of each of the pixels is shifted in a second horizontal direction that is opposite to the first direction.

The present application is a continuation application of U.S. patent application Ser. No. 12/508,207, filed on Jul. 23, 2009, which claims priority to U.S. Provisional Pat. Appl. No. 61/111,045, filed on Nov. 4, 2008. The disclosure of each of these documents, including the specification, drawings, and claims, is incorporated herein by reference in its entirety.

The disclosure of each of Japanese Pat. Appl. No. 2009-101846, filed on Apr. 20, 2009, and Japanese Pat. Appl. No. 2008-190525, filed on Jul. 24, 2008, including the specification, drawings, and claims, is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

(1) Field of the Invention

The present invention belongs to a technical field of a graphics composition technique.

(2) Description of the Related Art

The graphics composition technique is a technique of compositing graphics such as subtitles or a GUI, to each frame of video which constructs a video stream, then displaying the result. As a technical trend of display device, displays that can display stereoscopic video as well as planar video are becoming popular. Various methods of stereoscopic viewing are adopted in the stereoscopic display devices, however, nearly all of the methods use the basic principle of displaying each of left eye and a right eye a different image to create stereoscopic effect, using binocular disparity.

In order to allow the viewers to view the stereoscopic video at the same frame rate as normal planar video, response performance twice as high as response performance needed for the normal planar video is necessary. This means, for example, that it is necessary to switch-among at least 120 frames per second in order to display video consisting of 60 frames per second.

Accordingly, the video stream to be displayed needs to be encoded at 120 frames per second. A stereoscopic effect may also be obtained without increasing the frame rate, by using a side-by-side method as described in a Non-Patent Document 1 or checker pattern method as described in Patent Document 2.

There is also, a technique known to generate stereoscopic images, that extracts information indicating the number of objects from a 2D video, then creating the number of layers corresponding to the number of objects so stereoscopic images can be generated by changing the depth of each of the layers as disclosed in a Patent Document 3.

[Patent Document 1] International Publication No. 2005/119675 pamphlet [Patent Document 2] US Patent Application Publication No. 2008-0036854 [Patent Document 3] US Patent Application Publication No. 2002-0118275

[Non-Patent Document 1] FOUNDATIONS OF THE STEREOSCOPIC CINEMA A STUDY IN DEPTH (by Lenny LIPTON)

SUMMARY

As the situation now stands, application of a method of allowing viewers to enjoy viewing video streams stereoscopically is mainly engaged at theaters and the like. However, it is expected that it will be common for the viewers to enjoy viewing the stereoscopic video streams with use of playback apparatuses for household use in the future.

In package media such as BD-ROMs and DVD-videos, subtitles are not pre-embedded as part of video like the movies. In package media, plurality of subtitle data corresponding to plurality of languages is recorded separately from a video stream. This is because it is necessary to composite appropriate subtitle data with a video in accordance with a language setting in a playback apparatus, and to display data obtained as a result of the composition.

Also, according to graphics for realizing the GUI, a plurality of pieces of graphics data is recorded separately from the video stream. This is also because it is necessary to composite appropriate graphics data with a video in accordance with a language setting in a playback apparatus, and to display data obtained as a result of the composition. In this case, it is desirable that both of data for a left view (left-view data) and data for a right view (right-view data) are prepared for each of graphics showing subtitles, and graphics showing the GUI. However, since BD-ROMs have limited capacity, it may not be possible to prepare both the left-view data and the right-view data for each of the video and the graphics.

In a case where only one of the left-view data or the right-view data can be prepared for the graphics, the graphics will appear to be flat while the video corresponding to the stereoscopic video stream can be viewed stereoscopically. Suppose that the planar graphics is composited with the stereoscopic video stream without the consideration of spacing caused by the stereoscopic effect of displaying separate left-view video and right-view video data. In such case, a flat graphics will appear within a stereoscopic video as a result of composition. As a result, the graphics will appear to be buried into the stereoscopic video, which in turn will cause displeasure to the viewers. This is caused by the eyes not being able to adjust to an unnatural stereoscopic composition not seen in real life, leading to decrease in realistic sensation.

The present invention has an objective to provide a playback apparatus that is capable of executing stereoscopic playback by performing composition of the video and the graphics so as to prevent viewers from feeling uncomfortable despite when only graphics for one eye, either graphics for left view (left-view graphics) or graphics for right view (right-view graphics), is recorded on a recording medium.

In order to solve the above-stated problem, the present invention is a playback apparatus that executes stereoscopic playback, the playback apparatus comprising: a video decoder operable to decode a video stream to obtain video frames; a video plane that holds therein the video frames; a graphics plane that holds therein graphics data, the graphics data having a resolution of a predetermined number of horizontal and vertical pixels; a composition unit operable to composite the graphics data within the graphics plane with one of the video frames; and a shift engine operable to perform plane shifting of the graphics plane, wherein each of the video frames in the video plane is outputted as a right-view video frame or a left-view video frame, and the stated plane shifting of the graphics plane is defined as; prior to outputting the left-view video frame, shifting each of the pixels of the graphics data within the graphics plane in a right or a left direction, and prior to outputting the right-view video frame, shifting each of the pixels of the graphics data within the graphics plane in an opposite direction from left view, and then giving the resulted shifted graphics data within the graphics plane to the composition unit for the composition.

By shifting the coordinates of each of the pixel data pieces held in the graphics plane (hereinafter, referred to as “plane shifting of the graphics plane), it will become possible to adjust the depth position of the graphics to be displayed. The stated depth position meaning how close or far the objects on the plane will appear to the user in accordance to the display depth. Therefore, even in a case where subtitles and GUI appear to be buried in the stereoscopic video because both of the right-view graphics (right-view interactive graphics) and the left-view graphics (left-view interactive graphics) are not provided in a recording medium, a more natural stereoscopic composition (graphics depth position is closer to the viewer than the video depth position) can be achieved. Since the depth position of the subtitles and GUI are changed by such an adjustment, it is not necessary to prepare both the left-view graphics and the right-view graphics and only prepare 2D graphics for displaying the subtitles and the GUI stereoscopically or adding the depths thereto. Therefore, even in a case where a BD-ROM whose capacity is limited is a target for the playback, it is possible to provide the user with preferable stereoscopic view.

Also for authoring purposes, it will become possible to re-use the graphics for 2D playback, thus will be able to skip the conversion process of stereoscopic graphics (create a second view), resulting in a reduction of labor cost for generating stereoscopic contents.

Even when the depths of the graphics showing the subtitles and GUI are not appropriate, it is possible to change the depth of the graphics to an appropriate depth in accordance with the depth of video to be outputted, by performing the above-stated shifting. Therefore, the graphics to which the depth is added appears to be natural, and there will be no difference in how the video and the graphics showing the subtitles and GUI appear stereoscopically. Therefore, stress caused to the naked eyes of the viewers is reduced.

Also, since it is not necessary to load the left-view graphics and the right-view graphics in a memory separately, the memory capacity of the playback apparatus will not be consumed even when an image obtained by compositing the video with the graphics is displayed stereoscopically.

With the above-stated structure, the stereoscopic video playback apparatus pertaining to the present invention can stereoscopically display the subtitles and the GUI on the video even if both of the left-view graphics and right-view graphics are not prepared.

BRIEF DESCRIPTION OF THE DRAWINGS

These and the other objects, advantages and features of the invention will become apparent from the following description thereof taken in conjunction with the accompanying drawings which illustrate a specific embodiment of the invention. In the drawings:

FIG. 1 shows a usage pattern of a recording medium and a playback apparatus 200;

FIG. 2 shows an internal structure of a BD-ROM 100;

FIG. 3 shows an internal structure of a BD-J object;

FIG. 4 shows an internal structure of the playback apparatus 200;

FIG. 5 shows detailed structures of a front end unit 101, a system LSI, a nonvolatile memory 105, a memory device 13, a back end unit and a host microcomputer 106 in detail;

FIG. 6 shows switching between a 2D display mode and a 3D display mode;

FIG. 7 shows an example of composition processing when a stereo mode of each plane is ON, and an example of composition processing when a stereo mode of each plane is OFF;

FIG. 8 shows how data in a background plane, data in a video plane 6, data in an image plane 8 and data in an interactive graphics plane 10 are composited when stereo modes of all the planes are ON;

FIG. 9 shows how data in a background plane, data in a video plane 6 and data in an interactive graphics plane 10 are composited when stereo modes of all the planes are OFF;

FIG. 10 shows a composition result for each plane;

FIG. 11 shows an example where an image that is outputted when the stereo modes of all the planes are ON is viewed on a 3D display;

FIG. 12 shows an example of how the stereoscopic video appears when the viewer sees, through shutter glasses 500, an image that is outputted when the stereo mode of the video plane 6 is ON and the stereo modes of other planes are OFF is viewed;

FIG. 13A and FIG. 13B show an image in a left-view graphics plane shifted in a right direction, and an image in a right-view graphics plane shifted in a left direction, respectively;

FIG. 14A and FIG. 14B show an internal structure of the image plane 8;

FIG. 15A, FIG. 15B and FIG. 15C show pixel data pieces in a foreground area and pixel data pieces in a background area after a plane shift engine 19 shifts coordinates of each pixel data piece in the right direction, and shifts the coordinates of each pixel data piece in the left direction; Each of FIG. 16A and FIG. 16B shows an internal structure of the interactive graphics plane 10;

FIG. 17A, FIG. 17B and FIG. 17C show pixel data pieces in a foreground area and pixel data pieces in a background area after the plane shift engine 19 shifts the coordinates of each pixel data piece in the right direction, and shifts the coordinates of each pixel data piece in the left direction;

FIG. 18A, FIG. 18B and FIG. 18C show processing procedures for shifting the coordinates of each pixel data piece held in the image plane 8;

FIG. 19A, FIG. 19B and FIG. 19C show processing procedures for shifting the coordinates of each pixel data piece held in the interactive graphics plane 10;

FIG. 20A and FIG. 20B show how depth of subtitles changes by a negative offset and a positive offset;

FIG. 21 shows pixel data pieces in the graphics plane;

Each of FIG. 22A and FIG. 22B shows what is held in the graphics plane after the plane shift engine 19 shifts the coordinates of each of the pixel data pieces;

FIG. 23 shows an internal structure of a BD-J platform unit;

FIG. 24 shows what is stored in a display mode storage unit 29;

FIG. 25 shows, in a table, changes in display modes when a title is changed;

FIG. 26 shows switching between display modes when a playlist being played back is changed in each title;

FIG. 27A and FIG. 27B show API used for setting by the display mode storage unit 29;

FIG. 28 shows an internal structure of an offset setting unit;

FIG. 29 shows a flowchart showing processing procedures for display mode setting when a title is changed;

FIG. 30 shows a flowchart showing processing procedures for display mode setting in each title;

FIG. 31A, FIG. 31B and FIG. 31C describe principles of how a planer image appears to be in a position closer to a viewer than a position of a display screen;

FIG. 32 shows a flowchart showing main procedures for playing back a playlist;

FIG. 33 shows a flowchart showing playback procedures based on playitem information;

FIG. 34 shows a flowchart showing left-view processing of a 3D stream in the 3D display mode;

FIG. 35 shows a flowchart showing right-view processing for a 3D stream in the 3D display mode;

FIG. 36 shows a flowchart showing processing procedures for a 2D AV stream in the 2D display mode; and

FIG. 37 shows a flowchart showing processing procedures when the BD-J application gives an offset update instruction;

FIG. 38 shows a structure of a composition unit 15;

FIG. 39A, FIG. 38B and FIG. 38C show processes for executing a shifting of each line data in the right direction;

FIG. 40A, FIG. 40B and FIG. 40C show processes for executing shifting of coordinates of each pixel data piece in each line data in the plane in the left direction;

FIG. 41 shows a flowchart showing processing procedures for storing data in a line memory; and

FIG. 42A, FIG. 42B and FIG. 42C describe principles of how a planar image appears to be in a position more distant from the viewer than a position of the display screen.

The following describes the embodiments of a recording medium and a playback apparatus 200 having the above-stated Means to Solve the Problems with reference to the drawings.

FIG. 1 shows a usage pattern of the recording medium and the playback apparatus 200. As shown in FIG. 1, a BD-ROM 100 (taken as an example of recording medium) and a playback apparatus 200 compose a home theater system together with a remote control 300, TV 400 and liquid crystal shutter glasses 500, and are used by a user.

The BD-ROM 100 provides the above-stated home theater system with a movie, for example.

The playback apparatus 200 is connected to the TV 400, and plays back the BD-ROM 100.

The remote control 300 is a device that receives an operation for the hierarchized GUI from the user. In order to receive such an operation, the remote control 100 includes: a menu key for calling menus composing the GUI; an arrow key for moving a focus of the GUI parts composing each menu; a decision key for performing determined operation on the GUI parts composing each menu; a return key for returning to higher order hierarchized menus; and a numerical key.

The TV 400 provides the user with a dialogical operational environment by displaying the playback picture of the movie, a menu and the like.

The liquid crystal shutter glasses 500 are composed of crystal liquid shutters and a control unit, and realize stereoscopic view with use of binocular disparity of the viewer\'s eyes. Lenses having a feature that the transmittance of light is changed by changing applied voltage are used for the crystal liquid shutters of the liquid crystal shutter glasses 500. The control unit of the liquid crystal shutter glasses 500 receives a SYNC signal for switching between a right-view image and a left-eye image that are transmitted from the playback apparatus 200, and switches between a first state and a second state in accordance with this SYNC signal.

In the first state, the control unit adjusts the applied voltage such that light does not transmit through a liquid crystal lens corresponding to the right view, and adjust the applied voltage such that light transmits through a liquid crystal lens corresponding to the left view. In such a state, only the left-view image is viewed.

In the second state, the control unit adjusts the applied voltage such that the liquid crystal lens corresponding to the right view transmits light, and adjusts the applied voltage such that the liquid crystal lens corresponding to the left view does not transmit light. In such a state, the crystal liquid shutters provide view of the right-view image.

Generally, the right-view image and left-view image look a little different due to differences between angles. With use of such a difference, the user can recognize an image as a stereoscopic image. Thus, the user confuses a planar display with a stereoscopic display by synchronizing timing of switching between the above-stated first state and second state with timing of switching between the right-view image and the left-view image. Next, a description is given of a time interval in displaying right-view video and left-view video.

Specifically, there is a difference between the right-view image and the left-view image that corresponds to binocular disparity of the user in a planar image. By displaying theses images while switching the images at a short time interval, the images appear as if the images are displayed stereoscopically.

The short time interval may be a time period just enough to confuse the user on planar images with stereoscopic images when the switching and the displaying are performed as stated in the above.

This concludes the description of the home theater system.

The following describes a recording medium to be played back by the playback apparatus 200. The playback apparatus 200 plays back the BD-ROM 100. FIG. 2 shows the internal structure of the BD-ROM 100.

The BD-ROM is shown in a fourth tier from the top in the present figure, and a track on the BD-ROM is shown in a third tier. Although the track is usually formed in a spiral manner from an inner circumference to an outer circumference, the track is drawn in a laterally-expanded manner in the present figure. This track consists of a read-in area, a volume area and a read-out area. Also, in the read-in area exists a special area called BCA (Burst Cutting Area) that can be read only by a drive. Since this area cannot be read by an application, this area is often used in copyright protection technology.

The volume area in the present figure has a layer model having a physical layer, a file system layer and an application layer. Application data such as image data starting with file system information (volume) is stored in the volume area. The file system is UDF, ISO9660 or the like. In the file system, it is possible to read logic data recorded in the same manner as a normal PC, with use of a directory or a file structure. Also, a file name or a directory name consisting of 255 words can be read. A top tier of FIG. 2 shows an application layer format (application format) of the BD-ROM expressed using a directory structure. As shown in the first tier, in the BD-ROM, a CERTIFICATE directory and a BDMV directory exists below the Root directory.

Below the CERTIFICATE directory, a file of a root certificate (app.discroot.certificate) of a disc exists. This app.discroot.certificate is a digital certificate used in a process that checks whether an application has been tampered, or identifies the application (hereinafter, signature verification) when executing a program of a JAVA™ application that performs dynamic scenario control using a JAVA™ virtual machine.

The BDMV directory is a directory in which data such as AV content and management information used in the BD-ROM are recorded. Six directories called “PLAYLIST directory”, “CLIPINF directory”, “STREAM directory”, “BDJO directory”, “JAR directory” and “META directory” exist below the BDMV directory. Also, two types of files (i.e. INDEX.BDMV and MovieObject.bdmv) are arrayed.

The STREAM directory is a directory storing a file which is a so-called transport stream body. A file (000001.m2ts) to which an extension “m2ts” is given exists in the STREAM directory.

A file (000001.mpls) to which an extension “mpls” is given exists in the PLAYLIST directory.

A file (000001.clpi) to which an extension “clpi” is given exists in the CLIPINF directory.

A file (XXXXX.bdjo) to which an extension “bdjo” is given exists in the BDJO directory.

A file (YYYYY.jar) to which an extension “jar” is given exists in the JAR directory.

An XML file (ZZZZZ.xml) exists in the META directory.

The following describes these files.

Firstly, a description is given of the file to which the extension “m2ts” is given. The file to which the extension “m2ts” is given is a digital AV stream in the MPEG-TS (Transport Stream) method, and is acquired by multiplexing a video stream, one or more audio streams, a graphics stream, a text subtitle stream and the like. The video stream represents moving part of the movie, and the audio stream represents audio part of the movie. In the case of the 3D stream, both of left-eye data and right-eye data may be included in m2ts, or m2ts may be prepared separately for each of the left-eye data and the right-eye data. It is preferable to use a codec (e.g. MPEG-4 AVC MVC) in which a left-view stream and a right-view stream refer to one another in order to save disc capacity used for streams. Video streams compressed and encoded with use of such a codec are called MVC video streams.

There are 2 types of MVC video streams, base view video stream and enhanced view video stream. The base view video stream is a video stream composing either the left view or the right view video that realizes planar view display. Meanwhile, the “enhanced view video stream” is the video stream that is used with the base view to construct stereoscopic video, composing the left or right view video that is not composed in the base view video stream. Picture data pieces composing the enhanced view video stream are compressed and encoded based on frame correlativity with picture data pieces composing the base view stream.

The file to which the extension “mpls” is given is a file storing PlayList (PL) information. The PL information defines a playlist referring to the AVClip.

In the present embodiment, it is possible to determine whether streams to be played back include a 3D video stream, based on a structural format of the PlayList (PL) stored on the BD-ROM.

The PlayList information includes MainPath information, Subpath information and PlayListMark information.

1) The MainPath information defines a logic playback section by defining at least one pair of a time point (In_Time) and a time point (Out_Time) on a playback time axis of the AV stream. The MainPath information has a stream number table (STN_table) that stipulates which elementary streams that have been multiplexed into AV stream are permitted to be played back and are not permitted to be played back.

2) The PlayListMark information shows specification of a time point corresponding to a chapter in a part of the AV stream specified by the pair of the In_Time information and the Out_Time information.

3) The Subpath information is composed of at least one piece of SubPlayItem information. The SubPlayItem information includes information on specification of an elementary stream to be played back in synchronization with the AV stream, and includes a pair of In_Time information and Out_Time information on the playback time axis of the elementary stream. The Java™ application for controlling playback instructs a Java™ virtual machine to generate a JMF (Java Media Frame work) player instance that plays back this PlayList information. This starts the playback of the AV stream. The JMF player instance is actual data generated on a heap memory of the virtual machine based on the JMF player class.

Hereinafter, a stream including only a stream for 2D playback is referred to as a “2D stream”, and a stream including both a 2D and 3D streams is referred to as a “3D stream”.

Furthermore, according to a term definition, a 2D playlist is a playlist including only a stream for the 2D playback while a 3D playlist includes a stream for 3D viewing in addition to the 2D stream.

The file to which the extension “clpi” is given is Clip information which is in one to one correspondence with AVclip information. Since the Clip information is management information, the Clip information has an EP_map showing an encoding format of the stream in the AVClip, a frame rate, a bit rate, information on resolution and the like, and a starting point of GOPs. The Clip information and PL information are classified as “static scenario”.

The following describes the file to which the extension “BD-J object” is given. The file to which the extension “BD-J object” is given is a file storing a BD-J object. The BD-J object is information that defines a title by association an AVClip string defined by the PlayList information with an application.

The entity of the Java™ application corresponds to a Java™ archive file (YYYYY.jar) stored in the JAR directory under the BDMV directory in FIG. 2.

The application is a Java™ application, and is composed of one or more xlet programs loaded in the heap area (also called “work memory”) of the virtual machine. The application is composed of the xlet program loaded in the work memory and data.

In the meta file (ZZZZZ.xml) included in the META directory is stored various information pieces relating to the movie in the disc. Examples of information pieces stored in the meta file are a name of the disc and an image of the disc, information on who has created the disc, and a title name for each title. This concludes the description of the BD-RO 100. The meta file is not a prerequisite, and some BD-ROMs do not include this meta file. This concludes the description of the BD-ROM.

The following describes the BD-J object. FIG. 3 shows the internal structure of the BD-J object. As shown in FIG. 3, the BD-J object is composed of an “application management table”, a “GUI management table” and a “playlist management table”.

The following describes these elements.

The “application management table” is a table for causing the playback apparatus 200 to perform application signaling that runs the title as a life cycle. A lead line bj1 shows the internal structure of the application management table in closeup. As shown in this lead line, the application management table includes an “application identifier” and a “control code” that specify an application to be operated when a title corresponding to the BD-J object becomes a current title. When the control code is set to AutoRun, the control code shows that this application is automatically started after the application is loaded on the heap memory. When the control code is set to Present, the control code waits for a call from another application, and shows whether the application should be operated after the application is loaded on the heap memory.

The “GUI management table” is a table used for GUI related operations by the application. More specifically, resolution, font data used, masking information when GUI for executing the menu call, or a title call is instructed by the user, is included. A lead line bj2 shows the internal structure of the GUI management table in closeup. As shown by this lead line bj2, the GUI management table may be set to one of HD3D—1920×1080, HD—3D—1280×720, HD—1920×1080, HD—1280×720, QHD960×540, SD, SD—50 HZ—720—576 and SD—60 HZ—720—480.

A lead line bj3 shows the internal structure of the playlist management table in closeup. The playlist management table includes information on specification of the playlist to be operated by default when a title corresponding to the BD-J object becomes a current title. A lead line bj4 shows the internal structure of a auto start playlist in closeup. As shown by the lead line bj4, 3D playlist 1920*1080, 3D playlist 1280*720, 2D playlist 1920*1080, 2D playlist1 1280*720, 2D playlist 720*576 and 2D playlist 720*480 may be specified as information specifying the auto start playlist.

FIG. 4 shows the internal structure of the playback apparatus 200. A front end unit 101, a system LSI 102, a memory device 103, a back end unit 104, a nonvolatile memory 105, a host microcomputer 106 and a network I/F 107 mainly compose a playback apparatus 200 in FIG. 4.

The front end unit 101 is a data input source. In a figure described in the following, the front end unit 101 includes a BD drive 110 and a local storage 111, for example.

The system LSI 102 is composed of logic elements, and is a core part of the playback apparatus 200. This system LSI includes therein at least a demultiplexer 4, video decoders 5a and 5b, image decoders 7a and 7b, an audio decoder 9, a playback state/setting register (PSR: Player Status/Setting Register) set 12, a playback control engine 14, a composition unit 15, a plane shift engine 19 and an offset setting unit 20.

The memory device 103 is composed of arrays of memory devices such as an SDRAM.

The back end unit 104 is a connection interface that connects internal parts of the playback apparatus 200 with other devices.

The nonvolatile memory 105 is a readable and writable recording medium, and is a medium that can hold recorded contents without needing power source supply. The nonvolatile memory 105 is used for backup of information on a display mode stored in a display mode storage unit 24 (described in the following). A flash memory, an FeRAM or the like may be used as this nonvolatile memory 105.

The host microcomputer 106 is a core part of the playback apparatus 200, and is composed of an MPU, a ROM and a RAM. The host microcomputer 106 includes a BD-J platform 22 and a command interpreter 25 from among the elements specifically described in the following.

The network interface 107 is for performing communication with an external device outside the playback apparatus 200, and is capable of accessing a server accessible via the Web, and accessing a server connected by a local network. For example, the network interface 107 is used for downloading BD-ROM additional contents publicized on the web. The BD-ROM additional content is content that is not stored in an original BD-ROM, examples of which are additional sub audio, subtitles, special features and an application. It is possible to control the network interface107 from the BD-J platform, and to download, in the local storage 111, the additional content publicized on the web.

As stated in the above, the front end unit 101 includes the BD drive 110 and the local storage 111, for example. The BD drive 110 includes, for example, a semiconductor laser (not shown), collimator lenses (not shown), a beam splitter (not shown), an objective lens (not shown), a condenser lens (not shown), an optical head (not shown) including a light detector (not shown). Light beam outputted from the semiconductor laser is collected on a information side of the optical disc through the collimator lenses, the beam splitter and the objective lens. The collected light beam is reflected and diffracted on the optical disc, and then collected by the light detector through the objective lenses, the beam splitter and the condenser lenses. The generated signal corresponds to data read from the BD-ROM in accordance with the amount of light collected on the light detector.

The local, storage 111 includes a built-in media and a removable media, and is used for storing the downloaded additional contents and data used by the application. A storage area for the additional contents is provided for each BD-ROM, and an area that can be used for storing data is provided for each application. Also, merge management information, which is a merge rule regarding how the downloaded additional contents are merged with the data on the BD-ROM, is also stored in the built-in media and the removable media.

The built-in media is a writable recording medium such as a hard disk drive and a memory built in the playback apparatus 200.

The removable medium is a portable recording medium, for example, and is preferably a portable semiconductor memory card such as an SD card.

A description is given taking a case where the removable media is a semiconductor memory card as an example. The playback apparatus 200 is provided with a slot (not shown) into which the removable media is inserted, and an interface (e.g. memory card I/F) for reading the removable area inserted into the slot. When the semiconductor memory is inserted into the slot, the removable media and the playback apparatus 200 are electrically connected to each other, and it is possible to convert data recorded in the semiconductor memory into an electrical signal and read the electrical signal with use of the interface (e.g. memory card I/F).

The elements in the front end unit 101, the system LSI, the nonvolatile memory 105, the memory device 13, the back end unit and the host microcomputer 106 are further described. FIG. 5 shows the structure of the front end unit 101, the system LSI, the nonvolatile memory 105, the memory device 13, the back end unit and the host microcomputer 106 in detail. As shown in FIG. 5, the front end unit 101, the system LSI, the nonvolatile memory 105, the memory device 13, the back end unit and the host microcomputer 106 include a read buffers 1 and 2, a virtual file system 3, a demultiplexer 4, the video decoders 5a and 5b, a video plane 6, image decoders 7a and 7b, image memories 7c and 7d, an image plane 8, an audio decoder 9, an interactive graphics plane 10, a background plane 11, the playback state/setting register (PSR) set 12, a static scenario memory 13, the playback control engine 14, the composition unit 15, an HDMI transmission/reception unit 16, a left-right processing storage unit 17, a display function flag holding unit 18, a plane shift engine 19, the offset setting unit 20, the BD-J platform 22, a dynamic scenario memory 23, a mode management module 24, an HDMV module 25, a UO detection module 26, a still image memory 27a, a still image decoder 27b, a display mode setting initial display setting unit 28 and a display mode storage unit 29.

The read buffer 1 temporarily stores source packets composing extents that compose the base view stream read from the BD drive 110. The read buffer 1 transfers the source packets to the demultiplexer 4 after adjusting the transfer speed, and has a size “RB1” as stated in the above.

The read buffer 2 stores source packets composing extents that compose the enhanced view stream read from the BD drive 110. The read buffer 2 transfers the source packets to the demultiplexer after adjusting the transfer speed, and has a size “RB2” as stated in the above.

The virtual file system 3 configures a virtual BD-ROM (virtual package) in which the additional contents stored in the local storage are merged with the contents on the loaded BD-ROM based on the merge management information downloaded in the local storage 111 together with the additional contents.

The virtual package and the original BD-ROM can be referred to from a command interpreter which is a main operational part in the HDMV mode, and the BD-J platform which is a main operational part in the BD-J mode. The playback apparatus 200 performs the playback control with use of the data on the BD-ROM and the data on the local storage during the playback of the virtual package.

The demultiplexer 4 is composed of a source packet depacketizer and a PID filter. Receiving an instruction from a packet identifier corresponding to a stream to be played back (the stream is included in the structured virtual package (data on the loaded BD-ROM and the local storage corresponding to the loaded BD-ROM), the demultiplexer 4 executes packet filtering based on the packet identifier. In executing the packet filtering, the demultiplexer 4 extracts one of the left-view video stream and the right-view video stream that corresponds to a display method flag based on the flag in the left-right processing storage unit 31, and the demultiplexer 4 transfers the video stream to the video decoder 5a or the video decoder 5b.

When a stream separated from the stream to be played back is a subtitle stream, the demultiplexer 4 writes the separated subtitle stream in to the image memory. When the subtitle streams (a left-view subtitle stream, and a right-view subtitle stream) are included in the stream, the demultiplexer 4 writes the left-view subtitle stream in the image memory 7c, and writes the right-view subtitle stream in the image memory 7d.

When the 2D subtitle stream (subtitle stream used for the planar display) is included in the stream, the demultiplexer 4 writes the 2D subtitle stream in the image memory 7c.

The video decoder 5a decodes a TS packet outputted from the demultiplexer 4, and writes a compressed picture in a left-eye plane 6 (expressed as a code (L) in the video plane 6 in FIG. 5).

The video decoder 5b decodes the enhanced view video stream outputted from the demultiplexer 4, decodes the TS packet, and writes the uncompressed picture in a right view video plane 6 (expressed as a code (R) in the video plane 6 in FIG. 5).

The right-view video plane 6 is a plane memory that can store picture data having a resolution such as 1920*2160 (1280*1440). The video plane 6 has a left-eye plane (expressed as the code (L) in the video plane 6 in FIG. 5) having an area Capable of storing data with resolution such as 1920*1080 (1280*720), and a right-eye plane (expressed as the code (R) in the video plane 6 in FIG. 5) having an area capable of storing data with resolution such as 1920*1080 (1280*720).

When the display mode of the video plane is in 3D display mode, and the stereo mode is ON, the video decoder 5a decodes the left-view video stream, and writes the decoded left-view video stream in the left-eye plane (expressed as the code (L) in the video plane 6 in FIG. 5). The video decoder 5b decodes the right-view video stream, and writes the decoded right-view video stream in the right-eye plane (expressed as the code (R) in the video plane 6 in FIG. 5).

When the display mode of the video plane is a 3D display mode, and the stereo mode is OFF, the video decoder 5a decodes the left-view video stream, for example, and writes the decoded left-view video stream in the left-eye plane (expressed as the code (L) in the video plane 6 in FIG. 5) and in the right-eye plane (expressed as the code (R) in the video plane 6 in FIG. 5).

When the display mode of the video plane is a 2D display mode, the demultiplexer 4 transmits the 2D video stream to the video decoder 5a, and the video decoder 5a writes the decoded 2D video data held in the left-eye plane (expressed as the code (L) in the video plane 6 in FIG. 5).

Although an examples are shown for cases where each of the left-eye plane and the right-eye plane included in the video plane 6 shown in FIG. 5 is a physically separated memory, the structure of the video plane 6 is not limited to this. Therefore, areas for the left-eye plane and the right-eye plane may be provided together as one memory. In such case, the video data is written in each of the corresponding areas (left and right).

Each of the image decoders 7a and 7b decodes TS packets composing the subtitle stream that is outputted from the demultiplexer 4 and written in the image memories 7c and 7d, and writes the uncompressed graphics subtitles in the graphics plane 8a. The “subtitle streams” decoded from the image decoders 7a and 7b are data pieces each showing subtitles compressed by a run-length coding, and is defined by pixel codes showing a Y value, a Cr value, a Cb value and an a value and a run lengths of the pixel codes.

The image plane 8 is a graphics plane capable of storing graphics data (e.g. subtitle data) obtained by for example, decoding the subtitle stream with a resolution of 1920*1080 (1280×720). The image plane 8 has a left-eye plane (expressed as a code (L) in the image plane 8 in FIG. 5) having an area capable of storing data having a resolution of 1920*1080 (1280*720), for example, and a right-eye plane (expressed as a code (R) in the image plane 8 in FIG. 5) having an area capable of storing data having a resolution of 1920*1080 (1280*720), for example.

When the display mode of the plane with subtitle data is in 3D display mode, and the stereo mode is ON, the image decoder 7a decodes the left-view subtitle stream stored in the image memory 7c, and writes the decoded subtitle stream in the left-eye plane (expressed as a code (L) in the image plane 8 in FIG. 5). The image decoder 7b decodes the right-view subtitle stream stored in the image memory 7d, and writes the decoded right-view subtitle stream in the right-eye plane (expressed as a code (R) in the image plane 8 in FIG. 5).

When the display mode of the plane with subtitle data is a 3D display mode, and the stereo mode is OFF, the image decoder 7a decodes the left-view subtitle stream stored in the image memory 7c, and writes the decoded left-view subtitle stream in the left-eye plane (expressed as a code (L) in the image plane 8 in FIG. 5) and in the right-eye plane (expressed as a code (R) in the image plane 8 in FIG. 5).

When the display mode of the plane with subtitle data is a 2D display mode, the demultiplexer 4 stores the 2D subtitle stream in the image memory 7c, and the image decoder 7a decodes the 2D subtitle stream stored in the image memory 7c, and writes the left-eye video plane (expressed as a code (L) in the image plane 8 in FIG. 5).

Although an examples are shown for cases where each of the left-eye plane and the right-eye plane included in the image plane 8 shown in FIG. 5 is a physically separated memory, the structure of the image plane 8 is not limited to this. Therefore, areas for the left-eye plane and the right-eye plane may be provided together as one memory. In such case, the corresponding graphics data is written in each of the corresponding areas (left and right).

The audio decoder 9 decodes audio frames outputted from the demultiplexer 4, and outputs the uncompressed audio data.

The interactive graphics plane 10 is a graphics plane having a storage area capable of storing graphics data written by the BD-J application using the rendering engine 21 with resolutions such as 1920*2160 (1280*1440). The interactive graphics plane 10 has, for an example, a left-eye plane (expressed as a code (L) in the interactive graphics plane 10 in FIG. 5) having an area capable of storing data having a resolution of 1920*1080 (1280*720), and a right-eye plane (expressed as a code (R) in the interactive graphics plane 10 in FIG. 5) having an area capable of storing data having a resolution of 1920*1080 (1280*720).

When the display mode of the interactive graphics plane is in 3D display mode, and the stereo mode is ON, it is indicated that the BD-J application includes a program that writes an interactive graphics that is viewable by the left eye (left-eye interactive graphics) and an interactive graphics that is viewable by the right eye and is different from the left-eye interactive graphics (right-eye interactive graphics).

The left-eye interactive graphics and the right-eye interactive graphics written by this rendering program can be seen from different angles so as to allow the viewer to see stereoscopic graphics.

When the left-eye interactive graphics and the right-eye interactive graphics are displayed, the BD-J application writes the left-view interactive graphics in the left-eye plane (to which a code (L) is given in the interactive graphics plane 10 in FIG. 5) with use of the rendering engine 21, and writes the right-view interactive graphics in the right-eye plane (to which a code (R) is given in the interactive graphics plane 10 in FIG. 5).

When the display mode of the interactive graphics plane is in 3D display mode, and the stereo mode is OFF, the BD-J application writes the left-view interactive graphics in the each of the planes to which the code (L) and the code (R) are respectively given, with use of the rendering engine 21.

When the display mode of the interactive graphics plane is in 2D display mode, the BD-J application writes the 2D interactive graphics in the interactive graphics plane 10 (more specifically, the plane to which the code (L) is given in the interactive graphics plane 10) with use of the rendering engine 21.

Although examples are shown for cases where a left-eye area (to which the code (L) is given) and a right-eye area (to which the code (R) is given) of the interactive graphics plane 10 shown in FIG. 5 are provided in one plane memory, the structure of the interactive graphics plane 10 plane is not limited to this. Therefore, the left-eye area (to which the code (L) is given) and the right-eye area (to which the code (R) is given) of the interactive graphics plane 10 may be physically separated from one another.

The “graphics data” held in the interactive graphics plane 10 is graphics whose pixels each is defined by an R value, a G value, a B value, and an α value. The graphics written in the interactive graphics plane 10 is an image or a widget mainly used for composing the GUI.

Although the image data and the graphics data are different in terms of structure, they are collectively expressed as graphics data. There are two types of the graphics plane (i.e. the image plane 8 and interactive graphics plane 10). Hereafter, when the term “graphics plane” is used, it referrers to both or one of the image plane 8 and the interactive graphics plane 10.

The still image memory 27a stores still image data that is extracted from the structured virtual package and is to be a background image.

The background plane 11 is a plane memory capable of storing the still image data to be a background image having a resolution such as 1920*2160 (1280*1440). Specifically, the background plane 11 has a left-eye plane (expressed as a code (L) in the background plane 11 in FIG. 5) having an area capable of storing data having a resolution of 1920*1080 (1280*720), and a right-eye plane (expressed as a code (R) in the background plane 11 in FIG. 5) having an area capable of storing data having a resolution of 1920*1080 (1280*720).

When the display mode of the background plane is in 3D display mode, and the stereo mode is ON, the still image decoder 27a decodes left-view still image data and right-view still image data that are stored in the still image memory 27a. Then the image decoder 27a writes the left-view still image data and the right-view still image data held in the left-eye plane (to which a code (L) is given in the background plane 11 shown in FIG. 5) and the right-view plane (to which a code (R) is given in the background plane 11 shown in FIG. 5), respectively.

When the display mode of the background plane is in 3D display mode, and the stereo mode is OFF, the background plane 11 decodes the left-view still image data from among the 3D background images stored in the still image memory 27a (the left-view still image data and the right-view still image data), and writes the decoded left-view image data held in the left-eye plane (to which a code (L) is given in the background plane 11 shown in FIG. 5) and in the right-eye plane (to which a code (R) is given in the background plane 11 shown in FIG. 5).

When the display mode of the background image is a 2D display mode, the still image decoder 27b decodes the 2D still image data stored in the still image decoder 27a, and writes the decoded 2D still image data held in the left-eye plane (to which a code (L) is given in the background plane 11 shown in FIG. 5).

Although examples are shown for cases where the left-eye plane (to which the code (L) is given) and a right-eye plane (to which the code (R) is given) of the background plane 11 shown in FIG. 5 are provided in one plane memory, the structure of the background plane 11 is not limited to this. Therefore, the left-eye plane (to which the code (L) is given) and the right-eye plane (to which the code (R) is given) of the background plane 11 may be physically separated from one another.

Note that although it is disclosed that each of the video plane 6, the image plane 8, the interactive graphics plane 10 and the background plane 11 as shown in FIG. 5 is provided with a storage area for storing the left-eye data and a storage area for storing the right-eye data, the structures of these planes are not limited to this. Therefore, each of the planes may have only one memory area that is alternatively used as a left-eye area and a right-eye area.

The PSR set 12 is a collection of registers including a playback state register storing therein information on playback states of playlists, a playback setting register storing configuration information showing a configuration in the playback apparatus 200, and a general register capable of storing arbitrary information used by the contents. Each of the playback states of the playlists shows which of the AV data pieces in each kind of AV data information pieces that are written in the playlist are used, and at which position (time point) of the playlist the playback is performed.

When the state of each of the playlists changes, the playback control engine 14 stores information on what has been changed in the PSR set 12. Also, according to an instruction from the application executed by the command interpreter which is a main operational part in the HDMV mode or the main operational part in the BD-J mode, the playback control engine 14 is capable of storing a value specified by the application, and transferring the stored value to the application.

The static scenario memory 13 is a memory for storing current PlayList information or a current clip information. The current PlayList information is a current processing target from among a plurality of pieces of PlayList information accessible from the BD-ROM, a built-in media drive or a removable media drive. The current clip information is a current processing target from among a plurality of pieces of clip information accessible from the BD-ROM, a built-in media drive or a removable media drive.

The playback control engine 14 executes an AV playback function and a playback function of the playlist in response to a function call from the command interpreter which is the main operational part in the HDMV mode and the Java platform which is the main operational part in the BD-J function. The AV playback function is a set of functions used in DVD players and CD players, and includes playback start, playback stop, pause, release of pause, release of freeze frame function, fast forwarding at a playback speed specified by an immediate value, fast rewinding at a playback speed specified by an immediate value, audio conversion, sub image conversion, and angle conversion. The playlist playback function is to perform playback start or playback stop from among the above-stated AV playback functions according to the current PlayList information composing current playlists, and current clip information.

When a disc is inserted into the medium, a playlist and an AV stream that are playback processing targets by the playback control engine 12 are a auto start playlist (AutoStartPlaylist) and a default start stream respectively that are written in the current scenario on the BD-ROM. The playback of the AV stream starts due to a user operation (e.g. playback button) or automatically done by event triggered by the terminal (i.e such as resident application).

The composition unit 15 composites data held in the interactive graphics plane 10, data held in the image plane 8, data held in the video plane 6 and data held in the background plane 11.

Each of the interactive graphics plane 10, the image plane 8, the video plane 6 and the background plane 11 has a separate layer structure. Data held in each of the planes is composited (overlaid) in order of the background plane 11, the video plane 6, the image plane 8, then the interactive graphics plane 10. That is, even in the case where the planar graphics is composited with the stereoscopic video, the composition unit 15 composites video held in the video plane 6 with the background image held in the background plane 11, then composites the subtitles held in the image plane 8, then at last composites the graphics held in the interactive graphics plane 10. The composited image is displayed as a result. If done otherwise, graphics part may be hidden by the video, and thus will look unnatural.

The composition unit 15 also includes a scalar function. When the composition unit 15 composites shifted data pieces in the planes, the composition unit 15 is capable of performing scaling to make the image appear to be smaller or larger.

The purpose of performing the scaling is described in the following. In the real world, close objects appear to be larger, and distant objects appear to be smaller. However, just shifting the above-stated image data makes the object appear to be close to or distant from the viewer with its size unchanged. In such case, the viewer possibly may feel uncomfortable. The scaling is performed for the purpose of reducing such a sense of discomfort.

One example, would be to in order to display the image closer to the viewer with a large shift amount, the subtitles held in the image plane 8 and the image held in the interactive graphics plane 10 can be enlarged at the timing of scaling.

The HDMI transmission/reception unit 16 includes an interface that complies with the HDMI standard (HDMI: High Definition Multimedia Interface). The HDMI transmission/reception unit 16 performs transmission and reception such that the playback apparatus 200 and a device (in this example, a TV 400) that performs the HDMI connection with the playback apparatus 200 comply with the HDMI standard. The picture data stored in the video and audio data decoded from the uncompressed audio data by the audio decoder 9 are transmitted to the TV 400 via the HDMI transmission/reception unit 16. The TV 400 holds information such as, whether the TV 400 is capable of displaying data stereoscopically, information regarding-resolutions at which the planar display can be performed, and information regarding resolutions at which the stereoscopic display can be performed. When the playback apparatus 200 gives a request via the HDMI transmission/reception unit 16, the TV 400 gives the playback apparatus 200 necessary information (e.g. information regarding whether the TV 400 is capable of displaying data stereoscopically, information regarding resolutions at which the planar display can be performed, and information regarding resolutions at which the stereoscopic display can be performed) requested by the TV 400. Thus, the playback apparatus 200 is capable of obtaining, from the TV 400, the information regarding whether the TV 400 is capable of displaying data stereoscopically via the HDMI transmission/reception unit 16.

The left-right processing storage unit 17 stores information showing whether the current output processing is for left-view video or right-view video. A flag in the left-right processing storage unit 17 shows whether or not data to be outputted to a display device (TV in FIG. 1) connected to the playback apparatus 200 shown in FIG. 1 is the left-view video or the right-view video. While the left-view video is outputted, the flag in the left-right processing storage unit 17 is set as the left-view output. Also, while the right-view video is outputted, the flag in the left-right processing storage unit 17 is set as the right-view output.

The display function flag holding unit 18 stores a 3D display function flag showing whether the playback apparatus 200 is capable of performing the 3D display or not.

The plane shift engine 19 shifts coordinates of each of pixel data pieces held in the image plane 8 and/or coordinates of each of the pixel data pieces in the interactive graphics plane 10 in a predetermined direction (e.g. in a horizontal direction on a display screen) based on the flag in the left-right processing storage unit 31 and depth information on the data held in the graphics plane. That is, even if the objects of the graphics such as subtitles and the GUI used for data held in the image plane 8 and the data held in the interactive graphics plane 10 are not materials for the stereoscopic viewing, it is possible to obtain an effect that the objects are displayed in a position closer to the viewer than a position of the display screen. When the viewer wants only the graphics to have a stereoscopic effect with video displayed as 2 dimension, the composition unit 15 uses the left-view video for both the left view and the right view instead of using the set of the left-view video and the right-view video, and then composites the shifted data held in the image plane 8 and the shifted data held in the interactive graphics plane 10 with the video. The shifting targets do not have to be both the data held in the image plane 8 and the data held in the interactive graphics plane 10. Shifting target can be done for only one of the planes, either the data held in the interactive graphics plane 10 or the data held in the image plane 8.

The plane shift engine 19 includes a storage area for storing a “plane offset” (offset value) showing a direction in which data is shifted, along with distance by which the data is shifted, for performing the shifting. For example, when the playback apparatus 200 includes a setup function that can set a plane offset value, the plane shift engine 19 stores a value set using the setup function. The playback apparatus 200 may have two offset values, one for the image plane 8, and the other for the interactive graphics plane 10, and chooses one of the offset values for use according to a shifting target. When the playback apparatus 200 does not have the setup function, “0” may be specified as a default (in this case, graphics such as the subtitles and the GUI are displayed in the position of the display screen, and there is no effect that an object pops out from the display screen).

The rendering engine 21 includes base software (e.g. Java 2D, OPEN-GL), and writes graphics and a character string in the interactive graphics plane 10 in accordance with the instruction from the BD-J platform 22 in the BD-J mode. Also, in the HMV mode, the rendering engine 21 writes graphics data (e.g. graphics data corresponding to an input button) extracted from the graphics stream other than a stream corresponding to the subtitles (subtitle stream), and writes the extracted graphics data held in the interactive graphics plane 10.

The BD-J platform 22 is a Java platform which is a main operational part in the BD-J mode. The BD-J platform 22 is fully provided with the Java2 Micro_Edition (J2ME) Personal Basis Profile (PBP 1.0) and Globally Executable MHP specification (GEM1.0.2) for package media targets. The BD-J platform 22 reads byte codes from a class file in the JAR archive file, and stores the heap memory to start the BD-J application. Then, the BD-J platform 22 converts byte codes composing the BD-J application and byte codes composing a system application into native codes, and causes the MPU to execute the native codes.

The dynamic scenario memory 23 stores current dynamic scenario, and is used for processing by the HDMV module which is the main operational part in the HDMV mode, and the Java platform which is the main operational part in the BD-J mode. The current dynamic scenario is a current execution target which is one of Index.bdmv, the BD-J object and the movie object recorded in the BD-ROM, the built-in media, or the removable media.

The display mode management module 24 stores Index.bdmv read from the BD-ROM, the built-in media, or the removable media, and performs mode management and branching control. The mode management by the mode management module 24 is to perform allocation of the dynamic scenario to the module (i.e. to cause one of the BD-J platform 22 and the HDMV module 25 to execute the dynamic scenario).

The HDMV module 25 is a DVD virtual player to be a main operational part in the HDMV mode, and is a main execution part. This module includes a command interpreter, and executes control of the HDMV mode by reading and executing the navigation commands composing the movie object. The navigation commands are written by a syntax similar to a syntax for the DVD-Video. Therefore, the playback control like the DVD-Video can be realized by executing these navigation commands.

The UO detection module 26 receives the user operation on the GUI. The user operation received by the GUI includes title selection determining which of the titles recorded in the BD-ROM is selected, subtitle selection and audio selection. In particular, one of the user operations unique to the stereoscopic playback is to receive the depth of stereoscopic video. For example, there are three levels of the depth such as distant, usual and close, or levels of the depth may be expressed by the numerical values such as how many centimeter or how many millimeter.

The still image memory 27a stores still image data read from the BD-ROM for configured virtual package).

The still image decoder 27b decodes still image data read from the read buffer 27a, and writes the uncompressed background image data held in the background plane 11.

The display mode storage unit 29 stores information on a display mode and information on a stereo mode. When the 3D display function flag of the playback apparatus 200 shows that the playback apparatus 200 is capable of displaying 3D video, the display mode which is a terminal setting stored in the display mode storage unit 29 may be switched to one of a 2D mode and a 3D mode. Hereinafter, a state of the display mode shown as “3D” is referred to as a “3D display mode”, and a state of the display mode shown as “2D” is referred to as a “2D display mode”.

When the playback apparatus 200 is in the 3D playback mode, the stereo modes of the planes are either ON or OFF. The difference between ON and OFF of the stereo modes affects compositing methods for the planes.

“Stereo mode ON” is a 3D display mode in which composition is performed such that the playback apparatus 200 displays the left-view and the right-view that look different.

“Stereo mode OFF” is a 3D display mode in which composition is performed such that the playback apparatus 200 displays the left-view and the right-view that look the same. That is, when viewed by both of the eyes, the picture does not look stereoscopic (planar). However, when the data held in the graphics plane 8 is shifted in the horizontal direction by the plane offset, the plane graphics data (subtitle data) that is held in the graphics plane 8 is to be displayed can be displayed in a position closer to the viewer than a position of the display screen, or in a position more distant from the viewer than a position of the display screen. The same effect can be obtained when offsets of video data held in the video plane 6, interactive graphics data held in the interactive graphics plane 10, and background image data held in the background plane 11 are adjusted when the “stereo mode is in the OFF state”.

As described in the above, there are two modes, “Stereo mode ON” and “Stereo mode OFF” in the “3D display mode”. When the playback apparatus 200 is in the “Stereo mode ON” state in the 3D display mode, the left-view data and the right-view data (e.g. an image viewed by the left eye and the image viewed by the right eye can be seen from different angles) are held in the left-eye plane and the right-view plane, respectively, and displayed in accordance with the SYNC signal. This makes it possible to display the stereoscopic image.

Also, when the playback apparatus 200 is in the “Stereo mode OFF” state in the 3D display mode, one of the left-view data and the right-view data (the left-view data held in the present embodiment) is held in each of the left-eye plane and the right-view plane, and the offsets of the stored data pieces are adjusted. This makes it possible to display the planar image in a position closer to or more distant from the viewer than the position of the display screen.

In the present embodiment, the “Stereo mode ON” and the “stereo mode OFF” can be set for each plane (i.e. the video plane 6, the graphics plane 8, the interactive graphics plane 10 and the background plane 11).

The “2D display mode” is a normal display that displays the image in a position corresponding to the position of the display screen. In such case, a decoder and a plane used in a default setting is predetermined, and the composited image is displayed with use of the decoder and the plane.

For example, when the playback apparatus 200 is in the “2D display mode”, the composition unit 15 composites: the 2D video data written by the video decoder 5a in the left-eye video plane (expressed as the code (L) in the video plane 6 in FIG. 5); the 2D graphics data (subtitle data) written by the image decoder 7a in the left-eye plane (expressed as the code (L) in the image plane 8 in FIG. 5); the 2D interactive graphics written by the BD-J application in the left-eye plane (expressed as the code (L) in the interactive graphics plane 10 in FIG. 5) using the rendering engine 21; and the still image data written by the still image decoder 27b in the left-eye plane (expressed as the code (L) in the background plane 11 in FIG. 5).

At this time, the composition unit 15 performs the composition in the order of the 2D still image data, the 2D video data, the 2D graphics data (subtitle data) and the 2D interactive graphics data in order from the data from the bottom. The composition may be performed without compositing the 2D still image data when the video data is displayed on the whole screen. The flag in the display mode storage unit 29 showing whether the playback apparatus 200 is in the 2D display mode or the 3D display mode may also be stored in the playback state register 12, or may be stored in both the display mode storage unit 29 and the playback state 5, register 12.

The display mode setting initial display setting unit 28, sets the display mode and the resolutions based on the BD-J object in the current title provided with the BD-J platform unit.

This concludes a description of the internal structure of the playback apparatus 200. The following describes switching between the 2D display mode and the 3D display mode in the present embodiment in detail.

FIG. 6 shows switching between the 2D display mode and the 3D display mode. On the left side of FIG. 6, an output model in the 2D display mode is shown. In the plane structure on the left side of FIG. 6, only one each of the video plane 6, the image plane 8 (“Subtitle” in FIG. 6), the interactive graphics plane 10, the background plane 11 and an output is prepared.

Therefore, the same data is used for the left view and the right view in the 2D display mode. As a result, the same data is outputted.

On the right side of FIG. 6, an output model in the 3D display mode is shown. When the playback apparatus 200 is in the 3D display mode, the video plane 6, the image plane 8 (“Subtitle” in FIG. 6) and the interactive graphics plane 10 are prepared for each of the left view and the right view. The picture data and the graphics data to be played back are held in each of the video plane 6, the image plane 8 (“Subtitle” in FIG. 6), the interactive graphics plane 10 and the background plane 11 for the left eye and the right eye.

Therefore, the left-view output and the right-view output are performed separately in the 3D display mode. A different image can be provided for the left eye and the right eye. As a result, it is possible to obtain a 3D effect that the stereoscopic object in the screen appears to pop out closer to the viewer due to the binocular disparity.

(Specification of the Stereoscopic Effect)

FIG. 7 shows one example of composition processing when stereo modes of all the planes are ON, and when stereo modes of all the planes are OFF, in the 3D display mode.

Although FIG. 7 shows an example of a case where the stereo modes of the planes are the same, ON/OFF of the stereo mode may be changed for each of the planes.

On the left side of FIG. 7, the plane structure when the stereo modes of all the planes are ON is shown. On the right side of FIG. 7, the plane structure when the stereo modes of all the planes are ON.

The first row shows the background plane 11 and the outputted data before the composition.

The second row shows the video stream and the outputted data before the composition.

The third row shows the image plane 8 and the outputted data before the composition.

The fourth row shows the interactive graphics plane 10 and the outputted data before the composition.

When the stereo mode is ON, a left-eye background plane which is expressed as an area to which (L) is given is used for writing the left-view background data, and a right-eye background plane which is expressed as an area to which (R) is given is used for writing the right-view background data. Each of the background data pieces is composited with the corresponding left-eye or right-view picture. When the stereo mode of the background plane 11 is OFF, the left-view background data is written, by the application, in each of the areas to which (L) and (R) are given respectively in the background plane 11. Therefore, the right-view background data does not affect the display.

When the stereo mode is ON, picture data of the left-eye video in the video stream is held in the left-view video plane. Also, picture data of right-eye video in the video stream is held in the right-view video plane. When the video plane 6 is in the OFF state of the stereo mode, the picture data of the left-eye video is held in both the left-view video plane and the right-view video plane.

When the stereo mode is ON, in the image plane 8, left-view image data is written in a left-eye image plane expressed as the area to which (L) is given, and right-view image data is written in a right-eye image plane expressed as the area to which (R) is given. Each of the image data pieces is composited with the corresponding left-eye or right-view picture.

When the image plane 8 is in the OFF state of the stereo mode, the subtitle graphics corresponding to the right-view image data does not affect the display. Also, when the stereo mode is OFF, the content in the image plane 8 is a content shifted in the right or left direction (“Shifted Left” in FIG. 7).

When the stereo mode is ON, in the interactive graphics plane 10, left-view interactive graphics is written in a left-view interactive graphics plane expressed as an area to which (L) is given, and right-view interactive graphics is written in a right-view interactive graphics plane expressed as an area to which (R) is given. Each of the interactive graphics data pieces is composited with the corresponding left-eye or right-view picture.

When the interactive graphics plane 10 is in the OFF state of the stereo mode, the right-view interactive graphics by the application does not affect the display. Also, when the stereo mode is OFF, the content in the interactive graphics plane 10 is a content that is shifted in the right or left direction (“Shifted Left” in FIG. 7).

FIG. 8 shows how the data held in the background plane 11, the data held in the video plane 6, the data held in the image plane 8 and the data held in the interactive graphics plane 10 are composited with one another when the stereo modes of all the planes are ON. In the stereo mode, it can be seen that left-view background data u4, left-view video u3 read from the video stream, left-view graphics u2 in the image plane 8 and left-view graphics u1 in the interactive graphics plane 10 are composited as the left-view in this order.

Also, it can be seen that right-view background data u8, right-view video u7 read from the video stream, right-view graphics u6 in the image plane 8 and right-view graphics u5 in the interactive graphics plane 10 are composited as the right view in this order.

FIG. 9 shows how the data held in the background plane 11, the data held in the video plane 6, the data held in the image plane 8 and the data held in the interactive graphics plane 10 are composited with one another when the stereo modes of all the planes are OFF. When the stereo modes are OFF, it can be seen that left-view background data r4, left-view video r2 read from the video stream, Shifted Left graphics r3 which is left-view graphics in the image plane 8 that has been shifted in a predetermined direction (the right direction in FIG. 8) and Shifted Left graphics r1 which is left-view graphics in the interactive graphics plane 10 that has been shifted in a predetermined direction (the right direction in FIG. 9) are composited as the right view in this order.

Also, it can be seen that right-view background data r8, left-view video r6 read from the video stream, Shifted Left graphics r7 which is the left-view graphics in the image plane 8 that has been shifted in a direction opposite to the predetermined direction (the left direction in FIG. 8) and Shifted Left graphics r5 that is the left-view graphics in the interactive graphics plane 10 that has been shifted in a direction opposite to the predetermined direction (the left direction in FIG. 8) are composited as the right view in this order.

In FIG. 7 to FIG. 9, offsets of the background data and the video were not adjusted (i.e. an offset is 0, more specifically, the image is displayed in the position of the display screen), strictly for the purpose of simplifying the above-stated description. Therefore, the setting of the offsets is not limited to the above-stated descriptions, and the offsets may be adjusted such that the video is positioned in a position more distant from the viewer than a position of the graphics image (subtitles), and the background data is in a position more distant from the viewer than a position of the background data.

The following describes switching between the stereo modes in the present embodiment.