Parallax scanning through scene object position manipulation

This application claims the benefit of U.S. Provisional Application No. 60/429,358, filed Nov. 27, 2002, which is incorporated herein by reference.

The present invention relates to the visual arts field and more particularly to a system and method for using parallax scanning to generate and display autostereoscopic 3D computer graphics images.

The production of two-dimensional images that can be displayed to provide a three-dimensional illusion has been a long-standing goal in the visual arts field. Methods and apparatus for producing such three-dimensional illusions have to some extent paralleled the increased understanding of the physiology of human depth perception, as well as, developments in image manipulation through analog/digital signal processing and computer imaging software.

Binocular (i.e., stereo) vision requires two eyes that look in the same direction, with overlapping visual fields. Each eye views a scene from a slightly different angle and focuses it onto the retina, a concave surface at the back of the eye lined with nerve cells, or neurons. The two-dimensional retinal images from each eye are transmitted along the optic nerves to the brain\'s visual cortex, where they are combined, in a process known as stereopsis, to form a perceived three-dimensional model of the scene.

Perception of three-dimensional space depends on various kinds of information in the scene being viewed including monocular cues and binocular cues, for example. Monocular cues include elements such as relative size, linear perspective, interposition, light, and shadow. Binocular cues include retinal disparity, accommodation, convergence, and learned cues (e.g., familiarity with the subject matter). While all these factors may contribute to creating a perception of three-dimensional space in a scene, retinal disparity may provide one of the most important sources of information for creating the three-dimensional perception. Particularly, retinal disparity results in parallax information (i.e., an apparent change in the position, direction of motion, or other visual characteristics of an object caused by different observational positions) being supplied to the brain. Because each eye has a different observational position, each eye can provide a slightly different view of the same scene. The differences between the views represents parallax information that the brain can use to perceive three dimensional aspects of a scene.

A distinction exists between monocular depth cues and parallax information in the visual information received. Both eyes provide essentially the same monocular depth cues, but each provides different parallax depth information, a difference that is essential for producing a true three-dimensional view.

Depth information may be perceived, to a certain extent, in a two-dimensional image. For example, monocular depth may be perceived when viewing a still photograph, a painting, standard television and movies, or when looking at a scene with one eye closed. Monocular depth is perceived without the benefit of binocular parallax depth information. Such depth relations are interpreted by the brain from monocular depth cues such as relative size, overlapping, perspective, and shading. To interpret monocular depth information from a two-dimensional image (i.e., using monocular cues to indicate a three-dimensional space on a two-dimensional plane), the viewer is actually reading depth information into the image through a process learned in childhood.

True three dimensional images may differ from computer generated images commonly referred to as 3D or three-dimensional. Specifically, the term three-dimensional (3-D) has been expanded over the past several years by the computer-imaging industry to include images produced using depth cues that take advantage of perspective, shading, reflections, and motion. Although these images can be rendered with incredible results, they are nevertheless two-dimensional because they lack the parallax depth information found in true three dimensional images.

Several systems and methods exist for creating and/or displaying true three dimensional images. These methods may be divided into two main categories: stereoscopic display methods and autostereoscopic display methods. Stereoscopic techniques including stereoscopes, polarization, anaglyphic, Pulfrich, and shuttering technologies require the viewer to wear a special viewing apparatus such as glasses, for example. Auto-stereoscopic techniques such as holography, lenticular screens, and parallax barriers produce images with a three-dimensional illusion without the use of special glasses, but these methods generally require the use of a special screen.

Certain other systems and methods may use parallax scanning information to create autostereoscopic displays that allow a viewer to perceive an image as three-dimensional even when viewed on a conventional display. For example, at least one method has been proposed in which a single camera records images while undergoing parallax scanning motion. Thus, the optical axis of a single camera may be made to move in a repetitive pattern that causes the camera optical axis to be offset from a nominal stationary axis. This offset produces parallax information. The motion of the camera optical axis is referred to as parallax scanning motion. As the motion repeats over the pattern, the motion becomes oscillatory. At any particular instant, the motion may be described in terms of a parallax scan angle.

To generate an autostereoscopic display based on the parallax information, images captured during the scanning motion may be sequentially displayed. These images may be displayed at a view cycle rate of, for example, about 3 Hz to about 6 Hz. This frequency represents the rate at which the parallax image views in the sequence are changed. The displayed sequences of parallax images may provide an autostereoscopic display that conveys three-dimensional information to a viewer.

Parallax information may also be incorporated into computer generated images. For example, U.S. Pat. No. 6,324,347 (“the \'347 patent”), which is incorporated herein by reference, discloses a method for computer generating parallax images using a virtual camera having a virtual lens. The parallax images may be generated by simulating a desired parallax scanning pattern of the lens aperture, and a ray tracing algorithm, for example, may be used to produce the images. The images may be stored in computer memory on a frame-by-frame basis. The images may be retrieved from memory for display on a computer monitor, recorded on video tape for display on a TV screen, and/or recorded on film for projection on a screen.

Thus, in the method of the \'347 patent, the point of view of a camera (e.g., the lens aperture) is moved to produce the parallax scanning information. The ray tracing method of image generation, as may be used by one embodiment of the method of the \'347 patent, may be used to generate high-quality computer images, such as those used in movie special effects. Using this ray-tracing method to simulate optical effects such as depth of field variations, however, may require large amounts of computation and can place a heavy burden on processing resources. Therefore, such a ray tracing method may be impractical for certain applications, such as 3D computer games, animation, and other graphics applications, which require quick response.

Creating parallax image information by simulating the motion of a virtual lens may, in certain situations, create instability in the displayed parallax images. FIGS. 1A and 1B illustrate one embodiment of the method used in the \'347 patent to generate parallax information. FIG. 1A illustrates a condition where three objects, A, B, and C, reside on an optical axis 20 of camera 11. The method of the \'347 patent involves moving the point of view of camera 11 (e.g., the lens position) to generate parallax information. For example, as shown in FIG. 1B, camera 11 has been moved with respect to its original position in FIG. A. As a result, objects in front of and behind a convergence point 12 located on convergence plane 10 will appear to move with respect to the optical axis 20. Specifically, object A in front of convergence point 12 will appear to move in one direction, and object C will appear to move with respect to optical axis 20 in a direction opposite from the direction of motion of object A.

In the method of the \'347 patent in which the point of view is moved, the amount that objects A and C appear to move linearly depends on their respective distances from the lens. As illustrated in FIG. 2A, objects located beyond convergence point 12 will receive a linearly increasing amount of parallax offset as the distance from the point of view increases. This property, however, may cause instability in the displayed parallax images. Specifically, in the displayed parallax images, objects far from the point of view will appear to move by large distances compared to objects closer to the point of view. Because objects far from the point of view contribute less depth information than objects closer to the point of view, the motion of distant objects is less important and may even cause image instability (e.g., a jitter effect caused by the motion of objects between successive parallax image frames). Using the method of the \'347 patent (i.e., moving a virtual point of view to generate parallax information), under certain circumstances, direct control of object stability at depth extremes may be impractical.

The present invention is directed to overcoming one or more of the problems associated with the prior art.

One aspect of the invention includes a method of generating virtual autostereoscopic images using a computer. The method may include defining at least one object in a virtual three-dimensional space and defining a virtual point of view. Images may be generated by simulating a parallax scanning motion of the at least one object. The generated images can be displayed and stored.

Another aspect of the invention includes a method of generating an image. The method may include automatically monitoring at least one scene parameter associated with a scene to be imaged, determining a parallax scanning parameter based on the at least one scene parameter, applying the parallax scanning parameter to at least one object in the scene, and generating an image including the at least one object.