Conversion of a digital stereo image into multiple views with parallax for 3D viewing without glasses

• US 9,786,253 B2
• Filed: 01/25/2013
• Issued: 10/10/2017
• Est. Priority Date: 01/25/2012
• Status: Active Grant

• Current Assignee: Lumenco, LLC
• Original Assignee: Lumenco, LLC
• Inventors: Raymond, Mark A., Soto, Hector Andres Porras, Karsch, Kevin
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US13/749,861
• Publication Number:
  US 20130187910A1
• Time in Patent Office: 1,719 Days
• Field of Search: 345419, 348 43, 382154, 398 8
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/31 using parallax barriers
• CPC Class Codes: H04N 13/317 using slanted parallax optics
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/351 for displaying simultaneously
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

A method for generating additional views from a stereo image defined by a left eye image and a right eye image. The method includes receiving as input at least one stereo image. The method includes, for each of the stereo images, generating a plurality of additional images. The method includes interlacing the additional images for each of the stereo images to generate three dimensional (3D) content made up of multiple views of the scenes presented by each of the stereo images. The interlacing may be performed such that the generated 3D content is displayable on a 3D display device including a barrier grid or a lenticular lens array on the monitor screen. The additional images may include 12 to 40 or more frames providing views of the one or more scenes from differing viewing angles than provided by the left and right cameras used to generate the original stereo image.

Claims

1. A method of generating additional views from a stereo image, comprising:

executing a conversion module with a computer processor;

with the conversion module, receiving as input at least one stereo image of one or more scenes, wherein each stereo image includes a left image associated with a left camera and a right image associated with a right camera;

for each of the stereo images, generating a plurality of additional images of the one or more scenes with the conversion module, wherein the generating comprises identifying objects in the one or more scenes that are in a foreground and in a background relative to a focal plane of the right and left cameras and wherein the generating involves moving the identified objects from positions in the left and right images to new positions in the additional images based on whether the identified objects are in the foreground or in the background; and

interlacing the plurality of additional images for each of the stereo images to generate a 3D content comprising multiple views of the one or more scenes for each of the stereo images,

wherein the generating includes identifying a hole in one of the additional images and repairing the identified hole,

wherein the interlacing is performed such that the generated 3D content is displayable on a 3D display device including a barrier grid or a lenticular lens array on or associated with a monitor,

wherein the interlacing comprises displaying and replacing the additional images at a pixel level at an angle and position relative to pixel flow and replacement on a screen of the monitor, whereby the additional images are viewable by an observer of the monitor with frames in an order to be viewed in 3D, and

wherein pixels of the interlaced additional images are displayed and replaced at one or more of the following angle ranges;

negative to positive 3 degrees relative to vertical, negative 13 to 16 degrees to positive 13 to 16 degrees relative to vertical, negative 17 to 19 degrees to positive 17 to 19 degrees relative to vertical, and negative 24 to 28 degrees to positive 24 to 28 degrees relative to vertical.

2. The method of claim 1, wherein the plurality of additional images includes at least 12 frames providing views of the one or more scenes from differing viewing angles than provided by the left and right cameras.

3. The method of claim 2, wherein the differing viewing angles define an amount of 3D provided by the generating step and are viewing angles outside of a zone defined by the viewing angles of the left and right cameras.

4. The method of claim 3, wherein the differing viewing angles are selected from the range of 30 to 150 degrees relative to a focal plane.

5. The method of claim 1, wherein the generating uses a user-selected amount of parallax to create the plurality of additional images.

6. The method of claim 1, further comprising determining a depth map providing a depth of each pixel in each of the stereo images, wherein the generating is performed based on the depth map.

7. The method of claim 1, wherein the input comprises a 3D movie or a 3D animated video comprising 20 to 24 frames per second, the frames each providing a pair of the left and right images for the one or more scenes.

8. The method of claim 1, wherein the input includes anaglyph information and the method includes, prior to the generating step, processing input convert the anaglyph information into the left and right images.

9. A method of generating additional views from a stereo image, comprising:

executing a conversion module with a computer processor;

with the conversion module, receiving as input at least one stereo image of one or more scenes, wherein each stereo image includes a left image associated with a left camera and a right image associated with a right camera;

for each of the stereo images, generating a plurality of additional images of the one or more scenes with the conversion module; and

interlacing the plurality of additional images for each of the stereo images to generate a 3D content comprising multiple views of the one or more scenes for each of the stereo images,

wherein the generating includes identifying a region of missing pixels in one of the additional images and providing the missing pixels in the region by stretching or enlarging an object adjacent to the region,

wherein the interlacing is performed such that the generated 3D content is displayable on a 3D display device including a barrier grid or a lenticular lens array on or associated with a monitor,

wherein the interlacing comprises displaying and replacing the additional images at a pixel level at an angle and position relative to pixel flow and replacement on a screen of the monitor, whereby the additional images are viewable by an observer of the monitor with frames in an order to be viewed in 3D, and

wherein pixels of the interlaced additional images are displayed and replaced at one or more of the following angle ranges;

negative to positive 3 degrees relative to vertical, negative 13 to 16 degrees to positive 13 to 16 degrees relative to vertical, negative 17 to 19 degrees to positive 17 to 19 degrees relative to vertical, and negative 24 to 28 degrees to positive 24 to 28 degrees relative to vertical.

10. The method of claim 9, wherein the interlaced additional images are provided in multiples of 16 for angles in the range of negative 13 to 16 degrees to positive 13 to 16 degrees relative to vertical, wherein the interlaced additional images are provided in multiples of 9 for angles in the range of negative 17 to 19 degrees to positive 17 to 19 degrees relative to vertical, wherein the interlaced additional images are provided in multiples of 6 for angles in the range of negative 24 to 28 degrees to positive 24 to 28 degrees relative to vertical.

11. The method of claim 9, wherein the plurality of additional images includes frames providing views of the one or more scenes from differing viewing angles than provided by the left and right cameras, wherein the differing viewing angles define an amount of 3D provided by the generating step and are viewing angles outside of a zone defined by the viewing angles of the left and right cameras, wherein the differing viewing angles are selected from the range of 30 to 150 degrees relative to a focal plane.

12. The method of claim 9, wherein the generating uses a user-selected amount of parallax to create the plurality of additional images.

13. The method of claim 9, further comprising determining a depth map providing a depth of each pixel in each of the stereo images, wherein the generating is performed based on the depth map and further comprising identifying objects in the one or more scenes that are in a foreground and in a background relative to a focal plane of the right and left cameras based on the depth map, wherein the generating involves moving the identified objects based on whether the identified objects are in the foreground or in the background.

14. A method of generating additional views from a stereo image, comprising:

executing a conversion module with a computer processor;

with the conversion module, receiving as input at least one stereo image of one or more scenes, wherein each stereo image includes a left image associated with a left camera and a right image associated with a right camera;

determining a depth map providing a depth of each pixel in each of the stereo images;

for each of the stereo images, generating a plurality of additional images of the one or more scenes with the conversion module based on the depth map;

identifying objects in the one or more scenes that are in a foreground and in a background relative to a focal plane of the right and left cameras based on the depth map, wherein the generating involves moving the identified objects from positions in the left and right images to new positions in the additional images based on whether the identified objects are in the foreground or in the background; and

interlacing the plurality of additional images for each of the stereo images to generate a 3D content comprising multiple views of the one or more scenes for each of the stereo images,

wherein the interlacing is performed such that the generated 3D content is displayable on a 3D display device including a barrier grid or a lenticular lens array on or associated with a monitor,

wherein the interlacing comprises displaying and replacing the additional images at a pixel level at an angle and position relative to pixel flow and replacement on a screen of the monitor, whereby the additional images are viewable by an observer of the monitor with frames in an order to be viewed in 3D, and

wherein pixels of the interlaced additional images are displayed and replaced at one or more of the following angle ranges;

negative to positive 3 degrees relative to vertical, negative 13 to 16 degrees to positive 13 to 16 degrees relative to vertical, negative 17 to 19 degrees to positive 17 to 19 degrees relative to vertical, and negative 24 to 28 degrees to positive 24 to 28 degrees relative to vertical.

15. The method of claim 14, wherein the interlaced additional images are provided in multiples of 16 for angles in the range of negative 13 to 16 degrees to positive 13 to 16 degrees relative to vertical, wherein the interlaced additional images are provided in multiples of 9 for angles in the range of negative 17 to 19 degrees to positive 17 to 19 degrees relative to vertical, wherein the interlaced additional images are provided in multiples of 6 for angles in the range of negative 24 to 28 degrees to positive 24 to 28 degrees relative to vertical.

16. The method of claim 14, wherein the plurality of additional images includes frames providing views of the one or more scenes from differing viewing angles than provided by the left and right cameras, wherein the differing viewing angles define an amount of 3D provided by the generating step and are viewing angles outside of a zone defined by the viewing angles of the left and right cameras, wherein the differing viewing angles are selected from the range of 30 to 150 degrees relative to a focal plane.

17. The method of claim 14, wherein the generating uses a user-selected amount of parallax to create the plurality of additional images.

18. A method of generating additional views from a stereo image, comprising:

executing a conversion module with a computer processor;

with the conversion module, receiving as input at least one stereo image of one or more scenes, wherein each stereo image includes a left image associated with a left camera and a right image associated with a right camera;

for each of the stereo images, generating a plurality of additional images of the one or more scenes with the conversion module; and

interlacing the plurality of additional images for each of the stereo images to generate a 3D content comprising multiple views of the one or more scenes for each of the stereo images,

wherein the interlacing is performed such that the generated 3D content is displayable on a 3D display device including a barrier grid or a lenticular lens array on or associated with a monitor,

wherein the interlacing comprises displaying and replacing the additional images at a pixel level at an angle and position relative to pixel flow and replacement on a screen of the monitor, whereby the additional images are viewable by an observer of the monitor with frames in an order to be viewed in 3D, and

wherein pixels of the interlaced additional images are displayed and replaced at one or more of the following angle ranges;

negative to positive 3 degrees relative to vertical, negative 13 to 16 degrees to positive 13 to 16 degrees relative to vertical, negative 17 to 19 degrees to positive 17 to 19 degrees relative to vertical, and negative 24 to 28 degrees to positive 24 to 28 degrees relative to vertical.

19. The method of claim 18, wherein the interlaced additional images are provided in multiples of 16 for angles in the range of negative 13 to 16 degrees to positive 13 to 16 degrees relative to vertical, wherein the interlaced additional images are provided in multiples of 9 for angles in the range of negative 17 to 19 degrees to positive 17 to 19 degrees relative to vertical, wherein the interlaced additional images are provided in multiples of 6 for angles in the range of negative 24 to 28 degrees to positive 24 to 28 degrees relative to vertical.

20. The method of claim 18, wherein the plurality of additional images includes frames providing views of the one or more scenes from differing viewing angles than provided by the left and right cameras, wherein the differing viewing angles define an amount of 3D provided by the generating step and are viewing angles outside of a zone defined by the viewing angles of the left and right cameras, wherein the differing viewing angles are selected from the range of 30 to 150 degrees relative to a focal plane.

21. The method of claim 18, wherein the generating uses a user-selected amount of parallax to create the plurality of additional images.

22. The method of claim 18, wherein the generating includes identifying a hole in one of the additional images and repairing the identified hole, and wherein the identified hole comprises a plurality of missing pixels and the repairing comprises providing the missing pixels by cloning pixels proximate to the missing pixels in the one or more scenes or by stretching or enlarging objects in the left image or the right image adjacent to the missing pixels.