Image generation of a three-dimensional scene using multiple focal lengths

• US 10,212,406 B2
• Filed: 12/15/2016
• Issued: 02/19/2019
• Est. Priority Date: 12/15/2016
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Gallo, Orazio, Kautz, Jan, Badki, Abhishek Haridas
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US15/381,010
• Publication Number:
  US 20180176532A1
• Time in Patent Office: 796 Days
• Field of Search: 348 46
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

A system and method for computational zoom generates a resulting image having two or more effective focal lengths. A first surface within a three-dimensional (3D) scene including a first and second set of 3D objects defined by 3D information is identified. The first and second sets of 3D objects are located within first and second depth ranges of the 3D scene, respectively. The first set of 3D objects is projected onto the first surface according to a first projection mapping to produce a first portion of image components. The second set of 3D objects is projected onto the first surface according to a second projection mapping to produce a second portion of image components. The resulting image comprising the first portion of image components and the second portion of image components is generated based on a camera projection from the first surface to a camera view plane.

Claims

1. A method, comprising:

identifying a first surface within a three-dimensional (3D) scene, wherein the 3D scene includes a first set of 3D objects located within a first depth range of the 3D scene and a second set of 3D objects located within a second depth range of the 3D scene, and 3D information defines the first set of 3D objects and the second set of 3D objects;

projecting, by a processing unit, the first set of 3D objects onto the first surface according to a first projection mapping specified by a first focal length to produce a first portion of image components;

projecting, by the processing unit, the second set of 3D objects onto the first surface according to a second projection mapping specified by a second focal length to produce a second portion of image components; and

generating, by the processing unit, a first resulting image comprising the first portion of image components according to the first focal length and the second portion of image components according to the second focal length based on a camera projection from the first surface to a camera view plane.

2. The method of claim 1, wherein the first depth range is bounded by the camera view plane and the first surface.

3. The method of claim 1, further comprising:

receiving at least two images of the 3D scene;

obtaining depth map information for the at least two images; and

generating the 3D information based on the at least two images and the at least two depth maps.

4. The method of claim 3, wherein the at least two images comprise different video frames in a sequence of video frames captured by a digital camera.

5. The method of claim 3, wherein the at least two images comprise images rendered by a 3D rendering system.

6. The method of claim 1, wherein the 3D information defines a third set of 3D objects within a third depth range, and further comprising:

identifying a second surface within the 3D scene;

projecting the third set of 3D objects onto the second surface according to a third projection mapping to produce a third portion of image components; and

mapping the second surface into the 3D information.

7. The method of claim 6, wherein identifying the second surface comprises locating the second surface at a specified offset from the first surface.

8. The method of claim 1, wherein a view volume defines a portion of the 3D scene within a field of view for the camera view plane, the view volume comprising a near portion of a first view frustum and a far portion of a second view frustum.

9. The method of claim 1, wherein the first depth range and the second depth range are measured as a projection from the camera view plane into the 3D scene.

10. The method of claim 1, wherein identifying the first surface comprises determining a depth value for the first surface based on a specified screen location.

11. The method of claim 10, wherein the first surface comprises a plane that is parallel to the camera view plane.

12. The method of claim 10, further comprising adjusting the depth value for the first surface in response to a user interface input.

13. The method of claim 1, wherein at least one of the first projection mapping and the second projection mapping is specified in response to a user interface input.

14. The method of claim 1, wherein at least one of the first projection mapping and the second projection mapping comprises a homography operator.

15. The method of claim 1, further comprising generating a second resulting image according to a modified second projection mapping.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to:

identify a first surface within a three-dimensional (3D) scene, wherein the 3D scene includes a first set of 3D objects located within a first depth range of the 3D scene and a second set of 3D objects located within a second depth range of the 3D scene, and 3D information defines the first set of 3D objects and the second set of 3D objects;

project the first set of 3D objects onto the first surface according to a first projection mapping specified by a first focal length to produce a first portion of image components;

project the second set of 3D objects onto the first surface according to a second projection mapping specified by a second focal length to produce a second portion of image components; and

generate a first resulting image comprising the first portion of image components according to the first focal length and the second portion of image components according to the second focal length based on a camera projection from the first surface to a camera view plane.

17. The non-transitory computer-readable storage medium of claim 16, further comprising instructions that, when executed by a processor, cause the processor to:

receive at least two images of the 3D scene;

obtain depth map information for the at least two images; and

generate the 3D information based on the at least two images and the at least two depth maps.

18. A system, comprising:

a display device;

a touch input device;

a digital camera module;

a processing unit coupled to the display device, the touch input device, and the digital camera module, the processing unit configured to;

identify a first surface within a three-dimensional (3D) scene, wherein the 3D scene includes a first set of 3D objects located within a first depth range of the 3D scene and a second set of 3D objects located within a second depth range of the 3D scene, and 3D information defines the first set of 3D objects and the second set of 3D objects;

project the first set of 3D objects onto the first surface according to a first projection mapping specified by a first focal length to produce a first portion of image components;

project the second set of 3D objects onto the first surface according to a second projection mapping specified by a second focal length to produce a second portion of image components; and

generate a first resulting image comprising the first portion of image components according to the first focal length and the second portion of image components according to the second focal length based on a camera projection from the first surface to a camera view plane.

19. The system of claim 18, wherein the processing unit is further configured to:

receive at least two images of the 3D scene;

obtain depth map information for the at least two images; and

generate the 3D information based on the at least two images and the at least two depth maps,

wherein the at least two images comprise different video frames captured by the digital camera module.

20. The system of claim 18, wherein the 3D information defines a third set of 3D objects within a third depth range, and the processing unit is further configured to:

identify a second surface within the 3D scene;

project the third set of 3D objects onto the second surface according to a third projection mapping to produce a third portion of image components; and

map the second surface into the 3D information, based on user input detected by the touch input device.