Dynamic POV composite 3D video system

• US 9,392,248 B2
• Filed: 06/11/2013
• Issued: 07/12/2016
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Wheeler, Aaron Joseph, Plagemann, Christian, Dahlkamp, Hendrik, Chi, Liang-Yu, Zhao, Yong, Ganapathi, Varun, Kauffmann, Alejandro Jose
• Primary Examiner(s): Rao, Andy
• Application Number:
  US13/914,727
• Publication Number:
  US 20150172634A1
• Time in Patent Office: 1,127 Days
• Field of Search: 345/427, 345/424, 348 42- 60, 348135-139
• US Class Current: 1/1
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

Systems and techniques are disclosed for visually rendering a requested scene based on a virtual camera perspective request as well as a projection of two or more video streams. The video streams can be captured using two dimensional cameras or three dimensional depth cameras and may capture different perspectives. The projection may be an internal projection that maps out the scene in three dimensions based on the two or more video streams. An object internal or external to the scene may be identified and the scene may be visually rendered based on a property of the object. For example, a scene may be visually rendered based on where an mobile object is located within the scene.

Claims

1. A computer-implemented method comprising:

obtaining (i) a first image of a scene as viewed by a first depth camera from a first perspective, and (ii) a second image of the scene as viewed by a second depth camera from a different, second perspective;

generating, based on the first image that corresponds to the scene as viewed by the first depth camera from the first perspective, first data that references (i) a three-dimensional coordinate position associated with each point in the first image, (ii) a brightness characteristic associated with each point in the first image, and (iii) a color characteristic associated with each point in the first image;

generating, based on the second image that corresponds to the scene as viewed by the second depth camera from the second perspective, second data that references (i) a three-dimensional coordinate position associated with each point in the second image, (ii) a brightness characteristic associated with each point in the second image, and (iii) a color characteristic associated with each point in the second image;

generating, based at least on the first data and the second data, a three-dimensional projection of the scene;

determining a different, third perspective of the scene as the scene would be viewed from a particular position in three-dimensional space;

generating, based at least on the generated three-dimensional projection of the scene, a virtual image of the scene as the scene would be as viewed from the third perspective; and

providing the virtual image of the scene for output.

2. The method of claim 1, wherein each point in the first and second images corresponds to a pixel.

3. The method of claim 1, further comprising:

receiving data identifying a particular object having the particular position in three-dimensional space;

identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space; and

wherein determining the different, third perspective of the scene as would be viewed from the particular position in three-dimensional space comprises;

based on identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space, determining the different, third perspective of the scene as would be viewed from the particular object'"'"'s position in three-dimensional space.

4. The method of claim 3, wherein identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space comprises:

obtaining data from one or more sensors indicating the particular object'"'"'s position in three-dimensional space.

5. The method of claim 4, wherein obtaining data from one or more sensors indicating the particular object'"'"'s position in three-dimensional space comprises:

obtaining data from a GPS transmitter indicating the particular object'"'"'s geographical position.

6. The method of claim 4, wherein the particular object is shown in neither the first image of the scene nor the second image of the scene.

7. The method of claim 3, wherein identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space comprises:

identifying, based on one or more of the first and second images, the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space.

8. The method of claim 7, wherein identifying, based on one or more of the first and second images, the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space comprises:

applying one or more recognition processes to at least one of the first and second images to identify the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space.

9. The method of claim 7, wherein the particular object is shown in one or more of the first and second images of the scene.

10. The method of claim 3, further comprising:

identifying, at a particular point in time after providing the virtual image of the scene for output, another position in three-dimensional space as the particular object'"'"'s position in three-dimensional space at the particular point in time;

updating, based at least on the three-dimensional projection of the scene, the virtual image of the scene to reflect the scene as would be viewed from the particular object'"'"'s position in three-dimensional space at the particular point in time; and

providing the updated virtual image of the scene for output.

11. The method of claim 3, wherein:

receiving data identifying the particular object having the particular position in three-dimensional space comprises receiving, from a client device, indication of a user of the client device having selected the particular object; and

providing the virtual image of the scene for output comprises providing the virtual image of the scene for output at the client device.

12. The method of claim 3, wherein receiving data identifying the particular object having the particular position in three-dimensional space comprises:

in response to a set of criteria being satisfied, receiving data identifying the particular object having the particular position in three-dimensional space.

13. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

obtaining (i) a first image of a scene as viewed by a first depth camera from a first perspective, and (ii) a second image of the scene as viewed by a second depth camera from a different, second perspective;

generating, based on the first image that corresponds to the scene as viewed by the first depth camera from the first perspective, first data that references (i) a three-dimensional coordinate position associated with each point in the first image, (ii) a brightness characteristic associated with each point in the first image, and (iii) a color characteristic associated with each point in the first image;

generating, based on the second image that corresponds to the scene as viewed by the second depth camera from the second perspective, second data that references (i) a three-dimensional coordinate position associated with each point in the second image, (ii) a brightness characteristic associated with each point in the second image, and (iii) a color characteristic associated with each point in the second image;

generating, based at least on the first data and the second data, a three-dimensional projection of the scene;

determining a different, third perspective of the scene as the scene would be viewed from a particular position in three-dimensional space;

generating, based at least on the generated three-dimensional projection of the scene, a virtual image of the scene as the scene would be as viewed from the third perspective; and

providing the virtual image of the scene for output.

14. The system of claim 13, wherein each point in the first and second images corresponds to a pixel.

15. The system of claim 13, wherein the operations further comprise:

receiving data identifying a particular object having the particular position in three-dimensional space;

identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space; and

wherein determining the different, third perspective of the scene as would be viewed from the particular position in three-dimensional space comprises;

based on identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space, determining the different, third perspective of the scene as would be viewed from the particular object'"'"'s position in three-dimensional space.

16. The system of claim 15, wherein identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space comprises:

obtaining data from one or more sensors indicating the particular object'"'"'s position in three-dimensional space.

17. The system of claim 16, wherein obtaining data from one or more sensors indicating the particular object'"'"'s position in three-dimensional space comprises:

obtaining data from a GPS transmitter indicating the particular object'"'"'s geographical position.

18. The system of claim 16, wherein the particular object is shown in neither the first image of the scene nor the second image of the scene.

19. The system of claim 15, wherein identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space comprises:

identifying, based on one or more of the first and second images, the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space.

20. The system of claim 19, wherein identifying, based on one or more of the first and second images, the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space comprises:

applying one or more recognition processes to at least one of the first and second images to identify the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space.

21. The system of claim 19, wherein the particular object is shown in one or more of the first and second images of the scene.

22. The system of claim 15, wherein the operations further comprise:

identifying, at a particular point in time after providing the virtual image of the scene for output, another position in three-dimensional space as the particular object'"'"'s position in three-dimensional space at the particular point in time;

updating, based at least on the three-dimensional projection of the scene, the virtual image of the scene to reflect the scene as would be viewed from the particular object'"'"'s position in three-dimensional space at the particular point in time; and

providing the updated virtual image of the scene for output.

23. The system of claim 15, wherein:

receiving data identifying the particular object having the particular position in three-dimensional space comprises receiving, from a client device, indication of a user of the client device having selected the particular object; and

providing the virtual image of the scene for output comprises providing the virtual image of the scene for output at the client device.

24. The system of claim 15, wherein receiving data identifying the particular object having the particular position in three-dimensional space comprises:

in response to a set of criteria being satisfied, receiving data identifying the particular object having the particular position in three-dimensional space.

25. A non-transitory computer-readable storage device having instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations comprising:

obtaining (i) a first image of a scene as viewed by a first depth camera from a first perspective, and (ii) a second image of the scene as viewed by a second depth camera from a different, second perspective;

generating, based on the first image that corresponds to the scene as viewed by the first depth camera from the first perspective, first data that references (i) a three-dimensional coordinate position associated with each point in the first image, (ii) a brightness characteristic associated with each point in the first image, and (iii) a color characteristic associated with each point in the first image;

generating, based on the second image that corresponds to the scene as viewed by the second depth camera from the second perspective, second data that references (i) a three-dimensional coordinate position associated with each point in the second image, (ii) a brightness characteristic associated with each point in the second image, and (iii) a color characteristic associated with each point in the second image;

generating, based at least on the first data and the second data, a three-dimensional projection of the scene;

determining a different, third perspective of the scene as the scene would be viewed from a particular position in three-dimensional space;

generating, based at least on the generated three-dimensional projection of the scene, a virtual image of the scene as the scene would be as viewed from the third perspective; and

providing the virtual image of the scene for output.

26. The storage device of claim 25, wherein the operations further comprise:

receiving data identifying a particular object having the particular position in three-dimensional space;

identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space; and

wherein determining the different, third perspective of the scene as would be viewed from the particular position in three-dimensional space comprises;

based on identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space, determining the different, third perspective of the scene as would be viewed from the particular object'"'"'s position in three-dimensional space.

27. The storage device of claim 26, wherein identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space comprises:

obtaining data from one or more sensors indicating the particular object'"'"'s position in three-dimensional space.

28. The storage device of claim 26, wherein identifying the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space comprises:

identifying, based on one or more of the first and second images, the particular position in three-dimensional space as the particular object'"'"'s position in three-dimensional space.

29. The storage device of claim 26, wherein the operations further comprise:

identifying, at a particular point in time after providing the virtual image of the scene for output, another position in three-dimensional space as the particular object'"'"'s position in three-dimensional space at the particular point in time;

updating, based at least on the three-dimensional projection of the scene, the virtual image of the scene to reflect the scene as would be viewed from the particular object'"'"'s position in three-dimensional space at the particular point in time; and

providing the updated virtual image of the scene for output.

30. The storage device of claim 26, wherein:

receiving data identifying the particular object having the particular position in three-dimensional space comprises receiving, from a client device, indication of a user of the client device having selected the particular object; and

providing the virtual image of the scene for output comprises providing the virtual image of the scene for output at the client device.

31. The storage device of claim 26, wherein receiving data identifying the particular object having the particular position in three-dimensional space comprises:

in response to a set of criteria being satisfied, receiving data identifying the particular object having the particular position in three-dimensional space.