Dynamic POV Composite 3D Video System

• US 20150172634A1
• Filed: 06/11/2013
• Published: 06/18/2015
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Wheeler, Aaron Joseph, Plagemann, Christian, Dahlkamp, Hendrik, Chi, Liang-Yu, Zhao, Yong, Ganapathi, Varun, Kauffmann, Alejandro Jose
• Granted Patent: US 9,392,248 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

Systems and techniques are disclosed for visually rendering a requested scene based on a virtual camera perspective request as well as a projection of two or more video streams. The video streams can be captured using two dimensional cameras or three dimensional depth cameras and may capture different perspectives. The projection may be an internal projection that maps out the scene in three dimensions based on the two or more video streams. An object internal or external to the scene may be identified and the scene may be visually rendered based on a property of the object. For example, a scene may be visually rendered based on where an mobile object is located within the scene.

Claims

1. A method comprising:

receiving a first video stream from a first camera configured to capture a first perspective;

receiving a second video stream from a second camera configured to capture a second perspective;

receiving a virtual camera perspective request;

projecting the first video stream and the second video stream onto a three dimensional scene; and

visually rendering a requested scene based on the virtual camera perspective and the projection of the first video stream and second video stream onto the three dimensional scene.

2. The method of claim 1, wherein the first camera is a three dimensional depth camera.

3. The method of claim 1, wherein the first perspective is captured from a different location than the second perspective.

4. The method of claim 1, wherein the virtual camera perspective request is received from a user.

5. The method of claim 1, wherein the virtual camera perspective request is generated based on predetermined criteria.

6. The method of claim 5, wherein the predetermined criteria is selected from the group consisting of a time, an event, and a condition.

7. The method of claim 1, wherein the virtual camera perspective request is generated based on dynamic criteria.

8. The method of claim 1, wherein the virtual camera perspective request comprises identifying an object.

9. The method of claim 8, wherein the object is external to the projected scene.

10. The method of claim 9, wherein the virtual camera perspective is modified as an object position is modified.

11. The method of claim 8, further comprising visually rendering the requested scene based on an object property.

12. The method of claim 11, wherein the object property is selected from a group consisting of an object location, an object orientation, and an object history.

13. The method of claim 11, wherein the object property is determined based on an object sensor.

14. The method of claim 11, wherein the object property is automatically determined based on identifying an object shape.

15. The method of claim 8, wherein at least the first video stream comprises the object.

16. The method of claim 8, wherein neither the first video stream nor the second video stream comprises the object.

17. The method of claim 14, wherein the object shape is identified by object recognition software.

18. The method of claim 1, wherein the visual rendering of the requested scene is a two dimensional render.

19. The method of claim 1, wherein the visual rendering of the requested scene is a three dimensional render.

20. The method of claim 1, wherein the first video stream is a physical video stream.

21. The method of claim 1, wherein the first video stream is a virtual video stream.

22. A system comprising:

a server;

a first camera configured to capture a first perspective;

a second camera configured to capture a second perspective;

a processor in connection with the server, the processor configured to;

receive a first video stream from the first camera;

receive a second video stream from the second camera;

receive a virtual camera perspective request;

project the first video stream and the second video stream onto a three dimensional scene; and

visually render a requested scene based on the virtual camera perspective and the projection of the first video stream and second video stream onto the three dimensional scene.

23. The system of claim 22, wherein the virtual camera perspective request is received from a user.

24. The system of claim 22, wherein the virtual camera perspective request comprises identifying an object.

25. The system of claim 24, further comprising visually rendering the requested scene based on an object property.

26. The system of claim 25, wherein the object property is selected from a group consisting of an object location, an object orientation, and an object history.

27. The method of claim 25, wherein the object property is determined based on an object sensor.

28. The system of claim 25, wherein the object property is automatically determined based on identifying an object shape.

29. The system of claim 24, wherein at least the first video stream comprises the object.

30. The system of claim 24, wherein neither the first video stream nor the second video stream comprises the object.

31. The system of claim 28, wherein the object shape is identified by object recognition software.