REAL-TIME EXPLORATION OF VIDEO CONTENT

• US 20150262423A1
• Filed: 06/27/2014
• Published: 09/17/2015
• Est. Priority Date: 03/11/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: HEINZ, GERALD JOSEPH II, PESCE, MICHAEL SCHLEIF, DAVIS, COLLIN CHARLES, FRAZZINI, MICHAEL ANTHONY, ALKARMI, ASHRAF, GEORGE, MICHAEL MARTIN, LIMP, DAVID A., CARR, WILLIAM DUGALD Jr.
• Granted Patent: US 9,892,556 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

A real-time video exploration (RVE) system that allows users to pause, step into, and explore 2D or 3D modeled worlds of scenes in a video. The system may leverage network-based computation resources to render and stream new video content from the models to clients with low latency. A user may pause a video, step into a scene, and interactively change viewing positions and angles in the model to move through or explore the scene. The user may resume playback of the recorded video when done exploring the scene. Thus, rather than just viewing a pre-rendered scene in a movie from a pre-determined perspective, a user may step into and explore the scene from different angles, and may wander around the scene at will within the scope of the model to discover parts of the scene that are not visible in the original video.

Claims

1. A system, comprising;

one or more computing devices configured to implement a real-time video exploration (RVE) system comprising;

a playback module configured to begin playback of at least a portion of a pre-recorded video to a client device; and

a graphics processing and rendering module configured to;

receive input from the client device indicating an interaction with a scene of the video;

generate a model of the scene according to graphics data for the scene;

render new video of the scene from the model of the scene based at least in part on scene exploration input received from the client device; and

stream the new video of the scene to the client device.

2. The system as recited in claim 1, wherein the pre-recorded video shows the scene from a pre-determined perspective, and wherein the new video shows the scene from one or more different perspectives determined at least in part from the scene exploration input.

3. The system as recited in claim 1, wherein the scene exploration input moves a camera viewpoint within the model of the scene so that the new video shows portions of the model of the scene that are not visible in the pre-recorded video.

4. The system as recited in claim 1, wherein the one or more computing devices that implement the RVE system are on a provider network, and wherein the client device is configured to access the RVE system on the provider network via an intermediate network.

5. The system as recited in claim 4, wherein the graphics processing and rendering module is configured to leverage computing resources of the provider network to perform said rendering in real time in response to the scene exploration input from the client device.

6. The system as recited in claim 1, wherein the scene exploration input is received from the client device according to an application programming interface (API) of the RVE system.

7. The system as recited in claim 1, wherein the model of the scene is a three-dimensional (3D) model.

8. A method, comprising:

performing, by a real-time video exploration (RVE) system implemented on one or more computing devices;

playing back at least a portion of a pre-recorded video to a client device;

receiving input from the client device indicating an interaction with a current scene;

generating a model of the scene according to graphics data for the scene; and

rendering new video of the scene from the model of the scene based at least in part on scene interaction input received from the client device.

9. The method as recited in claim 8, further comprising streaming the new video of the scene to the client device.

10. The method as recited in claim 8, further comprising iteratively rendering new video of the scene from the model of the scene based at least in part on additional scene interaction input received from the client device.

11. The method as recited in claim 8, further comprising:

pausing playback of the current scene in response to said input from the client device; and

resuming playback of the video to the client device in response to resume input from the client device.

12. The method as recited in claim 8, wherein the pre-recorded video shows the scene from a pre-determined perspective, and wherein the new video shows the scene from one or more different perspectives determined at least in part from the scene interaction input.

13. The method as recited in claim 8, wherein the scene interaction input changes a viewing angle for the scene, and wherein the new video shows the scene from the perspective of the changed viewing angle.

14. The method as recited in claim 8, wherein the scene interaction input moves a viewing position within the scene, and wherein the new video shows the scene from the perspective of the moved viewing position.

15. The method as recited in claim 8, wherein the scene interaction input moves a camera viewpoint through the model of the scene so that the new video includes views of portions of the model of the scene that are not visible in the pre-recorded video.

16. The method as recited in claim 8, wherein the scene interaction input adds, modifies, or removes a graphics effect in the scene.

17. The method as recited in claim 16, wherein the graphics effect is one of a lens effect or a lighting effect.

18. The method as recited in claim 8, wherein the model of the scene is a three-dimensional (3D) model.

19. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement a real-time video exploration (RVE) system configured to:

begin playback of at least a portion of a pre-recorded to a client device;

pause playback of the video at a scene in response to input from the client device;

generate a three-dimensional (3D) model of the scene according to 3D graphics data for the scene;

render new video of the scene from the 3D model of the scene based at least in part on scene exploration input received from the client device; and

stream the new video of the scene to the client device.

20. The non-transitory computer-readable storage medium as recited in claim 19, wherein the RVE system is further configured to resume playback of the video to the client device in response to input from the client device.

21. The non-transitory computer-readable storage medium as recited in claim 19, wherein the pre-recorded video shows the scene from a pre-determined perspective, and wherein the new video shows the scene from one or more different perspectives determined at least in part from the scene exploration input.

22. The non-transitory computer-readable storage medium as recited in claim 19, wherein the scene exploration input changes a viewing angle for the 3D model of the scene or moves a viewing position within the 3D model of the scene, and wherein the new video shows the scene from the perspective of the changed viewing angle or the moved viewing position.

23. The non-transitory computer-readable storage medium as recited in claim 19, wherein the scene exploration input moves a viewpoint through the 3D model of the scene so that the portions of the 3D model of the scene that are not visible in the original video are rendered and streamed.