Real-time exploration of video content

• US 9,892,556 B2
• Filed: 06/27/2014
• Issued: 02/13/2018
• Est. Priority Date: 03/11/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Heinz, II, Gerard Joseph, Pesce, Michael Schleif, Davis, Collin Charles, Frazzini, Michael Anthony, Alkarmi, Ashraf, George, Michael Martin, Limp, David A., Carr, Jr., William Dugald
• Primary Examiner(s): Mazumder, Saptarshi
• Application Number:
  US14/318,042
• Publication Number:
  US 20150262423A1
• Time in Patent Office: 1,327 Days
• Field of Search: None
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

A real-time video exploration (RVE) system that allows users to pause, step into, and explore 2D or 3D modeled worlds of scenes in a video. The system may leverage network-based computation resources to render and stream new video content from the models to clients with low latency. A user may pause a video, step into a scene, and interactively change viewing positions and angles in the model to move through or explore the scene. The user may resume playback of the recorded video when done exploring the scene. Thus, rather than just viewing a pre-rendered scene in a movie from a pre-determined perspective, a user may step into and explore the scene from different angles, and may wander around the scene at will within the scope of the model to discover parts of the scene that are not visible in the original video.

Claims

1. A system, comprising:

one or more computing devices comprising one or more hardware processors and memory and configured to implement a real-time video exploration (RVE) system comprising;

a playback module implemented via the one or more hardware processors and memory and configured to begin playback of at least a portion of a pre-recorded video to a client device; and

a graphics processing and rendering module implemented via the one or more hardware processors and memory and configured to;

receive, after playback of the pre-recorded video has begun, scene exploration input from the client device indicating an interaction with a scene of the pre-recorded video, wherein the interaction comprises a modification of a camera viewpoint of the scene, and wherein the scene comprises a plurality of objects;

pause playback of the scene in response to the scene exploration input;

generate a model of the scene according to graphics data for the scene and the modified camera viewpoint;

render new video of the scene from the model of the scene based at least in part on the scene exploration input received from the client device;

stream the new video of the scene to the client device while playback of the scene is paused; and

resume playback of the pre-recorded video to the client device in response to resume input from the client device, wherein streaming the new video is stopped in response to the resume input.

2. The system as recited in claim 1, wherein the model of the scene is generated in response to the scene exploration input.

3. The system as recited in claim 1, wherein the scene exploration input directs movement of the camera viewpoint within the model of the scene so that the new video shows portions of the model of the scene that are not visible in the pre-recorded video.

4. The system as recited in claim 1, wherein the one or more computing devices that implement the RVE system are on a provider network, and wherein the client device is configured to access the RVE system on the provider network via an intermediate network.

5. The system as recited in claim 4, wherein the graphics processing and rendering module is configured to leverage computing resources of the provider network to perform said rendering in real time in response to the scene exploration input from the client device.

6. The system as recited in claim 1, wherein the scene exploration input is received from the client device according to an application programming interface (API) of the RVE system.

7. The system as recited in claim 1, wherein the model of the scene comprises a three-dimensional (3D) model.

8. A method, comprising:

performing, by a real-time video exploration (RVE) system implemented on one or more computing devices;

playing back at least a portion of a pre-recorded video to a client device;

receiving scene interaction input from the client device indicating an interaction with a current scene of the pre-recorded video, wherein the interaction comprises a modification of a camera viewpoint of the scene, wherein the scene comprises a plurality of objects;

pausing playback of the current scene in response to the scene interaction input;

generating a model of the scene according to graphics data for the scene and the modified camera viewpoint;

rendering new video of the scene from the model of the scene based at least in part on the scene interaction input received from the client device;

streaming the new video of the scene to the client device, wherein streaming the new video occurs while playback of the current scene is paused; and

resuming playback of the pre-recorded video to the client device in response to resume input from the client device, wherein streaming the new video is stopped in response to the resume input.

9. The method as recited in claim 8, wherein the model of the scene is generated in response to the scene exploration input.

10. The method as recited in claim 8, further comprising iteratively rendering new video of the scene from the model of the scene based at least in part on additional scene interaction input received from the client device.

11. The method as recited in claim 9, further comprising:

stopping rendering of the new video in response to the resume input.

12. The method as recited in claim 8, further comprising receiving additional scene interaction input indicating an additional interaction with the scene, wherein the new video is rendered while playback of the video is paused.

13. The method as recited in claim 8, wherein playback of the pre-recorded video is resumed at a time in which the playback was paused.

14. The method as recited in claim 8, wherein the scene interaction input moves a viewing position within the scene, and wherein the new video shows the scene from the perspective of the moved viewing position.

15. The method as recited in claim 8, wherein the scene interaction input moves a camera viewpoint through the model of the scene so that the new video includes views of portions of the model of the scene that are not visible in the pre-recorded video.

16. The method as recited in claim 8, wherein the scene interaction input adds, modifies, or removes a graphics effect in the scene.

17. The method as recited in claim 16, wherein the graphics effect is one of a lens effect or a lighting effect.

18. The method as recited in claim 8, further comprising receiving an object manipulation input from the client device indicating a manipulation of particular object of the plurality of objects in the scene, wherein the new video includes a manipulated model of the object based at least in part on the object manipulation input.

19. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement a real-time video exploration (RVE) system configured to:

begin playback of at least a portion of a pre-recorded video to a client device;

pause playback of the pre-recorded video at a scene in response to input from the client device;

generate a three-dimensional (3D) model of the scene according to 3D graphics data for the scene;

render new video of the scene from the 3D model of the scene based at least in part on scene exploration input received from the client device, wherein the scene exploration input indicates an interaction with the scene;

stream the new video of the scene to the client device while playback of the scene is paused; and

resume playback of the pre-recorded video to the client device in response to resume input from the client device, wherein streaming the new video is stopped in response to the resume input.

20. The non-transitory computer-readable storage medium as recited in claim 19, wherein the 3D model of the scene is generated in response to the scene interaction input.

21. The non-transitory computer-readable storage medium as recited in claim 19, wherein the pre-recorded video shows the scene from a pre-determined perspective, and wherein the new video shows the scene from one or more different perspectives determined at least in part from the scene exploration input.

22. The non-transitory computer-readable storage medium as recited in claim 19, wherein the scene exploration input changes a viewing angle for the 3D model of the scene or moves a viewing position within the 3D model of the scene, and wherein the new video shows the scene from the perspective of the changed viewing angle or the moved viewing position.

23. The non-transitory computer-readable storage medium as recited in claim 19, wherein the scene exploration input moves a viewpoint through the 3D model of the scene so that the portions of the 3D model of the scene that are not visible in the original video are rendered and streamed.