Virtual world generation engine

• US 10,332,311 B2
• Filed: 09/29/2014
• Issued: 06/25/2019
• Est. Priority Date: 09/29/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Davis, Collin Charles, Frazzini, Michael Anthony, Alkarmi, Ashraf, Heinz, II, Gerard Joseph, Pesce, Michael Schleif
• Primary Examiner(s): Mazumder, Saptarshi
• Application Number:
  US14/500,413
• Publication Number:
  US 20160093078A1
• Time in Patent Office: 1,730 Days
• Field of Search: None
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence

Abstract

A virtual world generation engine and methods for generating virtual worlds from images collected from various sources, including crowdsourced images. A virtual world generation engine may obtain images (e.g., digital photographs, digital video frames, etc.) related to a particular real-world scene, combine the images using image processing techniques such as image stitching techniques to generate composite images representing a view of the scene, and generate models from the composite images. The models may be used in rendering video content representing virtual worlds generated from the collected images of real-world scenes; the video content may be streamed to client device(s). Obtaining the images, generating models, rendering video content from the models, and streaming the video content may be performed in response to user interactions with video content on the client device(s) to allow interactive exploration of the virtual worlds in real-time or near-real time.

Claims

1. A system, comprising:

one or more computing devices comprising one or more respective processors and memory and configured to implement a video system comprising;

an image collection module implemented via the one or more respective processors and memory, configured to obtain digital images from one or more sources, each image capturing a portion of a real-world scene;

a virtual world generation engine implemented via the one or more respective processors and memory, configured to;

identify two or more of the images that include overlapping portions of the scene;

combine the identified images according to one or more digital image stitching techniques to generate a composite image representing a view of the scene; and

generate a model of the scene in a virtual world according to the composite image;

one or more video processing modules implemented via the one or more respective processors and memory, configured to;

stream a video including the scene to a client device;

receive user interactions from the client device exploring the scene in the video indicating a change in a viewpoint of the scene; and

in response to the user interactions, render and stream additional video including the scene from the changed viewpoint and the model;

receive additional user interactions from the client device to manipulate the scene in the video;

determine, based at least in part on the additional user interactions, that a change in the scene is required as a result of the manipulation; and

signal the virtual world generation engine to generate a model of the scene that includes the change in the scene that is required as a result of the manipulation.

2. The system as recited in claim 1, wherein the video system is configured to perform said identify, said combine, and said generate in response to the user interactions exploring content of video streamed to the client device.

3. The system as recited in claim 1, wherein the one or more sources for images include one or more crowdsourcing techniques for soliciting and obtaining digital images from a plurality of online contributors.

4. The system as recited in claim 1, wherein the image collection module is configured to:

obtain a plurality of images from a plurality of sources, the images capturing different portions of a plurality of scenes;

obtain or generate image metadata for the images, the metadata for a given image including information describing one or more characteristics of the respective image;

correlate the images into a plurality of image groups based on one or more of image content or image metadata, each image group corresponding to a particular scene;

determine spatial relationships among the images within each image group according to one or more of content of the images or image metadata indicating positioning of the images relative to the scene; and

maintain the images, image metadata, and image data describing the image groups for access by the virtual world generation engine.

5. The system as recited in claim 1, wherein, to identify two or more of the images that include overlapping portions of the scene, the virtual world generation engine is configured to determine overlapping images according to one or more of content of the images or image metadata indicating positioning of the images relative to the scene.

6. The system as recited in claim 1, wherein, to combine the identified images according to one or more digital image stitching techniques to generate a composite image representing a view of the scene, the virtual world generation engine is configured to:

align the identified images according to one or more of content of the images or image metadata indicating positioning of the images relative to each other; and

merge non-overlapping and overlapping portions of the aligned images to generate composited image content.

7. The system as recited in claim 1, wherein, to generate a model of the scene according to the composite image, the virtual world generation engine is configured to:

generate two-dimensional (2D) or three-dimensional (3D) graphics data from the composite image; and

generate a 2D or 3D model of the scene according to the graphics data.

8. The system as recited in claim 1, wherein the one or more computing devices that implement the video system are on a provider network, wherein the client device is configured to access the video system on the provider network via an intermediate network, and wherein the video system is configured to leverage one or more computing resources of the provider network to perform said identify, said combine, said generate, said render, and said stream in response to user input from the client device.

9. The system as recited in claim 1, wherein the video system is a real-time video exploration (RVE) system or an online game system that allows one or more client devices to interactive explore the virtual world.

10. A method, comprising:

performing, by a virtual world generation engine implemented on one or more computing devices;

obtaining two or more images that include overlapping portions of a real-world scene;

combining at least two of the obtained images according to one or more digital image stitching techniques to generate a composite image representing a view of the scene;

generating or updating a model of the scene in a virtual world according to the composite image; and

providing the model of the scene to a video processing module for rendering; and

performing, by the video processing module;

streaming a video including the scene to at least one client device;

receiving user interactions with the streamed video from the at least one client device indicating a change in a viewpoint of the scene;

rendering new video content based at least in part on the model of the scene and the changed viewpoint;

streaming the new video content to the at least one client device;

receiving additional user interactions with the streamed video from the at least one client device to manipulate the scene; and

responsive to a determination, based at least in part on the additional user interactions, that a change in the scene is required as a result of the manipulation, signaling the virtual world generation engine to generate a model of the scene that includes the change in the scene that is required as a result of the manipulation.

11. The method as recited in claim 10, further comprising:

obtaining, by an image collection module implemented on one or more computing devices, a plurality of images from a plurality of sources, the plurality of images capturing different portions of a plurality of scenes; and

obtaining or generating image data for the images, the image data including information indicating image groups corresponding to particular scenes and information indicating spatial relationships among the images within each image group;

wherein said obtaining two or more images that include overlapping portions of a real-world scene comprises obtaining the two or more images from the image collection module according to image data corresponding to the real-world scene.

12. The method as recited in claim 11, wherein the plurality of sources include a plurality of online contributors, wherein the method further comprises obtaining at least some of the images from the plurality of online contributors via one or more crowdsourcing techniques.

13. The method as recited in claim 10, wherein:

the receiving of the user interactions comprises receiving user interactions indicating a change in a viewing position within the scene; and

the rendering of the new video content is based at least in part on the changed viewing position.

14. The method as recited in claim 10, wherein the virtual world generation engine performs said obtaining, said combining, and said generating in response to input received from the video processing module indicating interactions exploring content of video sent to the client device.

15. The method as recited in claim 10, wherein said combining the images according to one or more digital image stitching techniques to generate a composite image representing a view of the scene comprises applying an image stitching technique to align the identified images according to one or more of content of the images or image metadata indicating positioning of the images relative to each other and merge the aligned images to generate the composite image.

16. The method as recited in claim 10, wherein the video processing module is a component of a real-time video exploration (RVE) system or of an online game system.

17. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement a real-time video exploration (RVE) system configured to:

receive input from a client device indicating user interactions with video content on the client device;

in response to the user interactions;

obtain a plurality of images that include content related to a real-world scene indicated by the user interactions with the video content;

select at least two of the obtained images according to the user interactions with the video content; and

generate a two-dimensional (2D) or three-dimensional (3D) model of the real-world scene from the obtained images; and

receive additional input from the client device indicating additional user interactions with the video content on the client device, wherein the addition user interactions indicate a change in a viewpoint of the real-world scene;

in response to the additional user interactions;

render new video content based at least in part on the changed viewpoint and the two-dimensional (2D) or three-dimensional (3D) model; and

stream video including the new video content to the client device;

receive further input from the client device indicating further user interactions with the new video content streamed to the client device to manipulate the real-world scene;

determine, based at least in part on the further user interactions,

that a change in the real-world scene is required as a result of the manipulation; and

generate a two-dimensional (2D) or three-dimensional (3D) model of the real-world scene that includes the change that is required as a result of the manipulation.

18. The non-transitory computer-readable storage medium as recited in claim 17, to generate the two-dimensional (2D) or three-dimensional (3D) model, the RVE system is further configured to:

combine the selected images according to one or more digital image stitching techniques to generate a composite image representing a view of the scene; and

generate the two-dimensional (2D) or three-dimensional (3D) model from the composite image.

19. The non-transitory computer-readable storage medium as recited in claim 18, wherein, to combine the selected images according to one or more digital image stitching techniques to generate a composite image representing a view of the scene, the RVE system is configured to:

align the identified images according to content or features of the images; and

merge non-overlapping and overlapping portions of the aligned images to generate the composite image.

20. The non-transitory computer-readable storage medium as recited in claim 17, wherein the RVE system is further configured to send the selected images to a client interface on the client device for display as two-dimensional (2D) images related to the scene.

21. The non-transitory computer-readable storage medium as recited in claim 18, wherein the RVE system is configured to

send the combined image to a client interface on the client device for display as a two-dimensional (2D) image related to the scene.

22. The non-transitory computer-readable storage medium as recited in claim 17, wherein the RVE system is further configured to obtain the plurality of images from one or more sources, wherein the one or more sources include at least one crowdsourcing technique for soliciting and obtaining digital images from one or more online communities or social media channels.

23. The non-transitory computer-readable storage medium as recited in claim 17, wherein the additional user interactions indicate a change in a viewing position within the real-world scene;

and wherein the RVE system is configured to render the new video content based at least in part on the changed viewing position.