VIRTUAL WORLD GENERATION ENGINE

• US 20160093078A1
• Filed: 09/29/2014
• Published: 03/31/2016
• Est. Priority Date: 09/29/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: DAVIS, COLLIN CHARLES, FRAZZINI, MICHAEL ANTHONY, ALKARMI, ASHRAF, HEINZ, GERARD JOSEPH II, PESCE, MICHAEL SCHLEIF
• Granted Patent: US 10,332,311 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence

Abstract

A virtual world generation engine and methods for generating virtual worlds from images collected from various sources, including crowdsourced images. A virtual world generation engine may obtain images (e.g., digital photographs, digital video frames, etc.) related to a particular real-world scene, combine the images using image processing techniques such as image stitching techniques to generate composite images representing a view of the scene, and generate models from the composite images. The models may be used in rendering video content representing virtual worlds generated from the collected images of real-world scenes; the video content may be streamed to client device(s). Obtaining the images, generating models, rendering video content from the models, and streaming the video content may be performed in response to user interactions with video content on the client device(s) to allow interactive exploration of the virtual worlds in real-time or near-real time.

Claims

1. A system, comprising:

one or more computing devices configured to implement a video system comprising;

an image collection module configured to obtain digital images from one or more sources, each image capturing a portion of a real-world scene;

a virtual world generation engine configured to;

identify two or more of the images that include overlapping portions of the scene;

combine the identified images according to one or more digital image processing techniques to generate a composite image representing a view of the scene; and

generate a model of the scene according to the composite image;

one or more video processing modules configured to;

render video of the scene from the model of the scene; and

stream the video of the scene to a client device.

2. The system as recited in claim 1, wherein the video system is configured to perform said identifying, said combining, said generating, said rendering, and said streaming in response to user interactions exploring content of video streamed to the client device.

3. The system as recited in claim 1, wherein the one or more sources for images include one or more crowdsourcing techniques for soliciting and obtaining digital images from a plurality of online contributors.

4. The system as recited in claim 1, wherein the image collection module is configured to:

obtain a plurality of images from a plurality of sources, the images capturing different portions of a plurality of scenes;

obtain or generate image metadata for the images, the metadata for a given image including information describing one or more characteristics of the respective image;

correlate the images into a plurality of image groups based on one or more of image content or image metadata, each image group corresponding to a particular scene;

determine spatial relationships among the images within each image group according to one or more of content of the images or image metadata indicating positioning of the images relative to the scene; and

maintain the images, image metadata, and image data describing the image groups for access by the virtual world generation engine.

5. The system as recited in claim 1, wherein, to identify two or more of the images that include overlapping portions of the scene, the virtual world generation engine is configured to determine overlapping images according to one or more of content of the images or image metadata indicating positioning of the images relative to the scene.

6. The system as recited in claim 1, wherein, to combine the identified images according to one or more digital image processing techniques to generate a composite image representing a view of the scene, the virtual world generation engine is configured to apply one or more image stitching techniques to:

align the identified images according to one or more of content of the images or image metadata indicating positioning of the images relative to each other; and

merge non-overlapping and overlapping portions of the aligned images according to one or more image processing techniques to generate composited image content.

7. The system as recited in claim 1, wherein, to generate a model of the scene according to the composite image, the virtual world generation engine is configured to:

generate two-dimensional (2D) or three-dimensional (3D) graphics data from the composite image; and

generate a 2D or 3D model of the scene according to the graphics data.

8. The system as recited in claim 1, wherein the one or more computing devices that implement the video system are on a provider network, wherein the client device is configured to access the video system on the provider network via an intermediate network, and wherein the video system is configured to leverage one or more computing resources of the provider network to perform said identifying, said combining, said generating, said rendering, and said streaming in response to user input from the client device.

9. The system as recited in claim 1, wherein the video system is a real-time video exploration (RVE) system or an online game system.

10. A method, comprising:

iteratively performing, by a virtual world generation engine implemented on one or more computing devices;

obtaining two or more images that include overlapping portions of a real-world scene;

combining at least two of the obtained images according to one or more digital image processing techniques to generate a composite image representing a view of the scene;

generating or updating a model of the scene according to the composite image; and

providing the model of the scene to a video processing module for rendering.

11. The method as recited in claim 10, further comprising:

obtaining, by an image collection module implemented on one or more computing devices, a plurality of images from a plurality of sources, the plurality of images capturing different portions of a plurality of scenes; and

obtaining or generating image data for the images, the image data including information indicating image groups corresponding to particular scenes and information indicating spatial relationships among the images within each image group;

wherein said obtaining two or more images that include overlapping portions of a real-world scene comprises obtaining the two or more images from the image collection module according to image data corresponding to the real-world scene.

12. The method as recited in claim 11, wherein the plurality of sources include a plurality of online contributors, wherein the method further comprises obtaining at least some of the images from the plurality of online contributors via one or more crowdsourcing techniques.

13. The method as recited in claim 10, further comprising:

iteratively performing, by the video processing module;

receiving a model of the scene from the virtual world generation engine;

rendering new video content at least in part from the model of the scene; and

sending the new video content to at least one client device.

14. The method as recited in claim 11, wherein the virtual world generation engine performs said obtaining, said combining, said generating, and said providing in response to input received from the video processing module indicating interactions exploring content of video sent to a client device.

15. The method as recited in claim 10, wherein said combining the images according to one or more digital image processing techniques to generate a composite image representing a view of the scene comprises applying an image stitching technique to the images, wherein the image stitching technique aligns the identified images according to one or more of content of the images or image metadata indicating positioning of the images relative to each other and merges the aligned images according to one or more image processing techniques to generate the composite image.

16. The method as recited in claim 10, wherein the video processing module is a component of a real-time video exploration (RVE) system or of an online game system.

17. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement a real-time video exploration (RVE) system configured to:

receive input from a client device indicating user interactions with video content on the client device;

in response to the user interactions;

obtain a plurality of images that include content related to a real-world scene indicated by the user interactions with the video content;

select at least two of the obtained images according to the user interactions with the video content; and

send the selected images to the client device for display to the user.

18. The non-transitory computer-readable storage medium as recited in claim 17, wherein, to send the selected images to the client device for display to the user, the RVE system is further configured to:

combine the selected images according to one or more digital image processing techniques to generate a composite image representing a view of the scene;

generate a two-dimensional (2D) or three-dimensional (3D) model of the scene according to the composite image;

render new video content at least in part from the model of the scene; and

stream video including the new video content to the client device.

19. The non-transitory computer-readable storage medium as recited in claim 18, wherein, to combine the selected images according to one or more digital image processing techniques to generate a composite image representing a view of the scene, the RVE system is configured to:

align the identified images according to content or features of the images; and

merge non-overlapping and overlapping portions of the aligned images to generate the composite image.

20. The non-transitory computer-readable storage medium as recited in claim 17, wherein, to send the selected images to the client device for display to the user, the RVE system is further configured to send the selected images to a client interface on the client device for display as two-dimensional (2D) images related to the scene.

21. The non-transitory computer-readable storage medium as recited in claim 17, wherein, to combine at least two of the obtained images according to one or more digital image processing techniques to:

combine the selected images according to one or more digital image processing techniques to generate a composite image representing a view of the scene;

send the combined image to a client interface on the client device for display as a two-dimensional (2D) image related to the scene.

22. The non-transitory computer-readable storage medium as recited in claim 17, wherein the RVE system is further configured to obtain the plurality of images from one or more sources, wherein the one or more sources include at least one crowdsourcing technique for soliciting and obtaining digital images from one or more online communities or social media channels.

23. The non-transitory computer-readable storage medium as recited in claim 17, wherein the user interactions include input exploring a scene in video being streamed to the client device, and wherein the RVE system is configured to iteratively perform said obtaining, said combining, said generating, said rendering, and said streaming in real time in response to the user interactions.