Active content in digital media within a media universe

• US 10,564,820 B1
• Filed: 09/29/2014
• Issued: 02/18/2020
• Est. Priority Date: 08/08/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Cabanero, Christian Robert, Van Gorder, Matthew James, Leonard, Thomas Richard, Nylund, Eric Stephen, MacDonald, Adam C. F.
• Primary Examiner(s): Barrett, Ryan
• Application Number:
  US14/500,600
• Time in Patent Office: 1,968 Days
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/20 Indexing scheme for editing of 3D models
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

A media universe system may leverage network-based computation resources and services, for example a streaming service, and a digital asset repository or repository service to dynamically provide active content in digital media of the media universe streamed to the client. An asset repository may contain digital assets including 3D models of objects and characters used in the digital media. Client interactions with digital media via a client interface may be mapped to particular objects, characters, locations, etc. within the media universe according to the digital assets from the repository. Most or all content within a rendered scene streamed to a client device may be rendered as active content. Clients may interact with the active content to, for example, obtain contextual information about the corresponding objects, manipulate and explore objects, discover hidden content, reveal hidden storylines, order merchandise (e.g., via 3D printing) related to the content, etc.

Claims

1. A system, comprising:

a digital asset repository including at least one or more processors and memory and to;

store a plurality of digital assets related to a world of a media universe; and

determine relationships between the digital assets and other content of the media universe;

a digital media module including at least one or more processors and memory and to;

render digital media related to the world of the media universe, the digital media comprising a video including one or more objects rendered from one or more digital assets of the plurality of digital assets from the digital asset repository, wherein the video comprises one or more masks generated from the one or more digital assets, and wherein the one or more masks indicate respective locations of respective ones of the one or more objects in the video; and

stream the digital media to a plurality of client devices; and

a graphics processing and rendering module including at least one or more processors and memory and to;

pause the digital media in response to receiving input indicating user interactions with a particular region in the video of the digital media streamed to the client devices;

identify at least one digital asset of the one or more digital assets used to render a particular object based at least in part on determining that a particular mask of the one or more masks corresponds to the particular region indicated in the input;

query the digital asset repository with the at least one digital asset to determine content relating to the at least one digital asset according to the relationships determined by the digital asset repository between the at least one digital asset and the other content; and

provide the determined content to users on respective client devices.

2. The system as recited in claim 1, wherein the graphics processing and rendering module is configured to:

determine a plurality of regions in one or more frames of the digital media streamed to the client devices, wherein the regions are determined according to the one or more masks that indicate locations where the one or more objects were inserted in the video of the digital media; and

determine correspondences between ones of the plurality of regions and ones of the plurality of digital assets in the repository.

3. The system as recited in claim 1, wherein the graphics processing and rendering module is configured to:

retrieve information about the particular object from a media universe database; and

present the information about the particular object via a user interface on the respective client devices.

4. The system as recited in claim 1, wherein the graphics processing and rendering module is further configured to identify the at least one digital asset based at least in part on metadata for the video, wherein the metadata indicates that the particular object is a particular type of object or physical information about the particular object.

5. The system as recited in claim 1, wherein, to provide the content corresponding to the digital assets to users on respective client devices, the graphics processing and rendering module is configured to:

render new digital media content including the content; and

stream the new digital media content to respective ones of the client devices.

6. The system as recited in claim 5, wherein the graphics processing and rendering module is further configured to:

obtain at least one of the digital assets from the repository;

render additional content including a rendering of the at least one digital asset; and

merge the additional content with the new digital media content.

7. The system as recited in claim 1, wherein the digital assets include two-dimensional (2D) or three-dimensional (3D) models of characters or objects of the world of the media universe, and wherein the content corresponding to the digital assets includes content related to the characters or objects of the world of the media universe modeled by the digital assets.

8. The system as recited in claim 1, wherein the digital asset repository, the digital media module and the graphics processing and rendering module are components of one or more computing devices that implement a media universe (MU) system on a provider network, wherein the plurality of client devices are configured to access the MU system on the provider network via an intermediate network, and wherein the MU system is configured to leverage one or more computing resources of the provider network to perform said streaming, said correlating, and said providing in real-time in response to input received from the client devices.

9. A method, comprising:

performing, by a media universe (MU) system implemented on one or more computing devices;

storing a plurality of digital assets related to a world of a media universe to a repository;

rendering digital media related to the world of the media universe, the digital media comprising a video including one or more objects rendered from one or more digital assets of the plurality of digital assets from the digital asset repository, wherein the video comprises one or more masks generated from the one or more digital assets, and wherein the one or more masks indicate respective locations of respective ones of the one or more objects in the video;

streaming the digital media to a plurality of client devices;

pause the digital media in response to receiving input indicating user interactions with a particular region in the video of the digital media streamed to the client devices;

identify one or more corresponding digital assets of the one or more digital assets used to render the particular object based at least in part on determining that a particular mask of the one or more masks corresponds to the particular region indicated in the input;

querying the repository with the one or more corresponding digital assets to determine relationships between the one or more corresponding digital assets and media universe content; and

sending the media universe content to respective client devices.

10. The method as recited in claim 9, further comprising:

determining a plurality of regions in the digital media streamed to the client devices according to one or more masks that indicate locations where the one or more objects were inserted in the video frame content of the digital media; and

determining correspondences between the plurality of regions and the plurality of digital assets in the repository.

11. The method as recited in claim 10, further comprising:

determining the corresponding digital assets according to the determined correspondences between the individual regions and the plurality of digital assets in the repository.

12. The method as recited in claim 9, further comprising determining the relationships between the corresponding digital assets and the media universe content at least in part according to mappings between the plurality of digital assets and other media universe content maintained in the repository.

13. The method as recited in claim 9, wherein said sending the media universe content to respective client devices comprises:

rendering new digital media content including the media universe content; and

streaming the new digital media content to respective ones of the client devices.

14. The method as recited in claim 13, further comprising:

rendering additional content including a rendering of at least one of the corresponding digital assets; and

merging the additional content with the new digital media content.

15. The method as recited in claim 9, wherein the digital assets include two-dimensional (2D) or three-dimensional (3D) models of characters or objects of the world of the media universe, and wherein the media universe content related to the corresponding digital assets includes content related to characters or objects of the world of the media universe modeled by the corresponding digital assets.

16. The method as recited in claim 15, wherein the one or more computing devices that implement the MU system are on a provider network, wherein the MU system is configured to leverage computing resources of the provider network to perform said streaming, said correlating, said determining, and said sending in real-time in response to input received from the client devices.

17. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement a graphics processing and rendering module for a media universe (MU) system, the graphics processing and rendering module configured to:

determine a plurality of regions in digital media streamed to one or more client devices according to one or more masks indicating locations where one or more objects were inserted in video frame content of the digital media during rendering of the digital media; and

determine correspondences between the plurality of regions and a plurality of digital assets stored in a digital asset repository based on the one or more masks;

pause the digital media in response to receiving input indicating a user interaction with an object of the one or more objects included in the video frame content of the digital media on a client device;

map the user interaction to a region of the regions;

determine a digital asset of the plurality of digital assets corresponding to the region based at least in part on determining that a particular mask of the one or more masks corresponds to the region, wherein the object is rendered in the digital media based at least in part on the determined digital asset;

determine media universe content corresponding to the determined digital asset according to mappings between the plurality of digital assets and other content of the media universe determined by the digital asset repository; and

send the media universe content to the respective client device.

18. The non-transitory computer-readable storage medium as recited in claim 17, wherein, to send the media universe content to the respective client device, the graphics processing and rendering module is configured to:

render new digital media content including the media universe content; and

stream the new digital media content to the respective client device.

19. The non-transitory computer-readable storage medium as recited in claim 17, wherein the graphics processing and rendering module is further configured to:

render additional content including a rendering of the determined digital asset; and

merge the additional content with the new digital media content.

20. The non-transitory computer-readable storage medium as recited in claim 17, wherein the digital assets include two-dimensional (2D) or three-dimensional (3D) models of characters or objects of the world of the media universe, and wherein the media universe content mapped to the digital assets includes content related to the characters or objects of the world of the media universe modeled by the digital assets.