Real-time rendering of targeted video content

• US 10,375,434 B2
• Filed: 06/27/2014
• Issued: 08/06/2019
• Est. Priority Date: 03/11/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Heinz, II, Gerard Joseph, Pesce, Michael Schleif, Davis, Collin Charles, Frazzini, Michael Anthony, Alkarmi, Ashraf, George, Michael Martin, Limp, David A., Carr, Jr., William Dugald
• Primary Examiner(s): Shang, Annan Q
• Application Number:
  US14/317,984
• Publication Number:
  US 20150264416A1
• Time in Patent Office: 1,866 Days
• Field of Search: 725 34- 36, 709217-232
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/25883 being end-user demographical data, e.g. age, family status or address arrangements for identifying locations of users in broadcast systems H04H60/52
• CPC Class Codes: H04N 21/2668 Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics

Abstract

A real-time video targeting (RVT) system may leverage network-based computation resources and services, available 2D or 3D model data, and available viewer information to dynamically personalize content of, or add personalized content to, video for particular viewers or viewer groups. When playing back pre-recorded video to viewers, at least some objects or other content in at least some of the scenes of the video may be replaced with objects or content targeted at particular viewers or groups according to profiles or preferences of the viewers or groups. Since the video is being rendered and streamed to different viewers or groups in real-time by the network-based computation resources and services, any given scene of a video may be modified and viewed in many different ways by different viewers or groups based on the particular viewers'"'"' or groups'"'"' profiles.

Claims

1. A system, comprising;

one or more network-based computing devices comprising one or more processors and one or more memories storing program instructions executable by the one or more processors to implement a real-time video targeting (RVT) system to;

begin playback of a pre-recorded video to a plurality of client devices; and

for each of two or more of the client devices to which the pre-recorded video is being played back, during playback of the pre-recorded video;

obtain information about a viewer associated with the respective client device;

generate one or more objects targeted at the viewer according to the viewer'"'"'s information;

render a plurality of new video frames for one or more scenes to add the one or more objects to the one or more scenes to produce a merged video that includes at least some content from the pre-recorded video and the one or more objects targeted at the viewer; and

stream, over a packet-based network, the merged video for the one or more scenes to the respective client device associated with the viewer, wherein the streamed merged video includes the at least some content from the pre-recorded video and the one or more objects targeted at the viewer;

wherein the one or more objects added to the one or more scenes in the pre-recorded video that is streamed is different for at least two of the two or more client devices.

2. The system as recited in claim 1, wherein the viewer'"'"'s information includes one or more preferences of the viewer, and wherein the program instructions are further executable to dynamically determine at least a portion of the one or more objects to be added to the one or more scenes according to the viewer'"'"'s preferences.

3. The system as recited in claim 1, wherein renderings of the one or more objects represent one or more particular products or brands, wherein the one or more particular products or brands are determined according to the viewer'"'"'s information.

4. The system as recited in claim 1, wherein, to generate one or more objects targeted at the viewer according to the viewer'"'"'s information, the program instructions are further executable to:

obtain graphics data for one or more objects in the pre-recorded video according to the viewer'"'"'s information;

generate models of the objects according to the graphics data; and

render the one or more objects according to the generated models of the objects.

5. The system as recited in claim 4, wherein the models of the objects are three-dimensional (3D) models.

6. The system as recited in claim 1, wherein the program instructions are further executable to, during playback of the pre-recorded video:

receive input from one of the plurality of client devices indicating interactions by the respective viewer with video content of a scene in the pre-recorded video being played back to the client device;

generate modified video content at least in part according to the viewer'"'"'s interactions with the video content of the scene in the pre-recorded video;

render new video of the scene to add the modified video content to the scene in the pre-recorded video; and

stream the new video of the scene in the pre-recorded video including the modified video content to the client device.

7. The system as recited in claim 6, wherein the input is received from the client device according to an application programming interface (API) of the RVT system.

8. The system as recited in claim 1, wherein the one or more computing devices that implement the RVT system are on a provider network, and wherein the RVT system is configured to leverage one or more computing resources of the provider network to perform said generating one or more objects and said rendering new video frames in real time during playback of the pre-recorded video to the plurality of client devices.

9. The system recited in claim 1, wherein the program instructions are further executable to perform:

obtain graphics data for a particular product or brand from an external data source comprising a manufacturer, vendor, dealer or distributor data source for the particular product or brand;

wherein said generate the one or more objects targeted at the viewer is based at least in part on the graphics data obtained from the external data source.

10. A method, comprising:

performing, by a real-time video targeting (RVT) system implemented on one or more network-based computing devices;

sending at least a portion of a pre-recorded video to two or more client devices;

obtaining profiles associated with the two or more client devices;

for each of the two or more client devices;

generating, during playback of the pre-recorded video to the respective client device, one or more objects targeted at the client device according to the respective profile;

rendering new video frames for one or more scenes in the pre-recorded video to add one or more objects to the one or more scenes to produce a merged video that includes at least some content from the pre-recorded video and the one or more objects targeted based on the respective profile; and

streaming, over a packet-based network, the merged video for the one or more scenes to the respective client device associated with the respective profile, wherein the merged video that is sent includes the at least some content from the pre-recorded video and the one or more objects targeted based on the respective profile;

wherein the one or more objects added to the one or more scenes in the pre-recorded video is different for at least two of the two or more client devices.

11. The method as recited in claim 10, wherein the profile indicates one or more preferences of a viewer or viewer group, and wherein at least a portion of the one or more objects is determined according to the one or more preferences.

12. The method as recited in claim 10, wherein the profile indicates demographic or historical information for a viewer or viewer group, and wherein at least a portion of the one or more objects is determined according to the demographic or historical information.

13. The method as recited in claim 10, wherein renderings of the one or more objects represent one or more particular products or brands, wherein the one or more particular products or brands are determined according to the profile.

14. The method as recited in claim 10, wherein said generating one or more objects targeted at the client device according to the profile comprises:

obtaining graphics data for one or more objects in the video according to the profile;

generating models of the objects according to the graphics data; and

rendering the one or more objects according to the generated models of the objects.

15. The method as recited in claim 14, wherein the models of the objects are three-dimensional (3D) models.

16. The method as recited in claim 10, further comprising:

receiving input from one of the two or more client devices indicating interactions by a viewer with video content of a scene in the pre-recorded video being played back to the client device;

generating modified video content at least in part according to the viewer'"'"'s interactions with the video content of the scene in the pre-recorded video;

rendering new video for the scene in the pre-recorded video to add the new video content to the pre-recorded video; and

sending the new video of the scene in the pre-recorded video including the modified video content to the client device.

17. The method recited in claim 10, further comprising:

obtaining graphics data for a particular product or brand from an external data source comprising a manufacturer, vendor, dealer or distributor data source for the particular product or brand;

wherein said generating the one or more objects targeted at the viewer comprises generating new video content based at least in part on the graphics data obtained from the external data source.

18. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement a real-time video targeting (RVT) system configured to:

begin playback of at least a portion of a pre-recorded video to a plurality of client devices;

for each of the plurality of client devices, during playback of the pre-recorded video;

obtain information about one or more viewers associated with the client device;

generate one or more objects targeted at the one or more viewers according to the one or more viewers'"'"' respective information;

render new video frames for one or more scenes to add the one or more objects to the one or more scenes to produce a merged video that includes at least some content from the pre-recorded video and the one or more objects targeted at the viewer; and

stream, over a packet-based network, the merged video for the one or more scenes to the client device associated with the viewer, wherein the streamed merged video includes the at least some content from the pre-recorded video and the one or more objects associated with the one or more viewers for the respective client device;

wherein the one or more objects added to the one or more scenes in the pre-recorded video is different for at least two of the plurality of client devices.

19. The non-transitory computer-readable storage medium as recited in claim 18, wherein the information about one or more viewers associated with a given client device indicates one or more preferences of the respective one or more viewers, and wherein at least a portion of the one or more objects targeted at the respective one or more viewers associated with a given client device is determined according to the one or more preferences of the respective one or more viewers.

20. The non-transitory computer-readable storage medium as recited in claim 18, wherein the information about one or more viewers associated with a given client device indicates demographic or historical information for the respective one or more viewers, and wherein at least a portion of the one or more objects targeted at the respective one or more viewers associated with a given client device is determined according to the demographic or historical information of the respective one or more viewers.

21. The non-transitory computer-readable storage medium as recited in claim 18, wherein renderings of the one or more objects targeted at the one or more viewers associated with a given client device include renderings that advertise or market particular products or brands to the respective one or more viewers, wherein the particular products or brands are determined according to the respective information.

22. The non-transitory computer-readable storage medium as recited in claim 18, wherein the RVT system is further configured to, during playback of the pre-recorded video:

generate modified video content at least in part according to viewer interaction with video content of a scene in the pre-recorded video received from one of the client devices;

render new video of the scene to add the modified video content to the scene in the pre-recorded video; and

stream the new video of the scene in the pre-recorded video including the modified video content to the respective client device.