Personalized video-based augmented reality

• US 9,965,900 B2
• Filed: 09/01/2016
• Issued: 05/08/2018
• Est. Priority Date: 09/01/2016
• Status: Active Grant

• Current Assignee: Avid Technology Incorporated
• Original Assignee: Avid Technology Incorporated
• Inventors: Wojdala, Andrzej, Borys, Piotr A., Benovici, Ofir, Sela, Tomer
• Primary Examiner(s): Wilson, Nicholas R
• Application Number:
  US15/253,953
• Publication Number:
  US 20180061130A1
• Time in Patent Office: 614 Days
• Field of Search: None
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8455 involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

In current systems, augmented reality graphics is generated at a central broadcast facility or studio where it is combined with the video that is transmitted to subscribers. By contrast, in the described system, the studio does not generate the graphics, but transmits video together with real-time metadata to the end-user set-top device. The end-user device generates the augmented reality graphics, using the metadata to determine positional and other parameters for displaying the graphics. Shifting the generation of augmented reality graphics to the consumer level facilitates end-user customization and individualized targeting of information by a broadcaster or advertiser.

Claims

1. A method performed by a user device, the method comprising:

receiving a broadcast video stream;

receiving time-based metadata that is associated with one or more video frames of the broadcast video stream, wherein;

the time-based metadata defines a spatial position within the one or more video frames at which the user device is to superpose onto the one or more video frames an element generated by the user device; and

the user device is able to display the one or more video frames of the broadcast video stream independently of the received time-based metadata;

generating the element and superposing the element onto the one or more video frames of the broadcast video stream at the spatial position within the one or more video frames defined by the time-based metadata, wherein an attribute of the generated element is based on a preselected preference by a user of the user device; and

displaying the one or more video frames and the superposed element on a display of the user device.

2. The method of claim 1, wherein the generated element is an augmented reality graphical element.

3. The method of claim 1, wherein the attribute is a shape of the generated element.

4. The method of claim 1, wherein the attribute is a color of the generated element.

5. The method of claim 1, wherein the metadata includes advertising information, and the generated element includes a rendering of the advertising information projected onto an object represented in the one or more video frames such that the advertising information appears to be displayed on a surface of the object.

6. The method of claim 5, wherein the advertising information is based in part on a geographical location of the user device.

7. The method of claim 5, wherein the advertising information is based in part on program viewing preferences of the user of the user device.

8. The method of claim 1, wherein the generated element represents data obtained by analyzing the one or more video frames.

9. The method of claim 8, wherein the analysis includes determining a position of an object that appears within the one or more video frames, and the generated element is based in part on the position of the object.

10. The method of claim 8, wherein the analysis includes determining a position of two objects that appear within the one or more video frames, and the generated element is based in part on a distance between the two objects.

11. The method of claim 8, wherein the analysis includes analyzing a first frame and a second frame of the broadcast video stream and determining a distance between a position of an object in the first video frame and a position of the object in the second video frame.

12. The method of claim 1 wherein the generating step is based in part on a preselection stored by the user device.

13. The method of claim 12, wherein the preselection includes a specification of a set of element types to be generated for display.

14. The method of claim 1, further comprising receiving input from the user of the user device, and wherein the attribute is based in part on the input from the user.

15. The method of claim 1, wherein the time-based metadata is received as part of the broadcast video stream, and the time-based metadata is synchronized with the one or more video frames with which the time-based metadata is associated within the broadcast video stream.

16. The method of claim 1, wherein the time-based metadata is timestamped and is received asynchronously from the broadcast video stream, and the user device uses timestamps of the metadata to associate the metadata with a corresponding one or more video frames of the broadcast video stream.

17. A computer program product comprising:

a non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed on a processor of a user device, cause the user device to perform a method comprising;

receiving a broadcast video stream;

receiving time-based metadata that is associated with one or more video frames of the broadcast video stream, wherein;

the time-based metadata defines a spatial position within the one or more video frames at which the user device is to superpose onto the one or more video frames an element generated by the user device; and

the user device is able to display the one or more video frames of the broadcast video stream independently of the received time-based metadata;

generating the element and superposing the element onto the one or more video frames of the broadcast video stream at the spatial position within the one or more video frames defined by the time-based metadata, wherein an attribute of the generated element is based on a preselected preference by a user of the end-user user device; and

displaying the one or more video frames and the superposed element on a display of the user device.

18. A user device comprising:

a CPU; and

a memory storing instructions for execution by the CPU, wherein execution of the instructions on the CPU implements a method comprising;

receiving a broadcast video stream;

receiving time-based metadata that is associated with one or more video frames of the broadcast video stream, wherein;

the time-based metadata defines a spatial position within the one or more video frames at which the user device is to superpose onto the one or more video frames an element generated by the user device; and

the user device is able to display the one or more video frames of the broadcast video stream independently of the received time-based metadata;

generating the element and superposing the element onto the one or more video frames of the broadcast video stream at the spatial position within the one or more video frames defined by the time-based metadata, wherein an attribute of the generated element is based on a preselected preference by a user of the user device; and

displaying the one or more video frames and the superposed element on a display of the user device.