Personalized video-based augmented reality

• US 10,078,920 B2
• Filed: 04/19/2018
• Issued: 09/18/2018
• Est. Priority Date: 09/01/2016
• Status: Active Grant

• Current Assignee: Avid Technology Incorporated
• Original Assignee: Avid Technology Incorporated
• Inventors: Wojdala, Andrzej, Borys, Piotr A., Benovici, Ofir, Sela, Tomer
• Primary Examiner(s): Wilson, Nicholas R
• Application Number:
  US15/956,863
• Publication Number:
  US 20180240277A1
• Time in Patent Office: 152 Days
• Field of Search: None
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8455 involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

In current systems, augmented reality graphics is generated at a central broadcast facility or studio where it is combined with the video that is transmitted to subscribers. By contrast, in the described system, the studio does not generate the graphics, but transmits video together with real-time metadata to the end-user set-top device. The end-user device generates the augmented reality graphics, using the metadata to determine positional and other parameters for displaying the graphics. Shifting the generation of augmented reality graphics to the consumer level facilitates end-user customization and individualized targeting of information by a broadcaster or advertiser.

Claims

1. A method performed by a user device, the method comprising:

receiving a broadcast video stream;

receiving time-based metadata that is associated with one or more video frames of the broadcast video stream, wherein;

the time-based metadata defines a spatial position within the one or more video frames of an object appearing within the one or more video frames; and

the user device is connected to a video display, and is able to display on the video display the one or more video frames of the broadcast video stream independently of the received time-based metadata;

automatically generating an element for display on the video display, wherein an attribute of the element is determined by the metadata that defines the spatial position of the object appearing within the one or more video frames; and

displaying the element on the video display.

2. The method of claim 1, wherein the attribute of the element is a spatial position of the element as displayed on the video display.

3. The method of claim 2, wherein the spatial position of the element as displayed on the video display corresponds to the spatial position within the one or more video frames of the object appearing within the one or more video frames.

4. The method of claim 2, wherein the element is a graphical object, and a feature of the graphical object corresponds to the spatial position of within the one or more video frames of the object appearing within the one or more video frames.

5. The method of claim 4, wherein the graphical object is a line indicating a trajectory of the object, and the feature is one of a starting point and an ending point of the trajectory.

6. The method of claim 2, wherein the spatial position of the element as displayed on the video display is offset from the spatial position of the object appearing within the one or more video frames.

7. The method of claim 2, wherein the element is a graphical object, and the graphical object is rendered so as to appear to be located within a scene depicted in the one or more video frames.

8. The method of claim 7, wherein the graphical object includes advertising information.

9. The method of claim 1, wherein the attribute of the element is an information content of the element.

10. The method of claim 9, wherein the information content of the element includes data pertaining to a difference between the spatial position of the object within the one or more frames and a spatial position of the object within a frame other than the first-mentioned one or more frames.

11. The method of claim 9, wherein the information content of the element includes data pertaining to a rate of change of the spatial position of the object within the one or more frames and a spatial position of the object within frames other than the first-mentioned one or more frames.

12. The method of claim 1, wherein the user device displays the one or more video frames on the video display and the generated element is superposed onto the displayed one or more video frames on the video display.

13. A computer program product comprising:

a non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed on a processor of a user device, cause the user device to perform a method comprising;

receiving a broadcast video stream;

receiving time-based metadata that is associated with one or more video frames of the broadcast video stream, wherein;

the time-based metadata defines a spatial position within the one or more video frames of an object appearing within the one or more video frames; and

the user device is connected to a video display, and is able to display on the video display the one or more video frames of the broadcast video stream independently of the received time-based metadata;

automatically generating an element for display on the video display, wherein an attribute of the element is determined by the metadata that defines the spatial position of the object appearing within the one or more video frames; and

displaying the element on the video display.

14. A user device comprising:

a CPU; and

a memory storing instructions for execution by the CPU, wherein execution of the instructions on the CPU implements a method comprising;

receiving a broadcast video stream;

receiving time-based metadata that is associated with one or more video frames of the broadcast video stream, wherein;

the time-based metadata defines a spatial position within the one or more video frames of an object appearing within the one or more video frames; and

the user device is connected to a video display, and is able to display on the video display the one or more video frames of the broadcast video stream independently of the received time-based metadata;

automatically generating an element for display on the video display, wherein an attribute of the element is determined by the metadata that defines the spatial position of the object appearing within the one or more video frames; and

displaying the element on the video display.