Method for estimating a timestamp in a video stream and method of augmenting a video stream with information

• US 10,721,431 B2
• Filed: 05/18/2018
• Issued: 07/21/2020
• Est. Priority Date: 06/01/2017
• Status: Active Grant

• Current Assignee: Accedo Broadband AB
• Original Assignee: Eyecandylab Corporation
• Inventors: Ahmann, Sascha, Hackl, Martin, Moser, Robin, Orzikowski, Thomas
• Primary Examiner(s): Prabhakher, Pritham D
• Application Number:
  US15/983,731
• Publication Number:
  US 20180352186A1
• Time in Patent Office: 795 Days
• Field of Search: 34823199
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/43079 of additional data with content streams on multiple devices
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4725 using interactive regions of the image, e.g. hot spots details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits

Abstract

The invention discloses a method for determining a timestamp t of an image of a video stream V(t) comprising a plurality of images, said method comprising the steps of determining an acquired image descriptor of at least one of a plurality of acquired images Si of an acquired video stream during a display phase and determining a similarity level of an original image descriptor and the acquired image descriptor and if the similarity level of the original image descriptor and the acquired image descriptor fulfills at least one criterion, determining the timestamp t of the original image descriptor and assigning the timestamp t to the acquired image corresponding to the acquired image descriptor as an estimated timestamp T. Augmented information can be displayed to a user depending on the time stamp.

Claims

1. A method for determining a timestamp t of an image of a video stream V(t) comprising a plurality of images, said method comprising the following steps:

receiving a plurality of original image descriptors at a monitoring unit, wherein each original image descriptor is associated to an image in the original video stream and wherein the original image

descriptors are generated during preparation phase;

displaying the original video stream V(t) on a display device during a display phase;

acquiring an acquired video stream by an acquisition device of the monitoring unit by acquiring images Si of the display device on which the original video stream V(t) is displayed;

determining an acquired image descriptor of at least a portion of at least one of a plurality of acquired images S of the acquired video stream during the display phase;

determining a similarity level of an original image descriptor and the acquired image descriptor; and

if the similarity level of the original image descriptor and the acquired image descriptor fulfills at least one criterion, determining the timestamp t of the original image descriptor and assigning the timestamp t to the acquired image corresponding to the acquired image descriptor as an estimated timestamp T.

2. The method according to claim 1, further comprising the step of

determining the uniqueness level of the acquired image;

storing the acquired image descriptor and the estimated timestamp T as a candidate point in a list comprising a plurality of candidate points;

if the number of candidate points exceed a predetermined number, removing the candidate point with the lowest uniqueness level from the list.

3. The method according to claim 1, wherein the step of determining the uniqueness level ω

i of the acquired image descriptor includes the step of comparing the acquired image descriptor with the acquired image descriptors of a plurality of neighboring acquired images.

4. The method according to claim 3, further comprising the steps of

determining the candidate point of all candidate points in the list with the highest uniqueness level ω

i; and

determining the timestamp t based on basis of the candidate point having the highest uniqueness level ω

i.

5. The method according to claim 3, wherein the step of determining the uniqueness level of a ω

i of the acquired image descriptor of the acquired image includes at least one of the following steps;

analyzing the variance of the timestamp Ti associated to the acquired image Si;

determining the inverse standard deviation of the timestamp Ti associated to the acquired image Si;

determining the inverse standard deviation of the timestamp Ti associated to the acquired image Si based on the median of the timestamp Ti associated to the acquired image Si.

6. The method according to claim 1, further comprising the following step:

determining the timestamp TO of the starting point of acquired video stream by the formula TO=T*−

t*;

wherein

T* is the timestamp of the acquired image; and

t* is the timestamp of the corresponding original image.

7. The method according to claim 6, further comprising the following step:

determining the timestamp TO of the starting point of acquired video stream by the formula TO=T*−

t*;

wherein

T* is the timestamp of the acquired image corresponding to the candidate point having the highest uniqueness level; and

t* is the timestamp of the corresponding original image.

8. The method according to claim 7, wherein the uniqueness level is

determined by the step of calculating a distance function Di(t|, V(t), Si), wherein

t is the timestamp in the original video steam;

V(t) is the original video steam; and

Si is the acquired image.

9. The method according to claim 8, further comprising the step of

determining the candidate point (S*, T*) having the highest uniqueness level of all candidate points (Si, Ti, i=1, . . . , n) by the step of minimizing the functional;

J(t)=Σ

ω

i Di(t|V(t), Si), i=1, . . . , n;

wherein

S* is the image having the highest uniqueness level of all candidate points;

T* is the timestamp of the image having the highest uniqueness level of all candidate points;

Si is the image i; and

Ti is the timestamp of the image i.

10. The method according to claim 1, wherein the step of determining an acquired image descriptor comprises at least one of the following steps:

performing an image histogram equalization;

detecting local feature points;

calculating local feature descriptors;

calculating a vector of locally aggregated descriptors; and

/or

decreasing the dimension of the locally aggregated descriptors.

11. The method according to claim 1, further comprising at least one of

the following steps:

detecting the corners of the display device by image processing;

detecting the orientation of the display device by determining the orientation of the mobile unit by gyroscopic data of the mobile unit;

detecting the edge of the display device by image processing;

determining the orientation of the display device by image processing;

transforming the orientation of the captured images of the captured video stream to the orientation of the mobile unit;

extracting the acquired image of the acquired video stream from the captured image of the captured image stream, wherein the acquired image corresponds to the portion of the captured image that depicts the original video stream.

12. The method according to claim 1, wherein

the original video stream is a television signal stream;

the original video stream is a broadcast signal;

the original video stream is a multicast signal;

the display device is a television set;

the display device is a display monitor;

the monitoring unit is a mobile computer and the acquisition device is a camera of the mobile computer;

the monitoring unit is a tablet computer and the acquisition device is a camera of the tablet computer;

the monitoring unit is a smart phone and the acquisition device is a camera of the smart phone; and

/or

the monitoring unit are data glasses and the acquisition device is a camera of the data glasses.

13. A method of augmenting a displayed video stream by displaying

information, comprising the following steps:

determining a timestamp t of an image of an original video stream V(t) according to claim 12;

receiving at the mobile unit at least one information that is to displayed at least one display timestamp td of the original video stream V(t); and

displaying the information at the mobile unit, if the mobile unit determines that the image corresponding to the display timestamp td of the original video stream V(t) is displayed on the display device.