Annotation framework for video

• US 7,559,017 B2
• Filed: 12/22/2006
• Issued: 07/07/2009
• Est. Priority Date: 12/22/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Datar, Mayur, Garg, Ashutosh, Mittal, Vibhu
• Primary Examiner(s): RIES, LAURIE ANNE
• Application Number:
  US11/615,771
• Publication Number:
  US 20080154908A1
• Time in Patent Office: 928 Days
• Field of Search: 715/201, 715/229, 715/230
• US Class Current: 715/230
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier

Abstract

A system and method for transferring annotations associated with a media file. An annotation associated with a media file is indexed to a first instance of that media file. By comparing features of the two instances, a mapping is created between the first instance of the media file and a second instance of the media file. The annotation can be indexed to the second instance using the mapping between the first and second instances. The annotation can be processed (displayed, stored, or modified) based on the index to the second instance.

Claims

1. A method for retrieving annotations, the method comprising:

receiving from a client device a request for annotations associated with a first segment of a first instance of a media file, the request including a feature derived from the content of the first segment of the first instance of the media file;

searching for the feature in a second instance of the media file to identify a second segment of the second instance of the media file, the second instance of the media file stored at a host device remotely located from the client device;

retrieving an annotation associated with the second segment of the second instance of the media file; and

transmitting the annotation to the client device for processing thereon.

2. The method of claim 1, wherein the media file comprises a video.

3. The method of claim 1, wherein searching for the feature in a second instance of the media file comprises mapping a first frame of the first segment of the first instance to a corresponding frame of the second segment of the second instance of the media file based on the feature.

4. The method of claim 1, wherein the annotation comprises a segment of audio.

5. The method of claim 1, wherein the annotation comprises a description of the media file.

6. The method of claim 1, wherein the annotation comprises an advertisement.

7. The method of claim 1, wherein the annotation comprises a command.

8. The method of claim 1, wherein searching for the feature in the second instance of the media file comprises:

receiving in the request a temporal position of the first segment within the first instance of the media file;

selecting from a plurality of segments of the second instance of the media file, a candidate set of segments based at least in part on the temporal position of the first segment, the candidate set including at least two segments of the second instance;

calculating, for each segment in the candidate set, a distance between the feature of the first segment and a feature of the segment in the candidate set; and

selecting the segment in the candidate set with a shortest calculated distance as the second segment.

9. A method for processing annotations associated with a media file, the method comprising:

determining a feature from content of a first segment of a first instance of the media file;

requesting from a server an annotation associated with the media file, the request including the feature for the server to search for the feature in a second instance of the media file to identify a second segment of the second instance of the media file;

receiving from the server a response to the request, said response comprising an annotation associated with the second segment of the second instance of the media file; and

processing the annotation.

10. The method of claim 9, wherein the media file comprises a video.

11. The method of claim 9, wherein the first segment comprises a first frame and wherein the second segment comprises a second frame.

12. The method of claim 9, wherein the annotation comprises a segment of audio.

13. The method of claim 9, wherein the annotation comprises a description of the media file.

14. The method of claim 9, wherein the annotation comprises an advertisement.

15. The method of claim 9, wherein the annotation comprises a command.

16. The method of claim 9, wherein requesting from the server an annotation comprises:

transmitting in the request a temporal position of the first segment for the server to;

selecting from a plurality of segments of the second instance of the media file a candidate set of segments based at least in part on the temporal position, the candidate set including at least two of the segments of the second instance;

calculate, for each segment in the candidate set, a distance between the feature of the first segment and a feature of the segment in the candidate set; and

select the segment in the candidate set with a shortest calculated distance as the second segment.

17. The method of claim 9, wherein processing the annotation comprises outputting the annotation in conjunction with the first segment of the first instance of the media file.

18. A method for storing annotations comprising:

receiving from a first client device a first annotation associated with a first segment of a first instance of a media file;

receiving a feature derived from the content of the first segment of the first instance of the media file;

searching for the feature in a second instance of the media file to identify a second segment of the second instance of the media file, the second instance of the media file stored at a host device remotely located from the first client device; and

storing the first annotation, wherein the first annotation is indexed to the second segment of the second instance of the media file.

19. The method of claim 18, wherein the media file comprises a video.

20. The method of claim 18, wherein searching for the feature in a second instance of the media file comprises mapping a first frame of the first segment of the first instance to a corresponding frame of the second segment of the second instance of the media file based on the feature.

21. The method of claim 18, wherein the first annotation comprises a segment of audio.

22. The method of claim 18, wherein the first annotation comprises a description of the media file.

23. The method of claim 18, wherein the first annotation comprises an advertisement.

24. The method of claim 18, wherein the first annotation comprises a command.

25. The method of claim 18, further comprising:

receiving from a second client device a second annotation; and

storing the second annotation, wherein the second annotation is indexed to the second segment of the second instance of the media file.

26. The method of claim 18, wherein searching for the feature in the second instance of the media file comprises:

receiving a temporal position of the first segment within the first instance of the media file;

selecting from a plurality of segments of the second instance of the media file, a candidate set of segments based at least in part on the temporal position of the first segment, the candidate set including at least two segments of the second instance;

calculating, for each segment in the candidate set, a distance between the feature of the first segment and a feature of the segment in the candidate set; and

selecting the segment in the candidate set with a shortest calculated distance as the second segment.

27. A system for indexing annotations comprising:

a feature detector, configured to search for a feature in a first instance of a media file to map a second instance of the media file to the first instance of the media file, the feature derived from the content of the second instance of the media file;

an annotation retriever, configured to retrieve an annotation indexed to the first instance of the media file; and

an annotation indexer, configured to index the annotation to the second instance of the media file using the mapping.

28. The system of claim 27, further comprising:

an annotation displayer, configured to display the annotation together with the second instance of the media file.

29. The system of claim 27, the feature detector further configured to:

receive a temporal position derived from the second instance of the media file;

select from a plurality of segments of the first instance of the media file, a candidate set of segments based at least in part on the temporal position derived from the second instance, the candidate set including at least two segments of the first instance;

calculating, for each segment in the candidate set, a distance between the feature of the second instance and a feature of the segment in the candidate set; and

map the second instance to the segment in the candidate set with a shortest calculated distance.

30. A system for retrieving annotations, the system comprising:

means for receiving from a client device a request for annotations associated with a first segment of a first instance of a media file, the request including a feature derived from the content of the first segment of the first instance of the media file;

means for searching for the feature in a second instance of the media file to identify a second segment of the second instance of the media file, the second instance of the media file stored at a host device remotely located from the client device;

means for retrieving an annotation associated with the second segment of the second instance of the media file; and

means for transmitting the annotation to the client device for display thereon.

31. The system of claim 30, further comprising:

means for receiving in the request a temporal position of the first segment within the first instance of the media file;

means for selecting from a plurality of segments of the second instance of the media file, a candidate set of segments based at least in part on the temporal position of the first segment, the candidate set including at least two segments of the second instance;

means for calculating, for each segment in the candidate set, a distance between the feature of the first segment and a feature of the segment in the candidate set; and

means for selecting the segment in the candidate set with a shortest calculated distance as the second segment.

32. A computer program product, comprising a computer-readable storage medium having computer-executable code for retrieving annotations, the computer-executable code configured to cause a computer processor to perform the operations of:

receiving from a client device a request for annotations associated with a first segment of a first instance of a media file, the request including a feature derived from the content of the first segment of the first instance of the media file;

searching for the feature in a second instance of the media file to identify a second segment of the second instance of the media file, the second instance of the media file stored at a host device remotely located from the client device;

retrieving an annotation associated with the second segment of the second instance of the media file; and

transmitting the annotation to the client device for processing thereon.

33. The computer program product of claim 32, wherein searching for the feature in the second instance comprises:

receiving in the request a temporal position of the first segment within the first instance of the media file;

selecting from a plurality of segments of the second instance of the media file, a candidate set of segments based at least in part on the temporal position of the first segment, the candidate set including at least two segments of the second instance;

calculating, for each segment in the candidate set, a distance between the feature of

the first segment and a feature of the segment in the candidate set; and

selecting the segment in the candidate set with a shortest calculated distance as the second segment.

34. A computer program product, comprising a computer-readable storage medium having computer-executable code for processing annotations associated with a media file, the computer-executable code configured to cause a computer processor to perform the operations of:

determining a feature from content of a first segment of a first instance of the media file;

requesting from a server an annotation associated with the media file, the request including the feature for the server to search for the feature in a second instance of the media file to identify a second segment of the second instance of the media file;

receiving from the server a response to the request, said response comprising an annotation associated with the second segment of the second instance of the media file; and

processing the annotation.

35. The computer program product of claim 34, wherein requesting from the server an annotation comprises:

transmitting in the request a temporal position of the first segment for the server to;

select from a plurality of segments of the second instance of the media file a candidate set of segments based at least in part on the temporal position, the candidate set including at least two of the segments of the second instance;

calculate, for each segment in the candidate set, a distance between the feature of the first segment and a feature of the segment in the candidate set; and

select the segment in the candidate set with a shortest calculated distance as the second segment.

36. The computer program product of claim 34, wherein searching for the feature in the second instance comprises:

receiving in the request a temporal position of the first segment within the first instance of the media file;

selecting from a plurality of segments of the second instance of the media file, a candidate set of segments based at least in part on the temporal position of the first segment, the candidate set including at least two segments of the second instance;

calculating, for each segment in the candidate set, a distance between the feature of the first segment and a feature of the segment in the candidate set; and

selecting the segment in the candidate set with a shortest calculated distance as the second segment.

37. The computer program product of claim 34, wherein processing the annotation comprises outputting the annotation in conjunction with the first segment of the first instance of the media file.

38. A computer program product, comprising a computer-readable storage medium having computer-executable code for storing annotations, the computer-executable code configured to cause a computer processor to perform the operations of:

receiving from a first client device a first annotation associated with a first segment of a first instance of a media file;

receiving a feature derived from the content of the first segment of the first instance of the media file;

searching for the feature in a second instance of the media file to identify a second segment of the second instance of the media file, the second instance of the media file stored at a host device remotely located from the first client device; and

storing the first annotation, wherein the first annotation is indexed to the second segment of the second instance of the media file.

39. The computer program product of claim 38, wherein searching for the feature in the second instance comprises:

receiving in the request a temporal position of the first segment within the first instance of the media file;

selecting from a plurality of segments of the second instance of the media file, a candidate set of segments based at least in part on the temporal position of the first segment, the candidate set including at least two segments of the second instance;

calculating, for each segment in the candidate set, the distance between the feature of the first segment and a feature of the segment in the candidate set; and

selecting the segment in the candidate set with a shortest calculated distance as the second segment.

40. A computer program product, comprising a computer-readable storage medium having computer-executable code configured to cause a computer processor to perform the operations of:

storing a first instance of a media file having audio-visual content;

storing a second instance of the media file, the second instance of the media file containing audio-visual content that is different at least in part from the first instance of the media file;

receiving a feature derived from the audio-visual content of a first segment of the first instance of the media file;

searching for the feature in the second instance of the media file to identify a second segment of the second instance of the media file; and

storing a mapping of the first segment to the second segment.