TAGGING VIDEO USING CHARACTER RECOGNITION AND PROPAGATION

• US 20100246965A1
• Filed: 03/31/2009
• Published: 09/30/2010
• Est. Priority Date: 03/31/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ofek, Eyal, Epshtein, Boris
• Granted Patent: US 8,433,136 B2
• Time in Patent Office: Days
• US Class Current: 382/187
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

In one example, video may be analyzed and divided into segments. Character recognition may be performed on the segments to determine what text appears in the segments. The text may be used to assign tags to the video and/or to the segments. Segments that appear visually similar to each other (e.g., segments that appear to be different views of the same person) may be grouped together, and a tag that is assigned to one segment may be propagated to another segment. The tags may be used to perform various types of tasks with respect to the video. One example of such a task is to perform a search on the video.

Claims

1. One or more computer-readable storage media that store executable instructions that, when executed by a computer, cause the computer to perform acts comprising:

receiving a video;

using optical character recognition to identify an item of text that appears in a first segment of said video;

applying a tag to said first segment based on said text;

determining which one or more second segments in said video meet one or more similarity criteria with respect to said first segment;

propagating said tag to said one or more second segments based on said second segments meeting said similarity criteria with respect to said first segment; and

storing said video with said tag in a tangible medium.

2. The one or more computer-readable storage media of claim 1, further comprising:

analyzing said video to divide said video into segments.

3. The one or more computer-readable storage media of claim 2, wherein said analyzing of said video comprises:

calculating histograms of frames in said video;

identifying a boundary between segments based on finding an abrupt transition between successive frames.

4. The one or more computer-readable storage media of claim 1, wherein said acts further comprise:

comparing said text with words in a dictionary;

wherein said tag is applied based on said text having been found in said dictionary.

5. The one or more computer-readable storage media of claim 1, wherein said acts further comprise:

determining that said text comprises a name of a person based on comparison of said text with words in a dictionary of person names.

6. The one or more computer-readable storage media of claim 1, wherein said acts further comprise:

determining that said text comprises a name of a place based on a comparison of said text with a dictionary of place names.

7. The one or more computer-readable storage media of claim 6, wherein said acts further comprise:

associating said first segment with a geographic location of said place based on said text having been found, by said optical character recognition, to be in said first segment, and further based on said text having been found to contain said place name.

8. The one or more computer-readable storage media of claim 1, further comprising:

determining that said second segments meet said similarity criteria with respect to said first segment based on a comparison of histograms of said first segment and said second segments.

9. The one or more computer-readable storage media of claim 1, wherein said acts further comprise:

applying an edge-preserving temporal filter to said video prior to said using of said optical character recognition.

10. A system for labeling video, the system comprising:

a processor;

one or more components that execute on said processor, said one or more components comprising;

a segmenter that receives said video and determines boundaries between one or more segments in said video;

an optical character recognition component that identifies text that appears in said video;

a labeler that applies a tag, based on said text, to a first one of said segments, said labeler applying said tag to said first one of said segments based on a finding that said text appears in said first one of said segments; and

a propagator that propagates said tag to a second one of said segments based on a finding that said second one of said segments meets a level of similarity with respect to said first one of said segments.

11. The system of claim 10, further comprising:

a grouper that groups a set of said segments together based on a finding that said set of said segments meet said level of similarity with respect to each other, wherein said propagator propagates tags in one segment in said set to other segments in said set.

12. The system of claim 10, further comprising:

one or more dictionaries, wherein said labeler looks up said text in said dictionaries and determines which tag to apply to said first one of said segments based on said text appearing in said one or more dictionaries.

13. The system of claim 12, wherein said one or more dictionaries comprise a dictionary of person names, and wherein said labeler determines to tag said first one of said segments as containing a person having a name based on a finding that said text contains said name and that said name appears in said dictionary.

14. The system of claim 10, wherein said segmenter calculates histograms of frames in said video and determines said boundaries based on a finding of which successive frames of said video have sufficiently different histograms from each other.

15. The system of claim 10, wherein said labeler determines that said first one of said segments and said second one of said segments meet said level of similarity with respect to each other based on a comparison of (a) a first histogram calculated from said first one of said segments, or from one or more frames in said first one of said segments, and (b) a second histogram calculated from said second one of said segments, or from one or more frames in said second one of said segments.

16. The system of claim 10, wherein said labeler determines that said first one of said segments and said second one of said segments meet said level of similarity based on a comparison of a first set of pixels that appears in said first one of said segments with a second set of pixels that appear in said second one of said segments.

17. The system of claim 10, wherein said labeler compares points in a three dimensional space from said first one of said segments with points in said three dimensional space from said second one of said segments to determine that said first one of said segments and said second one of said segments are different perspective views of a scene.

18. The system of claim 10, wherein said system uses said tag to respond to a query by indicating, based on a comparison of said query with said tag, that said first one of said segments is relevant to said query.

19. A method of applying tags to video, the method comprising:

using a processor to perform acts comprising;

segmenting the video into a plurality of segments;

using optical character recognition to find text that appears in said video;

comparing said text with a dictionary;

based on a finding that said text appears in said dictionary, and further based on said text appearing in a first one of said segments, applying a tag to said first one of said segments, said tag being based on said text; and

responding to a search by comparing a query with tags in said video.

20. The method of claim 19, further comprising:

using a processor to perform acts comprising;

calculating a first histogram based on said first one of said segments and a second histogram based on a second one of said segments, said second one of said segments not containing said text;

determining, based on a comparison of said first histogram with said second histogram, that said second one of said segments meets a level of similarity with respect to said first one of said segments; and

propagating said tag to said second one of said segments.