Using Audio Cues to Improve Object Retrieval in Video

• US 20150120726A1
• Filed: 10/30/2013
• Published: 04/30/2015
• Est. Priority Date: 10/30/2013
• Status: Active Grant

• Current Assignee: Texas Instruments, Inc.
• Original Assignee: Texas Instruments, Inc.
• Inventors: Barnum, Peter Charles, Jain, Eakta
• Granted Patent: US 10,108,617 B2
• Time in Patent Office: Days
• US Class Current: 707/736
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/30 on the same track as the main recording
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

A method of object retrieval from visual data is provided that includes annotating at least one portion of the visual data with a context keyword corresponding to an object, wherein the annotating is performed responsive to recognition of the context keyword in audio data corresponding to the at least one portion of the visual data, receiving a query to retrieve the object, wherein the query includes a query keyword associated with both the object and the context keyword, identifying the at least one portion of the visual data based on the context keyword, and searching for the object in the at least one portion of the visual data using an appearance model corresponding to the query keyword.

Claims

1. A method of object retrieval from visual data, the method comprising:

annotating at least one portion of the visual data with a context keyword corresponding to an object, wherein the annotating is performed responsive to recognition of the context keyword in audio data corresponding to the at least one portion of the visual data;

receiving a query to retrieve the object, wherein the query comprises a query keyword associated with both the object and the context keyword;

identifying the at least one portion of the visual data based on the context keyword; and

searching for the object in the at least one portion of the visual data using an appearance model corresponding to the query keyword.

2. The method of claim 1, wherein the appearance model, the corresponding query keyword, and the context keyword are predetermined.

3. A digital system configured to perform object retrieval from visual data, the digital system comprising:

means for capturing the visual data and corresponding audio data;

means for annotating at least one portion of the visual data with a context keyword corresponding to an object, wherein the annotating is performed responsive to recognition of the context keyword in audio data corresponding to the at least one portion of the visual data;

means for receiving a query to retrieve the object, wherein the query comprises a query keyword associated with both the object and the context keyword;

means for identifying the at least one portion of the visual data based on the context keyword; and

means for searching for the object in the at least one portion of the visual data using an appearance model corresponding to the query keyword.

4. The digital system of claim 3, wherein the appearance model, the corresponding query keyword, and the context keyword are predetermined.

5. A method of object retrieval from visual data in a digital system, the method comprising:

receiving video frames and corresponding audio frames;

annotating some of the video frames with audio cues recognized from corresponding audio frames; and

retrieving objects from annotated video frames using the audio cues.

6. The method of claim 5, wherein annotating some of the video frames comprises:

analyzing the audio frames to recognize predetermined context keywords; and

annotating a video frame with a predetermined context keyword when the predetermined context keyword is recognized in a corresponding audio frame.

7. The method of claim 6, wherein retrieving objects comprises:

receiving a query to retrieve an object;

determining one or more context keywords associated with a query keyword identified from the query, wherein the query keyword identifies the object;

identifying at least one video frame annotated with the one or more context keywords;

retrieving an appearance model for the object, wherein the appearance model is associated with the query keyword; and

searching the at least one frame for the object using the appearance model.