Multimodal access of meeting recordings

• US 7,298,930 B1
• Filed: 11/29/2002
• Issued: 11/20/2007
• Est. Priority Date: 11/29/2002
• Status: Expired due to Fees

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Hull, Jonathan J., Erol, Berna, Lee, Dar-Shyang
• Primary Examiner(s): Bella; Matthew C.
• Assistant Examiner(s): PERUNGAVOOR, SATHYANARAYA V
• Application Number:
  US10/307,235
• Time in Patent Office: 1,817 Days
• Field of Search: 382/305
• US Class Current: 382/305
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: Y10S 707/99931 Database or file accessing

Abstract

A meeting recorder captures multimodal information of a meeting. Subsequent analysis of the information produces scores indicative of visually and aurally significant events that can help identify significant segments of the meeting recording. Textual analysis can enhance searching for significant meeting segments and otherwise enhance the presentation of the meeting segments.

Claims

1. A method for retrieving content comprising multimedia information, the method comprising:

receiving a request to retrieve a portion of said multimedia information;

in response to said request, producing a first portion of said multimedia information and presenting said first portion of said multimedia information;

producing a term-frequency inverse document frequency (TF-IDF) score for each word in a plurality words contained in a transcript of said multimedia information;

producing a plurality of keywords by taking words from said plurality of words based on said TF-IDF scores; and

for each keyword, producing a document occurrence score for a plurality of segments of said transcript and associating a segment having a highest document occurrence score with said each keyword, said segments corresponding to portions of said multimedia information,

wherein producing said first portion includes incorporating portions of said multimedia information corresponding to one or more of said segments of said transcript,

said producing a first portion of said multimedia information comprising;

selecting a frame of video from a plurality of video frames, said selecting based on a visual significance score, said visual significance score indicating a responsiveness of said selected frame to said request;

selecting a segment of audio of said multimedia information, said selecting based on an audio significance score, said audio significance score indicating a responsiveness of said selected segment of audio to said request, said segment of audio having a corresponding video clip;

generating a video skim from said selected frame of video and said video clip; and

displaying said generated video skim.

2. The method of claim 1 wherein said transcript further comprises textual material from one or more additional sources comprising whiteboard input, slide presentation material, and transcribed notes.

3. The method of claim 1 wherein said video information is MPEG encoded video data, and said video frames comprises I-frames.

4. The method of claim 1 wherein said audio significance score is based on a measure of speaking activity.

5. The method of claim 1 further comprising:

determining a best shot image from among candidate images of a person'"'"'s face comprising, for each candidate image, steps of;

identifying a plurality of facial features;

determining a plurality of measurements among said facial features; and

comparing said measurements against template measurements to produce an error quantity, said template measurements representing a plurality of average measurements of facial features,

wherein a candidate image having a minimal error quantity is selected as a best shot image,

wherein one or more such best shot images of people can be displayed along with said video skim.

6. The method of claim 5 wherein said identifying a plurality of facial features includes:

determining a time index associated with said candidate image;

determining sound directions occurring during a period of time corresponding to said associated time index;

identifying pupils from said candidate image; and

if a positive determination of pupil identification occurs, confirming said positive determination based on said sound directions.

7. The method of claim 1 further comprising:

determining a best shot image from among candidate images of a person'"'"'s face comprising, for each candidate image, steps of;

identifying pupils;

identifying a head outline;

determining location metrics of the pupils relative to said head outline;

determining pupil orientation relative to a reference; and

determining head orientation relative to a reference,

wherein a best shot is selected from among said candidate image based on said location metrics, said pupil orientation, and said head orientation,

wherein one or more of such best shot can be displayed with said video skim.

8. The method of claim 7 wherein said identifying pupils includes:

determining a time index associated with said candidate image;

determining sound directions occurring during a period of time corresponding to said associated time index;

identifying pupils from said candidate image; and

if a positive determination of pupil identification occurs, confirming said positive determination based on said sound directions.

9. A system for accessing content from multimedia data, the system comprising:

a data component configured to provide said multimedia data;

a scoring component operative to produce scores corresponding to segments of said multimedia data, wherein said scores include visual significance scores and audio significance scores; and

a selection component operative to select segments of said multimedia data based on their corresponding scores, wherein said selection is performed in response to a request for a portion of said multimedia data,

wherein each of said scores indicates a responsiveness of the corresponding segment to the request,

wherein said scoring component is further operative to produce a video skim incorporating portions of said multimedia data corresponding to one or more of said selected segments,

wherein said scoring component comprises a computer program configured to perform operations comprising;

producing a term-frequency inverse document frequency (TF-IDF) score for each word in a plurality words contained in a transcript of said multimedia data;

producing a plurality of keywords by taking words from said plurality of words based on said TF-IDF scores; and

for each keyword, producing a document occurrence score for a plurality of segments of said transcript and associating a segment having a highest document occurrence score with said each keyword, said segments corresponding to portions of said multimedia data.

10. The system of claim 9 wherein said scoring component comprises a computer program configured to perform operations comprising:

obtaining a first video frame from video data contained in said multimedia data, said first video frame comprising a plurality of first subframes;

obtaining a second video frame from said video data, said second video frame comprising a plurality of second subframes, each of said first subframes associated with one of said second subframes;

for each of said first subframes computing a local luminance metric as a function of said first subframe and its associated second subframe, thereby generating a plurality of local luminance metrics associated with said first video frame; and

associating a visual score with said first video frame, said visual score based on one or more of said local luminance metrics,

wherein said foregoing steps are repeated for a plurality of video frames in said video data to produce a plurality of visual scores, each of said visual scores associated with each of said video frames.

11. The system of claim 10 wherein said local luminance metric is determined for successive I-frames.

12. The system of claim 9 wherein said multimedia data comprises MPEG (motion picture experts group) video.

13. The system of claim 12 wherein the video is omni-directional video.

14. The system of claim 9 wherein said multimedia data comprises video compressed using a block-based compression format including MPEG-1, MPEG-2, MPEG-4, H.263, H.263L, and JVT/H.26L.

15. The apparatus of claim 9 further including a display component operable to display said segments of said multimedia data, said display component including a computer program for determining a best shot image from among a plurality of images of a person'"'"'s face, said computer program configured to perform operations comprising, for each image:

identifying a plurality of facial features;

determining a plurality of measurements among said facial features; and

comparing said measurements against template measurements to produce an error quantity, said template measurements representing a plurality of average measurements of facial features,

wherein an image having a minimal error quantity is selected as a best shot image.

16. The apparatus of claim 15 wherein said identifying a plurality of facial features includes:

determining a time index associated with said candidate image;

determining sound directions occurring during a period of time corresponding to said associated time index;

identifying pupils from said candidate image; and

if a positive determination of pupil identification occurs, confirming said positive determination based on said sound directions.

17. The apparatus of claim 9 further including a display component operable to display said segments of said multimedia data, said display component including a computer program for determining a best shot image from among a plurality of images of a person'"'"'s face, said computer program configured to perform operations comprising, for each image:

identifying pupils and a head outline;

determining location metrics of said pupils relative to said head outline;

determining pupil orientation relative to a reference; and

determining head orientation relative to a reference,

wherein a best shot is selected from among said images based on said location metrics, said pupil orientation, and said head orientation.