INDEXING MULTIMEDIA COMMUNICATIONS

• US 20010042114A1
• Filed: 02/19/1998
• Published: 11/15/2001
• Est. Priority Date: 02/19/1998
• Status: Active Grant

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: SHUR, DAVID HILTON, MARKOWITZ, ROBERT E., ROSEN, KENNETH H., AGRAHARAM, SANJAY, WINTHROP, JOEL A.
• Granted Patent: US 6,377,995 B2
• Time in Patent Office: Days
• US Class Current: 709/223
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A network based platform uses face recognition, speech recognition, background change detection and key scene events to index multimedia communications. Before the multimedia communication begins, active participants register their speech and face models with a server. The process consists of creating a speech sample, capturing a sample image of the participant and storing the data in a database. The server provides an indexing function for the multimedia communication. During the multimedia communication, metadata including time stamping is retained along with the multimedia content. The time stamping information is used for synchronizing the multimedia elements. The multimedia communication is then processed through the server to identify the multimedia communication participants based on speaker and face recognition models. This allows the server to create an index table that becomes an index of the multimedia communication. In addition, through scene change detection and background recognition, certain backgrounds and key scene information can be used for indexing. Therefore, through this indexing apparatus and method, a specific participant can be recognized as speaking and the content that the participant discussed can also be used for indexing.

Claims

1. A method for indexing a multimedia communication, comprising:

receiving the multimedia communication, the multimedia communication including a plurality of multimedia data packets;

processing the plurality of multimedia data packets to identify distinguishing features; and

indexing the plurality of multimedia data packets based on the identified distinguishing features, wherein the processing step comprises associating each of the plurality of multimedia data packets with one of a plurality of objects within the multimedia communication.

2. The method of

claim 1

, wherein the distinguishing features are based on at least one of audio data and video data of the multimedia communication.

3. The method of

claim 1

, further comprising rebroadcasting the processed plurality of multimedia data packets.

4. The method of

claim 1

, wherein the objects include at least one of a person, an animal, a plant, and an inanimate object.

5. The method of

claim 4

, wherein a multimedia data packet of the plurality of multimedia data packets includes a payload having one of the audio and the video data that corresponds to an object, the associating step attaching a header identifier that identifies the object .

6. The method of

claim 5

, wherein the person is a speaker participating in the multimedia communication, audio speech patterns and video statistical sampling of a face of the speaker being a portion of the distinguishing features, the speaker being associated with the multimedia data packet if the portion of the distinguishing features is included in the payload of the multimedia data packet.

7. The method of

claim 5

, further comprising:

identifying background changes in the video data;

identifying key scene events based on the video data; and

attaching a second header identifier to each multimedia data packet containing the background change and the key scene event, the second header identifier identifying the multimedia data packet as containing a background change and a key scene event.

8. The method of

claim 7

, wherein the multimedia communication is a multicast multimedia communication, the rebroadcasting step including multicasting the processed plurality of multimedia data packets.

9. The method of

claim 8

, wherein a time stamp is provided to synchronize the audio and the video data.

10. The method of

claim 9

, further comprising storing the indexed plurality of multimedia data packets, wherein the indexed plurality of multimedia data packets can be searched to retrieve audio and video multimedia data packets corresponding to selected distinguishing features.

11. The method of

claim 10

, wherein the indexed plurality of multimedia data packets can be searched using key words.

12. The method of

claim 11

, wherein the multimedia communication is conducted using a local area network.

13. The method of

claim 1

, wherein the indexing and the processing steps are performed at a multicast network.

14. An apparatus for indexing a multimedia communication, comprising:

a server that receives multimedia communications in multimedia data packets including audio, visual and data communications and identifies distinguishing features in the multimedia communication based on audio and video recognition and a source of the multimedia communications;

a header function module connected to the server, the header function module entering metadata in a header segment corresponding to the multimedia data packets received by the server, the metadata being related to the distinguishing features; and

a storage device that stores the multimedia data packets.

15. The apparatus of

claim 14

, wherein the distinguishing features include audio voice and video face patterns of participants in the multimedia communications.

16. The apparatus of

claim 15

, wherein the metadata includes voice and face identifiers of the participants.

17. The apparatus of

claim 16

, wherein the server identifies background changes in the video multimedia data packets and wherein the header function module enters second metadata in the header segment corresponding to the multimedia data packets having background changes, the second metadata including scene identifiers.

18. The apparatus of

claim 17

, wherein the server is an multicast server.

19. The apparatus of

claim 18

, wherein the server comprises audio, video and data bridges.

20. A method of identifying participants to a multimedia communication, comprising:

comparing audio speech patterns for each participant to speech models;

comparing video face patterns for each participant to face models; and

determining an identity of a particular participant when both the audio speech patterns and the video face patterns match speech and face models for the particular participant.

21. The method of

claim 20

, further comprising creating an index of the participants based on identification of the speech and face patterns of the participants, the index being used to segment the multimedia communication.