System & method for integrative analysis of intrinsic and extrinsic audio-visual

• US 20070061352A1
• Filed: 11/30/2004
• Published: 03/15/2007
• Est. Priority Date: 12/05/2003
• Status: Abandoned Application

• Current Assignee: Koninklijke Philips N.V.
• Original Assignee: Koninklijke Philips N.V.
• Inventors: Dimitrova, Nevenka, Turetsky, Robert
• Application Number:
  US10/596,112
• Publication Number:
  US 20070061352A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49

Abstract

A system is provided for integrative analysis of intrinsic and extrinsic audiovisual information, such as a system for analysis and correlation of features in a film with features not present in the film but available through the Internet. The system comprises an intrinsic content analyser communicatively connected to an audio-visual source, e.g. a film source, for searching the film for intrinsic data and extracting the intrinsic data using an extraction algorithm. Further, the system comprises an extrinsic content analyser communicatively connected to an extrinsic information source, such as a film screenplay available through the Internet, for searching the extrinsic information source and retrieving extrinsic data using a retrieval algorithm. The intrinsic data and the extrinsic data are correlated in a multisource data structure. The multisource data structure being transformed into high-level information structure which is presented to a user of the system. The user may browse the high-level information structure for such information as the actor identification in a film.

Claims

1. A system (100) for integrative analysis of intrinsic (10) and extrinsic (11) audio-visual data, the system comprising:

an intrinsic content analyser, the intrinsic content analyser being communicatively connected to an audio-visual source, the intrinsic content analyser being adapted to search the audio-visual source for intrinsic data and being adapted to extract intrinsic data using an extraction algorithm,

an extrinsic content analyser, the extrinsic content analyser being communicatively connected to an extrinsic information source, the extrinsic content analyser being adapted to search the extrinsic information source and being adapted to retrieve extrinsic data using a retrieval algorithm,

wherein the intrinsic data and the extrinsic data are correlated, thereby providing a multisource data structure.

2. A system according to claim 1, wherein the retrieval of the extrinsic data is based on the extracted intrinsic data.

3. A system according to claim 1, wherein the extraction and/or retrieval algorithm(s) is/are provided by a module.

4. A system according to claim 1, wherein a query is provided by a user, the query being provided to the extraction algorithm and wherein the intrinsic data is extracted in accordance with the query.

5. A system according to claim 1, wherein a query is provided by a user, the query being provided to the retrieval algorithm and wherein the extrinsic data is retrieved in accordance with the query.

6. A system according to claim 1, wherein a feature reflected in the intrinsic and extrinsic data include textual, audio and/or visual features.

7. A system according to claim 1, wherein the audio-visual source is a film (101) and wherein the extracted data include textual (104), audio and/or visual features (105, 106).

8. A system according to claim 1, wherein the extrinsic information source is connected to and may be accessed via the Internet (103).

9. A system according to claim 1, wherein the extrinsic information source is a film screenplay (102).

10. A system according to claim 9, wherein the extrinsic content analyser include knowledge about screenplay grammar, and wherein the extrinsic data is retrieved based on information extracted from the screenplay by use of the screenplay grammar.

11. A system according to claim 9 wherein the identification (5) of persons in a film is obtained by means of the screenplay.

12. A system according to claim 9 wherein a feature in a film is analysed based on information included in the screenplay.

13. A system according to claim 1, wherein the correlation of the intrinsic and extrinsic data is time correlation (121), thereby providing a multisource data structure where a feature reflected in the intrinsic data is time correlated to a feature reflected in the extrinsic data.

14. A system according to claim 13, wherein the time correlation is obtained by an alignment of a dialogue (120) in the screenplay to the spoken text (104) in the film and thereby providing a timestamped transcript (121) of the film.

15. A system according to claim 14, wherein a speaker identification in the film is obtained from the timestamped transcript.

16. A system according to claim 9, wherein the screenplay is compared with the spoken text in the film by means of a self-similarity matrix (30).

17. A system according to claim 1, wherein a high-level information structure (5-9) is generated in accordance with the multisource data structure.

18. A system according to claim 17, wherein the high-level information structure may be stored on a storage medium.

19. A system according to claim 17, wherein an updated high-level information structure is generated, the updated high-level information structure being an already existing high-level information structure which is updated in accordance with the multisource data structure.

20. A system according to claim 1, wherein the retrieval algorithm is a dynamic retrieval algorithm adapted to dynamically update itself by including additional functionalities in accordance with retrieved extrinsic data.

21. A system according to claim 20, wherein the additional functionalities is obtained by training the retrieval algorithm on a set of features from intrinsic data using labels obtained from the extrinsic data.

22. A system according to claim 9, wherein the training is performed using at least one screenplay.

23. A system according to 1, wherein an automatic ground truth identification in a film is obtained based on the multisource data structure for use in benchmarking algorithms on audio-visual content.

24. A system according to 1, wherein an automatic scene content understanding in a film is obtained based on the textual description in the screenplay and the audio-visual features from the film content.

25. A system according to 1, wherein an automatic labelling in a film is obtained based on the multisource data structure.

26. A method for integrative analysis of intrinsic and extrinsic audio-visual information, the method comprising the steps of:

searching an audio-visual source for intrinsic data and extracting intrinsic data using an extraction algorithm,

searching an extrinsic information source and retrieving extrinsic data using a retrieval algorithm,

correlating the intrinsic data and extrinsic data, thereby providing a multisource data structure.

27. A method according to claim 26 further comprising the step of generating a high-level information structure in accordance with the multisource data structure.

28. A method according to claim 26, wherein the extrinsic content analyser include knowledge about screenplay grammar, and wherein the extrinsic data is retrieved using information extracted from the screenplay by use of the screenplay grammar.

29. A method according to claim 26, wherein the retrieval algorithm is updated by training the algorithm on a set of extrinsic data.

30. (canceled)

31. (canceled)

32. (canceled)