Context and content based information processing for multimedia segmentation and indexing

• US 20020157116A1
• Filed: 03/09/2001
• Published: 10/24/2002
• Est. Priority Date: 07/28/2000
• Status: Abandoned Application

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Jasinschi, Radu Serban
• Application Number:
  US09/803,328
• Publication Number:
  US 20020157116A1
• Time in Patent Office: Days
• US Class Current: 725/136
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Method and system are disclosed for information processing, for example, for multimedia segmentation, indexing and retrieval. The method and system includes multimedia, for example audio/visual/text (A/V/T), integration using a probabilistic framework. Both, multimedia content and context information are represented and processed via the probabilistic framework. This framework is represented, for example, by a Bayesian network and hierarchical priors, which is graphically described by stages, each having a set of layers with each layer including a number of nodes representing content or context information. At least the first layer of the first stage is processes multimedia content information such as objects in the A/V/T domains, or combinations of thereof. The other layers of the various stages describe multimedia context information, as further described below. Each layer is a Bayesian network, wherein nodes of each layer explain certain characteristics of the next “lower” layer and/or “lower” stages. Together, the nodes and connections there between form an augmented Bayesian network. Multimedia context is the circumstance, situation, underlying structure of the multimedia information (audio, visual, text) being processed. The multimedia information (both content and context) is combined at different levels of granularity and level of abstraction within the layers and stages.

Claims

1. A data processing device for processing an information signal comprising:

at least one stage, wherein a first stage includes,

a first layer having first plurality of nodes for extracting content attributes from the information signal; and

a second layer having at least one node for determining context information for the at least one node using the content attributes of selected nodes in an other layer or a next stage, and for integrating certain ones of the content attributes and the context information at the at least one node.

2. The data processing device according to claim 1, further including a second stage, the second stage having, at least one layer having at least one node for determining context information for the at least one node using the content attributes of selected nodes in an other layer or a next stage, and for integrating certain ones of the content attributes and the context information for the at least one node.

3. The data processing device according to claim 2, wherein the at least one node of the second layer of the first stage includes determining the context information from information cascaded from a higher layer or the second stage to the at least one node, and for integrating the information at the at least one node.

4. The data processing device according to claim 1, wherein each stage is associated with a set of a hierarchical priors.

5. The data processing device according to claim 1, wherein each stage is represented by a Bayesian network.

6. The data processing device according to claim 1, wherein the content attributes are selected from the group consisting of audio, visual, keyframes, visual text, and text.

7. The data processing device according to claim 1, wherein the integration of each layer is arranged to combine certain ones of the content attributes and the context information for the at least one node at different levels of granularity.

8. The data processing device according to claim 1, wherein the integration of each layer is arranged to combine certain ones of the content attributes and the context information for the at least one node at different levels of abstraction.

9. The data processing device according to claim 7, wherein the different levels of granularity are selected from the group consisting of the program, sub-program, scene, shot, frame, object, object parts and pixel level.

10. The data processing device according to claim 8, wherein the different level of abstraction is selected from the group consisting of the pixels in an image, objects in 3-D space and transcript text character.

11. The data processing device according to claim 1, wherein the selected nodes are related to each other by directed arcs in a directed acyclic graph (DAG).

12. The data processing device according to claim 11, wherein a selected node is associated with a cpd of an attribute defining the selected node being true given the truthfulness of the attribute associated with a parent node.

13. The data processing device according to claim 1, wherein the first layer is further arranged to group certain ones of the content attributes for the each one of the first plurality of nodes.

14. The data processing device according to claim 1, wherein the nodes of each layer correspond to stochastic variables.

16. The method according to claim 15, wherein the determining step includes using the context information from information cascaded from a higher layer or stage to the at least one node, and for integrating the information at the at least one node.

17. The method according to claim 15, wherein the extracting step includes extracting audio, visual, keyframes, visual text, and text attributes.

18. The method according to claim 15, wherein the integrating step includes combining certain ones of the content attributes and the context information for the at least one node at different levels of granularity.

19. The method according to claim 15, wherein the integrating step includes combining certain ones of the content attributes and the context information for the at least one node at different levels of abstraction.

20. The method according to claim 18, wherein the different levels of granularity are selected from the group consisting of the program, sub-program, scene, shot, frame, object, object parts and pixel level.

21. The method according to claim 19, wherein the different level of abstraction are selected from the group consisting of the pixels in an image, objects in 3-D space and character.

22. The method according to claim 15, wherein the determining step includes using directed acyclic graphs (DAGs) that relate the content attributes of selected nodes in an other layer or a next stage.

24. The memory medium according to claim 23, wherein the probability generating code further includes using context information cascaded from higher layers or stages to a node, and for integrating the cascaded information at the node.

25. The memory medium according to claim 23, wherein each stage is associated with a set of priors in a hierarchical prior system.

26. The memory medium according to claim 23, wherein the stages are represented by a Bayesian network.

27. The memory medium according to claim 23, wherein the content attributes are selected from the group consisting of audio, visual, keyframes, visual text, and text.

28. The memory medium according to claim 23, wherein each layer is arranged to combine certain ones of the content attributes and the context information for a node at different levels of granularity.

29. The memory medium according to claim 23, wherein each layer is arranged to combine certain ones of the content attributes and the context information for a node at different levels of abstraction.

30. The memory medium according to claim 28, wherein the different levels of granularity are selected from the group consisting of the program, sub-program, scene, shot, frame, object, object parts and pixel level.

31. The memory medium according to claim 29 wherein the different level of abstraction are selected from the group consisting of the pixels in an image, objects in 3-D space and character.

32. The memory medium according to claim 23, wherein the selected nodes are related to each other by directed arcs in a directed acyclic graph (DAG).

33. The memory medium according to claim 32, wherein a selected node is associated with a cpd of an attribute defining the selected node being true given the truthfulness of the attribute associated with a parent node in an other layer or a next stage.

34. The memory medium according to claim 23, wherein the nodes of each layer correspond to stochastic variables.

15. A method for processing an information signal comprising the steps of:

segmenting and indexing the information signal using a probabilistic framework, said framework including at least one stage having a plurality of layers with each layer having a plurality of nodes, wherein the segmenting and indexing includes,

extracting content attributes from the information signal for each node of a first layer;

determining context information, in a second layer, using the content attributes of selected nodes in an other layer or a next stage; and

integrating certain content attributes and the context information for at lest one node in the second layer.

23. A computer-readable memory medium including code for processing an information signal, the code comprising:

framework code said framework including at least one stage having a plurality of layers with each layer having a plurality of nodes, wherein the segmenting and indexing includes,

feature extracting code to extract content attributes from the information signal for each node of a first stage;

probability generating code to determine context information, in a node of a stage, using the content attributes of selected nodes in other layers or context information of a next stage; and

integrating code to combine certain content attributes and the context information for a node.

35. An apparatus for processing an information signal, the apparatus comprising:

a memory which stores process steps; and

a processor which executes the process steps stored in the memory so as (i) to use at least one stage with a plurality of layers with at least one node in each layer, (ii) extract content attributes from the information signal for each node of a first layer, (iii) to determine context information, in a second layer, using the content attributes of selected nodes in an other layer or context information of a next stage; and

(iv) to combine certain content attributes and the context information for a node.