Method for inferring scenes from test images and training data using probability propagation in a markov network

• US 6,496,184 B1
• Filed: 06/18/1999
• Issued: 12/17/2002
• Est. Priority Date: 11/30/1998
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Moghaddam, Baback, Freeman, William T., Pasztor, Egon C.
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): MANCUSO, HUEDUNG XUAN CAO
• Application Number:
  US09/335,943
• Time in Patent Office: 1,278 Days
• Field of Search: 345/419, 345/418, 345/473, 345/474, 345/433, 348/700, 348/699, 348/416.1, 382/236, 382/276, 382/284, 382/154, 382/107, 382/173
• US Class Current: 345/419
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/46 using random fields
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A method infers a scene from a test image. During a training phase, a plurality of images and corresponding scenes are acquired. Each of the images and corresponding scenes are partitioned respectively into a plurality of image patches and scene patches. Each image patch is represented as an image vector, and each scene patch is represented as a scene vector. The image vectors and scene vectors are modeled as a network. During an inference phase, the test image is acquired. The test image is partitioned into a plurality of test image patches. Each test image patch is represented as a test image vector. Candidate scene vectors corresponding to the test image vectors are located in the network. Compatibility matrices for the candidate scene vectors are determined, and probabilities of the compatibility matrices are propagated in the network until convergence to infer the scene from the test image.

Claims

1. A method for inferring a scene from a test image, comprising the steps of:

acquiring a plurality of images and corresponding scenes;

representing each image as a set of image vectors and each scene as a set of scene vectors;

modeling the image vectors and scene vectors as a network;

acquiring the test image;

representing the test image as a test set of vectors;

identifying candidate scene vectors corresponding to the test image vectors in the network;

determining compatibility matrices for the candidate scene vectors; and

propagating probabilities of the candidate scene vectors to infer the scene from the test image.

2. The method of claim 1 wherein the images, the scenes, and the test image are visual data.

3. The method of claim 1 wherein the images, the scenes, and the test image are audio data.

4. The method of claim 1 wherein the images and the scenes, and the test image are synthetically generated.

5. The method of claim 1 wherein the images and the scene are measured.

6. The method of claim 1 wherein there is a one-to-one correspondence between the images and the scenes.

7. The method of claim 1 wherein each of the image, scene, and test image is respectively partitioned into a plurality of image, scene, and test image patches.

8. The method of claim 1 wherein the vectors are low-dimensional vectors determined by principal component analysis.

9. The method of claim 8 wherein the dimensionality of the vectors is less than ten.

10. The method of claim 1 wherein the network is a Markov network having nodes and edges, and wherein the nodes represent the vectors and the edges represent statistical dependencies between the vectors.

11. The method of claim 1 wherein the network is multi-resolution.

12. The method of claim 1 further comprising the steps of determining local likelihood functions for the candidate scene vectors, and determining compatibility functions to indicate how neighboring vectors are probabilistically related to each other.

13. The method of claim 10 wherein a sum-product rule determines a marginalized mean of a posterior probability for each node of the network.

14. The method of claim 10 wherein a max-product determines a value of a variable which maximizes the posterior probability for each node.

15. The method of claim 12 wherein a loss function is used to modify the local likelihood functions.

16. The method of claim 1 wherein the test image is low-resolution, and the inferred scene is high-resolution.

17. The method of claim 1 wherein the test image is a picture, and the inferred scene is a set of intrinsic images having reflectance values and surface heights, and a description of overall lighting.

18. The method of claim 1 wherein the test image is a video of a moving body, and the inferred scene is a set of three-dimensional motion parameters of the moving body.

19. The method of claim 1 wherein the test image is a first video of a moving object, and the inferred scene is a second video of a second moving object moving according to the first moving object.

20. The method of claim 1 wherein the image vectors and scene vectors comprise training data arranged as a binary tree.

21. The method of claim 20 wherein the candidate scene vectors are precomputed at index nodes in the binary tree.

22. The method of claim 1 wherein the images and scenes are organized as a plurality of classes, each image and scene having a corresponding class label c.

23. The method of claim 22 wherein a class compatibility function ψ

(ci, cj) indicates a likelihood that a scene from class c₁ borders a scene from class c_j such that ψ

(c_i, c_i) is 1, and ψ

(c_i, c_j) is less than one, for i≠

j.

24. The method of claim 23 wherein a scene compatibility function is:

25. The method of claim 1 wherein the propagating probabilities of the candidate scene vectors are propagated until a termination condition is reached.