Method and system for estimating the accuracy of inference algorithms using the self-consistency methodology

• US 6,834,120 B1
• Filed: 11/15/2000
• Issued: 12/21/2004
• Est. Priority Date: 11/15/2000
• Status: Expired

• Current Assignee: Transpacific Kodex, LLC
• Original Assignee: SRI International, Inc.
• Inventors: LeClerc, Yvan G., Luong, Quang-Tuan, Fua, Pascal
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US09/714,345
• Time in Patent Office: 1,497 Days
• Field of Search: 382/168, 382/169, 382/170, 345/596, 345/690, 345/691, 345/692, 345/693, 345/694, 345/695, 345/696, 345/697, 358/3-1-
• US Class Current: 382/170
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

The present invention provides a method for measuring the self-consistency of inference algorithms. The present invention provides a method for measuring the accuracy of an inference algorithm that does not require comparison to ground truth. Rather, the present invention pertains to a method for measuring the accuracy of an inference algorithm by comparing the outputs of the inference algorithm against each other. Essentially, the present invention looks at how well the algorithm applied to many of the different observations gives the same answer. In particular, the present invention provides a method that is not time and labor intensive and is cost effective.

Claims

1. A method for estimating an accuracy of an inference process, said method comprising the steps of:

a) collecting a plurality of observations of a scene, wherein said scene is within a class of scenes and observational conditions;

b) applying said inference process to each observation of said scene;

c) measuring a self-consistency of said inference process by comparing a plurality of hypotheses of said inference process as a function of a nature of said observation; and

d) for a plurality of hypotheses of said scene, incrementing a histogram of a function of an estimate of an attribute of a hypothesis of said scene, said hypothesis normalized by the function of said estimate, resulting in a statistical analysis of the self-consistency of said inference process.

2. The method as recited in claim 1 wherein said scene is a static scene.

3. The method as recited in claim 1 wherein step d) comprises the step of

d1) conditionalizing said histogram on a score, said score being an appraisal of the confidence of the accuracy of said estimate.

4. The method as recited in claim 3 further comprising the step of:

e) adjusting the internal parameters of said inference process based on said statistical analysis.

5. The method as recited in claim 3 further comprising the step of:

e) comparing different inference processes based on said statistical analysis.

6. The method as recited in claim 1 further comprising the step of:

e) adjusting the internal parameters of said inference process based on said statistical analysis.

7. The method as recited in claim 1 further comprising the step of:

e) comparing different inference processes based on said statistical analysis.

8. A computer-readable medium having stored thereon instructions for causing a computer to implement a process for estimating an accuracy of an inference process to perform the steps of:

a) collecting a plurality of observations of a scene, wherein said scene is within a class of scenes and observational conditions;

b) applying said inference process to each observation of said scene;

c) measuring a self-consistency of said inference process by comparing a plurality of hypotheses of said inference process as a function of a nature of said observation; and

d) for a plurality of hypotheses of said scene, incrementing a histogram of a function of an estimate of an attribute of a hypothesis of said scene, said hypothesis normalized by the function of said estimate, resulting in a statistical analysis of the self-consistency of said inference process.

9. The computer-readable medium of claim 8 wherein said scene is a static scene.

10. The computer-readable medium of claim 8 wherein said instructions therein causes a computer to perform the step of:

d1) conditionalizing said histogram on a score, said score being an appraisal of the confidence of the accuracy of said estimate.

11. The computer-readable medium of claim 10 wherein said instructions therein causes a computer to perform the step of:

e) adjusting the internal parameters of said inference process based on said statistical analysis.

12. The computer-readable medium of claim 10 wherein said instructions therein causes a computer to perform the step of:

e) comparing different inference processes based on said statistical analysis.

13. The computer-readable medium of claim 8 wherein said instructions therein causes a computer to perform the step of:

e) adjusting the internal parameters of said inference process based on said statistical analysis.

14. The computer-readable medium of claim 8 wherein said instructions therein causes a computer to perform the step of:

e) comparing different inference processes based on said statistical analysis.

15. An computer system comprising:

a bus;

a processor coupled to said bus; and

a computer-readable memory unit coupled to said bus;

said processor for performing a method for estimating an accuracy of an inference process, said method comprising the steps of;

a) collecting a plurality of observations of a scene, wherein said scene is within a class of scenes and observational conditions;

b) applying said inference process to each observation of said scene;

c) measuring a self-consistency of said inference process by comparing a plurality of hypotheses of said inference process as a function of a nature of said observation; and

d) for a plurality of hypotheses of said scene, incrementing a histogram of a function of an estimate of an attribute of a hypothesis of said scene, said hypothesis normalized by the function of said estimate, resulting in a statistical analysis of the self-consistency of said inference process.

16. The computer system of claim 15, wherein said scene is a static scene.

17. The computer system of claim 15 wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:

d1) conditionalizing said histogram on a score, said score being an appraisal of the confidence of the accuracy of said estimate.

18. The computer system of claim 17 wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:

e) adjusting the internal parameters of said inference process based on said statistical analysis.

19. The computer system of claim 17 wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:

e) comparing different inference processes based on said statistical analysis.

20. The computer system of claim 15 wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:

e) adjusting the internal parameters of said inference process based on said statistical analysis.

21. The computer system of claim 15 wherein said processor performs said method for estimating an accuracy of an inference process, further comprising the step of:

e) comparing different inference processes based on said statistical analysis.