Method to aid object detection in images by incorporating contextual information

• US 6,578,017 B1
• Filed: 02/25/2000
• Issued: 06/10/2003
• Est. Priority Date: 02/26/1999
• Status: Expired due to Fees

• Current Assignee: Information Decision Technologies LLC
• Original Assignee: Information Decision Technologies LLC
• Inventors: Ebersole, John Franklin Jr., Ebersole, John Franklin
• Primary Examiner(s): Davis, George B.
• Application Number:
  US09/513,152
• Time in Patent Office: 1,201 Days
• Field of Search: 706/3, 706/4, 706/8, 706/9
• US Class Current: 706/3
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

The invention uses fuzzy logic and/or probability distributions to automatically calculate and display the effects of contextual information on the confidence that an object in an image is an object of interest. The goal is to assist in determining the location and type of target objects of interest in that imagery. The imagery can come from any kind of imaging sensor or can be non-sensor imagery (e.g., two-and three-dimensional maps), and can be live or archived imagery. The locations of context objects can be provided by a human or a computer. The resulting set of data, including the original imagery, the locations of context objects, any results from AOD, and predictions about target object type and location, can be combined into a display that helps a human better understand where target object appears in the imagery.

Claims

1. A method to use contextual information (location and characteristics of one or more “

context objects”

) to automatically calculate, at one or more image locations the confidence (“

target confidence”

) that any object located at such a location is an object of interest (“

target object”

) in order to enhance the ability of a human or an automated object detector (AOD) system to detect and/or evaluate the presence, type, and location of target objects in an image.

2. The method of claim 1, wherein imagery is defined as multi-dimensional sensor and non-sensor data, often associated with a two-dimensional display, intended for visual interpretation by a human or for automatic processing by an AOD.

3. The method of claim 1, wherein the “

sphere of influence”

of context objects (locations in the image where the impact of the context object on target confidence, as a function of distance, is non-zero or above some threshold) can be one-, two-, or three-dimensional.

4. The method of claim 3, wherein the sphere of influence is defined by a fuzzy logic membership function used to calculate target confidence.

5. The method of claim 3, wherein the sphere of influence is defined by a probability distribution function to calculate target confidence.

6. The method of claim 3, wherein genetic algorithms are used to optimize the sphere of influence of a context object.

7. The method of claim 1, wherein fuzzy logic and/or fuzzy set theory is used to combine context information from one or more context objects, to determine target confidence at said locations in said imagery.

8. The method of claim 1, wherein a context object'"'"'s influence on target confidence, as a function of distance from the object, is also a function of target object type.

9. The method of claim 1, wherein a context object'"'"'s influence on target confidence, as a function of distance from the object, takes into account known and potentially time varying errors in the sensors used to determine said distances.

10. The method of claim 1, wherein a context object'"'"'s influence on target confidence, as a function of distance from the object, takes into account errors incurred when incorporating the context objects into imagery.

11. The method of claim 1, which calculates target confidence at specific image locations identified by an AOD as likely to contain targets.

12. The method of claim 1, which evaluates imagery and supplements it with context object information that an AOD can use to alter the AOD'"'"'s probability of detecting targets, according to each image location'"'"'s target confidence.

13. The method of claim 1, which evaluates imagery and supplements it with context object information that an AOD can use to alter the AOD'"'"'s probability of detecting targets, according to each image location'"'"'s target confidence;

sends the imagery and information to an AOD; and

reports target confidence at the image locations where the AOD detects targets.

14. The method of claim 1, wherein said context information is stored in a database.

15. The method of claim 1, wherein said context information is encoded to include time dependencies of context object size, shape, and location, and automatically updated to account for said time dependencies.

16. The method of claim 1, wherein said context information is determined by a human from imagery from one or more sources.

17. The method of claim 1, wherein said context information is determined automatically by a computer system from imagery from one or more sources.

18. The method of claim 1, wherein said context information is entered by a human onto one or more displays of imagery.

19. The method of claim 1, wherein said context information is input by a human using a display consisting of a plurality of pre-defined menu tables.

20. The method of claim 1, wherein said context information is input by a human using a graphical user interface.

21. The method of claim 1, wherein said context information is entered by a human by specifying numerical coordinates of the context object.

22. The method of claim 1, wherein said context information can be transmitted to and/or received from a remote system.

23. The method of claim 1, wherein the human can manually change the size, shape, and/or location of a context object'"'"'s sphere of influence.

24. The method of claim 1, which displays overlays of contextual information, target confidence, and any available AOD output onto imagery, in order to enhance the ability of a human to detect and/or evaluate the presence, type, and location of target objects in an image.

25. The method of claim 24, wherein one or more images are so overlaid, and each displayed on a separate physical display.

26. The method of claim 24, wherein said overlay indicates the location, target type, and target confidence associated with one or more targets.

27. The method of claim 26, wherein said overlay identifies each target object location by overlaying a symbol at each such location.

28. The method of claim 27, wherein said symbol is a different shape for each type of target object or set of types.

29. The method of claim 27, wherein said symbol is a different shape or no symbol at all, for each of various levels of target confidence.

30. The method of claim 27, wherein said symbol is a different color for each of various levels of target confidence.

31. The method of claim 26, wherein text displayed near target objects identifies target type and/or target confidence for those objects.

32. The method of claim 24, wherein said overlay indicates the target confidence at each location in said image.

33. The method of claim 32, wherein said overlay of target confidence uses different colors, for different influences (including positive and negative) on target confidence respectively, with intensity of the color at each location proportional to influence at that location.

34. The method of claim 24, wherein a context object is used as a placeholder to which the human can return after searching an image.

35. The method of claim 24, wherein a context object of known size is used as a yardstick, allowing the human to better understand spatial relationships in the image, including determination of size and distances of context objects and targets in an image.

36. The method of claim 24, wherein a context object can be used as a reference point (spotlight/flashlight) to highlight regions of interest in the image.

37. The method of claim 24, wherein the display is automatically updated to account for any changing perspective of the image;

any manual or time-dependent changes in size, shape, and location of context information;

any resulting changes in target confidence; and

changes in any available output of the AOD.