Multi-Modal Sensor Fusion

• US 20130083959A1
• Filed: 09/29/2011
• Published: 04/04/2013
• Est. Priority Date: 09/29/2011
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Owechko, Yuri, Kim, Kyungnam, Medasani, Swarup S., Cheng, Shinko Y.
• Granted Patent: US 8,891,820 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/10044 Radar image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method and apparatus for processing images. A sequence of images for a scene is received from an imaging system. An object in the scene is detected using the sequence of images. A viewpoint of the imaging system is registered to a model of the scene using a region in the model of the scene in which an expected behavior of the object is expected to occur.

Claims

1. A method for processing images, the method comprising:

receiving a sequence of images for a scene from an imaging system;

detecting an object in the scene using the sequence of images; and

registering a viewpoint of the imaging system to a model of the scene using a region in the model of the scene in which an expected behavior of the object is expected to occur.

2. The method of claim 1 further comprising:

projecting a track for the object onto the model of the scene based on a registration of the viewpoint of the imaging system to the model.

3. The method of claim 1, wherein registering the viewpoint of the imaging system to the model of the scene using the region in the model of the scene in which the expected behavior of the object is expected to occur comprises:

estimating a state of the imaging system based on a probability of the state of the imaging system given an identification of the object in the sequence of images of the scene.

4. The method of claim 3, wherein the state of the imaging system comprises at least one of a position and an orientation of the imaging system with respect to a coordinate system for the model of the scene.

5. The method of claim 3, wherein estimating the state of the imaging system based on the probability of the state of the imaging system given the identification of the object in the sequence of images of the scene comprises:

estimating the state of the imaging system based on the probability of the state of the imaging system given the identification of the object in the sequence of images of the scene using a particle filter.

6. The method of claim 1, wherein the expected behavior is expected to occur in the region in the model of the scene for any object of a same type as the object.

7. The method of claim 6, wherein the imaging system is a first imaging system, the sequence of images is a first sequence of images, the object is a first object, and the viewpoint is a first viewpoint, and further comprising:

receiving a second sequence of images for a scene from a second imaging system;

detecting a second object in the scene using the second sequence of images, wherein the second object is the same type as the first object; and

registering a second viewpoint of the second imaging system to the model of the scene using the region in the scene in which the expected behavior for the second object is expected to occur.

8. The method of claim 7 further comprising:

projecting a first track for the first object onto the model of the scene to form a first projection based on a registration of the first viewpoint of the first imaging system to the model of the scene; and

projecting a second track for the first object onto the model of the scene to form a second projection based on a registration of the second viewpoint of the second imaging system to the model of the scene.

9. The method of claim 8, wherein the first projection and the second projection form a projection pair and further comprising:

determining whether a distance between the first projection and the second projection with respect to a coordinate system for the model is less than a selected threshold; and

identifying a centroid of the first projection and the second projection as a final projection when the distance between the first projection and the second projection with respect to the coordinate system for the model is less than the selected threshold.

10. The method of claim 3, wherein estimating the state of the imaging system based on the probability of the state of the imaging system given the identification of the object in the sequence of images of the scene comprises:

estimating the state of the imaging system using a condensation algorithm comprising a first function and a second function, wherein the first function takes into account noise of the imaging system and the second function determines whether a detection of the object in the scene is at a location in the region in the model of the scene in which the expected behavior of the object is expected to occur, and wherein the first function and the second function form a data likelihood function.

11. The method of claim 1, wherein the imaging system is selected from one of an electro-optical imaging system, an infrared imaging system, a radar imaging system, a thermal imaging system, and an ultrasound imaging system.

12. An apparatus comprising:

a computer system configured to receive a sequence of images for a scene from an imaging system;

detect an object in the scene using the sequence of images; and

register a viewpoint of the imaging system to a model of the scene using a region in the model of the scene in which an expected behavior of the object is expected to occur.

13. The apparatus of claim 12, wherein the computer system is further configured to project a track for the object onto the model of the scene based on a registration of the viewpoint of the imaging system to the model.

14. The apparatus of claim 12, wherein in being configured to register the viewpoint of the imaging system to the model of the scene using the region in the model of the scene in which the expected behavior of the object is expected to occur, the computer system is configured to estimate a state of the imaging system based on a probability of the state of the imaging system given an identification of the object in the sequence of images of the scene given.

15. The apparatus of claim 14, wherein the state of the imaging system comprises at least one of a position and an orientation of the imaging system with respect to a coordinate system for the model of the scene.

16. The apparatus of claim 14, wherein in being configured to estimate the state of the imaging system based on the probability of the state of the imaging system given the identification of the object in the sequence of images of the scene, the computer system is configured to estimate the state of the imaging system based on the probability of the state of the imaging system given the identification of the object in the sequence of images of the scene using a particle filter.

17. The apparatus of claim 12, wherein the expected behavior is expected to occur in the region in the model of the scene for any object of a same type as the object.

18. The apparatus of claim 12, wherein the imaging system is a first imaging system, the sequence of images is a first sequence of images, the object is a first object, and the viewpoint is a first viewpoint, and wherein the computer system is further configured to receive a second sequence of images for the scene from a second imaging system;

and detect a second object in the scene using the second sequence of images, wherein the second object is a same type as the first object; and

register a second viewpoint of the second imaging system to the model of the scene using the region in the scene in which the expected behavior for the second object is expected to occur.

19. The apparatus of claim 18, wherein the computer system is further configured to project a first track for the first object onto the model of the scene to form a first projection based on a registration of the first viewpoint of the first imaging system to the model of the scene;

and project a second track for the first object onto the model of the scene to form a second projection based on a registration of the second viewpoint of the second imaging system to the model of the scene.

20. The apparatus of claim 19, wherein the first projection and the second projection form a projection pair and wherein the computer system is configured to determine whether a distance between the first projection and the second projection with respect to a coordinate system for the model is less than a selected threshold;

and identify a centroid of the first projection and the second projection as a final projection when the distance between the first projection and the second projection with respect to the coordinate system for the model is less than the selected threshold.