Multi-modal sensor fusion

• US 8,891,820 B2
• Filed: 09/29/2011
• Issued: 11/18/2014
• Est. Priority Date: 09/29/2011
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Owechko, Yuri, Cheng, Shinko Y., Medasani, Swarup S., Kim, Kyungnam
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): YANG, WEI WEN
• Application Number:
  US13/249,064
• Publication Number:
  US 20130083959A1
• Time in Patent Office: 1,146 Days
• Field of Search: 382/128, 382/294, 381/28, 382/94
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/10044 Radar image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method and apparatus for processing images. A sequence of images for a scene is received from an imaging system. An object in the scene is detected using the sequence of images. A viewpoint of the imaging system is registered to a model of the scene using a region in the model of the scene in which an expected behavior of the object is expected to occur.

Claims

1. A method for processing images, the method comprising:

receiving a sequence of images for a scene from an imaging system;

detecting an object in the scene using the sequence of images, forming a detection; and

registering a viewpoint of the imaging system to a model of the scene using the detection of the object in the scene and a region in the model of the scene in which an expected behavior of the object is expected to occur, forming a registration, wherein the registering the viewpoint of the imaging system to the model of the scene comprises determining whether the detection of the object in the scene is at a location in the region in the model of the scene in which the expected behavior of the object is expected to occur.

2. The method of claim 1, wherein registering the viewpoint of the imaging system to the model of the scene using the detection of the object in the scene and the region in the model of the scene in which the expected behavior of the object is expected to occur comprises:

estimating a state of the imaging system based on a probability of the state of the imaging system given an identification of the object detected in the scene using the sequence of images of the scene and involved in forming the detection used in the registering the viewpoint of the imaging system to the model of the scene.

3. The method of claim 2, wherein estimating the state of the imaging system based on the probability of the state of the imaging system given the identification of the object in the sequence of images of the scene comprises:

estimating the state of the imaging system using a condensation algorithm comprising a first function and a second function, wherein the first function takes into account noise of the imaging system and the second function determines whether the detection of the object in the scene is at the location in the region in the model of the scene in which the expected behavior of the object is expected to occur, and wherein the first function and the second function form a data likelihood function.

4. The method of claim 2, further comprising:

tracking a movement of the object in the scene over time using the sequence of images, forming a track, wherein the track is generated when the detection of the object used in the registering the viewpoint of the imaging system to the model of the scene has been made for at least a selected period of time; and

wherein the registering the viewpoint of the imaging system to the model of the scene uses the detection of the object in the scene, the track of the movement of the object generated when the detection of the object has been made for at least the selected period of time, and the region in the model of the scene in which the expected behavior of the object is expected to occur.

5. The method of claim 4, further comprising:

identifying the object involved in the detection used in the registering the viewpoint of the imaging system to the model of the scene and the track used in the registering the viewpoint of the imaging system to the model of the scene, forming the identification;

wherein the object involved in the detection of the object in the scene, the track of the movement of the object, and the region in the model of the scene in which the expected behavior of the object is expected to occur all used in the registering the viewpoint of the imaging system to the model of the scene comprises at least one of a vehicle and a pedestrian;

wherein determining whether the detection of the object in the scene is at the location in the region in the model of the scene in which the expected behavior of the object is expected to occur comprises at least one of;

determining whether the detection of the vehicle in the scene is at the location in the region in the model of the scene in which the vehicle is expected to at least one of travel on a road, be parked in a parking lot, turn at an intersection, and change lanes in the road; and

determining whether the detection of the pedestrian in the scene is at the location in the region in the model of the scene in which the pedestrian is expected to walk along sidewalks in the scene; and

wherein the registering the viewpoint of the imaging system to the model of the scene uses the region in the model of the scene in which at least one of;

the vehicle is expected to at least one of travel on the road, be parked in the parking lot, turn at the intersection, and change the lanes in the road; and

the pedestrian is expected to walk along the sidewalks in the scene.

6. The method of claim 5, wherein the imaging system is a first imaging system, the sequence of images is a first sequence of images, the object is a first object, the viewpoint is a first viewpoint, the detection is a first detection, the registration is a first registration, the track is a first track, and the movement is a first movement, and further comprising:

receiving a second sequence of images for the scene from a second imaging system;

detecting a second object in the scene using the second sequence of images, forming a second detection, wherein the second object is a same type as the first object; and

registering a second viewpoint of the second imaging system to the model of the scene using the second detection of the second object in the scene and the region in the scene in which the expected behavior for the second object is expected to occur, forming a second registration.

7. The method of claim 6 further comprising:

projecting the first track for the first object onto the model of the scene to form a first projection based on the first registration of the first viewpoint of the first imaging system to the model of the scene; and

projecting a second track for the first object onto the model of the scene to form a second projection based on the second registration of the second viewpoint of the second imaging system to the model of the scene.

8. The method of claim 7, wherein the first projection and the second projection form a projection pair and further comprising:

determining whether a distance between the first projection and the second projection with respect to a coordinate system for the model is less than a selected threshold; and

identifying a centroid of the first projection and the second projection as a final projection when the distance between the first projection and the second projection with respect to the coordinate system for the model is less than the selected threshold, wherein the final projection tracks the first movement of the first object with respect to the model of the scene.

9. The method of claim 8, wherein:

the first imaging system and the second imaging system are different, and wherein the final projection is generated using multi-modal fusion by a fusion module in communication with the first imaging system and the second imaging system; and

further comprising;

using the fusion to improve the tracking of the movement of the first object in the scene over time and of another movement of the second object in the scene over time.

10. The method of claim 9, wherein the first imaging system comprises an electro-optical imaging system and the second imaging system comprises an infrared imaging system.

11. The method of claim 9 further comprising:

back-projecting the final projection into at least one of the first sequence of images and the second sequence of images, wherein a back-projection is formed; and

using the back-projection to improve detection of a feature in the at least one of the first sequence of images and the second sequence of images, wherein improvement is relative to detection of the feature using only the first sequence of images or the second sequence of images.

12. The method of claim 9 further comprising:

back-projecting the final projection into at least one of the first sequence of images and the second sequence of images, wherein a back-projection is formed.

13. The method of claim 12, wherein back-projecting is performed using an inverse homography algorithm.

14. The method of claim 12 further comprising:

using the back-projection to improve detection of a feature in the at least one of the first sequence of images and the second sequence of images, wherein improvement is relative to detection of the feature using only the first sequence of images or the second sequence of images.

15. An apparatus comprising:

a computer system configured to receive a sequence of images for a scene from an imaging system;

detect an object in the scene using the sequence of images, forming a detection; and

register a viewpoint of the imaging system to a model of the scene using the detection of the object in the scene and a region in the model of the scene in which an expected behavior of the object is expected to occur, wherein in being configured to register the viewpoint of the imaging system to the model of the scene, the computer system is configured to determine whether the detection of the object in the scene is at a location in the region in the model of the scene in which the expected behavior of the object is expected to occur.

16. The apparatus of claim 15, wherein in being configured to register the viewpoint of the imaging system to the model of the scene using the detection of the object in the scene and the region in the model of the scene in which the expected behavior of the object is expected to occur, the computer system is configured to estimate a state of the imaging system based on a probability of the state of the imaging system given an identification of the object detected in the scene using the sequence of images of the scene and involved in forming the detection used in the registering the viewpoint of the imaging system to the model of the scene.

17. The apparatus of claim 16, wherein:

in being configured to estimate the state of the imaging system based on the probability of the state of the imaging system given the identification of the object in the sequence of images of the scene, the computer system is configured to estimate the state of the imaging system using a condensation algorithm comprising a first function and a second function, wherein the first function takes into account noise of the imaging system and the second function determines whether the detection of the object in the scene is at the location in the region in the model of the scene in which the expected behavior of the object is expected to occur, and wherein the first function and the second function form a data likelihood function.

18. The apparatus of claim 15, wherein:

the computer system is further configured to track a movement of the object in the scene over time using the sequence of images, forming a track;

the computer system is configured to generate the track when the detection of the object used to register the viewpoint of the imaging system to the model of the scene has been made for at least a selected period of time; and

wherein in being configured to register the viewpoint of the imaging system to the model of the scene, the computer is configured to use the detection of the object in the scene, the track of the movement of the object generated when the detection of the object has been made for at least the selected period of time, and the region in the model of the scene in which the expected behavior of the object is expected to occur.

19. The apparatus of claim 18, wherein the imaging system is a first imaging system, the sequence of images is a first sequence of images, the object is a first object, the viewpoint is a first viewpoint, and the detection is a first detection, and wherein the computer system is further configured to receive a second sequence of images for the scene from a second imaging system;

and detect a second object in the scene using the second sequence of images, forming a second detection, wherein the second object is a same type as the first object;

measure a first distance between the first object and the second object;

identify the first object and the second object; and

register a second viewpoint of the second imaging system to the model of the scene using the second detection and the region in the scene in which the expected behavior for the second object is expected to occur.

20. The apparatus of claim 19, wherein the computer system is further configured to project a first track for the first object onto the model of the scene to form a first projection based on a first registration of the first viewpoint of the first imaging system to the model of the scene;

and project a second track for the first object onto the model of the scene to form a second projection based on a second registration of the second viewpoint of the second imaging system to the model of the scene, wherein the first projection and the second projection form a projection pair and wherein the computer system is configured to determine whether a second distance between the first projection and the second projection with respect to a coordinate system for the model is less than a selected threshold; and

identify a centroid of the first projection and the second projection as a final projection when the distance between the first projection and the second projection with respect to the coordinate system for the model is less than the selected threshold, wherein the final projection tracks the movement of the first object with respect to the model of the scene.