Method and system for detection and tracking employing multi-view multi-spectral imaging

• US 8,761,445 B2
• Filed: 03/10/2011
• Issued: 06/24/2014
• Est. Priority Date: 02/21/2010
• Status: Active Grant

• Current Assignee: Elbit Systems Limited
• Original Assignee: Elbit Systems Limited
• Inventors: Shamir, Hanan, Berger, Liat, Tell, Eliyahu, Efrat, Ilan
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US13/580,247
• Publication Number:
  US 20130034266A1
• Time in Patent Office: 1,202 Days
• Field of Search: 382/100, 382/103, 382106-107, 382116-118, 382/155, 382/162, 382/168, 382/173, 382/181, 382189-199, 382/209, 382/232, 382/254, 382/274, 382/276, 382/291, 382/305, 382/312, 348/149, 348/143, 348/158, 342/52, 340/539.26
• US Class Current: 382/103
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 2207/10036 Multispectral image; Hyperspectral image
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/292 Multi-camera tracking

Abstract

Multi view multi spectral detection and tracking system comprising at lease one imager, at least one of the at least one imager being a multi spectral imager, the at least one imager acquiring at least two detection sequences, and at least two tracking sequences, each sequence including at least one image, each acquired image being associated with respective image attributes, an object detector, coupled with the at least one imager, detecting objects of interest in the scene, according to the detection sequence of images and the respective image attributes, an object tracker coupled with the object detector, the object tracker tracking the objects of interest in the scene and determining dynamic spatial characteristics and dynamic spectral characteristics for each object of interest according to the tracking sequences of images and the respective image attributes and an object classifier, coupled with the object tracker, classifying the objects of interest according to the dynamic spatial characteristics and the dynamic spectral characteristics.

Claims

1. Multi-view multi-spectral detection and tracking system comprising:

at least one imager, at least one of said at least one imager being a multi-spectral imager, said multi-spectral imager acquiring images in a plurality of spectral bands, at least one of said spectral bands extending beyond the visible spectral band, said at least one imager acquiring at least two detection sequences, and at least two tracking sequences, each sequence including at least one image, each acquired image being associated with respective image attributes;

an object detector, coupled with said at least one imager, detecting objects of interest in said scene, according to said detection sequence of images and said respective image attributes;

an object tracker coupled with said object detector, said object tracker tracking said objects of interest in said scene and determining dynamic spatial characteristics and dynamic spectral characteristics for each object of interest according to said tracking sequences of images and said respective image attributes, said image attributes including at least the spectrum of the image;

an object classifier, coupled with said object tracker, classifying said objects of interest according to said dynamic spatial characteristics and said dynamic spectral characteristics; and

wherein said object detector comprises a pre-processor couples with a database, said pre-processor at least including;

an image registration processor, registering the images acquired from different positions in said scene;

an image segmentation processor, segmenting the acquired images into segments, each segment is associated with at least a portion of an object in said scene; and

a segments of interest identifier, couples with said image registration processor and with said image segmentation processor, said segments of interest identifier identifies segments of interest in the scene according to the segmented images and a background model of said scene.

2. The system according to claim 1, wherein said image attributes further include at least one of the group consisting of:

a respective time stamp; and

respective imager characterizing parameters corresponding to the imager that acquired the image.

3. The system according to claim 2, wherein said imager characterizing parameters are selected from the group consisting of:

the Pan;

the Tilt;

the Zoom; and

the relative position of the imager.

4. The system according to claim 2, wherein said spectral band is selected from the group consisting of:

visible;

Near InfraRed;

Short Wave InfraRed;

Long Wave InfraRed;

Mid wave infrared;

Ultraviolet; and

Millimeter wave.

5. The system according to claim 1, wherein said system includes at least two imagers, at least one of said at least two imagers being a multi-spectral imager.

6. The system according to claim 1, wherein said object detector further comprises:

an initial spatial characteristics determinator, coupled with said pre-processor, determining initial spatial characteristics of objects corresponding to said segments of interest, according to said detection sequences of images and the respective image attributes thereof;

an initial spectral characteristics determinator, coupled with said pre-processor determining initial spectral characteristics of the objects corresponding to said segments of interest according to said detection sequences of images and the respective image attributes thereof; and

object of interest designator, coupled with said spatial characteristics determinator and said with spectral characteristics determinator, said object of interest designator identifying objects in the scene corresponding to said segments of interest, and selecting objects of interest from said objects in the scene according to said initial spatial characteristics and said initial spectral characteristics of said segments of interest,

wherein said initial spectral characteristics include the spectral signature of said objects corresponding to the segments of interest,

wherein said initial spatial characteristics at least includes 3D information related to the objects corresponding to the segments of interest,

wherein said 3D information includes the size of said objects corresponding to said segments of interest, the trajectory of said objects corresponding to said segments of interest in a reference coordinate system associated with said scene.

7. The system according to claim 6, wherein said pre-processor further filters and stabilizes the acquired images.

8. The system according to claim 6, wherein said segments of interest identifier identifies segments, which appear in an image acquired from one position but do not appear in a background image which exhibits similar image attributes, as segments of interest;

and

wherein said segments of interest identifier identifies segments of interest in different images, exhibiting different respective image attributes, as corresponding to at least a portion of the same object in said scene.

9. The system according to claim 6, wherein said object of interest designator identifies an object in said scene according to at least one of the following criteria:

an object, corresponding to at least one segment of interest, which exhibit a 3D trajectory which is substantially different than the 3D trajectory of background objects, is identified as an object in said scene

objects, corresponding to a group of segments of interest, said objects exhibiting substantially the same 3D trajectory, are identified as a single object is said scene;

a group of segments of interest exhibiting substantially the same intra-image trajectory are identified as a single object in said scene; and

a segment of interest which appear in at least a selected portion of the spectral bands associated with the acquired images, as objects of in the scene.

10. The system according to claim 6, where said 3D information is may further include the rigidity of said objects corresponding to said segments of interest.

11. The system according to claim 6, wherein said initial spatial characteristics further include the intra-image size, intra-image location and the intra-image trajectory of the segments of interest.

12. The system according to claim 6, wherein said initial spectral characteristics further include the temperature and the heat capacity of said objects corresponding to the segments of interest.

13. The system according to claim 1, wherein said dynamic spatial characteristics include the size, the rigidity and the trajectory of the selected objects in the scene determined according to a tracking sequence,

wherein said dynamic spectral characteristics include the spectral signature of the object determined according to said tracking sequence, and

wherein said tracking sequence is a portion of a tracking and classification sequence.

14. The system according to claim 1, wherein said object tracker superimposes a representation of the tracked objects on a representation of the scene.

15. The system according to claim 1, wherein said classifier comprises:

a dynamic spatial characteristics analyzer analyzing said dynamic spatial characteristic associated with the objects of interest;

a dynamic spectral characteristics analyzer analyzing said dynamic spectral characteristic associated with the objects of interest;

an object model determinator, coupled with said dynamic spatial characteristics analyzer and with said dynamic spectral characteristics analyzer, determining an object model for each object of interest; and

a classifier, coupled with said object model and with a database classifying said object according to the respective object models associated with the objects of interest and object models of known objects stored in a database.

16. The system according to claim 15, wherein said dynamic spatial characteristics analyzer statistically analyzes the motion of the object of interest.

17. The system according to claim 15, wherein said dynamic spectral characteristics analyzer performs at least one of following:

determines the spectral signature of the objects of interest in one spectral band relative to the spectral signature in other spectral bands;

statistical analysis of the spectral signature of the objects of interest during the tracking period.

18. The system according to claim 15, wherein said object model is a table of object features, and

wherein said features at least include the trajectory and the spectral signature.

19. The system according to claim 18, wherein said features are further selected from the group consisting of:

average velocity;

standard deviation of the velocity;

size; and

rigidity.

20. The system according to claim 15, wherein said object classifier is further coupled with said object detector and performs initial classification of said objects of interest the according to the initial spatial and spectral characteristics and provides said initial classification to said object tracker.

21. The system according to claim 15, wherein said object classifier updates the stored object models according to the determined object models.

22. The system according to claim 1, further comprising a background model determinator, said background model determinator determines a background model of said scene by identifying background objects in said scene and determined the respective spatial and spectral characteristics of said background objects according to a background determination sequence of images,

wherein said background model determinator is further coupled with a database.

23. The system according to claim 22, further comprising a geographical information system, providing a terrain model of said scene, said background model determinator associating between background objects and said terrain model, and

wherein said object model determinator refines said terrain model according to trajectory of an object moving on the ground.

24. A multi-view multi-spectral detection and tracking method comprising the procedures of:

acquiring at least two detection sequences of images of the scene, each sequence including at least one image, each image being associated with respective image attributes, at least a portion of said images being acquired in a plurality of spectral bands, at least one of said spectral bands extending beyond the visible spectral band;

detecting objects of interest from objects in the scene and determining initial object characteristics according to said detection sequence of images and said respective image attributes;

acquiring at least two tracking sequences of images of the scene, each sequence including at least one image, each image being associated with respective image attributes;

tracking the objects of interest in the scene according to the sequences of images and determining dynamic spatial and spectral characteristics of each said objects of interest according to tracking sequences of images and said respective image attributes, said image attributes including at least the spectrum of the image,

wherein said objects in said scene are identified according to at least one of the following criteria;

an object, corresponding to at least one segment of interest, which exhibit a 3D trajectory which is substantially different that the 3D trajectory of background objects, is identified as an object in said scene; and

objects, corresponding to a group of segments of interest, said objects exhibiting substantially to same 3D trajectory, are identified as a single object in said scene; and

classifying each of the objects of interest according to a respective object classification model respective of said each object of interest.

25. The method according to claim 24, wherein said image attributes further include at least one of the group consisting of:

a respective time stamp; and

respective imager characterizing parameters corresponding to the imager that acquired the image.

26. The method according to claim 25, wherein said imager characterizing parameters are selected from the group consisting of:

the Pan;

the Tilt;

the Zoom; and

the relative position of the imager.

27. The method according to claim 25, wherein said spectral band is selected from the group consisting of:

visible;

Near InfraRed;

Short Wave InfraRed;

Long Wave InfraRed;

Mid wave infrared;

Ultraviolet; and

Millimeter wave.

28. The method according to claim 24, wherein said procedure of detecting objects of interest includes the sub-procedures of:

pre-processing the acquired images and identifying segments of interest;

determining initial spatial characteristics of the segments of interest;

determining initial spectral characteristics of the segments of interest;

identifying said objects in the scene according to the spatial and spectral characteristics of the identified segments; and

selecting objects of interest from identified objects in the scene according to the spatial and spectral characteristics thereof.

29. The method according to claim 28, wherein said procedure of pre-processing at least includes registering and segmenting the acquired images,

wherein said procedure of pre-processing further includes filtering and stabilizing said acquired images,

wherein said procedure of pre-processing includes identifying segments of interest according to segments which appear in an image acquired from one position in but do not appear in a background image which exhibits similar image attributes, and

wherein said segments of interest in different images, exhibiting different respective image attributes, are identified as corresponding to at least a portion of the same object in said scene.

30. The method according to claim 29, wherein said objects in said scene are alternatively identified according to at least one of the following criteria:

a group of segments of interest exhibiting substantially the same intra-image trajectory are identified as a single object in said scene; and

a segment of interest which appear in at least a selected portion of the spectral bands associated with the acquired images, as objects of in the scene.

31. The method according to claim 28, wherein said initial spatial characteristics at least includes 3D information related to the objects corresponding to the segments of interest, and

wherein said 3D information includes the size of said objects corresponding to said segments of interest and trajectory of said objects corresponding to said segments of interest in a reference coordinate system associated with said scene.

32. The method according to claim 28, wherein said initial spectral characteristics include the spectral signature of said objects corresponding to the segments of interest.

33. The method according to claim 32, wherein said initial spectral characteristics further the temperature and the heat capacity of said objects corresponding to the segments of interest.

34. The method according to claim 24, wherein said 3D information is may further include the rigidity of said objects corresponding to said segments of interest.

35. The method according to claim 24, wherein said initial spatial characteristics further include the intra-image size, intra-image location and intra-image trajectory of the segments of interest.

36. The method according to claim 24, wherein said detection sequence is a portion of a tracking and classification sequence.

37. The method according to claim 24, wherein said dynamic spatial characteristics include the size, the rigidity and the trajectory of the selected objects in the scene determined according to a tracking sequence,

wherein said dynamic spectral characteristics include the spectral signature of the object determined according to said tracking sequence, and

wherein said tracking sequence is a portion of a tracking and classification sequence.

38. The method according to claim 24, wherein a representation of the tracked objects is superimposed on a representation of the scene.

39. The method according to claim 24, wherein said procedure of classifying said objects of interest includes the sub-procedures of:

analyzing the dynamic spatial characteristics of the objects of interest;

analyzing the dynamic spectral characteristics of the objects of interest;

determining a classification model for each object of interest; and

classifying the objects of interest by comparing the classification model of each object of interest to stored object models.

40. The method according to claim 39, further includes the sub-procedure of updating the stored object models according to the determined object models.

41. The method according to claim 39, wherein said analyzing said dynamic spatial characteristics includes statistically analyzing the motion of the object of interest.

42. The method according to claim 39, wherein analyzing said dynamic spectral characteristics includes performing any combination of following:

determining the spectral signature of the objects of interest in one spectral band relative to the other spectral bands;

statistically analyzing of the spectral signature of the objects of interest during the tracking period;

spatially analyzing of the spectral signature of the object; and

temporally analyzing the dynamic spectral characteristics.

43. The method according to claim 39, wherein said object model is a table of object features.

44. The method according to claim 43, wherein said features at least include the following:

average velocity;

standard deviation of the velocity;

size;

rigidity;

spectral signature; and

heat capacity.

45. The method according to claim 39, wherein said object classifier updates the stored object models according to the determined object models.

46. The method according to claim 24, further including the procedure of initially classifying said object of interest according to the initial spatial characteristics and said initial spectral characteristics prior to said procedure of tracking said objects of interest.

47. The method according to claim 24, further including the preliminary procedure of determining a background model of said scene.

48. The method according to claim 47, wherein said preliminary procedure of determining a background model of said scene includes the sub-procedures of:

acquiring at least two background detection sequences of images of the scene each image is associated with respective image attributes;

detecting background objects in the scene and determining initial object characteristics;

acquiring at least two background tracking sequences of images of the scene, each image is associated with respective image attributes;

tracking the objects in the scene according to the background tracking sequences of images and determining dynamic spatial and spectral characteristics of each object;

classifying each of the objects of interest according to a respective object classification model, the object model being determined according to the respective dynamic spatial and spectral characteristics; and

determining a background model of the scene according to the detected objects and the dynamic spatial and spectral characteristics thereof.

49. The method according to claim 48, wherein said object model includes a table of object features determined according to the background object spatial and spatial characteristics.

50. The method according to claim 48, wherein said determining a background model of said scene further includes associating between background objects and a terrain model, and

wherein said terrain model is refined according to trajectory of an object moving on the ground.