Method and apparatus for stereo, multi-camera tracking and RF and video track fusion

• US 7,929,017 B2
• Filed: 07/28/2005
• Issued: 04/19/2011
• Est. Priority Date: 07/28/2004
• Status: Expired due to Fees

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Hanna, Keith, Zhao, Tao, Aggarwal, Manoj, Kalokitis, David, Patterson, David R., Kumar, Rakesh, Sawhney, Harpreet
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US11/192,682
• Publication Number:
  US 20060028552A1
• Time in Patent Office: 2,091 Days
• Field of Search: None
• US Class Current: 348/169
• CPC Class Codes: G01S 13/767 Responders; Transponders teaching or practice apparatus for gun-aiming or gun-laying using reflecting targets or active targets F41G3/26
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G01S 5/0294 Trajectory determination or predictive filtering, e.g. target tracking or Kalman filtering
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G07C 9/257 electronically G07C9/26 takes precedence
• CPC Class Codes: G07C 9/28 the pass enabling tracking or indicating presence
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19673 Addition of time stamp, i.e. time metadata, to video stream
• CPC Class Codes: G08B 13/19691 Signalling events for better perception by user, e.g. indicating alarms by making display brighter, adding text, creating a sound
• CPC Class Codes: G08B 13/19697 Arrangements wherein non-video detectors generate an alarm themselves
• CPC Class Codes: G08B 13/2462 Asset location systems combined with EAS inventory, tracking, logistic G06Q10/00; entrance control systems G07C9/00
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/368 : for two or more viewers
• CPC Class Codes: H04N 2013/0085 : Motion estimation from stereoscopic image signals

Abstract

A unified approach, a fusion technique, a space-time constraint, a methodology, and system architecture are provided. The unified approach is to fuse the outputs of monocular and stereo video trackers, RFID and localization systems and biometric identification systems. The fusion technique is provided that is based on the transformation of the sensory information from heterogeneous sources into a common coordinate system with rigorous uncertainties analysis to account for various sensor noises and ambiguities. The space-time constraint is used to fuse different sensor using the location and velocity information. Advantages include the ability to continuously track multiple humans with their identities in a large area. The methodology is general so that other sensors can be incorporated into the system. The system architecture is provided for the underlying real-time processing of the sensors.

Claims

1. A method for tracking an object, comprising:

receiving, by a plurality of camera trackers, camera video streams from a plurality of cameras;

providing, by the camera trackers, camera video tracks;

providing, by at least one radio frequency identification (RFID) and localization system, RF tracks and a plurality of RF identities;

selectively fusing, by a fusion module, at least one of two camera video tracks and a camera video track to an RF track to generate at least one fused track, wherein the fusing is performed by matching tracks using association of space-time and appearance cues, and wherein the appearance cues include shape and appearance;

associating, by the fusion module, the at least one fused track with the RF identities;

displaying, by a visualization display, the at least one fused track and the RF identities; and

providing time-synchronized access to the camera video tracks, by the visualization display.

2. The method of claim 1, wherein the camera video tracks are overlapping.

3. The method of claim 1, wherein the camera video tracks are non-overlapping.

4. The method of claim 1, wherein the fusing includes matching camera video tracks with new objects.

5. The method of claim 1, wherein the camera video tracks are from monocular cameras, stereo cameras, or any combination thereof.

6. The method of claim 1, wherein continuous coverage is provided by propagating the RF identities and by performing handoffs between any two cameras.

7. The method of claim 1, wherein the fusion module selectively associates a plurality of identities from at least one biometric system with the at least one fused track.

8. A system for tracking an object, comprising:

a plurality of camera trackers to receive camera video streams from a plurality of cameras and to provide camera video tracks;

at least one radio frequency identification (RFID) and localization system to provide RF tracks and a plurality of RF identities;

a fusion module to selectively fuse at least one of two camera video tracks and a camera video track to an RF track to generate at least one fused track, wherein the fusing is performed by matching tracks using association of space-time and appearance cues, and wherein the appearance cues include shape and appearance; and

a visualization display to display the at least one fused track and the RF identities and to provide access to the camera video tracks.

9. The system of claim 8, wherein the camera video tracks are overlapping.

10. The system of claim 8, wherein the camera video tracks are non-overlapping.

11. The system of claim 8, wherein the fusing includes matching camera video tracks with new objects.

12. The system of claim 8, wherein the camera video tracks are from monocular cameras, stereo cameras, or any combination thereof.

13. The system of claim 8, wherein continuous coverage is provided by propagating the RF identities and by performing handoffs between any two of:

the cameras and the RFID and localization system.

14. The system of claim 8, further comprising:

at least one biometric system to provide a plurality of identities;

wherein the fusion module selectively associates the identities with the at least one fused track tracks.

15. A non-transitory computer readable storage medium having stored thereon instructions for performing a method for tracking an object, the method comprising;

receiving, by a plurality of camera trackers, camera video streams from a plurality of cameras;

providing, by the camera trackers, camera video tracks;

providing, by at least one radio frequency identification (RFID) and localization system, RF tracks and a plurality of RF identities;

selectively fusing, by a fusion module, the at least one of two camera video tracks and a camera video track to an RF track to generate at least one fused track, wherein the fusing is performed by matching tracks using association of space-time and appearance cues, and wherein the appearance cues include shape and appearance;

associating, by the fusion module, the at least one fused track with the RF identities;

displaying, by a visualization display, the at least one fused track and the RF identities; and

providing access to the camera video tracks, by the visualization display.

16. The computer readable medium of claim 15, wherein the camera video tracks are overlapping.

17. The computer readable medium of claim 15, wherein the camera video tracks are non-overlapping.

18. The computer readable medium of claim 15, wherein the fusing includes matching camera video tracks with new objects.

19. The computer readable medium of claim 15, wherein the camera video tracks are from monocular cameras, stereo cameras, or any combination thereof.

20. The computer readable medium of claim 15, wherein continuous coverage is provided by propagating the RF identities and by performing handoffs between any two of:

the cameras and the RFID and localization system.