Automatic camera calibration and geo-registration using objects that provide positional information

• US 7,949,150 B2
• Filed: 04/02/2007
• Issued: 05/24/2011
• Est. Priority Date: 04/02/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: ObjectVideo Incorporated (Alarm.com Holdings, Inc.)
• Inventors: Taylor, Geoffrey, Cao, Xiao Chun, Hassan-Shafique, Khurram, Liu, Haiying, Hakeem, Asaad, Rasheed, Zeeshan, Gupta, Himaanshu, Javed, Omar, Madden, Donald G., Chosak, Andrew J., Yu, Li, Haering, Niels, Lipton, Alan J.
• Primary Examiner(s): Tabatabai; Abolfazl
• Application Number:
  US11/730,518
• Publication Number:
  US 20080240616A1
• Time in Patent Office: 1,513 Days
• Field of Search: 382/103, 382/107, 382/104, 382/294, 348/169, 348/170, 348/172, 348/173, 348/174, 348/143, 348/171, 342/444, 342/357.22, 342/357.24, 342/357.39, 342/357.44, 386/227, 701/213, 701/215, 702/85
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A video sequence of a field of view within an environment is received. Targets are detected in the video sequence. Target geo-positional information is received. Correspondences between the targets detected in the video sequence and the target geo-positional information are determined and used to calibrate the camera and to geo-register a field of view of the camera.

Claims

1. A non-transitory computer-readable medium comprising software for video processing, which when executed by a computer system, cause the computer system to perform operations comprising a method of:

receiving a video sequence of a field of view within an environment;

detecting targets in the video sequence;

receiving target geo-positional information transmitted by a geo-positional sensor; and

determining, using at least one processor of the computer system, correspondences between the targets detected in the video sequence and the target geo-positional information,

wherein determining the correspondences comprises;

determining corresponding pairs of targets detected in the video sequence and received target geo-positional information based on a predetermined criteria;

determining a probability density function of joint distribution of spatial positions of each determined corresponding pair; and

estimating high joint density modes.

2. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional information regarding a video camera that generated the video sequence, wherein the correspondences are further determined based on the geo-positional information of the video camera that generated the video sequence.

3. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from a global positioning system (GPS) transmitter.

4. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from an automatic identification system (AIS) transmitter.

5. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from a blue force tracker (BFT) transmitter.

6. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from a cell phone.

7. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from a radio frequency identification (RFID) tag.

8. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

estimating at least one intrinsic parameter or at least one extrinsic parameter of a video camera that generated the video sequence based on the determined correspondences.

9. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

geo-registering the field of view of a video camera that generated the video sequence based at least on the determined correspondences.

10. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

geo-registering the field of view of a video camera that generated the video sequence based on geographic imaging system imagery of the environment.

11. The non-transitory computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises satellite imagery.

12. The non-transitory computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises nautical charts.

13. The non-transitory computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises electronic nautical charts.

14. The non-transitory computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises Digital Elevation Maps.

15. The non-transitory computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises a geospatial information system input.

16. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

determining a measurement of at least one of the target or the environment based on the determined correspondences.

17. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

weighting the density modes with correlation coefficients;

selecting a predetermined number of higher weighted density modes; and

determining homography for the selected higher weighted density modes.

18. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

based on the determined correspondences, at least one of;

detecting a target,

identifying a target,

classifying a target,

verifying a target, or

tracking a target.

19. The non-transitory computer-readable medium as set forth in claim 1, wherein the method further comprises:

comparing the target geo-positional information to meta information of a corresponding target detected in the video sequence to obtain a comparison result; and

at least one of identifying or verifying at least one of the target detected in the video sequence or target geo-positional information based on the comparison result.

20. The method as set forth in claim 1, further comprising:

weighting the density modes with correlation coefficients;

selecting a predetermined number of higher weighted modes; and

determining homography for the selected higher weighted modes.

21. A computer-based system to perform a method for video processing, the method comprising:

receiving a video sequence of a field of view within an environment;

detecting targets in the video sequence;

receiving target geo-positional information transmitted by a geo-positional sensor; and

determining, using at least one processor of the computer-based system, correspondences between the targets detected in the video sequence and the target geo-positional information,

wherein determining the correspondences comprises;

determining corresponding pairs of targets detected in the video sequence and received target geo-positional information based on a predetermined criteria;

determining a probability density function of joint distribution of spatial positions of each determined corresponding pair; and

estimating high joint density modes.

22. The system as set forth in claim 21, wherein the method further comprises:

receiving geo-positional information regarding a video camera that generated the video sequence, wherein the correspondences are further determined based on the geo-positional information of the video camera that generated the video sequence.

23. The system as set forth in claim 21, wherein the method further comprises:

estimating at least one intrinsic, or at least one extrinsic parameter of a video camera that generated the video sequence based on the determined correspondences.

24. The system as set forth in claim 21, wherein the method further comprises:

geo-registering the field of view of a video camera that generated the video sequence based at least on the determined correspondences.

25. The system as set forth in claim 21, further comprising:

geo-registering the field of view of a video camera that generated the video sequence based on geographic imaging system imagery of the environment.

26. The system as set forth in claim 25, wherein the geographic imaging system imagery comprises satellite imagery.

27. The system as set forth in claim 21, wherein the method further comprises:

determining measurements of at least one of the target or the environment based on the determined correspondences.

28. The system as set forth in claim 21, wherein the method further comprises:

weighting the density modes with correlation coefficients;

selecting a predetermined number of higher weighted density modes; and

determining homography for the selected higher weighted density modes.

29. The system as set forth in claim 21, wherein method further comprises:

based on the determined the determined correspondences, at least one of;

detecting a target,

identifying a target,

classifying a target,

verifying a target, or

tracking a target.

30. The system as set forth in claim 21, wherein the method further comprises:

comparing the target geo-positional information to meta information of a corresponding target detected in the video sequence to obtain a comparison result; and

at least one of identifying or verifying at least one of the target detected in the video sequence or target geo-positional information based on the comparison result.

31. A method for video processing, comprising:

receiving a video sequence of a field of view within an environment;

detecting targets in the video sequence;

receiving target geo-positional information transmitted by a geo-positional sensor; and

determining, using at least one processor, correspondences between the targets detected in the video sequence and the target geo-positional information,

wherein determining the correspondences comprises;

determining corresponding pairs of targets detected in the video sequence and received target geo-positional information based on a predetermined criteria;

determining a probability density function of joint distribution of spatial positions of each determined corresponding pair; and

estimating high joint density modes.

32. The method as set forth in claim 31, further comprising:

receiving geo-positional information regarding a video camera that generated the video sequence, wherein the correspondences are further determined based on the geo-positional information of the video camera that generated the video sequence.

33. The method as set forth in claim 31, further comprising:

estimating at least one intrinsic parameter, or at least one extrinsic parameter of a video camera that generated the video sequence based on the determined correspondences.

34. The method as set forth in claim 31, further comprising:

geo-registering the field of view of a video camera that generated the video sequence based at least on the determined correspondences.

35. The method as set forth in claim 31, further comprising:

geo-registering the field of view of a video camera that generated the video sequence based on geographic imaging system imagery of the environment.

36. The method as set forth in claim 35, wherein the geographic imaging system imagery comprises satellite imagery.

37. The method as set forth in claim 31, further comprising:

determining a measurement of at least one of the target or the environment based on the determined correspondences.

38. The method as set forth in claim 31, further comprising:

based on the determined the determined correspondences, at least one of;

detecting a target,

identifying a target,

classifying a target,

verifying a target, or

tracking a target.

39. The method as set forth in claim 31, further comprising:

comparing the target geo-positional information and meta information of a corresponding target detected in the video sequence to obtain a comparison result; and

at least one of identifying or verifying at least one of the target detected in the video sequence or target geo-positional information based on the comparison result.

40. An apparatus to perform the video processing method of claim 31.

41. A system to perform video processing of a video sequence generated by at least one video camera having a field of view within an environment, comprising:

a correspondence module to determine, using application specific hardware or at least one processor, correspondences between targets detected in the video sequence and corresponding target geo-positional information transmitted by a geo-positional sensor; and

at least one of;

a calibration and geo-registration module to determine a relationship between the video camera and geo-positional sensor and geo-register the video camera based at least on the determined correspondences;

or

a metric rules and video understanding module to obtain measurement information regarding at least one of the environment or target detected in the video sequence,

wherein determining the correspondences comprises;

determining corresponding pairs of targets detected in the video sequence and received target geo-positional information based on a predetermined criteria;

determining a probability density function of joint distribution of spatial positions of each determined corresponding pair; and

estimating high joint density modes.

42. The system as set forth in claim 41, comprising both of:

the calibration and geo-registration module to determine the relationship between the video camera and geo-positional sensor and geo-register the video camera based at least on the determined correspondences; and

the metric rules and video understanding module to obtain measurement information regarding at least one of the environment or target detected in the video sequence.