Automatic camera calibration and geo-registration using objects that provide positional information

• US 20080240616A1
• Filed: 04/02/2007
• Published: 10/02/2008
• Est. Priority Date: 04/02/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: ObjectVideo Incorporated (Alarm.com Holdings, Inc.)
• Inventors: Chosak, Andrew J., Yu, Li, Haering, Niels, Lipton, Alan J., Taylor, Geoffrey, Hassan-Shafique, Khurram, Liu, Haiying, Hakeem, Asaad, Rasheed, Zeeshan, Gupta, Himaanshu, Javed, Omar, Madden, Donald G., Cao, Xiao C.
• Granted Patent: US 7,949,150 B2
• Time in Patent Office: Days
• US Class Current: 382/294
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A video sequence of a field of view within an environment is received. Targets are detected in the video sequence. Target geo-positional information is received. Correspondences between the targets detected in the video sequence and the target geo-positional information are determined and used to calibrate the camera and to geo-register a field of view of the camera.

Claims

1. A computer-readable medium comprising software for video processing, which when executed by a computer system, cause the computer system to perform operations comprising a method of:

receiving a video sequence of a field of view within an environment;

detecting targets in the video sequence;

receiving target geo-positional information transmitted by a geo-positional sensor; and

determining correspondences between the targets detected in the video sequence and the target geo-positional information.

2. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional information regarding a video camera that generated the video sequence, wherein the correspondences are further determined based on the geo-positional information of the video camera that generated the video sequence.

3. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from a global positioning system (GPS) transmitter.

4. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from an automatic identification system (AIS) transmitter.

5. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from a blue force tracker (BFT) transmitter.

6. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from a cell phone.

7. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

receiving geo-positional target information from a radio frequency identification (RFID) tag.

8. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

estimating at least one intrinsic parameter or at least one extrinsic parameter of a video camera that generated the video sequence based on the determined correspondences.

9. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

geo-registering the field of view of a video camera that generated the video sequence based at least on the determined correspondences.

10. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

geo-registering the field of view of a video camera that generated the video sequence based on geographic imaging system imagery of the environment.

11. The computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises satellite imagery.

12. The computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises nautical charts.

13. The computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises electronic nautical charts.

14. The computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises Digital Elevation Maps.

15. The computer-readable medium as set forth in claim 10, wherein the geographic imaging system imagery comprises a geospatial information system input.

16. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

determining a measurement of at least one of the target or the environment based on the determined correspondences.

17. The computer-readable medium as set forth in claim 1, wherein determining the correspondences comprises:

determining corresponding pairs of targets detected in the video sequence and received target geo-positional information based on a predetermined criteria;

determining a probability density function of joint distribution of spatial positions of each determined corresponding pair; and

estimating high joint density modes.

18. The computer-readable medium as set forth in claim 17, wherein the method further comprises:

weighting the density modes with correlation coefficients;

selecting a predetermined number of higher weighted density modes; and

determining homography for the selected higher weighted density modes.

19. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

based on the determined correspondences, at least one of;

detecting a target,

identifying a target,

classifying a target,

verifying a target, or

tracking a target.

20. The computer-readable medium as set forth in claim 1, wherein the method further comprises:

comparing the target geo-positional information to meta information of a corresponding target detected in the video sequence to obtain a comparison result; and

at least one of identifying or verifying at least one of the target detected in the video sequence or target geo-positional information based on the comparison result.

21. A computer-based system to perform a method for video processing, the method comprising:

receiving a video sequence of a field of view within an environment;

detecting targets in the video sequence;

receiving target geo-positional information transmitted by a geo-positional sensor; and

determining correspondences between the targets detected in the video sequence and the target geo-positional information.

22. The system as set forth in claim 21, wherein the method further comprises:

receiving geo-positional information regarding a video camera that generated the video sequence, wherein the correspondences are further determined based on the geo-positional information of the video camera that generated the video sequence.

23. The system as set forth in claim 21, wherein the method further comprises:

estimating at least one intrinsic, or at least one extrinsic parameter of a video camera that generated the video sequence based on the determined correspondences.

24. The system as set forth in claim 21, wherein the method further comprises:

geo-registering the field of view of a video camera that generated the video sequence based at least on the determined correspondences.

25. The system as set forth in claim 21, further comprising:

geo-registering the field of view of a video camera that generated the video sequence based on geographic imaging system imagery of the environment.

26. The system as set forth in claim 25, wherein the geographic imaging system imagery comprises satellite imagery.

27. The system as set forth in claim 21, wherein the method further comprises:

determining measurements of at least one of the target or the environment based on the determined correspondences.

28. The system as set forth in claim 21, wherein determining the correspondences comprises:

determining corresponding pairs of targets detected in the video sequence and received target geo-positional information based on a predetermined criteria;

determining a probability density function of joint distribution of spatial positions of each determined corresponding pair; and

estimating high joint density modes.

29. The system as set forth in claim 28, wherein the method further comprises:

weighting the density modes with correlation coefficients;

selecting a predetermined number of higher weighted density modes; and

determining homography for the selected higher weighted density modes.

30. The system as set forth in claim 21, wherein method further comprises:

based on the determined the determined correspondences, at least one of;

detecting a target,

identifying a target,

classifying a target,

verifying a target, or

tracking a target.

31. The system as set forth in claim 21, wherein the method further comprises:

comparing the target geo-positional information to meta information of a corresponding target detected in the video sequence to obtain a comparison result; and

at least one of identifying or verifying at least one of the target detected in the video sequence or target geo-positional information based on the comparison result.

32. A method for video processing, comprising:

receiving a video sequence of a field of view within an environment;

detecting targets in the video sequence;

receiving target geo-positional information transmitted by a geo-positional sensor; and

determining correspondences between the targets detected in the video sequence and the target geo-positional information.

33. The method as set forth in claim 32, further comprising:

receiving geo-positional information regarding a video camera that generated the video sequence, wherein the correspondences are further determined based on the geo-positional information of the video camera that generated the video sequence.

34. The method as set forth in claim 32, further comprising:

estimating at least one intrinsic parameter, or at least one extrinsic parameter of a video camera that generated the video sequence based on the determined correspondences.

35. The method as set forth in claim 32, further comprising:

geo-registering the field of view of a video camera that generated the video sequence based at least on the determined correspondences.

36. The method as set forth in claim 32, further comprising:

geo-registering the field of view of a video camera that generated the video sequence based on geographic imaging system imagery of the environment.

37. The method as set forth in claim 36, wherein the geographic imaging system imagery comprises satellite imagery.

38. The method as set forth in claim 32, further comprising:

determining a measurement of at least one of the target or the environment based on the determined correspondences.

39. The method as set forth in claim 32, wherein determining the correspondences comprises:

determining corresponding pairs of targets detected in the video sequence and received target geo-positional information based on a predetermined criteria;

determining a probability density function of joint distribution of spatial positions of each determined corresponding pair; and

estimating high joint density modes.

40. The method as set forth in claim 39, further comprising:

weighting the density modes with correlation coefficients;

selecting a predetermined number of higher weighted modes; and

determining homography for the selected higher weighted modes.

41. The method as set forth in claim 32, further comprising:

based on the determined the determined correspondences, at least one of;

detecting a target,

identifying a target,

classifying a target,

verifying a target, or

tracking a target.

42. The method as set forth in claim 32, further comprising:

comparing the target geo-positional information and meta information of a corresponding target detected in the video sequence to obtain a comparison result; and

at least one of identifying or verifying at least one of the target detected in the video sequence or target geo-positional information based on the comparison result.

43. An apparatus to perform the video processing method of claim 32.

44. A system to perform video processing of a video sequence generated by at least one video camera having a field of view within an environment, comprising at least one of:

a correspondence module to determine correspondences between targets detected in the video sequence and corresponding target geo-positional information transmitted by a geo-positional sensor;

a calibration and geo-registration module to determine a relationship between the video camera and geo-positional sensor and geo-register the video camera based at least on the determined correspondences;

or

a metric rules and video understanding module to obtain measurement information regarding at least one of the environment or target detected in the video sequence.

45. The system as set forth in claim 44, comprising at least two of:

the correspondence module to determine correspondences between the targets detected in the video sequence and corresponding target geo-positional information transmitted by the geo-positional sensor;

the calibration and geo-registration module to determine the relationship between the video camera and geo-positional sensor and geo-register the video camera based at least on the determined correspondences;

or

the metric rules and video understanding module to obtain measurement information regarding at least one of the environment or target detected in the video sequence.