Multi-sensor location and identification

• US 8,531,523 B2
• Filed: 12/08/2009
• Issued: 09/10/2013
• Est. Priority Date: 12/08/2009
• Status: Active Grant

• Current Assignee: Skyhook Holding, Inc. (Liberty Broadband Corp.)
• Original Assignee: TruePosition Incorporated
• Inventors: Anderson, Robert J, Bolon, Brian R
• Primary Examiner(s): SERRAO, RANODHI N
• Application Number:
  US12/633,672
• Publication Number:
  US 20110134240A1
• Time in Patent Office: 1,372 Days
• Field of Search: 348/143, 348/158, 348/169, 455/456.1, 382/103
• US Class Current: 348/143
• CPC Class Codes: G01S 2205/01 specially adapted for specific applications
• CPC Class Codes: G01S 5/0264 at least one of the systems being a non-radio wave positioning system
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/38 for collecting sensor information

Abstract

By combining imaging systems with wireless location functionality, a subject'"'"'s videometric signature can be linked to a public identity, thus enabling continuous surveillance outside or between the coverage area of video surveillance networks. In addition to extending the surveillance coverage area, the combination of computerized video surveillance with wireless location determination may also allow for identification of mobile device users via the existing mobile equipment and user identifiers used in the wireless network.

Claims

1. A computer-implemented method of tracking a subject associated with a mobile device, comprising:

using a computer, processing an image data set and a mobile device location data set to determine a match, wherein the image data set comprises image, time and location information, and the mobile device location data set comprises mobile device identification, time and location information, and wherein a match is indicated when the time and location information from the image and mobile device data sets meet a prescribed degree of correlation; and

in response to determining a match between the image and mobile device data sets, associating a specific mobile device with a specific image;

wherein at least one of the subject'"'"'s personal identity and location is unknown;

wherein the image data set is received from an image capture system comprising a plurality of image capture devices; and

wherein the mobile device location data set is received from a wireless location system configured to locate mobile devices; and

further comprising analyzing the specific image to determine a specific personal identity to associate with the specific image and the specific mobile device, determining that the subject to be tracked corresponds to the specific personal identity, and tracking the location of the specific mobile device as a proxy for tracking the subject.

2. The method of claim 1, wherein the image data set comprises video surveillance data.

3. The method of claim 1, wherein the mobile device comprises a wireless communications transceiver for communicating with a wireless communications network.

4. The method of claim 3, wherein the wireless communications network is a local wireless communications network.

5. The method of claim 3, wherein the wireless communications network is a wide area wireless communications network.

6. The method of claim 1, further comprising monitoring entry and exit of the subject with respect to a geo-fence boundary, wherein the geo-fence boundary is at least partially defined by a field-of-view of an image capture device.

7. The method of claim 1, further comprising storing the analyzed specific image, image data set and mobile device location data set and accumulating statistical data based on the analyzed specific image, image data set and mobile device location data set.

8. The method of claim 1, further comprising using the image data set to track the subject.

9. The method of claim 1, further comprising using the mobile device location data set to track the subject.

10. The method of claim 1, wherein the subject is a vehicle.

11. The method of claim 1, wherein said image data set is received from a first and second image capture device, further comprising tracking the subject as the subject moves from the first image capture device'"'"'s field of view to the second image capture device'"'"'s field of view, and using image data from the first and second image capture devices to track the subject.

12. The method of claim 1, further comprising determining a direction of movement of the mobile device based on the image data set.

13. The method of claim 1, wherein said image data set is received from a first and second image capture device with overlapping fields of view.

14. The method of claim 13, further comprising determining a velocity of the mobile device based on the image data set received from the first and second image capture devices.

15. The method of claim 1, wherein the method is performed at a choke point.

16. A computer-implemented method of tracking a subject associated with a mobile device, wherein at least one of the subject'"'"'s personal identity and location is unknown, the mobile device having a wireless communications transceiver for communicating with a wireless communications network, the method comprising:

receiving image data from at least one image capture device, wherein the image data comprises at least a portion of an image representing said subject;

receiving identification information and location information for the mobile device, wherein the identification information comprises a mobile device ID and a public ID associated with the mobile device, and wherein the identification information and location information for the mobile device are received from a network-based wireless location system configured to locate mobile devices using radio emissions from the mobile devices;

analyzing the received image data; and

determining that the received image data is associated with said mobile device based on the analyzed image data and previously stored image data, wherein said determining comprises determining that the received image data is correlated to the previously stored image data according to a predetermined correlation level; and

further comprising monitoring entry and exit of the subject with respect to a geo-fence boundary, wherein said at least one image capture device comprises a first and second image capture device, and further comprising tracking the subject as the subject moves from the first image capture device'"'"'s field of view to the second image capture device'"'"'s field of view, and using image data from the first and second image capture devices to track the subject.

17. The method of claim 16, wherein the image data comprises video surveillance data.

18. The method of claim 16, wherein the wireless communications network is a local wireless communications network.

19. The method of claim 16, wherein the wireless communications network is a wide area wireless communications network.

20. The method of claim 16, further comprising storing the analyzed image data, identification information, and location information and accumulating statistical data based on the stored image data, identification information, and location information.

21. The method of claim 16, further comprising using the received image data to track the mobile device when the mobile device is not located by the wireless location system.

22. The method of claim 16, further comprising using the identification information and location information to track the mobile device when the subject is out of a field of view of the image capture device.

23. The method of claim 16, wherein the subject is a vehicle.

24. The method of claim 16, further comprising determining a direction of movement of the mobile device based on the received image data.

25. The method of claim 16, wherein said at least one image capture device comprises a first and second image capture device with overlapping fields of view.

26. The method of claim 25, further comprising determining a velocity of the mobile device based on the received image data from the first and second image capture devices.

27. The method of claim 16, wherein the method is performed at a choke point.

28. A computer-implemented method of tracking a subject'"'"'s location, wherein the subject is associated with a mobile device having a wireless communications transceiver for communicating with a wireless communications network, the method comprising:

receiving image data from an image capture device associated with a geographic location, wherein the image data comprises an image capturing at least a portion of said subject;

analyzing the received image data and determining that the received image data is associated with the subject;

determining that location information from a location determination service is not available for the mobile device; and

using the geographic location of the image capture device as a location for the subject until said location information becomes available; and

further comprising monitoring entry and exit of the subject with respect to a geo-fence boundary, wherein said at least one image capture device comprises a first and second image capture device, and further comprising tracking the subject as the subject moves from the first image capture device'"'"'s field of view to the second image capture device'"'"'s field of view, and using image data from the first and second image capture devices to track the subject.

29. The method of claim 28, wherein the location information is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices.

30. A method of correlating a subject'"'"'s location with a mobile device, the mobile device having a wireless communications transceiver for communicating with a wireless communications network, the method comprising:

receiving image data from at least one image capture device, wherein the image data comprises an image capturing at least a portion of said subject;

analyzing the received image data and determining a signature for said subject based on the analyzed image data;

requesting identification information and location information for the mobile device; and

associating the signature with said identification information and location information, wherein the location information is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices; and

further comprising monitoring entry and exit of the subject with respect to a geo-fence boundary, analyzing a specific image to determine a specific personal identity to associate with the specific image and a specific mobile device, determining that the subject to be tracked corresponds to the specific personal identity, and tracking the location of the specific mobile device as a proxy for tracking the subject.

31. A system configured to track a subject associated with a mobile device, wherein at least one of the subject'"'"'s personal identity and location is unknown, the system comprising at least one processor and at least one storage medium communicatively coupled to said at least one processor, the storage medium having stored therein computer-executable instructions for instructing the processor in causing the following steps:

processing an image data set and a mobile device location data set to determine a match, wherein the image data set comprises image, time and location information, and the mobile device location data set comprises mobile device identification, time and location information, and wherein a match is indicated when the time and location information from the image and mobile device data sets meet a prescribed degree of correlation;

in response to determining a match between the image and mobile device data sets, associating a specific mobile device with a specific image;

wherein the image data set is received from an image capture system comprising a plurality of image capture devices; and

wherein the mobile device location data set is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices; and

further comprising analyzing the specific image to determine a specific personal identity to associate with the specific image and the specific mobile device, determining that the subject to be tracked corresponds to the specific personal identity, and tracking the location of the specific mobile device as a proxy for tracking the subject.

32. The system of claim 31, wherein the image data set comprises video surveillance data.

33. The system of claim 31, wherein the mobile device comprises a wireless communications transceiver for communicating with a wireless communications network.

34. The system of claim 33, wherein the wireless communications network is a local wireless communications network.

35. The system of claim 33, wherein the wireless communications network is a wide are wireless communications network.

36. The system of claim 31, further comprising monitoring entry and exit of the subject with respect to a geo-fence boundary, wherein the geo-fence boundary is at least partially defined by a field-of-view of an image capture device.

37. The system of claim 31, further comprising storing the analyzed specific image, image data set and mobile device location data set and accumulating statistical data based on the analyzed specific image, image data set and mobile device location data set.

38. The system of claim 31, further comprising using the image data set to track the subject when the mobile device is not located by the wireless location system.

39. The system of claim 31, further comprising using the mobile device location data set to track the subject when the image data set is not available.

40. The system of claim 31, wherein the subject is a vehicle.

41. The system of claim 31, wherein said image data set is received from a first and second image capture device, further comprising tracking the subject as the subject moves from the first image capture device'"'"'s field of view to the second image capture device'"'"'s field of view, and using image data from the first and second image capture devices to track the subject.

42. The system of claim 31, further comprising determining a direction of movement of the mobile device based on the image data set.

43. The system of claim 31, wherein said image data set is received from a first and second image capture device with overlapping fields of view.

44. The system of claim 43, further comprising determining a velocity of the mobile device based on the image data set received from the first and second image capture devices.

45. The system of claim 31, wherein the method is performed at a choke point.

46. A system configured to track a subject associated with a mobile device, wherein at least one of the subject'"'"'s personal identity and location is unknown, the mobile device having a wireless communications transceiver for communicating with a wireless communications network, the system comprising at least one processor and at least one storage medium communicatively coupled to said at least one processor, the storage medium having stored therein computer-executable instructions for instructing the processor in causing the following steps:

receiving image data from at least one image capture device, wherein the image data comprises at least a portion of an image representing said subject;

receiving identification information and location information for the mobile device, wherein the identification information comprises a mobile device ID and a public ID associated with the mobile device;

analyzing the received image data; and

determining that the received image data is associated with said mobile device based on the analyzed image data and previously stored image data, wherein said determining comprises determining that the received image data is correlated to the previously stored image data according to a predetermined correlation level; and

further comprising monitoring entry and exit of the subject with respect to a geo-fence boundary, wherein said at least one image capture device comprises a first and second image capture device, and further comprising tracking the subject as the subject moves from the first image capture device'"'"'s field of view to the second image capture device'"'"'s field of view, and using image data from the first and second image capture devices to track the subject.

47. The system of claim 46, wherein the image data comprises video surveillance data.

48. The system of claim 46, wherein the location information is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices.

49. The system of claim 48, further comprising using the received image data to track the mobile device when the mobile device is not located by the wireless location system.

50. The system of claim 48, further comprising using the identification information and location information to track the mobile device when the subject is out of a field of view of the at least one image capture device.

51. The system of claim 46, wherein the wireless communications network is a local wireless communications network.

52. The system of claim 46, wherein the wireless communications network is a wide area wireless communications network.

53. The system of claim 46, further comprising monitoring entry and exit of the subject with respect to a geo-fence boundary based on the location information and the identification information.

54. The system of claim 49, further comprising storing the analyzed image data, identification information, and location information and accumulating statistical data based on the stored image data, identification information, and location information.

55. The system of claim 46, wherein the subject is a vehicle.

56. The system of claim 46, further comprising determining a direction of movement of the mobile device based on the received image data.

57. The system of claim 46, wherein said at least one image capture device comprises a first and second image capture device with overlapping fields of view.

58. The system of claim 57, further comprising determining a velocity of the mobile device based on the received image data from the first and second image capture devices.

59. The system of claim 46, wherein the method is performed at a choke point.

60. A system configured to track a subject'"'"'s location, wherein the subject is associated with a mobile device having a wireless communications transceiver for communicating with a wireless communications network, the system comprising at least one processor and at least one storage medium communicatively coupled to said at least one processor, the storage medium having stored therein computer-executable instructions for instructing the processor in causing the following steps:

receiving image data from an image capture device, wherein the image data comprises an image capturing at least a portion of said subject;

analyzing the received image data and determining that the received image data is associated with the subject;

determining that location information from a location determination service is not available for the mobile device;

using a geographic location of the image capture device as a location for the subject until said location information becomes available; and

monitoring entry and exit of the subject with respect to a geo-fence boundary, wherein said at least one image capture device comprises a first and second image capture device, and tracking the subject as the subject moves from the first image capture device'"'"'s field of view to the second image capture device'"'"'s field of view, and using image data from the first and second image capture devices to track the subject.

61. The system of claim 60, wherein the location information is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices.

62. A system configured to correlate a subject'"'"'s location with a mobile device, the mobile device having a wireless communications transceiver for communicating with a wireless communications network, the system comprising at least one processor and at least one storage medium communicatively coupled to said at least one processor, the storage medium having stored therein computer-executable instructions for instructing the processor in causing the following steps:

receiving image data from at least one image capture device, wherein the image data comprises an image capturing at least a portion of said subject;

analyzing the received image data and determining a signature for said subject based on the analyzed image data;

requesting identification information and location information for the mobile device; and

associating the signature with said identification information and location information, wherein the location information is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices; and

monitoring entry and exit of the subject with respect to a geo-fence boundary, analyzing a specific image to determine a specific personal identity to associate with the specific image and a specific mobile device, determining that the subject to be tracked corresponds to the specific personal identity, and tracking the location of the specific mobile device as a proxy for tracking the subject.

63. A non-transitory computer readable storage medium storing thereon computer executable instructions for tracking a subject associated with a mobile device, said computer executable instructions for:

processing an image data set and a mobile device location data set to determine a match, wherein the image data set comprises image, time and location information, and the mobile device location data set comprises mobile device identification, time and location information, and wherein a match is indicated when the time and location information from the image and mobile device data sets meet a prescribed degree of correlation;

in response to determining the match between the image and mobile device data sets, associating a specific mobile device with a specific image;

wherein at least one of the subject'"'"'s personal identity and location is unknown;

wherein the image data set is received from an image capture system comprising a plurality of image capture devices; and

wherein the mobile device location data set is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices; and

analyzing the specific image to determine a specific personal identity to associate with the specific image and the specific mobile device, determining that the subject to be tracked corresponds to the specific personal identity, and tracking the location of the specific mobile device as a proxy for tracking the subject.

64. The computer readable storage medium of claim 63, wherein the image data set comprises video surveillance data.

65. The computer readable storage medium of claim 63, wherein the mobile device comprises a wireless communications transceiver for communicating with a wireless communications network.

66. The computer readable storage medium of claim 65, wherein the wireless communications network is a local wireless communications network.

67. The computer readable storage medium of claim 65, wherein the wireless communications network is a wide area wireless communications network.

68. The computer readable storage medium of claim 63, further comprising instructions for monitoring entry and exit of the subject with respect to a geo-fence boundary, wherein the geo-fence boundary is at least partially defined by a field-of-view of an image capture device.

69. The computer readable storage medium of claim 63, further comprising instructions for storing the analyzed specific image, image data set and mobile device location data set and accumulating statistical data based on the analyzed specific image, image data set and mobile device location data set.

70. The computer readable storage medium of claim 63, further comprising instructions for using the image data set to track the mobile device when the mobile device is not located by the wireless location system.

71. The computer readable storage medium of claim 63, further comprising instructions for using the mobile device location data set to track the mobile device when the image data set is not available.

72. The computer readable storage medium of claim 63, wherein the subject is a vehicle.

73. The computer readable storage medium of claim 63, wherein said image data set is received from a first and second image capture device, further comprising tracking the subject as the subject moves from the first image capture device'"'"'s field of view to the second image capture device'"'"'s field of view, and using image data from the first and second image capture devices to track the subject.

74. The computer readable storage medium of claim 63, further comprising instructions for determining a direction of movement of the mobile device based on the image data set.

75. The computer readable storage medium of claim 63, wherein said image data set is received from a first and second image capture device with overlapping fields of view.

76. The computer readable storage medium of claim 75, further comprising instructions for determining a velocity of the mobile device based on the received image data set from the first and second image capture devices.

77. The computer readable storage medium of claim 63, wherein the method is performed at a choke point.

78. A non-transitory computer readable storage medium storing thereon computer executable instructions for tracking a subject associated with a mobile device, wherein at least one of the subject'"'"'s personal identity and location is unknown, the mobile device having a wireless communications transceiver for communicating with a wireless communications network, said computer executable instructions for:

receiving image data from at least one image capture device, wherein the image data comprises at least a portion of an image representing said subject;

receiving identification information and location information for the mobile device, wherein the identification information comprises a mobile device ID and a public ID associated with the mobile device;

analyzing the received image data; and

determining that the received image data is associated with said mobile device based on the analyzed image data and previously stored image data, wherein said determining comprises determining that the received image data is correlated to the previously stored image data according to a predetermined correlation level; and

monitoring entry and exit of the subject with respect to a geo-fence boundary, wherein said at least one image capture device comprises a first and second image capture device, and tracking the subject as the subject moves from the first image capture device'"'"'s field of view to the second image capture device'"'"'s field of view, and using image data from the first and second image capture devices to track the subject.

79. The computer readable storage medium of claim 78, wherein the image data comprises video surveillance data.

80. The computer readable storage medium of claim 78, wherein the location information is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices.

81. The computer readable storage medium of claim 80, further comprising instructions for using the received image data to track the mobile device when the mobile device is not located by the wireless location system.

82. The computer readable storage medium of claim 80, further comprising instructions for using the identification information and location information to track the mobile device when the subject is out of a field of view of the at least one image capture device.

83. The computer readable storage medium of claim 78, wherein the wireless communications network is a local wireless communications network.

84. The computer readable storage medium of claim 78, wherein the wireless communications network is a wide area wireless communications network.

85. The computer readable storage medium of claim 78, further comprising instructions for monitoring entry and exit of the subject with respect to a geo-fence boundary based on the location information and the identification information.

86. The computer readable storage medium of claim 78, further comprising instructions for storing the analyzed image data, identification information, and location information and accumulating statistical data based on the stored imaged data, identification information, and location information.

87. The computer readable storage medium of claim 78, wherein the subject is a vehicle.

88. The computer readable storage medium of claim 78, further comprising instructions for determining a direction of movement of the mobile device based on the received image data.

89. The computer readable storage medium of claim 78, wherein said at least one image capture device comprises a first and second image capture device with overlapping fields of view.

90. The computer readable storage medium of claim 89, further comprising instructions for determining a velocity of the mobile device based on the received image data from the first and second image capture devices.

91. The computer readable storage medium of claim 90, wherein the method is performed at a choke point.

92. A non-transitory computer readable storage medium storing thereon computer executable instructions for tracking a subject'"'"'s location, wherein the subject is associated with a mobile device having a wireless communications transceiver for communicating with a wireless communications network, said computer executable instructions for:

receiving image data from an image capture device, wherein the image data comprises an image capturing at least a portion of said subject;

analyzing the received image data and determining that the received image data is associated with the subject;

determining that location information from a location determination service is not available for the mobile device;

using a geographic location of the image capture device as a location for the subject until said location information becomes available; and

monitoring entry and exit of the subject with respect to a geo-fence boundary, wherein said at least one image capture device comprises a first and second image capture device, and tracking the subject as the subject moves from the first image capture device'"'"'s field of view to the second image capture device'"'"'s field of view, and using image data from the first and second image capture devices to track the subject.

93. The computer readable storage medium of claim 92, wherein the location information is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices.

94. A non-transitory computer readable storage medium storing thereon computer executable instructions for correlating a subject'"'"'s location with a mobile device, the mobile device having a wireless communications transceiver for communicating with a wireless communications network, said computer executable instructions for:

receiving image data from at least one image capture device, wherein the image data comprises an image capturing at least a portion of said subject;

analyzing the received image data and determining a signature for said subject based on the analyzed image data;

requesting identification information and location information for the mobile device; and

associating the signature with said identification information and location information, wherein the location information is received from a network-based wireless location system configured to locate wireless devices using radio emissions from the wireless devices; and

monitoring entry and exit of the subject with respect to a geo-fence boundary, analyzing a specific image to determine a specific personal identity to associate with the specific image and a specific mobile device, determining that the subject to be tracked corresponds to the specific personal identity, and tracking the location of the specific mobile device as a proxy for tracking the subject.