Hybrid multi-camera based positioning

• US 9,785,857 B2
• Filed: 08/30/2016
• Issued: 10/10/2017
• Est. Priority Date: 07/18/2013
• Status: Expired due to Fees

• Current Assignee: Golba LLC
• Original Assignee: Golba LLC
• Inventors: Moshfeghi, Mehran
• Primary Examiner(s): SAINI, AMANDEEP SINGH
• Application Number:
  US15/252,180
• Publication Number:
  US 20170053419A1
• Time in Patent Office: 406 Days
• Field of Search: 382115
• CPC Class Codes: G01S 5/0264 at least one of the systems being a non-radio wave positioning system
• CPC Class Codes: G01S 5/0289 of multiple transceivers, e.g. in ad hoc networks
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A positioning server is connected to a collection of access points, base stations, NFC stations, and image or video cameras and the collected data is used for positioning objects. A plurality of electronic devices are paired with an object by tracking the position of the object based on imaging and the position of electronic devices based on RF signals in vicinity of the object. Once a device is paired with an object, the propagation channel profile measured through the electronic device is used to develop and tune a database of channel profiles versus location. This database is used based on signature/profile matching and correlation for positioning devices and objects that do not have pairing or have poor image-based positioning accuracy or reliability. When a device is detected that cannot be paired with any object, or a device that is unpaired from a previously associated object, a theft or loss alert is generated.

Claims

1. A method of providing a position of an object, the method comprising:

determining the position of the object using a plurality of images taken by a set of cameras;

identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device in the set of electronic devices;

associating one or more electronic devices in the set of electronic devices with the object based on a distance between the object and each of said one or more electronic devices at one or more instances of time; and

sending the position of the object determined by using the plurality of images to each of said one or more associated electronic devices.

2. The method of claim 1 further comprising:

receiving RF measurements corresponding to the determined position of the object from each associated electronic device; and

storing the RF measurements received from each associated electronic device in a database as an RF-profile signature for the determined position.

3. The method of claim 1, wherein the determined position of the object is a first position, the method further comprising:

determining that the object has moved from the first position;

determining a second position of the object using a plurality of images taken by the set of cameras;

determining that an accuracy of the second position of the object is below a threshold;

determining a channel propagation profile from the RF signals received from each electronic device associated with the object;

comparing the channel propagation profile determined for each electronic device associated with the object with a set of RF-profile signature retrieved from a database;

determining a position of each electronic device associated with the object based on the comparison; and

revising the second position of the object based on the determined positions of each electronic device associated with the object.

4. The method of claim 1, wherein identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device comprises:

determining a channel propagation profile from the set of RF signals received from each electronic device;

comparing the channel propagation profile determined for each electronic device with a set of channel propagation profiles retrieved from a database;

determining a position of each electronic device based on the comparison; and

determining that a distance between the determined position of each electronic device and the position of the object is within the predetermined distance.

5. The method of claim 4 further comprising:

determining that a plurality of channel propagation profiles from the set of channel propagation profiles retrieved from a database matches the measured propagation profile received from a particular electronic device, the plurality of channel propagation profiles corresponding to a plurality of positions retrieved from the database;

determining a search region for the particular electronic device centered around a previous position of the particular electronic device, the search region radius determined based on (i) a maximum speed of the particular electronic device and (ii) an elapsed time since the previous position was determined; and

determining a position of the particular electronic device from one or more of the positions retrieved from the database that are within the search region.

6. The method of claim 5, wherein determining a position of the particular electronic device from one or more of the positions retrieved from the database that are within the search region comprises:

determining that only one position from the positions retrieved from the database is within the search region; and

setting the position of the particular electronic device to the only position from the positions retrieved from the database that is within the search region.

7. The method of claim 6, wherein the search region is a first search region, wherein the previous position of the particular electronic device is a first previous position, wherein determining a position of the particular electronic device from one or more of the positions retrieved from the database that are within the search region comprises:

determining that more than one positions from the positions retrieved from the database are within the search region;

determining a second search region for the particular electronic device centered around a second previous position of the particular electronic device, the search region radius determined based on (i) a maximum speed of the particular electronic device and (ii) an elapsed time since the second previous position was determined; and

determining the position of the particular electronic device from one or more of the positions retrieved from the database that are within the second search region.

8. The method of claim 1 further comprising:

identifying the object in a plurality of camera feeds using one of image processing, pattern recognition, and feature detection on said plurality of images; and

extracting a two-dimensional position of the object from the plurality of the images,

wherein determining the position of the object using a plurality of images comprises determining the position of the object based on the extracted two-dimensional position of the object, a position of each camera in the set of cameras, and an orientation of each camera in the set of cameras.

9. The method of claim 1, wherein associating one or more electronic devices in the set of electronic devices with the object based on the distance between each of said one or more electronic devices and the object at one or more instances of time comprises:

determining the position of the object at said one or more instances of time;

determining the position of each associated electronic device at said one or more instances of time; and

determining that each associated electronic device is within a predetermined distance of the object at said one or more instances of time.

10. The method of claim 1, wherein associating each electronic device in the set of electronic devices with the object further comprises:

determining a communication standard used by the associated electronic device; and

determining that the associated electronic device is the only electronic device using said standard within said predetermined distance of the object in all said one or more instances of time.

11. The method of claim 10, wherein the communication standard is one of an 802.11 communication protocol version, a Bluetooth version, a long-term evolution (LTE) version, and a multiple-input multiple-output (MIMO) capability of the associated electronic device.

12. The method of claim 1, wherein the cameras are infrared cameras.

13. The method of claim 1, wherein each associated device comprises a display, wherein the position of object is for each associated device to display on the display of associated device.

14. A non-transitory machine readable medium storing a program for providing a position of an object, the program executable by at least one processing unit, the program comprising sets of instructions for:

determining the position of the object using a plurality of images taken by a set of cameras;

identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device in the set of electronic devices;

associating one or more electronic devices in the set of electronic devices with the object based on a distance between the object and each of said one or more electronic devices at one or more instances of time; and

sending the position of the object determined by using the plurality of images to each of said one or more associated electronic devices.

15. The non-transitory machine readable medium of claim 14, wherein the set of instructions for identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device comprises sets of instructions for:

determining a channel propagation profile from the set of RF signals received from each electronic device;

comparing the channel propagation profile determined for each electronic device with a set of channel propagation profiles retrieved from a database;

determining a position of each electronic device based on the comparison; and

determining that a distance between the determined position of each electronic device and the position of the object is within the predetermined distance.

16. The non-transitory machine readable medium of claim 15, the program further comprising sets of instructions for:

determining that a plurality of channel propagation profiles from the set of channel propagation profiles retrieved from a database matches the measured propagation profile received from a particular electronic device, the plurality of channel propagation profiles corresponding to a plurality of positions retrieved from the database;

determining a search region for the particular electronic device centered around a previous position of the particular electronic device, the search region radius determined based on (i) a maximum speed of the particular electronic device and (ii) an elapsed time since the previous position was determined; and

determining a position of the particular electronic device from one or more of the positions retrieved from the database that are within the search region.

17. The non-transitory machine readable medium of claim 14, wherein the cameras are infrared cameras.

18. The non-transitory machine readable medium of claim 14, wherein each associated device comprises a display, wherein the position of object is for each associated device to display on the display of associated device.

19. An apparatus comprising:

a set of processing units for executing sets of instructions;

a non-transitory machine readable medium storing a program which when executed by one of the processing units provides a position of an object, the program comprising sets of instructions for;

determining the position of the object using a plurality of images taken by a set of cameras;

identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device in the set of electronic devices;

associating one or more electronic devices in the set of electronic devices with the object based on distances between the object and each of said one or more electronic devices at one or more instances of time; and

sending the position of the object determined by using the plurality of images to each of said one or more associated electronic devices.

20. The apparatus of claim 19, wherein the set of instructions for identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device comprises sets of instructions for:

determining a channel propagation profile from the set of RF signals received from each electronic device;

comparing the channel propagation profile determined for each electronic device with a set of channel propagation profiles retrieved from a database;

determining a position of each electronic device based on the comparison; and

determining that a distance between the determined position of each electronic device and the position of the object is within the predetermined distance.

21. The apparatus of claim 19, wherein the cameras are infrared cameras.

22. The apparatus of claim 19, wherein each associated device comprises a display, wherein the position of object is for each associated device to display on the display of associated device.