HYBRID MULTI-CAMERA BASED POSITIONING

• US 20170053419A1
• Filed: 08/30/2016
• Published: 02/23/2017
• Est. Priority Date: 07/18/2013
• Status: Active Grant

• Current Assignee: Golba LLC
• Original Assignee: Golba LLC
• Inventors: Moshfeghi, Mehran
• Granted Patent: US 9,785,857 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01S 5/0264 at least one of the systems being a non-radio wave positioning system
• CPC Class Codes: G01S 5/0289 of multiple transceivers, e.g. in ad hoc networks
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A positioning server is connected to a collection of access points, base stations, NFC stations, and image or video cameras and the collected data is used for positioning objects. A plurality of electronic devices are paired with an object by tracking the position of the object based on imaging and the position of electronic devices based on RF signals in vicinity of the object. Once a device is paired with an object, the propagation channel profile measured through the electronic device is used to develop and tune a database of channel profiles versus location. This database is used based on signature/profile matching and correlation for positioning devices and objects that do not have pairing or have poor image-based positioning accuracy or reliability. When a device is detected that cannot be paired with any object, or a device that is unpaired from a previously associated object, a theft or loss alert is generated.

Claims

1. A method of providing a position of an object, the method comprising:

determining the position of the object using a plurality of images taken by a set of cameras;

identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device in the set of electronic devices;

pairing a plurality of electronic devices in the set of electronic devices with the object based on distances between the object and each electronic device in the set of electronic devices at a plurality of instances of time; and

sending the position of the object determined by using the plurality of images to at least one of the paired electronic devices.

2. The method of claim 1 further comprising:

receiving RF measurements corresponding to the determined position of the object from the plurality of paired electronic devices; and

storing the RF measurement received from the plurality of paired electronic devices in a database as an RF-profile signature for the determined position.

3. The method of claim 1, wherein the determined position of the object is a first position, the method further comprising:

determining that the object has moved from the first position;

determining a second position of the object using a plurality of images taken by the set of cameras;

determining that an accuracy of the second position of the object is below a threshold;

determining a channel propagation profile from the RF signals received from each electronic device paired with the object;

comparing the channel propagation profile determined for each paired electronic device paired with the object with a set of RF-profile signature retrieved from a database;

determining a position of each electronic device paired with the object based on the comparison; and

revising the second position of the object based on the determined positions of each electronic device paired with the object.

4. The method of claim 1, wherein identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device comprises:

determining a set of channel propagation profiles from the set of RF signals received from each electronic device;

comparing the propagation profile determined for each electronic device with a set of channel propagation profiles retrieved from a database;

determining a position of each electronic device based on the comparison; and

determining that a distance between the determined position of each electronic device and the position of the object is within the predetermined distance.

5. The method of claim 4 further comprising:

determining that a plurality of channel propagation profiles from the set of channel propagation profiles retrieved from a database matches the measured propagation profile received from a particular electronic device, the plurality of channel propagation profiles corresponding to a plurality of positions retrieved from the database;

determining a search region for the particular electronic device centered around a previous position of the particular electronic device, the search region radius determined based on (ii) a maximum speed of the particular electronic device and (iii) an elapsed time since the previous position was determined; and

determining a position of the particular electronic device from one or more of the positions retrieved from the database that are within the search region.

6. The method of claim 5, wherein determining a position of the particular electronic device from one or more of the positions retrieved from the database that are within the search region comprises:

determining that only one position from the positions retrieved from the database is within the search region; and

setting the position of the particular electronic device to the only position from the positions retrieved from the database that is within the search region.

7. The method of claim 6, wherein the search region is a first search region, wherein the previous position of the particular electronic device is a first previous position, wherein determining a position of the particular electronic device from one or more of the positions retrieved from the database that are within the search region comprises:

determining that more than one positions from the positions retrieved from the database are within the search region;

determining a second search region for the particular electronic device centered around a second previous position of the particular electronic device, the search region radius determined based on (ii) a maximum speed of the particular electronic device and (iii) an elapsed time since the second previous position was determined; and

determining the position of the particular electronic device from one or more of the positions retrieved from the database that are within the second search region.

8. The method of claim 1 further comprising:

identifying the object in a plurality of camera feeds using one of image processing, pattern recognition, and feature detection on said plurality of images; and

extracting a two-dimensional position of the object from the plurality of the images,

wherein determining the position of the object using a plurality of images comprises determining the position of the object based on the extracted two-dimensional position of the object, a position of each camera in the set of cameras, and an orientation of each camera in the set of cameras.

9. The method of claim 1 further comprising:

identifying a set of features of the object in the plurality of images;

tracking the object across a plurality of cameras in the set of cameras using the identified features; and

handing off the object from at least one camera coverage to another camera coverage.

10. The method of claim 9, wherein the object is a human, wherein identifying the set of features comprises at least one of a face feature, a cloth'"'"'s color, a height, and a carry-on item.

11. The method of claim 1 further comprising:

determining an accuracy region for the object position; and

determining an accuracy region for each electronic device in the set of electronic devices,

wherein pairing based on proximity of the electronic devices and the object at a plurality of instances of time comprises determining that the accuracy region of the object and the accuracy regions of the plurality of the electronic devices overlap at the plurality of instances of time.

12. The method of claim 11, wherein the accuracy region for the object position indicates a probability of the object being within the accuracy region, wherein the accuracy region for each electronic device indicates a probability of the electronic device being within the corresponding accuracy region.

13. The method of claim 1, wherein identifying the set of electronic devices within a predetermined distance of the object comprises:

receiving a set of signal measurements from each electronic device in the set of electronic devices;

determining a position of each electronic device based on the corresponding set of signal measurements; and

determining that the position of each electronic device in the set of electronic devices is within the predetermined distance of the object.

14. The method of claim 13, wherein the signal measurements comprise at least one of time-of-arrival and angle-of-arrival of wireless signals exchanged between each electronic device in the set of electronic devices and a set of access points.

15. The method of claim 1, wherein pairing an electronic device in the set of electronic devices with the object based on proximity of the electronic device and the object at a plurality of instances of time comprises:

determining the position of the object at said plurality of instances of time;

determining the position of the paired electronic device at said plurality of instances of time; and

determining that the electronic device is within a predetermined distance of the object in said plurality of instances of time.

16. The method of claim 1, wherein pairing a plurality of electronic devices in the set of electronic devices with the object further comprises:

determining a communication standard used by the paired electronic device; and

determining that the paired electronic device is the only electronic device using said standard within said predetermined distance of the object in all said plurality of instances of time.

17. The method of claim 16, wherein the communication standard is one of an 802.11 communication protocol version, a Bluetooth version, a long-term evolution (LTE) version, and a multiple-input multiple-output (MIMO) capability of the paired electronic device.

18. A non-transitory machine readable medium storing a program for providing a position of an object, the program executable by at least one processing unit, the program comprising sets of instructions for:

determining the position of the object using a plurality of images taken by a set of cameras;

identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device in the set of electronic devices;

pairing a plurality of electronic devices in the set of electronic devices with the object based on distances between the object and each electronic device in the set of electronic devices at a plurality of instances of time; and

sending the position of the object determined by using the plurality of images to at least one of the paired electronic devices.

19. The non-transitory machine readable medium of claim 18, wherein the set of instructions for identifying a set of electronic devices within a predetermined distance of the object based on a set of radio frequency (RF) signals received from each electronic device comprises sets of instructions for:

determining a set of channel propagation profiles from the set of RF signals received from each electronic device;

comparing the propagation profile determined for each electronic device with a set of channel propagation profiles retrieved from a database;

determining a position of each electronic device based on the comparison; and

determining that a distance between the determined position of each electronic device and the position of the object is within the predetermined distance.

20. The non-transitory machine readable medium of claim 19, the program further comprising sets of instructions for:

determining that a plurality of channel propagation profiles from the set of channel propagation profiles retrieved from a database matches the measured propagation profile received from a particular electronic device, the plurality of channel propagation profiles corresponding to a plurality of positions retrieved from the database;

determining a search region for the particular electronic device centered around a previous position of the particular electronic device, the search region radius determined based on (ii) a maximum speed of the particular electronic device and (iii) an elapsed time since the previous position was determined; and

determining a position of the particular electronic device from one or more of the positions retrieved from the database that are within the search region.

21-27. -27. (canceled)