Method of tracking moveable objects by combining data obtained from multiple sensor types

• US 10,641,604 B1
• Filed: 05/26/2017
• Issued: 05/05/2020
• Est. Priority Date: 06/22/2013
• Status: Active Grant

• Current Assignee: IntelliVision Technologies Corp. (Melrose Industries Plc)
• Original Assignee: IntelliVision Technologies Corp. (Melrose Industries Plc)
• Inventors: Nathan, Vaidhi, Gope, Chandan, Afraimovich, Lev, Soprykin, Sergey, Valerievich, Roman Kholodov
• Primary Examiner(s): Henson, Mischita L
• Application Number:
  US15/606,675
• Time in Patent Office: 1,075 Days
• CPC Class Codes: G01B 21/00 Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Method of tracking moveable objects (typically tagged objects that are moved by actors e.g. people, vehicles) by combining and analyzing data obtained from multiple types of sensors, such as video cameras, RFID tag readers, GPS sensors, and WiFi transceivers. Objects may be tagged by RFID tags, NFC tags, bar codes, or even tagged by visual appearance. The system operates in near real-time, and compensates for errors in sensor readings and missing sensor data by modeling object and actor movement according to a plurality of possible paths, weighting data from some sensors higher than others according to estimates of sensor accuracy, and weighing the probability of certain paths according to various other rules and penalty cost parameters. The system can maintain a comprehensive database which can be queried as to which actors associate with which objects, and vice versa. Other data pertaining to object location and association can also be obtained.

Claims

1. A method of automatically estimating locations of a plurality of different objects that are capable of being moved by a plurality of different actors, said method comprising:

using a set of first sensors and at least one processor to track and store in memory approximate paths over time of said plurality of different actors, thereby determining actor path data, wherein said set of first sensors comprise a set of video sensors;

using said at least one processor and computer vision algorithms to determine the approximate paths over time of said plurality of different actors;

using a set of second sensors and said at least one processor to track and store in memory approximate locations over time of said plurality of different objects, thereby determining object location over time data;

wherein said first sensors and said second sensors need not be the same types of sensors, and at least said set of second sensors is only able to approximately determine the locations over time of said plurality of different objects;

using said at least one processor to use both said actor path data and said object location over time data to determine if the location of a given object and the path of a given actor from the approximate paths of said plurality of different actors correlate over time, thereby determining at least one tentative association probability between said given object and said given actor;

wherein the probability that an object, once moved by an actor, remains associated with said actor, is less than or equal to 1, and changing a tentative association probability between an object and an actor is assigned at least one penalty cost parameter;

using said at least one processor, said actor path data, said object location over time data, said at least one tentative association probability between said given object and said given actor, and said at least one penalty cost parameter to determine the at least one of a probable location of said given object at one or more times and the association probability between said given object and said given actor at one or more times,

determining that the at least one of the probable location of said given object at one or more times and the association probability between said given object and said given actor at one or more times is outside a threshold that comprises a combination of preset object location and preset actor location; and

generating an alert notification in response to determining that the at least one of the probable location of said given object at one or more times and the association probability between said given object and said given actor at one or more times is outside the threshold.

2. The method of claim 1, further used to determine which various different objects are likely to be associated with a given actor;

using said at least one processor to further determine, for said given actor, at least one conflict resolution time when a sum of the association probabilities between said given objects and said given actor is at a local maximum value;

and reporting, for said given actor, at least the various different objects associated with said actor at said conflict resolution time.

3. The method of claim 1, wherein at least some of said actors are further tagged with a unique ID wireless device carried by said actors, further using a set of third wireless device sensors and said at least one processor to associate said unique ID with at least some of said actors;

and

further using said at least one processor to determine association probability between said given object and said given actor at one or more times, wherein said given actor is also associated with said unique ID.

4. The method of claim 3, further using said at least one processor to determine identity or demographic information pertaining to said given actor associated with said unique ID;

and

storing said association probability between said given object and said identity or demographic information pertaining to said given actor associated with said unique ID in a computer database.

5. The method of claim 3, wherein said unique ID wireless device is a Smartphone, and wherein said unique ID is supplied by at least one of an app running on said Smartphone;

or

wherein said Smartphone additionally comprises a WiFi transceiver with a WiFi address, and said unique ID is supplied by said Smartphone'"'"'s WiFi address;

or

wherein said Smartphone additionally comprises a Bluetooth transceiver with a Bluetooth address, and unique ID is supplied by said Smartphone'"'"'s Bluetooth address.

6. The method of claim 1, wherein said different objects are tagged with RFID tags, and said set of second sensors comprise a set of RFID sensors;

or

wherein said second sensors comprise wireless devices carried by said actors.

7. The method of claim 1, wherein said actors are selected from the group consisting of people, vehicles, robots, and shopping carts.

8. The method of claim 1, wherein said objects are selected from the group consisting of products, badges, tags, inventory, merchandise, and vehicles.

9. The method of claim 1, wherein using said at least one processor to use both said actor path data and said object location over time data to determine if the location of a given object and the path of a given actor correlate over time, thereby determining at least one tentative association probability between said given object and said given actor further comprises the steps of:

using said at least one processor to minimize noise induced errors by combining and filtering data from at least some second sensors in said set of second sensors;

using said at least one processor to determine if the location of a given object and the path of a given actor correlate over time by computing likely proximity between said given object and said path of said given actor over a plurality of time points;

using said at least one processor to compute a likely track of said given object over said plurality of time points, and compute if said likely track of said given object over time correlates with said path of said given actor over said plurality of time points.

10. A method of automatically estimating locations of a plurality of different objects that are capable of being moved by a plurality of different actors, said method comprising:

using a set of first sensors and at least one processor to track and store in memory approximate paths over time of said plurality of different actors, thereby determining actor path data;

wherein said set of first sensors comprise a set of video sensors; and

further using said at least one processor and computer vision algorithms to determine the approximate paths over time of said plurality of different actors;

using a set of second sensors and said at least one processor to track and store in memory approximate locations over time of said plurality of different objects, thereby determining object location over time data;

wherein said different objects are tagged with RFID tags, and said set of second sensors comprise a set of RFID sensors;

wherein said first sensors and said second sensors need not be the same types of sensors, and at least said set of second sensors is only able to approximately determine the locations over time of said plurality of different objects;

using said at least one processor to use both said actor path data and said object location over time data to determine if the location of a given object and the path of a given actor correlate over time, thereby determining at least one tentative association probability between said given object and said given actor;

wherein using said at least one processor to use both said actor path data and said object location over time data to determine if the location of a given object and the path of a given actor correlate over time, thereby determining at least one tentative association probability between said given object and said given actor further comprises the steps of;

using said at least one processor to minimize noise induced errors by combining and filtering data from at least some second sensors in said set of second sensors;

using said at least one processor to determine if the location of a given object and the path of a given actor correlate over time by computing likely proximity between said given object and said path of said given actor over a plurality of time points;

using said at least one processor to compute a likely track of said given object over said plurality of time points, and compute if said likely track of said given object over time correlates with said path of said given actor over said plurality of time points;

wherein the probability that an object, once moved by an actor, remains associated with said actor, is less than or equal to 1, and changing a tentative association probability between an object and an actor is assigned at least one penalty cost parameter; and

using said at least one processor, said actor path data, said object location over time data, said at least one tentative association probability between said given object and said given actor, and said at least one penalty cost parameter to determine the at least one of a probable location of said given object at one or more times and the association probability between said given object and said given actor at one or more times.

11. The method of claim 10, further used to determine which various different objects are likely to be associated with a given actor;

using said at least one processor to further determine, for said given actor, at least one conflict resolution time when a sum of the association probabilities between said given objects and said given actor is at a local maximum value;

and reporting, for said given actor, at least the various different objects associated with said actor at said conflict resolution time.

12. The method of claim 10, wherein at least some of said actors are further tagged with a unique ID wireless device carried by said actors, further using a set of third wireless device sensors and said at least one processor to associate said unique ID with at least some of said actors;

and

further using said at least one processor to determine association probability between said given object and said given actor at one or more times, wherein said given actor is also associated with said unique ID.

13. The method of claim 12, further using said at least one processor to determine identity or demographic information pertaining to said given actor associated with said unique ID;

and

storing said association probability between said given object and said identity or demographic information pertaining to said given actor associated with said unique ID in a computer database.

14. The method of claim 12, wherein said unique ID wireless device is a Smartphone, and wherein said unique ID is supplied by at least one of an app running on said Smartphone;

or

wherein said Smartphone additionally comprises a WiFi transceiver with a WiFi address, and said unique ID is supplied by said Smartphone'"'"'s WiFi address;

or

wherein said Smartphone additionally comprises a Bluetooth transceiver with a Bluetooth address, and unique ID is supplied by said Smartphone'"'"'s Bluetooth address.

15. The method of claim 10, wherein said actors are selected from the group consisting of people, vehicles, robots, and shopping carts.

16. The method of claim 10, wherein said objects are selected from the group consisting of products, badges, tags, inventory, merchandise, and vehicles.

17. A method of automatically estimating locations of a plurality of different objects that are capable of being moved by a plurality of different actors, said method comprising:

using a set of first sensors and at least one processor to track and store in memory approximate paths over time of said plurality of different actors, thereby determining actor path data;

wherein said set of first sensors comprise a set of video sensors; and

further using said at least one processor and computer vision algorithms to determine the approximate paths over time of said plurality of different actors;

using a set of second sensors and said at least one processor to track and store in memory approximate locations over time of said plurality of different objects, thereby determining object location over time data;

wherein said different objects are tagged with RFID tags, and said set of second sensors comprise a set of RFID sensors;

wherein at least some of said actors are further tagged with a unique ID wireless device carried by said actors, further using a set of third wireless device sensors and said at least one processor to associate said unique ID with at least some of said actors; and

wherein said unique ID wireless device is a Smartphone, and wherein said unique ID is supplied by at least one of an app running on said Smartphone;

or

wherein said Smartphone additionally comprises a WiFi transceiver with a WiFi address, and said unique ID is supplied by said Smartphone'"'"'s WiFi address;

or

wherein said Smartphone additionally comprises a Bluetooth transceiver with a Bluetooth address, and unique ID is supplied by said Smartphone'"'"'s Bluetooth address;

wherein said first sensors and said second sensors need not be the same types of sensors, and at least said set of second sensors is only able to approximately determine the locations over time of said plurality of different objects;

using said at least one processor to use both said actor path data and said object location over time data to determine if the location of a given object and the path of a given actor correlate over time, thereby determining at least one tentative association probability between said given object and said given actor;

wherein using said at least one processor to use both said actor path data and said object location over time data to determine if the location of a given object and the path of a given actor correlate over time, thereby determining at least one tentative association probability between said given object and said given actor further comprises the steps of;

using said at least one processor to minimize noise induced errors by combining and filtering data from at least some second sensors in said set of second sensors;

using said at least one processor to determine if the location of a given object and the path of a given actor correlate over time by computing likely proximity between said given object and said path of said given actor over a plurality of time points;

using said at least one processor to compute a likely track of said given object over said plurality of time points, and compute if said likely track of said given object over time correlates with said path of said given actor over said plurality of time points;

wherein the probability that an object, once moved by an actor, remains associated with said actor, is less than or equal to 1, and changing a tentative association probability between an object and an actor is assigned at least one penalty cost parameter; and

using said at least one processor, said actor path data, said object location over time data, said at least one tentative association probability between said given object and said given actor, and said at least one penalty cost parameter to determine the at least one of a probable location of said given object at one or more times and the association probability between said given object and said given actor at one or more times;

further using said at least one processor to determine association probability between said given object and said given actor at one or more times, wherein said given actor is also associated with said unique ID;

further using said at least one processor to determine identity or demographic information pertaining to said given actor associated with said unique ID; and

storing said association probability between said given object and said identity or demographic information pertaining to said given actor associated with said unique ID in a computer database.

18. The method of claim 17, further used to determine which various different objects are likely to be associated with a given actor;

using said at least one processor to further determine, for said given actor, at least one conflict resolution time when a sum of the association probabilities between said given objects and said given actor is at a local maximum value;

and reporting, for said given actor, at least the various different objects associated with said actor at said conflict resolution time.

19. The method of claim 17, wherein said actors are selected from the group consisting of people, vehicles, robots, and shopping carts;

and

wherein said objects are selected from the group consisting of products, badges, tags, inventory, merchandise, and vehicles.