System for verifying physical object absences from assigned regions using video analytics

• US 10,372,753 B2
• Filed: 03/15/2017
• Issued: 08/06/2019
• Est. Priority Date: 03/16/2016
• Status: Active Grant

• Current Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Original Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Inventors: High, Donald, Jones, Matthew Allen, Mattingly, Todd Davenport, Natarajan, Chandrashekar
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Mangialaschi, Tracy
• Application Number:
  US15/459,451
• Publication Number:
  US 20170270360A1
• Time in Patent Office: 874 Days
• Field of Search: None
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04N 1/32117 in a separate transmission or protocol signal prior to or subsequent to the image data transmission, e.g. in digital identification signal [DIS], in non standard setup [NSS] or in non standard field [NSF] for mode signalling H04N1/333
• CPC Class Codes: H04N 2201/3253 Position information, e.g. geographical position at time of capture, GPS data

Abstract

Described in detail herein are systems and methods for verification of a report of absence of a physical object from an assigned region. A mobile device may capture an image of a vacant region and transmit the image and GPS coordinates of the location of the vacant region to a video analysis module within a computing device. The video analysis module may perform video analytics on the image and verify the location of the vacant region. The video analysis module may verify the physical object is absent from the vacant region. In response to verification that the physical object is absent from the vacant region the video analysis module may trigger an event.

Claims

1. A detection system comprising:

a computing device equipped with a processor and a video analysis module and a confirmation module;

an image capture device affixed to a fixture of a facility; and

a database accessible by the video analysis module;

wherein the video analysis module is configured to;

receive a first image of a vacant region to which a first plurality of like physical objects are assigned, the first image received from a mobile device executing an application associated with the facility and including an identifier of the first plurality of like physical objects,

receive, from the mobile device, GPS coordinates of a geographic location at which the first image was taken;

query the database for data associated with the first plurality of like physical objects using the identifier of the first plurality of like physical objects from the first image;

perform video analytics on the first image of the vacant region using data received in response to the query;

verify a location of the vacant region based on the video analytics and the GPS coordinates;

retrieve a stored image of at least one of the first plurality of like physical objects, from the database;

compare the stored image of the at least one of the first plurality of like physical objects with the first image of the vacant region;

verify the first plurality of like physical objects are absent from the vacant region in the image based on the data associated with the first plurality of like physical objects and the location of the vacant region and the comparison of the stored image of the at least one of the first plurality of like physical objects with the first image of the vacant region,

wherein, in response to verifying that the first plurality of like physical objects are absent from the vacant region, an event is triggered,

wherein the confirmation module is configured to confirm that the first image taken by the mobile device is legitimate by capturing a second image of the mobile device taking the first image of the vacant region, the second image taken by the image capturing device affixed to the fixture of the facility.

2. The system of claim 1, wherein the location is a location within a retail facility.

3. The system of claim 1, wherein the video analytics include determining at least one of the size, shape, and dimensions of the vacant region.

4. The system of claim 1, wherein the data associated with the first plurality of like physical objects includes at least one of a shape, size, dimensions and image of at least one of the first plurality of like physical objects.

5. The system of claim 4, wherein the video analysis module is configured to verify that the first plurality of like physical objects are absent from the vacant region by comparing at least one of the size, shape, and dimensions of the vacant region to at least one of the size, shape and dimensions of the at least one of the first plurality of like physical objects.

6. The system of claim 4, wherein the video analysis module is configured to verify that the first plurality of like physical objects are absent from the vacant region by retrieving from the database a quantity of the first plurality of like physical objects.

7. The system of claim 1, wherein the video analysis module is further configured to verify the geographic location by performing video analytics on the second image captured by the image capturing device affixed to a fixture in the facility and determining a second plurality of objects adjacent to the vacant region.

8. The system of claim 1, wherein the event is issuing an alert regarding the absence of the first plurality of like physical objects from the vacant region and storing a record of a user of the mobile device.

9. A detection method comprising:

receiving, with a computing device equipped with a video analysis module and a confirmation module, in communication with a database, a first image of a vacant region within a facility, to which a first plurality of like physical objects are assigned, the first image received from a mobile device executing an application associated with the facility and including an identifier of the first plurality of like physical objects;

receiving, from the mobile device, GPS coordinates of a geographic location at which the first image was taken; and

analyzing the first image with the video analysis module, the video analysis module;

querying the database for data associated with the first plurality of like physical objects using the identifier of the first plurality of like physical objects from the first image,

performing video analytics on the first image of the vacant region using data received in response to the query,

verifying a location of the vacant region based on the video analytics and the GPS coordinates,

retrieving a stored image of at least one of the first plurality of like physical objects, from the database;

comparing the stored image of the at least one of the first plurality of like physical objects with the first image of the vacant region;

verifying the first plurality of like physical objects are absent from the vacant region in the image based on the data associated with the first plurality of like physical objects and the location of the vacant region and the comparison of the stored image of the at least one of the first plurality of like physical objects with the first image of the vacant region; and

confirming, with the confirmation module that the first image taken by the mobile device is legitimate by capturing a second image of the mobile device taking the first image of the vacant region, the second image taken by an image capturing device affixed to a fixture of the facility,

wherein an event is triggered in response to verifying that the first plurality of like physical objects are absent from the vacant region.

10. The method of claim 9, wherein the geographic location is a geographic location within a retail facility.

11. The method of claim 9, wherein performing the video analytics comprises determining, via the computing device, at least one of the size, shape, and dimensions of the vacant region.

12. The method of claim 9, wherein the data associated with the first plurality of like physical objects includes at least one of a shape, size, dimensions and image of at least one of the first plurality of like physical objects.

13. The method of claim 12, wherein verifying that the first plurality of like physical objects are absent from the vacant region further includes comparing, at least one of the size, shape, and dimensions of the vacant region to at least one of the size, shape and dimensions of the at least one of the first plurality of like physical objects.

14. The method of claim 12, wherein verifying the first plurality of like physical objects are absent from the vacant region further comprising retrieving, from the database a quantity of first plurality of like physical objects.

15. The method of claim 9, wherein verifying the geographic location further includes performing, with the video analysis module, video analytics on the second image captured by the image capturing device affixed to a fixture in the facility and determining a second plurality of objects adjacent to the vacant region.

16. The method of claim 9, wherein the event is an issued alert regarding the absence of the first plurality of like physical objects from the vacant region and a storing of a user of the image capturing device.

17. One or more non-transitory computer readable memory media storing instructions, wherein the instructions are executable by one or more processors to:

receive, with a computing device equipped with a video analysis module and a confirmation module and in communication with a database, a first image of a vacant region within a facility, to which a first plurality of like physical objects are assigned, the first image received from a mobile device executing an application associated with the facility and including an identifier of the first plurality of like physical objects;

receive, from the mobile device, GPS coordinates of a geographic location at which the first image was taken; and

analyze the first image with the video analysis module, the video analysis module configured to;

query the database for data associated with the first plurality of like physical objects using the identifier of the first plurality of like physical objects from the first image,

perform video analytics on the first image of the vacant region using data received in response to the query,

verify a location of the vacant region based on the video analytics and the GPS coordinates,

retrieve a stored image of at least one of the first plurality of like physical objects, from the database;

compare the stored image of the at least one of the first plurality of like physical objects with the first image of the vacant region;

verify the first plurality of like physical objects are absent from the vacant region in the image based on the data associated with the first plurality of like physical objects and the location of the vacant region and the comparison of the stored image of the at least one of the first plurality of like physical objects with the first image of the vacant region; and

confirm, with the confirmation module that the first image taken by the mobile device is legitimate by capturing a second image of the mobile device taking the first image of the vacant region, the second image taken by an image capturing device affixed to a fixture of the facility, wherein an event is triggered in response to verifying that the first plurality of like physical objects are absent from the vacant region.