Withholding notifications due to temporary misplaced products

• US 10,628,660 B2
• Filed: 03/14/2019
• Issued: 04/21/2020
• Est. Priority Date: 01/10/2018
• Status: Active Grant

• Current Assignee: Trax Technology Solutions Pte. Ltd. (Trax Technologies (SG) Pte Ltd.)
• Original Assignee: Trax Technology Solutions Pte. Ltd. (Trax Technologies (SG) Pte Ltd.)
• Inventors: Adato, Yair, Bronicki, Youval, Mhabary, Ziv, Pomeranz, Dolev
• Primary Examiner(s): Gart, Matthew S
• Assistant Examiner(s): Holzmacher, Derick J
• Application Number:
  US16/353,452
• Publication Number:
  US 20190213534A1
• Time in Patent Office: 404 Days
• Field of Search: None
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2115 by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 10/063112 Skill-based matching of a person or a group to a task
• CPC Class Codes: G06Q 10/06316 Sequencing of tasks or work
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 10/0875 Itemisation or classification of parts, supplies or services, e.g. bill of materials
• CPC Class Codes: G06Q 20/203 Inventory monitoring
• CPC Class Codes: G06Q 30/0629 for generating comparisons
• CPC Class Codes: G06Q 30/0639 Item locations
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 7/0002 : Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/521 : from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/17 : using hand-held instruments
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/62 : Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 20/68 : Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G07G 1/0036 : Checkout procedures
• CPC Class Codes: G08B 21/18 : Status alarms G08B21/02 takes precedence
• CPC Class Codes: G08B 21/182 : Level alarms, e.g. alarms responsive to variables exceeding a threshold
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/65 : Control of camera operation in relation to power supply
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A system for processing images captured in a retail store and automatically identifying misplaced products is provided. The system may comprise at least one processor configured to receive one or more images captured by one or more image sensors from an environment of a retail store, detect in the one or more images a first product, determine that the first product is not located in the first correct display location, cause an issuance of a user-notification associated with the first product, detect in the one or more images a second product, determine that the second product is not located in the second correct display location, and after determining that the second product is not located in the second correct display location and when the second urgency level is lower than the first urgency level, withhold issuance of a user-notification associated with the second product.

Claims

1. A system for processing images captured in a retail store and automatically identifying misplaced products, the system comprising:

at least one processor configured to;

receive one or more images captured by one or more image sensors from an environment of a retail store and depicting a plurality of products displayed on at least one store shelf;

detect in the one or more images a first product, wherein the first product has an associated first correct display location, and wherein detecting the first product comprises;

retrieving, from a database, a product model associated with the first product, the product model including parameters for a neural network configured to identify the first product; and

detecting the first product based on the neural network and the product model;

determine, based on analysis of the one or more images, that the first product is not located in the first correct display location;

determine an indicator of a first level of urgency for returning the first product to the first correct display location, wherein the indicator of the first level of urgency for returning the first product to the first correct display location is associated with a product type of the first product;

cause an issuance of a user-notification associated with the first product, wherein the user-notification is issued within a first period of time from when the first product was determined not to be located in the first correct display location, and wherein when the product type includes frozen food, the first period of time is shorter than when the product type includes fresh produce;

detect in the one or more images a second product, wherein the second product has an associated second correct display location;

determine, based on analysis of the one or more images, that the second product is not located in the second correct display location;

determine an indicator of a second level of urgency for returning the second product to its associated second correct display location; and

after determining that the second product is not located in the second correct display location and when the second urgency level is lower than the first urgency level, withhold issuance of a user-notification associated with the second product within a time duration equal to the first period of time.

2. The system of claim 1, wherein the at least one processor is configured to:

cause an issuance of a user-notification associated with the second product within a second period of time from when the second product was determined not to be located in the second correct display location, wherein the second period of time is longer than the first period of time when the second level of urgency is lower than the first level of urgency.

3. The system of claim 1, wherein the user-notification is included in a product-related task assigned to a store employee and includes at least one of:

information about a correct display location of a misplaced product, information about a store shelf associated with the misplaced product, information about a product type of the misplaced product, or a visual depiction of the misplaced product.

4. The system of claim 1, wherein the at least one processor is configured to detect in the one or more images the second product at a time within the first period of time from when the first product was determined not to be located in the first correct display location.

5. The system of claim 1, wherein when the product type includes canned goods, the first period of time is longer than when the product type includes fresh produce.

6. The system of claim 1, wherein the at least one processor is further configured to determine, based on context information derived through analysis of the one or more images, whether the first product determined not to be located in the first correct display requires repositioning.

7. The system of claim 6, wherein the at least one processor is configured to forego causing the issuance of the user-notification after determining that the first product does not require repositioning.

8. The system of claim 6, wherein the at least one processor is configured to determine that the first product does not require repositioning when the context information derived from analysis of the one or more images indicates that the first product is located in the hand of a person.

9. The system of claim 6, wherein the at least one processor is configured to determine that the first product does not require repositioning when the context information derived from analysis of two or more captured images indicates that the first product is moving.

10. The system of claim 6, wherein the at least one processor is configured to determine that the first product requires repositioning when the context information derived from analysis of the one or more images indicates an absence of people located in a vicinity of the first product, and wherein the at least one processor is configured to determine that the first product does not require repositioning when the context information derived from analysis of the one or more images indicates a presence of at least one person located in a vicinity of the first product.

11. The system of claim 10, wherein the at least one person is determined, based on an analysis of the one or more images, to have at least one characteristic associated with a customer.

12. The system of claim 1, wherein the at least one processor is further configured to determine, after the time duration, that the second product remains in a location other than the second correct display location, and cause issuance of a user-notification associated with the second product.

13. The system of claim 1, wherein the at least one processor is configured to determine the first and second correct display locations based on product types for the first and second products determined based on analysis of the one or more images.

14. The system of claim 13, wherein the first and second correct display locations are determined based on information including at least one of:

a planogram, a store map, a database, a label associated with the at least one store shelf, other products associated with the at least one store shelf, and an aisle associated with at least one store shelf.

15. The system of claim 1, wherein the at least one processor is further configured to determine the first correct display location for the first product and to include in the user-notification associated with the first product an instruction for use in re-locating the first product to the associated first correct display location.

16. A computer-implemented method for processing images captured in a retail store and automatically identifying misplaced products, the method comprising:

receiving, by at least one processor, one or more images captured by one or more image sensors from an environment of a retail store and depicting a plurality of products displayed on at least one store shelf;

detecting, by the at least one processor, in the one or more images a first product, wherein the first product has an associated first correct display location, and wherein detecting the first product comprises;

retrieving, from a database, a product model associated with the first product, the product model including parameters for a neural network configured to identify the first product; and

detecting the first product based on the neural network and the product model;

determining, by the at least one processor and based on analysis of the one or more images, that the first product is not located in the first correct display location;

determining an indicator of a first level of urgency for returning the first product to the associated first correct display location, wherein the indicator of the first level of urgency for returning the first product to the first correct display location is associated with a product type of the first product;

causing an issuance of a user-notification associated with the first product, wherein the user-notification is issued within a first period of time from when the first product was determined not to be located in the first correct display location, and wherein when the product type includes frozen food, the first period of time is shorter than when the product type includes fresh produce;

detecting in the one or more images a second product, wherein the second product has an associated second correct display location;

determining, based on analysis of the one or more images, that the second product is not located in the second correct display location;

determining an indicator of a second level of urgency for returning the second product to its associated second correct display location; and

after determining that the second product is not located in the second correct display location and when the second urgency level is lower than the first urgency level, withholding issuance of a user-notification associated with the second product within a time duration equal to the first period of time.

17. The method of claim 16, further comprising:

causing an issuance of a user-notification associated with the second product within a second period of time from when the second product was determined not to be located in the second correct display location, wherein the second period of time is longer than the first period of time when the second level of urgency is lower than the first level of urgency.

18. The method of claim 16, further comprising:

determining, based on context information derived through analysis of the one or more images, whether the first product determined not to be located in the first correct display requires repositioning, wherein the context information indicates at least one of;

the first product is located in a hand of a person, the first product is moving, or the first product is located in a vicinity of at least one person.

19. A computer program product for processing images captured in a retail store and automatically identifying misplaced products, the computer program being embodied in a non-transitory computer-readable medium and executable by at least one processor, the computer program product including instructions for causing the at least one processor to execute a method comprising:

receiving one or more images captured by one or more image sensors from an environment of a retail store and depicting a plurality of products displayed on at least one store shelf;

detecting in the one or more images a first product, wherein the first product has an associated first correct display location, and wherein detecting the first product comprises;

retrieving, from a database, a product model associated with the first product, the product model including parameters for a neural network configured to identify the first product; and

detecting the first product based on the neural network and the product model;

determining, based on analysis of the one or more images, that the first product is not located in the first correct display location;

determining an indicator of a first level of urgency for returning the first product to the associated first correct display location, wherein the indicator of the first level of urgency for returning the first product to the first correct display location is associated with a product type of the first product;

causing an issuance of a user-notification associated with the first product, wherein the user-notification is issued within a first period of time from when the first product was determined not to be located in the first correct display location, and wherein when the product type includes frozen food, the first period of time is shorter than when the product type includes fresh produce;

detecting in the one or more images a second product, wherein the second product has an associated second correct display location;

determining, based on analysis of the one or more images, that the second product is not located in the second correct display location;

determining an indicator of a second level of urgency for returning the second product to its associated second correct display location; and

after determining that the second product is not located in the second correct display location and when the second urgency level is lower than the first urgency level, withholding issuance of a user-notification associated with the second product within a time duration equal to the first period of time.