Withholding alerts due to temporary shelf occlusion

• US 10,452,924 B2
• Filed: 03/14/2019
• Issued: 10/22/2019
• Est. Priority Date: 01/10/2018
• Status: Active Grant

• Current Assignee: Trax Technology Solutions Pte. Ltd. (Trax Technologies (SG) Pte Ltd.)
• Original Assignee: Trax Technology Solutions Pte. Ltd. (Trax Technologies (SG) Pte Ltd.)
• Inventors: Adato, Yair, Mhabary, Ziv, Michael, Yotam, Devir, Yohai
• Primary Examiner(s): Liew, Alex Kok S
• Application Number:
  US16/353,499
• Publication Number:
  US 20190213421A1
• Time in Patent Office: 222 Days
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2115 by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 10/063112 Skill-based matching of a person or a group to a task
• CPC Class Codes: G06Q 10/06316 Sequencing of tasks or work
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 10/0875 Itemisation or classification of parts, supplies or services, e.g. bill of materials
• CPC Class Codes: G06Q 20/203 Inventory monitoring
• CPC Class Codes: G06Q 30/0629 for generating comparisons
• CPC Class Codes: G06Q 30/0639 Item locations
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 7/0002 : Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/521 : from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/17 : using hand-held instruments
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/62 : Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 20/68 : Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G07G 1/0036 : Checkout procedures
• CPC Class Codes: G08B 21/18 : Status alarms G08B21/02 takes precedence
• CPC Class Codes: G08B 21/182 : Level alarms, e.g. alarms responsive to variables exceeding a threshold
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/65 : Control of camera operation in relation to power supply
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A system for processing images to automatically identify occlusions in a field of view of one or more cameras in retail stores is provided. The system may comprise at least one processor configured to receive one or more images from an environment of a retail store, detect in the one or more images a first occlusion event, analyze the plurality of images to determine an indicator of a first level of urgency for resolving the first occlusion event, cause issuance of a user-notification associated with the first occlusion event, detect in the one or more images a second occlusion event, analyze the plurality of images to determine an indicator of a second level of urgency for resolving the second occlusion event, and when the second urgency level is lower than the first urgency level, withhold issuance of a user-notification associated with the second occlusion event.

Claims

1. A system for processing images to automatically identify occlusions in a field of view of one or more cameras in retail stores, the system comprising:

at least one processor configured to;

receive one or more images captured by one or more image sensors from an environment of a retail store and depicting a plurality of products displayed on at least one store shelf;

detect in the one or more images a first occlusion event, wherein the first occlusion event is associated with a first occluding object in the retail store;

analyze the plurality of images to determine an indicator of a first level of urgency for resolving the first occlusion event;

cause issuance of a user-notification associated with the first occlusion event, wherein the user-notification is issued within a first period of time from when the first occlusion event was detected;

detect in the one or more images a second occlusion event, wherein the second occlusion event is associated with a second occluding object in the retail store;

analyze the plurality of images to determine an indicator of a second level of urgency for resolving the second occlusion event; and

when the second urgency level is lower than the first urgency level, withhold issuance of a user-notification associated with the second occlusion event within a time duration equal to the first period of time.

2. The system of claim 1, wherein the at least one processor is configured to:

cause an issuance of a user-notification associated with the second occlusion event within a second period of time from when the occlusion event was detected, wherein the second period of time is longer than the first period of time when the second level of urgency is lower than the first level of urgency.

3. The system of claim 1, wherein the user-notification is part of a product-related task assigned to a store employee and includes at least one of:

information about a type of an occlusion event, information about an identify of an occluding object, information about an identify of an occluded product, information about a location associated with an occlusion event, and a visual depiction of the occlusion event.

4. The system of claim 1, wherein the at least one processor is further configured to detect in the one or more images the second occlusion event at a time within the first period of time from when the first occlusion event was detected.

5. The system of claim 1, wherein the indicator of the first level of urgency for resolving the first occlusion event is associated with an occlusion type of the first occlusion event.

6. The system of claim 5, wherein when the occlusion type of the first occlusion event is a customer occlusion event, the first period of time is shorter than when the occlusion type of the first occlusion event is an object occlusion event.

7. The system of claim 5, wherein when the occlusion type of the first occlusion event is a robot occlusion event, the first period of time is longer than when the occlusion type of the first occlusion event is an object occlusion event.

8. The system of claim 5, wherein when the first occlusion event and the second occlusion event are associated with a same occlusion type, the at least one processor is configured to determine the indicator of a second level of urgency for resolving the second occlusion event based on information determined about the second occluding object.

9. The system of claim 1, wherein the at least one processor is further configured to determine the indicator of the first level of urgency for resolving the first occlusion event based on a type of the first occluding object and to determine the indicator of the second level of urgency for resolving the second occlusion event based on a type of the second occluding object.

10. The system of claim 1, wherein the first and second occluding objects are persons, and wherein the at least one processor is further configured to:

determine the indicator of the first level of urgency for resolving the first occlusion event based on a determined identity of the first occluding object, and

determine the indicator of the second level of urgency for resolving the second occlusion event based on a determined identity of the second occluding object.

11. The system of claim 1, wherein the at least one processor is further configured to:

determine the indicator of the first level of urgency for resolving the first occlusion event based on detected motion of the first occluding object, and

determine the indicator of the second level of urgency for resolving the second occlusion event based on detected motion of the second occluding object.

12. The system of claim 1, wherein the at least one processor is further configured to:

determine the indicator of the first level of urgency for resolving the first occlusion event based on a determined location of the first occluding object, and

determine the indicator of the second level of urgency for resolving the second occlusion event based on a determined location of the second occluding object.

13. The system of claim 1, wherein the at least one processor is further configured to:

determine the indicator of the first level of urgency for resolving the first occlusion event based on a determined lack of people in proximity to the first occluding object, and

determine the indicator of the second level of urgency for resolving the second occlusion event based on an identification of people in proximity of the second occluding object.

14. The system of claim 1, wherein the at least one processor is further configured to determine, after the time duration, that the second occluding object continues to cause the second occlusion event, and, in response, cause issuance of a user-notification associated with the second occlusion event.

15. The system of claim 1, wherein the at least one processor is further configured to:

detect in the plurality of images a change associated with the second occlusion event;

determine the indicator of the second level of urgency for resolving the second occlusion event based on the change associated with the second occlusion event; and

cause issuance of a user-notification associated with the second occlusion within the time duration equal to the first period of time.

16. The system of claim 1, wherein the at least one processor is further configured to determine a location of the first occlusion event and to direct an employee to the determined location.

17. A method for processing images to automatically identify occlusions in a field of view of one or more cameras in retail stores, the method comprising:

receiving one or more images captured by one or more image sensors from an environment of a retail store and depicting a plurality of products displayed on at least one store shelf;

detecting in the one or more images a first occlusion event, wherein the first occlusion event is associated with a first occluding object in the retail store;

analyzing the plurality of images to determine an indicator of a first level of urgency for resolving the first occlusion event;

causing issuance of a user-notification associated with the first occlusion event, wherein the user-notification is issued within a first period of time from when the first occlusion event was detected;

detecting in the one or more images a second occlusion event, wherein the second occlusion event is associated with a second occluding object in the retail store;

analyzing the plurality of images to determine an indicator of a second level of urgency for resolving the second occlusion event; and

when the second urgency level is lower than the first urgency level, withhold issuance of a user-notification associated with the second occlusion event within a time duration equal to the first period of time.

18. The method of claim 17, further comprising:

determining the indicators of the first and second levels of urgency based on at least one of;

determined types of one or more of the first and second occluding objects, determined identities of one or more of the first and second occluding objects, detected motion associated with one or more of the first and second occluding objects, determined locations of one or more of the first and second occluding objects, or identification of people in proximity to one or more of the first and second occluding objects.

19. The method of claim 17, further comprising:

determining, after the time duration, that the second occluding object continues to cause the second occlusion event, and, in response, causing issuance of a user-notification associated with the second occlusion event.

20. A computer program product for processing images captured in retail stores embodied in a non-transitory computer-readable medium and executable by at least one processor, the computer program product including instructions for causing the at least one processor to execute a method comprising:

receiving one or more images captured by one or more image sensors from an environment of a retail store and depicting a plurality of products displayed on at least one store shelf;

detecting in the one or more images a first occlusion event, wherein the first occlusion event is associated with a first occluding object in the retail store;

analyzing the plurality of images to determine an indicator of a first level of urgency for resolving the first occlusion event;

causing issuance of a user-notification associated with the first occlusion event, wherein the user-notification is issued within a first period of time from when the first occlusion event was detected;

detecting in the one or more images a second occlusion event, wherein the second occlusion event is associated with a second occluding object in the retail store;

analyzing the plurality of images to determine an indicator of a second level of urgency for resolving the second occlusion event; and

when the second urgency level is lower than the first urgency level, withhold issuance of a user-notification associated with the second occlusion event within a time duration equal to the first period of time.