Inspecting store shelf capacity

• US 10,643,059 B2
• Filed: 03/14/2019
• Issued: 05/05/2020
• Est. Priority Date: 01/10/2018
• Status: Active Grant

• Current Assignee: Trax Technology Solutions Pte. Ltd. (Trax Technologies (SG) Pte Ltd.)
• Original Assignee: Trax Technology Solutions Pte. Ltd. (Trax Technologies (SG) Pte Ltd.)
• Inventors: Adato, Yair, Mhabary, Ziv, Michael, Yotam, Yudkin, Paul
• Primary Examiner(s): Masud, Rokib
• Application Number:
  US16/353,621
• Publication Number:
  US 20190213535A1
• Time in Patent Office: 418 Days
• Field of Search: 382141, 705 26, 705 28
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2115 by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 10/063112 Skill-based matching of a person or a group to a task
• CPC Class Codes: G06Q 10/06316 Sequencing of tasks or work
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 10/0875 Itemisation or classification of parts, supplies or services, e.g. bill of materials
• CPC Class Codes: G06Q 20/203 Inventory monitoring
• CPC Class Codes: G06Q 30/0629 for generating comparisons
• CPC Class Codes: G06Q 30/0639 Item locations
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 7/0002 : Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/521 : from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/17 : using hand-held instruments
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/62 : Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 20/68 : Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G07G 1/0036 : Checkout procedures
• CPC Class Codes: G08B 21/18 : Status alarms G08B21/02 takes precedence
• CPC Class Codes: G08B 21/182 : Level alarms, e.g. alarms responsive to variables exceeding a threshold
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/65 : Control of camera operation in relation to power supply
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A system for processing images captured in a retail store is provided. The system may comprise at least one processor configured receive at least one image depicting part of at least one store shelf having a plurality of products displayed thereon, the plurality of products including a group of first products associated with a first product size and a group of second products associated with a second product size, analyze the at least one image to detect the plurality of products, identify one or more vacant spaces associated with the plurality of products, determine that by rearranging the group of second products to partially eliminate the one or more identified vacant spaces, at least one additional first product may be displayed on the at least one store shelf next to the group of first products, and provide information to a user indicative of the identified vacant spaces.

Claims

1. A system for processing images captured in a retail store, the system comprising:

at least one processor configured to;

receive at least one image depicting at least part of at least one store shelf having a plurality of products displayed thereon, wherein the plurality of products includes a group of first products associated with a first product size and a group of second products associated with a second product size, the second product size differing from the first product size;

analyze, based on at least one image analysis technique, the at least one image to detect the plurality of products;

identify, based on the image analysis, one or more vacant spaces associated with the plurality of products on the at least one store shelf;

based on the one or more identified vacant spaces, determine that by rearranging the group of second products on the at least one store shelf to at least partially eliminate the one or more identified vacant spaces, at least one additional first product may be displayed on the at least one store shelf next to the group of first products; and

based on the determination that at least one additional first product may be displayed, provide information to a user indicative of the identified vacant spaces.

2. The system of claim 1, wherein the identified vacant spaces associated with the plurality of products on the at least one store shelf include at least one of:

horizontal spaces between adjacent first products, horizontal spaces between adjacent second products, a horizontal space between a first and a second product, a horizontal space between a first product and a frame associated with the at least one store shelf, and a horizontal space between a second product and a frame associated with the at least one store shelf.

3. The system of claim 1, wherein the identified vacant spaces associated with the plurality of products on the at least one store shelf include at least one of:

vertical spaces between first products and a frame associated with the at least one store shelf, and vertical spaces between second products and a frame associated with the at least one store shelf.

4. The system of claim 1, wherein, when a width associated with first products is narrower than a width associated with second products, the at least one processor is configured to determine that a first product may be added horizontally to the group of first products based on identified horizontal spaces between two or more second products.

5. The system of claim 4, wherein the at least one processor is further configured to:

aggregate the identified horizontal spaces between two or more second products;

compare the aggregated spaces to the width associated with first products to identify a potential for rearrangement; and

assign the user, based on the identified potential for rearrangement, a product-related task for rearranging the group of second products to add an additional first product horizontally to the group of first products.

6. The system of claim 1, wherein the at least one processor is further configured to:

identify that at least some of the group of first products are displayed in a nonstandard orientation; and

determine that by rearranging the group of second products on the at least one store shelf at least one additional first product may be displayed in a standard position.

7. The system of claim 1, wherein the at least one processor is further configured to identify the vacant spaces on the at least one store shelf by analyzing the at least one image.

8. The system of claim 1, wherein the at least one processor is further configured to identify the vacant spaces on the at least one store shelf based on input from one or more sensors positioned on the at least one store shelf.

9. The system of claim 8, wherein the one or more sensors include pressure detectors, and the input is indicative of pressure levels detected by the pressure detectors when one or more products are placed on the at least one store shelf.

10. The system of claim 8, wherein the one or more sensors include light detectors, and the input is indicative of light measurements made with respect to one or more products placed on the at least store shelf.

11. The system of claim 1, wherein the at least one processor is further configured to:

identify a plurality of possible rearrangements of the plurality of products based on the identified vacant spaces;

access product assortment rules associated with the plurality of products; and

determine a recommended rearrangement based on the product assortment rules, wherein the information provided to the user is part of a product-related task assigned to a store employee and includes details of the recommended rearrangement.

12. The system of claim 1, wherein the at least one processor is further configured to generate a visualization of a recommended rearrangement, and wherein the information provided to the user is part of a product-related task assigned to a store employee and includes the generated visualization of the recommended rearrangement.

13. The system of claim 1, wherein the at least one processor is further configured to generate a product-related task associated with an identified potential for product rearrangement, and wherein the information provided to the user includes identification of a location of the at least one store shelf where the potential for product rearrangement exists.

14. The system of claim 1, wherein the at least one processor is further configured to:

measure a metric associated with identified vacant spaces related to at least one product type across a plurality of shelves in the retail store;

compare the measured metric with metrics associated with identified vacant spaces for the at least one product type in other retail stores; and

rank the retail store based on the comparison.

15. The system of claim 14, wherein the at least one product type is associated with at least one of:

a product category, a brand of products, and a region of the retail store.

16. The system of claim 1, wherein the at least one processor is further configured to:

receive a plurality of images;

identify an area of the retail store displaying products from at least one product type associated with at least one of;

a product category, a brand of products, and a region of the retail store;

identify vacant spaces across a plurality of shelves in the area of the retail store; and

based on the identified vacant spaces, determine that by rearranging products in the area of the retail store, at least two or more additional products from at least one product type may be displayed in the area of the retail store.

17. The system of claim 1, wherein the image analysis technique includes at least one of object recognition, object detection, image segmentation, feature extraction, optical character recognition (OCR), object-based image analysis, shape region techniques, edge detection techniques, pixel-based detection, artificial neural networks, and convolutional neural networks.

18. A system for processing images captured in a retail store, the system comprising:

at least one processor configured to;

receive at least one image depicting at least part of a shelving unit with a plurality of store shelves having a plurality of products displayed thereon;

analyze, based on at least one image analysis technique, the at least one image to detect the plurality of products;

identify, based on the image analysis, one or more vertical vacant spaces associated with the shelving unit;

based on the one or more identified vacant spaces, determine that by rearranging the plurality of products, at least one shelf may be added to the shelving unit; and

based on the determination that at least one shelf may be added to the shelving unit, provide information to a user indicative of the identified vacant spaces.

19. The system of claim 18, wherein the at least one processor is further configured to:

aggregate identified vertical spaces associated with a plurality of store shelves;

compare the aggregated space to an average height of the products to identify a potential for rearrangement; and

assign the user a product-related task for rearranging the shelving unit to add the at least one shelf.

20. A method for processing images captured in a retail store, the method comprising:

receiving at least one image depicting at least part of at least one store shelf having a plurality of products displayed thereon, wherein the plurality of products includes a group of first products associated with a first product size and a group of second products associated with a second product size, the second product size differing from the first product size;

analyzing, based on at least one image analysis technique, the at least one image to detect the plurality of products;

identifying, based on the image analysis, one or more vacant spaces associated with the plurality of products on the at least one store shelf;

based on the one or more identified vacant spaces, determining that by rearranging the group of second products on the at least one store shelf to at least partially eliminate the one or more identified vacant spaces, at least one additional first product may be displayed on the at least one store shelf next to the group of first products; and

based on the determination that at least one additional first product may be displayed, providing information to a user indicative of the identified vacant spaces.

21. The method of claim 20, further comprising:

aggregating identified horizontal spaces between two or more second products;

comparing the aggregated space to an average width of first products to identify a potential for rearrangement;

determining that a first product may be added horizontally to the group of first products; and

assigning the user, based on the identified potential for rearrangement, a product-related task for rearranging the group of second products to add an additional first product horizontally to the group of first products.

22. A computer program product for processing images captured in retail stores embodied in a non-transitory computer-readable medium and executable by at least one processor, the computer program product including instructions for causing the at least one processor to execute a method comprising:

receiving at least one image depicting at least part of at least one store shelf having a plurality of products displayed thereon, wherein the plurality of products includes a group of first products associated with a first product size and a group of second products associated with a second product size, the second product size differing from the first product size;

analyzing, based on at least one image analysis technique, the at least one image to detect the plurality of products;

identifying, based on the image analysis, one or more vacant spaces associated with the plurality of products on the at least one store shelf;

based on the one or more identified vacant spaces, determining that by rearranging the group of second products on the at least one store shelf to at least partially eliminate the one or more identified vacant spaces, at least one additional first product may be displayed on the at least one store shelf next to the group of first products; and

based on the determination that at least one additional first product may be displayed, providing information to a user indicative of the identified vacant spaces.