Monitoring a display and shelf of the same product type

• US 10,521,645 B2
• Filed: 03/15/2019
• Issued: 12/31/2019
• Est. Priority Date: 01/10/2018
• Status: Active Grant

• Current Assignee: Trax Technology Solutions Pte. Ltd. (Trax Technologies (SG) Pte Ltd.)
• Original Assignee: Trax Technology Solutions Pte. Ltd. (Trax Technologies (SG) Pte Ltd.)
• Inventors: Adato, Yair, Gottlieb, David M.
• Primary Examiner(s): Tsai, Tsung Yin
• Application Number:
  US16/354,175
• Publication Number:
  US 20190213390A1
• Time in Patent Office: 291 Days
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2115 by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 10/063112 Skill-based matching of a person or a group to a task
• CPC Class Codes: G06Q 10/06316 Sequencing of tasks or work
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 10/0875 Itemisation or classification of parts, supplies or services, e.g. bill of materials
• CPC Class Codes: G06Q 20/203 Inventory monitoring
• CPC Class Codes: G06Q 30/0629 for generating comparisons
• CPC Class Codes: G06Q 30/0639 Item locations
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 7/0002 : Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/521 : from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/17 : using hand-held instruments
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/62 : Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 20/68 : Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G07G 1/0036 : Checkout procedures
• CPC Class Codes: G08B 21/18 : Status alarms G08B21/02 takes precedence
• CPC Class Codes: G08B 21/182 : Level alarms, e.g. alarms responsive to variables exceeding a threshold
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/65 : Control of camera operation in relation to power supply
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

The present disclosure relates to systems and methods for identifying products in retail stores based on analysis of image data and for automatically generating performance indicators relating to the identified products. In one implementation, the system may include a processor configured to receive a first set of images depicting a first plurality of products associated with single product type displayed on a first shelving unit in a retail store; analyze the first set of images to determine first product turnover data associated with the first shelving unit; receive a second set of images depicting a second plurality of products also associated with the single product type displayed on a second shelving unit in the retail store nonadjacent to the first shelving unit; and analyze the second set of images to determine second product turnover data associated with the second shelving unit.

Claims

1. A system for identifying products in retail stores based on analysis of image data and for automatically generating performance indicators relating to the identified products, the system comprising:

at least one processor configured to;

receive a first set of images depicting a first plurality of products associated with single product type displayed on a first shelving unit in a retail store;

analyze the first set of images to determine first product turnover data associated with the first shelving unit;

receive a second set of images depicting a second plurality of products also associated with the single product type displayed on a second shelving unit in the retail store nonadjacent to the first shelving unit;

analyze the second set of images to determine second product turnover data associated with the second shelving unit;

automatically generate a performance indicator associated with the single product type using the first product turnover data and the second product turnover data, wherein the performance indicator is determined based on a number of return events associated with the first shelving unit and a number of return events associated with the second shelving unit; and

use image data to identify a return event where a customer placed a product on at least one of the first shelving unit and the second shelving unit.

2. The system of claim 1, wherein the performance indicator accounts for a contribution of each of the first shelving unit and the second shelving unit to overall sales of the single product type by the retail store.

3. The system of claim 2, wherein the at least one processor is further configured to:

store records of the performance indicator associated with a period of time; and

generate a report of the contribution of each of the first shelving unit and the second shelving unit to the overall sales of the single product type during the period of time.

4. The system of claim 2, wherein the at least one processor is further configured to:

determine, based on image analysis of the first and second set of images, demographics of customers that picked up the single product type; and

generate a report showing the demographics of a first group of customers that picked up the single product type from the first shelving unit and the demographics of a second group of customers that picked up the single product type from the second shelving unit.

5. The system of claim 4, wherein the at least one processor is further configured to:

determine other types of products that customers buy in addition to the single product type; and

generate a report listing types of products that the first group of customers picked up and types of products that the second group of customers picked up.

6. The system of claim 2, wherein the at least one processor is further configured to:

receive performance indicators from multiple retail stores; and

generate a report showing an average contribution of each of the first shelving unit and the second shelving unit to the overall sales of the single product type.

7. The system of claim 2, wherein the at least one processor is further configured to:

identify existence of promotions for the single product type in proximity to each of the first shelving unit and the second shelving unit, based on image analysis of the first and second set of images; and

generate a report showing promotional effectiveness on the overall sales of the single product type at each of the first shelving unit and the second shelving unit.

8. The system of claim 2, wherein the at least one processor is further configured to provide a recommendation to increase the overall sales of the single product type by changing a display parameter associated with at least one of the first shelving unit and the second shelving unit.

9. The system of claim 8, wherein the display parameter includes at least one of a display location, a display size, and adjacent product types.

10. The system of claim 1, wherein the first shelving unit is located in a different area of the retail store than the second shelving unit.

11. The system of claim 1, wherein the first shelving unit has a different display size than the second shelving unit and wherein the first shelving unit is located adjacent to at least one product type that is different from product types adjacent to the second shelving unit.

12. The system of claim 1, wherein the at least one processor is further configured to:

analyze the first set of images to recognize a first price listed on a label associated with the first shelving unit;

analyze the second set of images to recognize a second price listed on a label associated with the second shelving unit; and

inform a user when there is a price mismatch between the first price and the second price.

13. The system of claim 1, wherein the at least one processor is further configured to use image data to identify a plurality of sell events where customers picked up products associated with the single product type, and wherein the performance indicator is determined based on a number of sell events associated with the first shelving unit and a number of sell events associated with the second shelving unit.

14. The system of claim 1, wherein the at least one processor is further configured to use image data to identify a plurality of near-sell events where customers showed interest in the single product type but did not pick up a product, and wherein the performance indicator is determined based on a number of identified near-sell events associated with the first shelving unit and a number of identified near-sell events associated with the second shelving unit.

15. The system of claim 1, wherein the product is associated with the single product type.

16. The system of claim 1, wherein the product is associated with a product type other than the single product type.

17. A method identifying products in a retail store based on image analysis of captured images and for automatically generating performance indicators relating to the identified products, the method comprising:

receiving a first set of images depicting a first plurality of products associated with single product type displayed on a first shelving unit in a retail store;

analyzing the first set of images to determine first product turnover data associated with the first shelving unit;

receiving a second set of images depicting a second plurality of products also associated with the single product type displayed on a second shelving unit in the retail store nonadjacent to the first shelving unit;

analyzing the second set of images to determine second product turnover data associated with the second shelving unit;

automatically generating a performance indicator associated with the single product type using the first product turnover data and the second product turnover data, wherein the performance indicator is determined based on a number of return events associated with the first shelving unit and a number of return events associated with the second shelving unit; and

using image data to identify a return event where a customer placed a product on at least one of the first shelving unit and the second shelving unit.

18. The method of claim 17, wherein the performance indicator accounts for a contribution of each of the first shelving unit and the second shelving unit to overall sales of the single product type by the retail store.

19. A computer program product for identifying products in a retail store based on image analysis of captured images embodied in a non-transitory computer-readable medium and executable by at least one processor, the computer program product including instructions for causing the at least one processor to execute a method comprising:

receiving a first set of images depicting a first plurality of products associated with single product type displayed on a first shelving unit in a retail store;

analyzing the first set of images to determine first product turnover data associated with the first shelving unit;

receiving a second set of images depicting a second plurality of products also associated with the single product type displayed on a second shelving unit in the retail store nonadjacent to the first shelving unit;

analyzing the second set of images to determine second product turnover data associated with the second shelving unit;

automatically generating a performance indicator associated with the single product type using the first product turnover data and the second product turnover data, wherein the performance indicator is determined based on a number of return events associated with the first shelving unit and a number of return events associated with the second shelving unit; and

using image data to identify a return event where a customer placed a product on at least one of the first shelving unit and the second shelving unit.

20. A computer program product for identifying products in a retail store based on image analysis of captured images embodied in a non-transitory computer-readable medium and executable by at least one processor, the computer program product including instructions for causing the at least one processor to execute a method comprising:

receiving a first set of images depicting a first plurality of products associated with single product type displayed on a first shelving unit in a retail store;

analyzing the first set of images to determine first product turnover data associated with the first shelving unit;

receiving a second set of images depicting a second plurality of products also associated with the single product type displayed on a second shelving unit in the retail store nonadjacent to the first shelving unit;

analyzing the second set of images to determine second product turnover data associated with the second shelving unit;

automatically generating a performance indicator associated with the single product type using the first product turnover data and the second product turnover data, wherein the performance indicator is determined based on a number of identified near-sell events associated with the first shelving unit and a number of identified near-sell events associated with the second shelving unit; and

using image data to identify a plurality of near-sell events where customers showed interest in the single product type but did not pick up a product.