IMAGE PROCESSING OF A RETAIL SHELF AREA

• US 20160224857A1
• Filed: 01/30/2015
• Published: 08/04/2016
• Est. Priority Date: 01/30/2015
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Zhang, Tong, Fan, Jian
• Granted Patent: US 9,710,926 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/457 by analysing connectivity, e.g. edge linking, connected component analysis or slices
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A system and method for processing an image of a retail shelf area. According to one aspect, a portion of an image that depicts the retail shelf area at an angled and depthwise perspective is determined to depict shelves. The image can be modified so that the portion of the image determined to depict shelves is substantially parallel to a predefined horizontal reference frame. Products can then be detected in the remainder portion of the image, separate from the portion of the image determined to depict shelves. An applicable planogram can be used to generate compliance information based on the detected products and the determined portion of the image depicting shelves.

Claims

1. A method for processing an image of a retail shelf area, the method being implemented by one or more processors and comprising:

receiving the image of the retail shelf area, the image depicting the retail shelf area at an angled and depthwise perspective in relation to the point of image capture;

determining a portion of the image that depicts one or more shelves in the retail shelf area; and

detecting one or more products in a remainder portion of the image, separate from the portion of the image determined to depict the one or more shelves.

2. The method of claim 1, further comprising:

based on the determination of the portion of the image depicting the one or more shelves, modifying the image so that the portion of the image depicting the one or more shelves is substantially parallel to a predefined horizontal reference frame.

3. The method of claim 1, wherein detecting the one or more products in the remainder portion of the modified image includes:

identifying, in the remainder portion, a plurality of candidate image portions for depicting the one or more products; and

analyzing each of the plurality of candidate image portions to determine the one or more products based on a plurality of stored product images.

4. The method of claim 1, further comprising:

identifying an applicable planogram for the retail shelf area in the image;

identifying one or more discrepancies in the applicable planogram based on the detected one or more products and the determination of the portion of the image depicting the one or more shelves; and

generating information regarding compliance of the applicable planogram based on the identified one or more discrepancies.

5. The method of claim 1, wherein the detecting one or more products in a remainder portion of the modified image includes:

identifying an applicable planogram for the retail shelf area in the image; and

determining one or more products in the remainder portion of the modified image based on the applicable planogram.

6. The method of claim 3, wherein the analyzing each of the plurality of candidate image portions is based on a dominant color in each of the plurality of candidate images.

7. The method of claim 3, wherein the analyzing each of the plurality of candidate image portions is based on a number of key points identifiable in each of the plurality of candidate images.

8. The method of claim 3, wherein the analyzing each of the plurality of candidate image portions includes:

identifying a dominant color in each of the plurality of candidate image portions; and

determining a shelved product in each of the plurality of candidate image portions from the plurality of the stored product images after excluding the stored product images that do not contain the dominant color.

9. The method of claim 3, wherein analyzing each of the plurality of candidate image portions includes:

identifying a number of key points in each of the candidate image portions; and

determining a shelved product in each of the plurality of candidate image portions from the plurality of stored product images based on the identified key points.

10. The method of claim 5, wherein the determining one or more products in the remainder portion of the modified image based on the applicable planogram includes:

identifying one or more target products in the applicable planogram;

identifying, in the remainder portion, a plurality of candidate image portions for depicting the one or more products; and

analyzing each of the plurality of candidate image portions to determine the one or more products, from the stored product images in the plurality of stored product images that correspond to the one or more target products.

11. The method of claim 8, wherein the determining a shelved product in each of the plurality of candidate image portions from the plurality of the stored product images after excluding the stored product images that do not contain the dominant color includes:

identifying a number of key points in each of the candidate image portions; and

based on the identified key points, determining the shelved product in each of the plurality of candidate image portions from the plurality of stored product images after excluding the stored product images that do not contain the dominant color.

12. The method of claim 9, wherein the number of key points is a predetermined threshold number.

13. The method of claim 10, wherein the analyzing each of the plurality of candidate image portions includes:

identifying a dominant color in each of the plurality of candidate image portions;

identifying a number of key points in each of the candidate image portions; and

based on the identified key points, determining a shelved product in each of the plurality of candidate image portions, from the stored product images in the plurality of stored product images that correspond to the one or more target products, after excluding the stored product images that do not contain the dominant color.

14. A system for processing an image of a retail shelf area, comprising:

a memory resource to store instructions; and

one or more processors using the instructions stored in the memory resource to;

receive the image of the retail shelf area, the image depicting the retail shelf area at an angled and depthwise perspective in relation to the point of image capture;

determine a portion of the image that depicts one or more shelves in the retail shelf area;

based on the determination of the portion of the image depicting the one or more shelves, modify the image so that the portion of the image depicting the one or more shelves is substantially parallel to a predefined horizontal reference frame;

detect one or more products in a remainder portion of the modified image, separate from the portion of the image determined to depict the one or more shelves; and

generate information regarding compliance of an applicable planogram based on the detected one or more products and the determination of the portion of the image depicting the one or more shelves.

15. A non-transitory computer-readable medium for processing an image of a retail shelf area, the computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations that include:

capturing the image of the retail shelf area, the image depicting the retail shelf area at an angled and depthwise perspective in relation to the point of image capture;

determining a portion of the image that depicts one or more shelves in the retail shelf area;

based on the determination of the portion of the image depicting the one or more shelves, modifying the image so that the portion of the image depicting the one or more shelves is substantially parallel to a predefined horizontal reference frame;

detecting one or more products in a remainder portion of the modified image, separate from the portion of the image determined to depict the one or more shelves; and

generating information regarding compliance of an applicable planogram based on the detected one or more products and the determination of the portion of the image depicting the one or more shelves.