Method for the automated extraction of a planogram from images of shelving

• US 9,141,886 B2
• Filed: 03/05/2012
• Issued: 09/22/2015
• Est. Priority Date: 04/01/2011
• Status: Active Grant

• Current Assignee: CVDM Solutions
• Original Assignee: CVDM Solutions
• Inventors: Auclair, Adrien, Tousch, Anne-Marie
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US14/004,714
• Publication Number:
  US 20140003729A1
• Time in Patent Office: 1,296 Days
• Field of Search: 382/100, 382/209, 382/218, 382/219, 382/224, 382/284, 382/103, 382/159, 382/239, 382/167, 382/118, 382/135, 382/283
• US Class Current: 1/1
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables

Abstract

A method for automatically constructing a planogram from photographs of shelving, replacing laborious manual construction includes the following steps: a step (1) in which the images are received, a step (2) in which the images are assembled, a step (3) in which the structure is automatically constructed, a step (4) in which the products are automatically detected, and a step (5) in which the products are positioned in the structure. The product detection step (4) enhances traditional image recognition techniques, using artificial learning techniques to incorporate characteristics specific to the planograms. This product detection step (4) also includes at least two successive classification steps, namely: an initialization step (41) with detection of product categories; and a classification step (42) with the classification of the products themselves, each of these steps including a first image recognition step, followed by a statistical filtering step based on the characteristics specific to the planograms.

Claims

1. A method for automatic construction of a planogram, comprising:

receiving one or more shelving images,

assembling said images, in the case where there are several,

automatic construction of a structure constituting the planogram,

automatic recognition of products contained in the images,

positioning the products in the planogram according to the results of the previous automatic recognition of the products and the results of the previous detection of the structure, wherein the automatic recognition step comprises at a minimum;

a) an initial recognition step, resulting in a set of possibilities, each possibility being in a form of an ordered pair consisting of a hypothesis and a a probability, called a detection probability, that said hypothesis is true, in which the hypothesis is an ordered pair consisting of a position in the image and an identified product, wherein this initial recognition step consists of a first classification step for classifying products according to several categories, wherein the number of categories is less than the number of products, and a first filtering step of global filtering of these results by re-evaluating the detection probability of each hypothesis in each possibility using probabilistic methods for integrating information specific to the products, and

b) a second recognition step, using the resulting set of possibilities from the original recognition step to make a new detection, wherein this second recognition step consists of a second classification step based on matchings of points of interest, and a second filtering step at the planogram level that consists of reevaluating the detection probability of each hypothesis in each possibility using probabilistic methods for integrating global information on a context previously estimated on a base of planograms,

each of the first classification step, first filtering step, second classification step and second filtering step being followed immediately by a selection step of selecting the best candidates, each selection step respectively corresponding to a thresholding of the detection probabilities.

2. The method according to claim 1, to which is added, following the second recognition step, a specialised recognition step comprising a step of classification and then selection of the best candidates and a step of global filtering and then selection of the best candidates, wherein the specialised recognition step uses the image colour information.

3. The method according to claim 1, to which is added, following the second recognition step, a specialised recognition step comprising a step of classification and then selection of the best candidates and a step of global filtering and then selection of the best candidates, wherein the specialised recognition step distinguishes between products that are identical to within a few details, using visual identification algorithms.

4. The method according to claim 1, wherein the specific information used in the first filtering step of the initial recognition step are of a scale type.

5. The method according to claim 1, wherein the global information used in the second filtering step of the second recognition step comprises the following measurements:

measurements of probabilities of co-occurrences between products,

measurements of probabilities of vertical positioning of the products,

measurements of frequency of the products, previously estimated on a base of planograms.

6. The method according to claim 5, wherein in the second filtering step of the second recognition step, the integration of said global information with the detection results issuing from the second classification step comprises the following steps:

a step of estimation of the probabilities that is independent for each of the probability measurements defined in the global information,

a parameterised probabilistic combination step where the parameters are previously determined by optimisation of a global recognition level.

7. The method according to claim 1, wherein the step of automatic construction of the structure:

precedes the automatic recognition step and uses image processing techniques, combined with the results of the assembling step,

is used in calculating the global information.

8. The method according to claim 1, wherein the step of automatic construction of the structure of the planogram is performed using the results of the automatic recognition step, relying on the positions of the products detected, and wherein the calculation of the global information is independent of said structure.

9. The method according to claim 1, wherein the categories of the products used in the first classification step of the initial recognition step issue from a semi-supervised classification of the products, in which:

the supervision serves only to preserve the partitioning of the images according to the products,

a category groups together products showing the same geometric features.

10. The method according to claim 1, wherein the categories of the products used in the first classification step of the initial recognition step correspond to semantic categories of products, that is to say the first classification step is supervised.

11. The method according to claim 2, to which is added, following the second recognition step, a specialised recognition step comprising a step of classification and then selection of the best candidates and a step of global filtering and then selection of the best candidates, wherein the specialised recognition step distinguishes between products that are identical to within a few details, using visual identification algorithms.

12. The method according to claim 2, wherein the specific information used in the first filtering step of the initial recognition step are of the scale type.

13. Method The method according to claim 2, wherein the global information used in the second filtering step of the second recognition step comprises the following measurements:

measurements of probabilities of co-occurrences between products,

measurements of probabilities of vertical positioning of the products,

measurements of frequency of the products, previously estimated on a base of planograms.

14. The method according to claim 2, wherein the step of automatic construction of the structure:

precedes the automatic recognition step and uses image processing techniques, combined with the results of the assembling step,

is used in calculating the global information.

15. The method according to claim 2, wherein the step of automatic construction of the structure of the planogram is performed using the results of the automatic recognition step, relying on the positions of the products detected, and wherein the calculation of the global information is independent of said structure.

16. The method according to claim 2, wherein the categories of the products used in the first classification step of the initial recognition step issue from a semi-supervised classification of the products, in which:

the supervision serves only to preserve the partitioning of the images according to the products,

a category groups together products showing the same geometric features.

17. The method according to claim 1, wherein the categories of the products used in the first classification step of the initial recognition step correspond to semantic categories of products, that is to say the first classification step is supervised.