METHOD FOR THE AUTOMATED EXTRACTION OF A PLANOGRAM FROM IMAGES OF SHELVING

• US 20140003729A1
• Filed: 03/05/2012
• Published: 01/02/2014
• Est. Priority Date: 04/01/2011
• Status: Active Grant

• Current Assignee: CVDM Solutions
• Original Assignee: CVDM Solutions
• Inventors: Auclair, Adrien, Tousch, Anne-Marie
• Granted Patent: US 9,141,886 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables

Abstract

A method for automatically constructing a planogram from photographs of shelving, replacing laborious manual construction includes the following steps: a step (1) in which the images are received, a step (2) in which the images are assembled, a step (3) in which the structure is automatically constructed, a step (4) in which the products are automatically detected, and a step (5) in which the products are positioned in the structure. The product detection step (4) enhances traditional image recognition techniques, using artificial learning techniques to incorporate characteristics specific to the planograms. This product detection step (4) also includes at least two successive classification steps, namely: an initialization step (41) with detection of product categories; and a classification step (42) with the classification of the products themselves, each of these steps including a first image recognition step, followed by a statistical filtering step based on the characteristics specific to the planograms.

Claims

1. Method for the automatic construction of a planogram, characterised in that it comprises:

the reception of one or more shelving images (1),

a step of assembling (2) said images, in the case where there are several,

a step of automatic construction of the structure (3) constituting the planogram,

a step of automatic recognition (4) of the products contained in the images,

a step of positioning the products (5) according to the results of the previous automatic recognition of the products (4) and the results of the previous detection of the structure (3),

the automatic recognition step (4) being itself characterised in that it comprises at a minimum the following two steps;

a) an initial recognition step (41), giving a set of possibilities, in the form of ordered pairs (hypothesis, probability of detection), in which a hypothesis is an ordered pair (position in the image, identified product). This step (41) consists of a category classification step (4171) and a step (4173) of global filtering of these results using probabilistic methods for integrating information specific to the products (I12),

b) a second recognition step (42), using the results of (41) to make a new detection, consisting of a classification step (4271) based on matchings of points of interest, and a second filtering step (4273) at the planogram level, integrating global information on the context (I2) previously estimated on a base of planograms (I3),

each of the classification and filtering steps (4171, 4173, 4271, 4273) being followed immediately by a step of selecting the best candidates, respectively (4172, 4174, 4272, 4274) corresponding to a thresholding of the detection probabilities.

2. Method according to claim 1, to which is added, following the classification step (42), a specialised recognition step (43) of type (7), that is to say comprising a step (71) of classification and then selection of the best candidates (72) and a step of global filtering (73) and then selection of the best candidates (74), characterised in that the specialised recognition step (43) uses the image colour information.

3. Method according to claim 1, to which is added, following the classification step (42), a specialised recognition step (43) of type (7), that is to say comprising a step (71) of classification and then selection of the best candidates (72) and a step (73) of global filtering and then selection of the best candidates (74), characterised in that the specialised recognition step (43) distinguishes between products that are identical to within a few details, using visual identification algorithms.

4. Method according to claim 1, characterised in that the specific information (I12) used in the filtering step (4173) of the initial recognition (41) are of the scale type.

5. Method according to claim 1, characterised in that the global information (I2) used in the filtering step (4273) of the second recognition step (42) comprise the following measurements:

measurements of probabilities of co-occurrences between products,

measurements of probabilities of vertical positioning of the products,

measurements of frequency of the products, previously estimated on a base of planograms (I3).

6. Method according to claim 5, characterised in that, in the filtering step (4273) of the second recognition step (42), the integration of said global information (I2) with the detection results issuing from the classification step (4271) comprises the following steps:

a step of estimation of the probabilities that is independent for each of the probability measurements defined in the global information (I2),

a parameterised probabilistic combination step of type (B), where the parameters are previously determined by optimisation of a global recognition level.

7. Method according to claim 1, characterised in that the step of automatic construction of the structure (3):

precedes the recognition step (4) and uses image processing techniques (1), combined with the results of the assembly (2),

is used in calculating the global information (I2).

8. Method according to claim 1, characterised in that the step of automatic construction of the structure of the planogram (3) is performed using the results of the recognition step (4), relying on the positions of the products detected, and in that the calculation of the global information (I2) is independent of said structure (3O).

9. Method according to claim 1, characterised in that the categories of the products used at (4171) issue from a semi-supervised classification of the products, in which:

the supervision serves only to preserve the partitioning of the images according to the products,

a category groups together products showing the same geometric features.

10. Method according to claim 1, characterised in that the categories of the products used at (4171) correspond to semantic categories of products (family, segment, sub-segment, etc.), that is to say the classification (4171) is supervised.

11. Method according to claim 2, to which is added, following the classification step (42), a specialised recognition step (43) of type (7), that is to say comprising a step (71) of classification and then selection of the best candidates (72) and a step (73) of global filtering and then selection of the best candidates (74), characterised in that the specialised recognition step (43) distinguishes between products that are identical to within a few details, using visual identification algorithms.

12. Method according to claim 2, characterised in that the specific information (I12) used in the filtering step (4173) of the initial recognition (41) are of the scale type.

13. Method according to claim 2, characterised in that the global information (I2) used in the filtering step (4273) of the second recognition step (42) comprise the following measurements:

measurements of probabilities of co-occurrences between products,

measurements of probabilities of vertical positioning of the products,

measurements of frequency of the products, previously estimated on a base of planograms (I3).

14. Method according to claim 2, characterised in that the step of automatic construction of the structure (3):

precedes the recognition step (4) and uses image processing techniques (1), combined with the results of the assembly (2),

is used in calculating the global information (I2).

15. Method according to claim 2, characterised in that the step of automatic construction of the structure of the planogram (3) is performed using the results of the recognition step (4), relying on the positions of the products detected, and in that the calculation of the global information (I2) is independent of said structure (3O).

16. Method according to claim 2, characterised in that the categories of the products used at (4171) issue from a semi-supervised classification of the products, in which:

the supervision serves only to preserve the partitioning of the images according to the products,

a category groups together products showing the same geometric features.

17. Method according to claim 2, characterised in that the categories of the products used at (4171) correspond to semantic categories of products (family, segment, sub-segment, etc.), that is to say the classification (4171) is supervised.