Interactive product auditing with a mobile device

• US 10,796,262 B2
• Filed: 09/30/2015
• Issued: 10/06/2020
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Original Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Inventors: Morate, Diego Garcia, Garcia, Antonio Hurtado
• Primary Examiner(s): Eskridge, Cory W
• Application Number:
  US14/894,901
• Publication Number:
  US 20170255891A1
• Time in Patent Office: 1,833 Days
• Field of Search: None
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Interactive product auditing with a mobile device is described. Example methods disclosed herein include performing, with an auditing device, image recognition based on a first set of candidate patterns accessed by the auditing device to identify a first product in a first region of interest of a segmented image. The disclosed example methods also include prompting, with the auditing device, a user to enter input associated with a first grid of the first region of interest displayed on a display, the first grid including the first product. The disclosed example methods further include determining, with the auditing device, a second set of candidate patterns to use to identify a second product in a second region of interest of the segmented image, the second set of candidate patterns determined based on the user input and a group of products identified in a neighborhood of the first region of interest.

Claims

1. An interactive product auditing method comprising:

performing, with a processor of an auditing device, image recognition based on a first set of candidate patterns accessed by the auditing device to identify a first product depicted in a first region of interest of a segmented image;

presenting, via a display of the auditing device, a message requesting user input associated with a first grid of the first region of interest displayed on a display of the auditing device, the first grid including the first product;

selecting, with the processor of the auditing device, a second set of candidate patterns from a pattern database based on the user input and a group of products identified in the segmented image in a neighborhood of the first region of interest;

identifying a second product in a second region of interest of the segmented image based on the second set of candidate patterns; and

displaying, on the display of the auditing device, an image-based result for the first and second products identified in the first and second regions of interest.

2. The method of claim 1, further including

modifying the image-based result based on the user input.

3. The method of claim 2, wherein the modifying of the image-based result includes:

(a) changing an identification of at least one of;

(1) the first product or (2) the second product, or (b) changing a segment of the segmented image.

4. The method of claim 1, further including;

displaying, on the display of the auditing device, key performance indicators providing data related to the first and second products;

displaying, on the display of the auditing device, a prompt to the user for a second user input while the key performance indicators are displayed on the display of the auditing device; and

modifying at least one of the key performance indicators based on the second user input.

5. The method of claim 4, wherein the key performance indicators include at least one of respective shares of shelf space or respective positions of the first and second products in the segmented image.

6. The method of claim 1, further including at least one of determining a location of the first and second products in the segmented image, determining respective numbers of facings of the first and second products in the segmented image, detecting a grid in the first region of interest that is empty, or detecting a grid in the first region of interest that is associated with an out-of-stock product.

7. The method of claim 1, wherein the first set of candidate patterns is selected based on at least one of a store being audited, a product category, or an auditor performing the audit.

8. The method of claim 1, wherein the second set of candidate patterns is further selected based on at least one of a product category or a store.

21. The method of claim 1, wherein the pattern database is stored on a remote server, and further including downloading, at the auditing device, the second set of candidate patterns from the remote server.

22. The method of claim 1, wherein the selecting of the second set of candidate patterns is further based on a product category of the group of products.

23. The method of claim 1, wherein the selecting the second set of the candidate patterns is further based on patterns identified in the first region of interest.

9. An auditing device comprising:

a product identifier to perform image recognition based on a first set of candidate patterns accessed by the auditing device to identify a first product depicted in a first region of interest of a segmented image;

a display to present a message requesting user input associated with a first grid of the first region of interest displayed on the display, the first grid including the first product;

an input/output interface to obtain the user input;

a candidate pattern selector to;

select a second set of candidate patterns from a pattern database based on the user input and a group of products identified in the segmented image in a neighborhood of the first region of interest, and

identify a second product in a second region of interest of the segmented image based on the second set of candidate patterns; and

a results analyzer to display, via the display, an image-based result for the first and second products identified in the first and second regions of interest.

10. The auditing device of claim 9, wherein the results analyzer is further to

enable the user to modify the image-based result.

11. The auditing device of claim 10, wherein the results analyzer is to modify the image-based result by:

(a) changing an identification of;

at least one of (1) the first product or (2) the second product, or (b) changing a segment of the segmented image.

12. The auditing device of claim 9, further including a key performance indicator definer to:

determine key performance indicators providing data related to the first and second products;

display the key performance indicators displayed on the display; and

prompt for a second user input to modify at least one of the key performance indicators.

13. The auditing device of claim 12, wherein the key performance indicators include at least one of respective shares of shelf space or respective positions of the first and second products in the segmented image.

14. The auditing device of claim 12, wherein the key performance indicator definer is further to:

determine a location of the first and second products in the segmented image; and

determine respective numbers of facings of the first and second products in the segmented image.

15. The auditing device of claim 9, wherein an image segmentor is further to:

detect a grid in the first region of interest that is empty; and

detect a grid in the first region of interest that is associated with an out-of-stock product.

16. The auditing device of claim 9, wherein the candidate pattern selector is further to select the first set of candidate patterns based on at least one of a store being audited, a product category, or a user performing the audit.

17. The auditing device of claim 9, wherein the candidate pattern selector is further to select the second set of candidate patterns based on at least one of a product category or a store.

18. A tangible computer readable storage medium comprising instructions that, when executed, cause a processor of an auditing device to at least:

perform image recognition based on a first set of candidate patterns accessed by the auditing device to identify a first product depicted in a first region of interest of a segmented image;

present a message requesting user input associated with a first grid of the first region of interest displayed on a display of the auditing device, the first grid including the first product;

select a second set of candidate patterns from a pattern database based on the user input and a group of products identified in the segmented image in a neighborhood of the first region of interest;

identify a second product in a second region of interest of the segmented image based on the second set of candidate patterns; and

display, on the display of the auditing device, an image-based result for the first and second products identified in the first and second regions of interest.

19. The machine-readable instructions of claim 18, wherein the instructions, when executed, further cause the processor to

modify the image-based results based on the user input.

20. The machine-readable instructions of claim 18, wherein the instructions, when executed, further cause the processor to:

display, on the display of the auditing device, key performance indicators providing data related to the first and second products;

display, on the display of the auditing device, a prompt to the user for a second user input while the key performance indicators are displayed on the display of the auditing device; and

modify at least one of the key performance indicators based on the second user input.