Identifying consumer products in images

• US 9,990,561 B2
• Filed: 11/23/2015
• Issued: 06/05/2018
• Est. Priority Date: 11/23/2015
• Status: Active Grant

• Current Assignee: Lexmark International Incorporated (Ninestar Corporation)
• Original Assignee: Lexmark International Incorporated (Ninestar Corporation)
• Inventors: Booth, Robert Reed
• Primary Examiner(s): DULANEY, KATHLEEN YUAN
• Application Number:
  US14/948,505
• Publication Number:
  US 20170147900A1
• Time in Patent Office: 925 Days
• Field of Search: 382205, 382141, 382203
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/09 Recognition of logos

Abstract

Systems and methods identify consumer products in images. Known consumer products are captured as grayscale or color images. They are converted to binary at varying thresholds. Connected components in the binary images identify image features according to pixels of a predetermined size, shape, solidity, aspect ratio, and the like. The image features are stored and searched for amongst image features similarly extracted from unknown images of consumer products. Identifying correspondence between the features of the images lends itself to identifying or not known consumer products.

Claims

1. In a grayscale or color image having pluralities of pixels corresponding to a consumer product, a method of normalizing image features thereof, comprising:

converting to binary the pluralities of pixels of the grayscale or color image according to three different thresholds to obtain three binary images each having a plurality of binary pixels;

identifying a plurality of connected components in the plurality of binary pixels of each of the three binary images, each of the connected components being a subset of the plurality of binary pixels of the three binary images;

filtering the plurality of connected components in each of the three binary images to exclude from a set of image features of each of the three binary images one or more connected components from the plurality of connected components not meeting certain predefined limits based on a predefined size of each of the plurality of connected components; and

in the set of image features, standardizing a size of each image feature to be equal to other image features of the set of image features,

wherein the standardizing the size further includes making said each of the image features into a 30×

30 pixel region.

2. The method of claim 1, further including determining a centroid of said each of the image features in the three binary images.

3. The method of claim 2, further including determining a pixel distance away from the centroid for each pixel of said each of the image features.

4. The method of claim 3, further including determining an average pixel distance away from said centroid for said each of the image features.

5. The method of claim 1, further including filtering the plurality of connected components in said each of the three binary images to exclude from the set of image features of the binary images the connected components not meeting a predefined aspect ratio, a predefined shape or a predefined limit of solidity in the subset of binary pixels.

6. The method of claim 1, wherein the converting to binary further includes defining the three different thresholds as a threshold of twenty-five percent, fifty percent, and seventy-five percent of pixel intensity values of the pluralities of pixels in the grayscale or color image.

7. The method of claim 6, wherein the converting to binary further includes defining as white pixels those with light pixel intensities relative to the three different thresholds in the pluralities of pixels in the grayscale or color image and as black pixels those with dark pixel intensities relative to the three different thresholds in the pluralities of pixels in the grayscale or color image.

8. The method of claim 1, wherein a first set of images correspond to known consumer products and a second set of images correspond to unknown consumerproducts, each image of the first and second sets of images having been converted into said three binary images, and further including finding three image features in the set of image features corresponding to one another in each of the three binary images corresponding to each image of the first and second sets of images.

9. The method of claim 8, wherein the finding the three image features includes restricting a pixel distance between the three image features.

10. The method of claim 8, further including determining whether or not the three image features reside in a corresponding portion of the first and second sets of images.

11. The method of claim 8, further including determining whether or not the three image features have a corresponding orientation in the first and second sets of images.

12. The method of claim 8, further including determining whether or not the three image features correspond mostly to black or white image features.

13. The method of claim 8, further including determining an aspect ratio in pixels for the three image features in the first and second sets of images.

14. The method of claim 8, wherein each of the binary pixels are defined in bits and the finding of the three image features corresponding to one another further includes XORing bits of two of the three image features one each from the first and second sets of images and counting a number of 1 bits, wherein a match between the two of the three image features exists if the number of the 1 bits does not exceed ten percent of a total number of the bits in the XORing operation.

15. The method of claim 8, further including devising triangles between the three image features in the set of image features corresponding to one another in said each of the three binary images corresponding to each image of the first and second sets of images and determining similarity between the triangles to identify or not possible matches of the known consumer products of the first set of images within the second set of images of the unknown consumer products.

16. In a grayscale or color image having pluralities of pixels corresponding to a consumer product, a method of normalizing image features thereof, comprising:

converting to binary the pluralities of pixels of the grayscale or color image according to three different thresholds to obtain three binary images each having a plurality of binary pixels;

identifying a plurality of connected components in the plurality of binary pixels of each of the three binary images, each of the connected components being a subset of the plurality of binary pixels of the three binary images;

filtering the plurality of connected components in each of the three binary images to exclude from a set of image features of each of the three binary images one or more connected components from the plurality of connected components not meeting certain predefined limits based on a predefined size of each of the plurality of connected components; and

in the set of image features, standardizing a size of each image feature to be equal to other image features of the set of image features,

wherein the filtering the connected components further includes excluding from the set of image features the connected components not having a minimum pixel width or pixel height of at least eighteen pixels.

17. In a grayscale or color image having pluralities of pixels corresponding to a consumer product, a method of normalizing image features thereof, comprising:

converting to binary the pluralities of pixels of the grayscale or color image according to three different thresholds to obtain three binary images each having a plurality of binary pixels;

identifying a plurality of connected components in the plurality of binary pixels of each of the three binary images, each of the connected components being a subset of the plurality of binary pixels of the three binary images;

filtering the plurality of connected components in each of the three binary images to exclude from a set of image features of each of the three binary images one or more connected components from the plurality of connected components not meeting certain predefined limits based on a predefined size of each of the plurality of connected components; and

in the set of image features, standardizing a size of each image feature to be equal to other image features of the set of image features,

wherein the filtering the connected components further includes excluding from the set of image features the connected components having a maximum pixel width or pixel height exceeding three hundred fifty pixels.

18. In a grayscale or color image having pluralities of pixels corresponding to a consumer product, a method of normalizing image features thereof, comprising:

converting to binary the pluralities of pixels of the grayscale or color image according to three different thresholds to obtain three binary images each having a plurality of binary pixels;

identifying a plurality of connected components in the plurality of binary pixels of each of the three binary images, each of the connected components being a subset of the plurality of binary pixels of the three binary images;

filtering the plurality of connected components in each of the three binary images to exclude from a set of image features of each of the three binary images one or more connected components from the plurality of connected components not meeting certain predefined limits based on a predefined size of each of the plurality of connected components; and

in the set of image features, standardizing a size of each image feature to be equal to other image features of the set of image features,

wherein the filtering the connected components further includes excluding from the set of image features the connected components having a ratio of pixel height to pixel width of less than five.

19. In a grayscale or color image having pluralities of pixels corresponding to a consumer product, a method of normalizing image features thereof, comprising:

converting to binary the pluralities of pixels of the grayscale or color image according to three different thresholds to obtain three binary images each having a plurality of binary pixels;

identifying a plurality of connected components in the plurality of binary pixels of each of the three binary images, each of the connected components being a subset of the plurality of binary pixels of the three binary images;

filtering the plurality of connected components in each of the three binary images to exclude from a set of image features of each of the three binary images one or more connected components from the plurality of connected components not meeting certain predefined limits based on a predefined size of each of the plurality of connected components; and

in the set of image features, standardizing a size of each image feature to be equal to other image features of the set of image features,

wherein the filtering the connected components further includes excluding from the set of image features the connected components having a ratio of pixel height to pixel width of more than eighteen.

20. In a grayscale or color image having pluralities of pixels corresponding to a consumer product, a method of normalizing image features thereof, comprising:

converting to binary the pluralities of pixels of the grayscale or color image according to three different thresholds to obtain three binary images each having a plurality of binary pixels;

identifying a plurality of connected components in the plurality of binary pixels of each of the three binary images, each of the connected components being a subset of the plurality of binary pixels of the three binary images;

filtering the plurality of connected components in each of the three binary images to exclude from a set of image features of each of the three binary images one or more connected components from the plurality of connected components not meeting certain predefined limits based on a predefined size of each of the plurality of connected components; and

in the set of image features, standardizing a size of each image feature to be equal to other image features of the set of image features,

wherein a first set of images correspond to known consumer products and a second set of images correspond to unknown consumer products, each image of the first and second sets of images having been converted into said three binary images, and further including finding three image features in the set of image features corresponding to one another in each of the three binary images corresponding to each image of the first and second sets of images, and wherein each of the binary pixels are defined in bits and the finding of the three image features corresponding to one another further includes XORing bits of two of the three image features one each from the first and second sets of images and counting a number of 1 bits, wherein a match between the two of the three image features exists if the number of the 1 bits does not exceed ten percent of a total number of the bits in the XORing operation.