Methods and arrangements for identifying objects

• US 10,192,087 B2
• Filed: 06/07/2016
• Issued: 01/29/2019
• Est. Priority Date: 08/30/2011
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Davis, Bruce L., Rodriguez, Tony F., Rhoads, Geoffrey B., Lord, John D., Evans, Eric D., Gerlach, Rebecca L., Bai, Yang, Stach, John
• Primary Examiner(s): Stanford, Christopher
• Application Number:
  US15/175,997
• Publication Number:
  US 20160364634A1
• Time in Patent Office: 966 Days
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06K 7/10861 sensing of data fields affixed to objects or articles, e.g. coded labels postal sorting B07C3/14, conveying articles B65G47/48
• CPC Class Codes: G06K 7/1456 determining the orientation of the optical code with respect to the reader and correcting therefore
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 20/201 Price look-up processing, e.g. updating
• CPC Class Codes: G06Q 20/208 Input by product or record sensing, e.g. weighing or scanner processing
• CPC Class Codes: G06Q 30/00 Commerce
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06T 2201/0065 Extraction of an embedded watermark; Reliable detection
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/243 by compensating for image skew or non-uniform image deformations
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/30 Noise filtering
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/63 : Scene text, e.g. street names
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 2201/09 : Recognition of logos
• CPC Class Codes: G06V 30/19173 : Classification techniques
• CPC Class Codes: G06V 30/224 : of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/2247 : Characters composed of bars, e.g. CMC-7
• CPC Class Codes: G07G 1/0036 : Checkout procedures
• CPC Class Codes: G07G 1/0045 : with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
• CPC Class Codes: G07G 1/0054 : with control of supplementary check-parameters, e.g. weight or number of articles
• CPC Class Codes: G07G 1/0063 : with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
• CPC Class Codes: G09G 2340/12 : Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2380/04 : Electronic labels
• CPC Class Codes: G09G 2380/06 : Remotely controlled electronic signs other than labels

Abstract

In some arrangements, product packaging is digitally watermarked over most of its extent to facilitate high-throughput item identification at retail checkouts. Imagery captured by conventional or plenoptic cameras can be processed (e.g., by GPUs) to derive several different perspective-transformed views—further minimizing the need to manually reposition items for identification. Crinkles and other deformations in product packaging can be optically sensed, allowing such surfaces to be virtually flattened to aid identification. Piles of items can be 3D-modelled and virtually segmented into geometric primitives to aid identification, and to discover locations of obscured items. Other data (e.g., including data from sensors in aisles, shelves and carts, and gaze tracking for clues about visual saliency) can be used in assessing identification hypotheses about an item. A great variety of other features and arrangements are also detailed.

Claims

1. An image processing method to identify a canned, barcoded item selected for purchase by a first shopper at a retail store, the store having a shopping portion where items are stocked for sale, and a checkout portion, the method including using information from a first sensor, and then using information from second sensor different than the first sensor, in a data fusion manner to narrow a universe of possible item identifications, the method more particularly comprising the acts:

based on information from the first sensor, developing a set of plural item identification hypotheses about the canned barcoded item selected by the shopper for purchase, each of said item identification hypotheses having an associated confidence score that respectively indicates certainty about said hypothesis;

refining the set of plural identification hypotheses about the selected barcoded item, by applying data fusion based on other information, the other information including information from the second sensor, said refining including revising at least certain of said associated confidence scores;

if need be, successively repeating said refining act, until one of said revised confidence scores has a value exceeding a particular, predetermined threshold value, said threshold value having been established prior to said refining act, and serving as a basis for comparison in judging the revised confidence scores; and

adding, to a purchase tally for said shopper, an item associated with said one revised confidence score that has a value exceeding the threshold value, said item being the canned barcoded item selected for purchase by the shopper;

wherein one of said sensors is a fixed sensor in an aisle of the shopping portion of the store, the information from said fixed sensor that is used in said data fusion manner more particularly comprising (a) information indicating locations visited by the shopper during a track of the shopper through the store, or (b) information indicating both removal of an object from a store shelf location and a time of such removal; and

the other of said sensors is a camera at the checkout portion of the retail store, the information from said camera that is used in said data fusion manner more particularly-comprising imagery depicting the canned, barcoded item, or a person'"'"'s interaction with said item.

2. The method of claim 1 in which the information from the fixed sensor comprises information indicating locations visited by the shopper during a track of the shopper through the store.

3. The method of claim 2 in which the data fusion includes revising at least certain of said confidence scores based on information about a layout of the store, indicating locations of different items in the store.

4. The method of claim 1 in which the information from the fixed sensor comprises information from a ceiling-, floor- or shelf-mounted camera.

5. The method of claim 1 in which the data fusion includes revising at least certain of said confidence scores based on information identifying a store aisle through which the shopper did not pass.

6. The method of claim 1 in which the data fusion includes revising at least certain of said confidence scores based on information about store locations at which the shopper, or the shopper'"'"'s cart, paused during the shopper'"'"'s track through the store.

7. The method of claim 1 in which the data fusion includes revising at least certain of said confidence scores based on information about a store location at which an item was placed into the shopper'"'"'s cart.

8. The method of claim 1 in which the information from the camera at the checkout portion of the store comprises information from a camera at an item bagging area of the store.

9. The method of claim 1 in which the information from the fixed sensor comprises information indicating both removal of an object from the store shelf location, and a time of such removal.

10. The method of claim 9 that includes sensing removal of stock from the store shelf location with a weight transducer or a camera.

11. The method of claim 10 that includes sensing removal of stock from the store shelf location with a camera, and sensing information about locations visited by the shopper with a camera.

12. The method of claim 9 wherein a weight given to the information from the fixed sensor in determining a confidence score depends, at least in part, on the time data.

13. The method of claim 1 in which the data fusion includes revising at least certain of said confidence scores based on information derived from one of the following:

sensing disruption of air blown through a pile of items, sensing temperature, sensing vibration, sensing sound, ultrasonic or millimeter wave scanning, sensing ultraviolet or infrared radiation, weight sensing, haptic sensing, 3D shape sensing, chemical or olfactory sensing, inertial modeling, statistical item correlations, or crowdsourced human assessment of item imagery.

14. The method of claim 1 that further includes:

determining an item of information relating to the shopper from the list consisting of;

sensed shopper demographic, sensed shopper gaze, sensed pose of a shopper hand grasping an item, shopper purchasing history, and shopper shopping list; and

revising at least one of said confidence scores based on said determined item of information.

15. The method of claim 1 that further includes:

determining one of the following types of information;

volumetric product shape information, color histogram data, incomplete or ambiguous barcode data, digital watermark data, NFC/RFID information, image fingerprint data, recognized text data, weight information, temperature information, and information about relative placement of different items in a bag; and

revising at least one of said confidence scores based on said determined item of information.

16. The method of claim 1 wherein both said first shopper and a further shopper both select a second item for purchase, the method including:

attempting to identify the second item selected for purchase by the first shopper, but failing to do so because the continued refinement of the set of plural identification hypotheses does not yield any revised confidence score having a value exceeding the threshold value;

flagging the second item, selected for purchase by the first shopper, to a human operator for follow-up;

attempting to identify the second item selected for purchase by the second shopper, but failing to do so because the continued refinement of the set of plural identification hypotheses does not yield any revised confidence score having a value exceeding the threshold value; and

enabling the second shopper to self-identify the second item;

wherein the second shopper, but not the first shopper, is permitted to self-identify the second item, because the second shopper is determined to have a heightened level of trust with the store.

17. The method of claim 1 that includes sensing, from imagery captured by the camera at the checkout portion of the retail store:

image fingerprint data for the item, color histogram data for the item, optical character recognition data for the item, volumetric product configuration data for the item, incomplete or ambiguous barcode or watermark data for the item, barcode position information for the item, barcode aspect ratio for the item, information indicating an order in which items are placed into a bag, information indicating a person'"'"'s hand pose as the item is grasped, or information indicating a person'"'"'s gaze towards the item; and

employing said sensed information in determining at least one of said confidence scores.

18. The method of claim 1 that further includes identifying the shopper by a smartphone app that exchanges data with a store computer, and using shopper identification information produced thereby in determining at least one of said confidence scores.

19. A system, configured to identify a packaged, barcoded item selected for purchase by a shopper in a retail store, the store having a shopping portion where items are stocked for sale, and a checkout portion, the system employing information from a first sensor, and then information from second sensor different than the first sensor, in a data fusion manner to narrow a universe of possible item identifications, the system including:

a first, fixed, sensor in an aisle of the shopping portion of the store, the first sensor being adapted to produce (a) information indicating locations visited by the shopper during a track of the shopper through the store, or (b) information indicating both removal of an object from a store shelf location and a time of such removal;

a second, camera, sensor, at the checkout portion of the retail store, the second sensor being adapted to produce imagery depicting the packaged, barcoded item, or a person'"'"'s interaction with said item;

a memory; and

one or more processors operating in accordance with instructions stored in the memory;

wherein said instructions stored in the memory configure the system to perform acts including;

based on information from one of said sensors, developing a set of plural item identification hypotheses about the packaged barcoded item selected by the shopper for purchase, each of said item identification hypotheses having an-associated confidence score that respectively indicates certainty about said hypothesis;

refining the set of plural identification hypotheses about the selected barcoded item, by applying data fusion based on other information, the other information including information from the other of said sensors, said refining including revising at least certain of said associated confidence scores;

if need be, successively repeating said refining act until one of said revised confidence scores has a value exceeding a particular, predetermined threshold value, said threshold value having been established prior to said refining act, and serving as a basis for comparison in judging the revised confidence scores; and

adding, to a purchase tally for said shopper, an item associated with said one revised confidence score that has a value exceeding the threshold value, said item being the packaged barcoded item selected for purchase by the shopper.

20. A tangible, non-transitory computer readable medium containing instructions for configuring a system, including one or more processors, to identify a packaged, barcoded item selected for purchase by a shopper in a retail store, the store having a shopping portion where items are stocked for sale, and a checkout portion, the system including first and second sensors, one of said sensors comprising a fixed sensor in an aisle of the shopping portion of the store that is adapted to produce (a) information indicating locations visited by the shopper during a track of the shopper through the store, or (b) information indicating both removal of an object from a store shelf location and a time of such removal, the other of said sensors comprising a camera at the checkout portion of the retail store that is adapted to produce imagery depicting the packaged, barcoded item, or a person'"'"'s interaction with said item, said instructions configuring the system to perform acts including:

based on information from one of said sensors, developing a set of plural item identification hypotheses about the packaged barcoded item selected by the shopper for purchase, each of said item identification hypotheses having an associated confidence score that respectively indicates certainty about said hypothesis;

refining the set of plural identification hypotheses about the selected barcoded item, by applying data fusion based on other information, the other information-including information from the other of said sensors, said refining including revising at least certain of said associated confidence scores;

if need be, successively repeating said refining act, until one of said revised confidence scores has a value exceeding a particular, predetermined threshold value, said threshold value having been established prior to said refining act, and serving as a basis for comparison in judging the revised confidence scores; and

adding, to a purchase tally for said shopper, an item associated with said one revised confidence score that has a value exceeding the threshold value, said item being the packaged, barcoded item selected for purchase by the shopper.