REDUCING THE SEARCH SPACE FOR RECOGNITION OF OBJECTS IN AN IMAGE BASED ON WIRELESS SIGNALS

• US 20160086029A1
• Filed: 08/28/2015
• Published: 03/24/2016
• Est. Priority Date: 08/28/2014
• Status: Active Grant

• Current Assignee: Retailmenot Incorporated (J2 Global, Inc.)
• Original Assignee: Retailmenot Incorporated (J2 Global, Inc.)
• Inventors: Dubuque, Shaun F.
• Granted Patent: US 9,684,826 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06Q 30/0207 Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/80 Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level multimodal speaker identification or verification G10L17/10
• CPC Class Codes: G06V 10/87 using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

Provided is a process including: determining that a mobile computing device has crossed a geofence associated with a merchant store; sending, to a remote classifier server, a request for object-recognition classifiers for objects in the merchant store; receiving a set of object-recognition classifiers; receiving with the mobile computing device from user a request to search for offers; capturing an image with a camera of the mobile computing device; receiving one or more wireless beacon identifiers with the mobile computing device; based on the wireless beacon identifiers, selecting a subset of the object-recognition classifiers in the set of object-recognition classifiers; and recognizing an object in the captured image based on the selected subset of the object-recognition classifiers; and requesting, from a remote offer publisher server, offers corresponding to the recognized object; and receiving offers from the remote offer publisher server; and displaying the received offers to the user.

Claims

1. A method of searching for electronic coupons or other offers based on an image captured by a camera of a mobile user device of a product for sale in a merchant'"'"'s store, wherein the product is recognized in the image by the mobile user device without sending the image to a remote computer for image recognition, the method comprising:

obtaining, in memory of a mobile user device, a set of object-recognition classifiers, the object-recognition classifiers each specifying one or more algorithms and parameters of those algorithms by which respective products among a plurality of products in a merchant store are recognized in images;

commanding, with one or more processors of the mobile user device, a camera of the mobile user device to capture an image;

receiving the captured image;

receiving, via an antenna of the mobile user device, a wirelessly transmitted beacon identifier that indicates a location of the mobile user device within the merchant store, wherein indicating the location of the mobile user device includes distinguishing the location of the mobile user device from a plurality of other locations in the merchant store where other wireless beacon identifiers are transmitted;

selecting a subset of the object-recognition classifiers based on the received beacon identifier, the subset of the object-recognition classifiers being configured to indicate whether at least some products positioned within range of the received wireless beacon identifier appear in the image;

recognizing a product in the image by executing the algorithms specified by the subset of object-recognition classifiers with the one or more processors of the mobile user device; and

obtaining a coupon or other offer based on the recognized product.

2. The method of claim 1, wherein:

obtaining the set of object-recognition classifiers comprises, before commanding the camera of the mobile user device to capture the image;

determining that a mobile user device has crossed a geofence associated with the merchant store based on cellular or satellite navigation signals received by the mobile user device while outside of the merchant store, beacons received by the mobile computer device, or a combination thereof;

sending, to a remote classifier server, a request for object-recognition classifiers for products in the merchant store; and

receiving, from the remote classifier server, a set of object-recognition classifiers;

at least one of the object-recognition classifiers comprises;

an identifier of a given product;

identifiers of a plurality of object-recognition functions stored in the memory of the mobile user device before obtaining the set of object-recognition classifiers;

weightings and a sequence that specify a cascaded arrangement of the object-recognition functions; and

parameters of the object-recognition functions that cause the image recognition function to output a value indicative of whether the given product appears in the image;

receiving the captured image comprises receiving a plurality of video frames captured over a trailing duration;

receiving the wirelessly transmitted beacon identifier comprises receiving a Low-Energy Bluetooth beacon and measuring a beacon signal strength that together specify the location of the mobile user device in an aisle of the merchant store to within less than plus-or-minus five meters;

selecting the subset of the object-recognition classifiers based on the received beacon identifier comprises;

determining a product section of the merchant store based on the received beacon identifier; and

filtering from the set of object-recognition classifiers those classifiers that identify a product in the product section of the merchant store to exclude those object-recognition classifiers that do not pertain to the product section of the merchant store;

recognizing the product in the image comprises;

for each video frame among the plurality of video frames, calculating a score with each classifier among the subset of the object-recognition classifiers;

calculating a measure of central tendency among the plurality of video frames for each classifier among the subset of the object-recognition classifiers; and

determining that the measure of central tendency of the classifier corresponding to the product satisfies a threshold;

obtaining the coupon or other offer based on the recognized product comprises;

parsing from the classifier corresponding to the product a text string;

sending a query including the text string to a remote offers engine;

receiving one or more offers responsive to the query; and

presenting at least some of the one or more offers on a display screen of the mobile user device.

3. The method of claim 1, wherein the wirelessly transmitted beacon identifier is a Low-Energy Bluetooth beacon identifier transmitted by one of a plurality of battery-powered Low-Energy Bluetooth beacons distributed in the merchant store, the method further comprising:

measuring a signal strength of the signal with which the wirelessly transmitted beacon identifier is received; and

determining a location of the mobile user device in the merchant store based on both the signal strength and the received beacon identifier.

4. The method of claim 1, wherein selecting a subset of the object-recognition classifiers based on the received beacon identifier comprises:

determining a product section of the merchant store in which the mobile user device is located and filtering from the set of object-recognition classifiers those classifiers associated with a product in the product section of the merchant store.

5. The method of claim 1, wherein obtaining, in memory of the mobile user device, the set of object-recognition classifiers comprises:

obtaining one or more cascades of classifiers, each stage of the cascades corresponding to a different classifier from either an upstream classifier or a downstream classifier in the respective cascade, and each cascade of classifiers being configured to output a score indicative of whether a given product is depicted in a given image.

6. The method of claim 1, wherein

receiving the captured image comprises storing a plurality of video frames with the camera of the mobile device in a video buffer;

and

recognizing a product in the image comprises;

calculating scores with each of the subset of object-recognition classifiers for each of the frames of video in the video buffer;

calculating a measure of central tendency of the scores for each of the classifiers among the subset of object-recognition classifiers; and

determining whether a product is present in the video frames by determining whether at least one of the measures of central tendency satisfies a threshold.

7. The method of claim 1, comprising:

polling an accelerometer of the mobile user device to receive a value indicative of movement of the mobile user device; and

determining to recognize the product in the image in response to the value indicating less than a threshold amount of movement.

8. The method of claim 1, wherein at least some of the object-recognition classifiers each comprise:

an identifier of a given product;

an identifier of an object-recognition function stored in the memory of the mobile user device before obtaining the set of object-recognition classifiers; and

parameters of the object-recognition function that cause the object-recognition function to output a value indicative of whether the given product appears in an image.

9. The method of claim 1, wherein obtaining the set of object-recognition classifiers comprises:

determining that the mobile user device is in a geographic area associated with the merchant store and, in response;

sending, to a remote classifier server, a request for object-recognition classifiers for objects in the merchant store; and

receiving, from the remote classifier server, the set of object-recognition classifiers associated with the merchant store.

10. The method of claim 1, comprising updating the set of object-recognition classifiers for the merchant store with a merchant mobile device, wherein updating the set of object-recognition classifiers comprises:

capturing another image of a given product in the merchant store with the merchant mobile device;

receiving, via an antenna of the merchant mobile device, the wirelessly transmitted beacon identifier that indicates the location of the merchant mobile device within the merchant store; and

calculating a classifier operative to indicative whether the given product is depicted in a given image based on the other image.

11. The method of claim 10, comprising:

designating a subset of the other image that depicts the given product by receiving human input identifying the subset of the other image, wherein calculating a classifier comprises calculating a classifier based on the subset of the other image.

12. The method of claim 10, comprising:

detecting, in the other image, an object placed in a scene with the given product to indicate the location of the given product in the other image;

designating a subset of the other image that depicts the given product based on a position of the detected object by detecting the object placed in the scene, wherein calculating a classifier comprises calculating a classifier based on the subset of the other image.

13. The method of claim 1, comprising:

sending a text query to a remote computer system for an offer, wherein the text query corresponds to a classifier among the subset of the classifiers that output a score satisfying a threshold;

wherein obtaining a coupon or other offer based on the recognized product comprises receiving an offer responsive to the query; and

sending, to an offer issuer or affiliate network, data indicating that the offer was sent to the mobile user device.

14. A tangible, machine-readable, non-transitory medium storing instructions that when executed by one or more data processing apparatuses effectuate operations comprising:

obtaining, in memory of a mobile user device, a set of object-recognition classifiers, the object-recognition classifiers each specifying one or more algorithms and parameters of those algorithms by which respective products among a plurality of products in a merchant store are recognized in images;

commanding, with one or more processors of the mobile user device, a camera of the mobile user device to capture an image;

receiving the captured image;

receiving, via an antenna of the mobile user device, a wirelessly transmitted beacon identifier that indicates a location of the mobile user device within the merchant store, wherein indicating the location of the mobile user device includes distinguishing the location of the mobile user device from a plurality of other locations in the merchant store where other wireless beacon identifiers are transmitted;

selecting a subset of the object-recognition classifiers based on the received beacon identifier, the subset of the object-recognition classifiers being configured to indicate whether at least some products positioned within range of the received wireless beacon identifier appear in the image;

recognizing a product in the image by executing the algorithms specified by the subset of object-recognition classifiers with the one or more processors of the mobile user device; and

obtaining a coupon or other offer based on the recognized product.

15. The medium of claim 14, wherein:

obtaining the set of object-recognition classifiers comprises, before commanding the camera of the mobile user device to capture the image;

determining that a mobile computing device has crossed a geofence associated with the merchant store based on cellular or satellite navigation signals received by the mobile user device while outside of the merchant store;

sending, to a remote classifier server, a request for object-recognition classifiers for products in the merchant store; and

receiving, from the remote classifier server, a set of object-recognition classifiers;

at least one of the object-recognition classifiers comprises;

an identifier of a given product;

identifiers of a plurality of object-recognition functions stored in the memory of the mobile user device before obtaining the set of object-recognition classifiers;

weightings and a sequence that specify a cascaded arrangement of the object-recognition functions; and

parameters of the object-recognition functions that cause the image recognition function to output a value indicative of whether the given product appears in the image;

receiving the captured image comprises receiving a plurality of video frames captured over a trailing duration;

receiving the wirelessly transmitted beacon identifier comprises receiving a Low-Energy Bluetooth beacon and measuring a beacon signal strength that together specify the location of the mobile user device in an aisle of the merchant store to within less than plus-or-minus five meters;

selecting the subset of the object-recognition classifiers based on the received beacon identifier comprises;

determining a product section of the merchant store based on the received beacon identifier; and

filtering from the set of object-recognition classifiers those classifiers that identify a product in the product section of the merchant store to exclude those object-recognition classifiers that do not pertain to the product section of the merchant store;

recognizing the product in the image comprises;

for each video frame among the plurality of video frames, calculating a score with each classifier among the subset of the object-recognition classifiers;

calculating a measure of central tendency among the plurality of video frames for each classifier among the subset of the object-recognition classifiers; and

determining that the measure of central tendency of the classifier corresponding to the product satisfies a threshold;

obtaining the coupon or other offer based on the recognized product comprises;

parsing from the classifier corresponding to the product a text string;

sending a query including the text string to a remote offers engine;

receiving one or more offers responsive to the query; and

presenting at least some of the one or more offers on a display screen of the mobile user device.

16. The medium of claim 14, wherein the wirelessly transmitted beacon identifier is a Low-Energy Bluetooth beacon identifier transmitted by one of a plurality of battery-powered Low-Energy Bluetooth beacons distributed in the merchant store, the operations further comprising:

measuring a signal strength of the signal with which the wirelessly transmitted beacon identifier is received; and

determining a location of the mobile user device in the merchant store based on both the signal strength and the received beacon identifier.

17. The medium of claim 14, wherein selecting a subset of the object-recognition classifiers based on the received beacon identifier comprises:

determining a product section of the merchant store in which the mobile user device is located and filtering from the set of object-recognition classifiers those classifiers associated with a product in the product section of the merchant store.

18. The medium of claim 14, wherein obtaining, in memory of the mobile user device, the set of object-recognition classifiers comprises:

obtaining one or more cascades of classifiers, each stage of the cascades corresponding to a different classifier from either an upstream classifier or a downstream classifier in the respective cascade, and each cascade of classifiers being configured to output a score indicative of whether a given product is depicted in a given image.

19. The medium of claim 14, wherein

receiving the captured image comprises storing a plurality of video frames with the camera of the mobile device in a video buffer;

and

recognizing a product in the image comprises;

calculating scores with each of the subset of object-recognition classifiers for each of the frames of video in the video buffer;

calculating a measure of central tendency of the scores for each of the classifiers among the subset of object-recognition classifiers; and

determining whether a product is present in the video frames by determining whether at least one of the measures of central tendency satisfies a threshold.

20. The medium of claim 14, wherein the operations further comprise:

polling an accelerometer of the mobile user device to receive a value indicative of movement of the mobile user device; and

determining to recognize the product in the image in response to the value indicating less than a threshold amount of movement.

21. The medium of claim 14, wherein at least some of the object-recognition classifiers each comprise:

an identifier of a given product;

an identifier of an object-recognition function stored in the memory of the mobile user device before obtaining the set of object-recognition classifiers; and

parameters of the object-recognition function that cause the object-recognition function to output a value indicative of whether the given product appears in an image.

22. The medium of claim 14, wherein obtaining the set of object-recognition classifiers comprises:

determining that the mobile user device is in a geographic area associated with the merchant store and, in response;

sending, to a remote classifier server, a request for object-recognition classifiers for objects in the merchant store; and

receiving, from the remote classifier server, the set of object-recognition classifiers associated with the merchant store.

23. The medium of claim 14, wherein the operations comprise updating the set of object-recognition classifiers for the merchant store with a merchant mobile device, wherein updating the set of object-recognition classifiers comprises:

capturing another image of a given product in the merchant store with the merchant mobile device;

receiving, via an antenna of the merchant mobile device, the wirelessly transmitted beacon identifier that indicates the location of the merchant mobile device within the merchant store; and

calculating a classifier operative to indicative whether the given product is depicted in a given image based on the other image.

24. The medium of claim 23, wherein the operations comprise:

designating a subset of the other image that depicts the given product by receiving human input identifying the subset of the other image, wherein calculating a classifier comprises calculating a classifier based on the subset of the other image.

25. The medium of claim 23, wherein the operations comprise:

detecting, in the other image, an object placed in a scene with the given product to indicate the location of the given product in the other image;

designating a subset of the other image that depicts the given product based on a position of the detected object by detecting the object placed in the scene, wherein calculating a classifier comprises calculating a classifier based on the subset of the other image.

26. The medium of claim 14, wherein the operations comprise:

sending a text query to a remote computer system for an offer, wherein the text query corresponds to a classifier among the subset of the classifiers that output a score satisfying a threshold;

wherein obtaining a coupon or other offer based on the recognized product comprises receiving an offer responsive to the query; and

sending, to an offer issuer or affiliate network, data indicating that the offer was sent to the mobile user device.

27. A mobile computing device, comprising:

a cellular radio;

a camera;

a satellite navigation module configured to sense the geolocation of the mobile computing device;

a Bluetooth radio;

one or more processors communicatively coupled to the cellular radio, the camera, the satellite navigation module, and the Bluetooth radio; and

memory communicatively coupled to the one or more processors and storing instructions that when executed by at least some of the one or more processors cause operations comprising;

determining that the mobile computing device has crossed a geofence associated with a merchant store;

sending, to a remote classifier server, a request for object-recognition classifiers for objects in the merchant store;

receiving, from the remote classifier server, a set of object-recognition classifiers;

receiving with the mobile computing device from the user a request to search for offers;

capturing an image with a camera of the mobile computing device;

receiving one or more wireless beacon identifiers with the mobile computing device;

based on the wireless beacon identifiers, selecting a subset of the object-recognition classifiers in the set of object-recognition classifiers; and

recognizing an object in the captured image based on the selected subset of the object-recognition classifiers; and

requesting, from a remote offer publisher server, offers corresponding to the recognized object; and

receiving offers from the remote offer publisher server; and

displaying the received offers to the user.