Methods and systems for a reminder servicer using visual recognition

• US 9,183,546 B2
• Filed: 10/07/2011
• Issued: 11/10/2015
• Est. Priority Date: 10/07/2011
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Blanchflower, Sean Mark, Loxam, James
• Primary Examiner(s): Zimmerman, Brian
• Assistant Examiner(s): Lau, Kevin
• Application Number:
  US13/268,256
• Publication Number:
  US 20130088337A1
• Time in Patent Office: 1,495 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G08B 3/00 Audible signalling systems; Audible personal calling systems
• CPC Class Codes: H04L 67/55 Push-based network services
• CPC Class Codes: H04N 21/25841 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; arrangements for identifying locations of receiving stations in broadcast systems H04H60/51; location of the user terminal in data switching networks H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/25866 Management of end-user data customer care in data networks H04L41/5077
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/47214 for content reservation or setting reminders; for requesting event notification, e.g. of sport results or stock market stock exchange data over packet-switching network H04L12/1804; push services over packet-switching network H04L12/1859; notification of incoming messages in packet switching networks H04L51/224
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4882 for displaying messages, e.g. warnings, reminders arrangements for providing short real-time information to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/6131 involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/6181 involving transmission via a mobile phone network arrangements for providing broadcast or conference services to substation in data switching networks in combination with wireless systems H04L12/189; wireless uplink channel access H04W74/004
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8173 End-user applications, e.g. Web browser, game
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04W 4/185 by embedding added-value information into content, e.g. geo-tagging

Abstract

A downloadable reminder application uses object recognition technology to recognize one or more objects appearing on a display screen of the mobile computing device and matches them to a set of known target objects. The downloadable reminder application process any associated trigger conditions for the target object. The downloadable reminder application then sets the future reminder for the target object to activate. The downloadable reminder application when it detects that a mobile computing device is satisfying the trigger condition for a target object for which it has set a future reminder on, then the downloadable reminder application in the mobile computing device fires the future reminder that was set for that target object and provides the alert to the user along with any triggerable contributions associated with that target object.

Claims

1. An apparatus, comprising:

a processor; and

a memory on which is stored machine readable instructions that when executed by a processor cause the processor to;

transmit data pertaining to a flagged object from an internet browsing session to an object recognition engine in a backend server system, wherein the object is flagged by a user from a web browser application displayed on a screen of the apparatus;

receive a list of candidate target objects from the object recognition engine, wherein the object recognition engine determines the list based on data pertaining to a plurality of objects stored in the backend server system that are determined to be similar to the flagged object;

display the list of candidate target objects on the display screen;

receive a user selection of a target object from the displayed list of candidate target objects;

set a future reminder on the target object to be triggered based on at least one of a local and a temporal trigger condition;

determine whether at least one of the local and the temporal trigger conditions for the target object has been satisfied; and

in response to a determination that at least one of the local and the temporal trigger conditions for the target object has been satisfied, notify a user that a triggerable contribution associated with the target object has been triggered, wherein the triggerable contribution includes price comparison information.

2. The apparatus of claim 1, wherein the machine readable instructions are further to cause the processor to:

identify an object in a frame of a captured video stream; and

determine whether the target object is contained in the frame of the captured video stream.

3. The apparatus of claim 2, wherein to determine whether the target object is contained in the frame of the captured video stream, the machine readable instructions are further to cause the processor to compare the identified object in the frame of the captured video stream to target objects stored in a local target object database of the apparatus to determine if there is a match between the identified object and the target objects stored in the local target object database, wherein the local target object database stores a subset of target objects stored in a backend target object database in the backend server system.

4. The apparatus of claim 2, wherein to determine whether the target object is contained in the frame of the captured video stream, the machine readable instructions are further to cause the processor to:

determine distinguishing features of the identified object using optical character recognition (OCR);

extract the determined distinguishing features of the identified object; and

code the distinguishing features of the identified object into a digital fingerprint; and

compare the digital fingerprint with a set of digital fingerprints of target objects stored in a local target object database of the apparatus.

5. The apparatus of claim 1, wherein the machine readable instructions are further to cause the processor to:

request a packet of digital fingerprints of target objects that have actionable trigger conditions from the backend server system; and

update target objects stored in a local target object database of the apparatus with the packet of digital fingerprints received from the backend server system.

6. The apparatus of claim 5, wherein, in response to the determination that at least one of the local and the temporal trigger conditions for the target object has been satisfied, the machine readable instructions are further to cause the processor to notify the user of the triggerable contribution including one or more of identification or descriptive information, advertisement information, link information, user review information, and coupons.

7. The apparatus of claim 1, wherein the machine readable instructions are further to cause the processor to:

in response to a communication between the apparatus and the backend server system being interrupted,

store the flagged object in a local target object database;

determine a local list of candidate target objects based on data pertaining to objects stored in the local target object database that are determined to be similar to the flagged object;

display the local list of candidate target objects on the display screen;

receive a user selection of a local target object from the displayed local list of candidate target objects;

set a future reminder on the local target object in the local target object database; and

transmit the flagged object, the selected target object, and the future reminder to the backend server system in response to the communication between the apparatus and the backend server system being reestablished.

8. A computer-implemented method comprising:

displaying, by a processor of a mobile device, an image of an object in a web browser application of the mobile device;

receiving an input from a user to flag the image of the object in the web browser application;

transmitting the image of the flagged object to a recognition engine in a backend server system;

receiving a list of candidate target objects from the recognition engine, wherein the recognition engine determines the list based on data pertaining to a plurality of objects stored in the backend server system that are determined to be similar to the flagged object;

displaying the list of candidate target objects on a display screen;

receiving a user selection of a target object from the displayed list of candidate target objects;

setting a future reminder on the target object to be triggered based on at least one of a local or temporal trigger condition;

determine whether at least one of the local and the temporal trigger condition for the target object has been satisfied; and

in response to a determination that at least one of the local and the temporal trigger conditions for the target object has been satisfied, notifying a user that a triggerable contribution associated with the target object has been triggered, wherein the triggerable contribution includes price comparison information.

9. The computer-implemented method of claim 8, further comprising:

in response to a communication between the mobile device and the backend server system being interrupted,

storing the flagged object in a local target object database of the mobile device;

determining a local list of candidate target objects based on data pertaining to objects stored in the local target object database that are determined to be similar to the flagged object;

displaying the local list of candidate target objects on the display screen;

receiving a user selection of a local target object from the displayed local list of candidate target objects;

setting a future reminder on the local target object in the local target object database; and

transmitting the flagged object, the selected target object, and the future reminder to the backend server system in response to the communication between the mobile device and the backend server system being reestablished.

10. The computer-implemented method of claim 8, further comprising providing a user interface to a third party to select the target object, set the trigger conditions for the target object, and send packets containing data pertaining to the target objects and their trigger conditions to a local target object database in the mobile device.

11. The computer-implemented method of claim 8, wherein the backend server system includes a database to store a backup copy of the future reminders and a history of expired reminders.

12. A non-transitory computer readable storage medium storing machine-readable instructions that when executed by a processor, cause the processor to:

transmit data pertaining to a flagged object from an internet browsing session to an object recognition engine in a backend server system, wherein the object is flagged by a user from a web browser displayed on a display screen of a mobile device;

receive a list of candidate target objects from the object recognition engine, wherein the object recognition engine determines the list based on data pertaining to a plurality of objects stored in the backend server system that are determined to be similar to the flagged object;

display the list of candidate target objects on the display screen;

receive a user selection of a target object from the displayed list of candidate target objects;

set a future reminder on the target object to be triggered based on at least one of a local and a temporal trigger condition;

determine whether at least one of the local and the temporal trigger condition for the target object has been satisfied; and

in response to a determination that at least one of the local and the temporal trigger conditions for the target object has been satisfied, notify a user that a triggerable contribution associated with the target object has been triggered, wherein the triggerable contribution includes price comparison information.

13. The non-transitory computer readable storage medium of claim 12, wherein the machine readable instructions are further to cause the processor to:

in response to the determination that at least one of the local and the temporal trigger conditions for the target object has been satisfied, notify the user of the triggerable contribution including one or more of identification or descriptive information, advertisement information, link information, user review information, and coupons.

14. The non-transitory computer readable storage medium of claim 12, wherein the machine readable instructions are further to cause the processor to:

provide a user interface to a third party to select the target object, set the trigger conditions for the target object, and send packets containing data pertaining to the target objects and their trigger conditions to a local target object database in the mobile device.

15. The non-transitory computer readable storage medium of claim 12, wherein the machine readable instructions are further to cause the processor to:

in response to a communication between the mobile device and the backend server system being interrupted,

store the flagged object in a local target object database;

determine a local list of candidate target objects based on data pertaining to objects stored in the local target object database that are determined to be similar to the flagged object;

display the local list of candidate target objects on the display screen;

receive a user selection of a local target object from the displayed local list of candidate target objects;

set a future reminder on the local target object in the local target object database; and

transmit the flagged object, the selected target object, and the future reminder to the backend server in response to the communication between the mobile device and the backend server system being reestablished.

16. The non-transitory computer readable storage medium of claim 12, wherein the machine readable instructions are further to cause the processor to:

request a packet of digital fingerprints of target objects that have actionable trigger conditions from the backend server system; and

update target objects stored in a local target object database of the mobile device with the packet of digital fingerprints received from the backend server system.