METHODS AND SYSTEMS FOR A REMINDER SERVICER USING VISUAL RECOGNITION

• US 20130088337A1
• Filed: 10/07/2011
• Published: 04/11/2013
• Est. Priority Date: 10/07/2011
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Autonomy Corp Ltd (HP Inc.)
• Inventors: Blanchflower, Sean Mark, Loxam, James
• Granted Patent: US 9,183,546 B2
• Time in Patent Office: Days
• US Class Current: 340/384.1
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G08B 3/00 Audible signalling systems; Audible personal calling systems
• CPC Class Codes: H04L 67/55 Push-based network services
• CPC Class Codes: H04N 21/25841 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; arrangements for identifying locations of receiving stations in broadcast systems H04H60/51; location of the user terminal in data switching networks H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/25866 Management of end-user data customer care in data networks H04L41/5077
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/47214 for content reservation or setting reminders; for requesting event notification, e.g. of sport results or stock market stock exchange data over packet-switching network H04L12/1804; push services over packet-switching network H04L12/1859; notification of incoming messages in packet switching networks H04L51/224
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4882 for displaying messages, e.g. warnings, reminders arrangements for providing short real-time information to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/6131 involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/6181 involving transmission via a mobile phone network arrangements for providing broadcast or conference services to substation in data switching networks in combination with wireless systems H04L12/189; wireless uplink channel access H04W74/004
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8173 End-user applications, e.g. Web browser, game
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04W 4/185 by embedding added-value information into content, e.g. geo-tagging

Abstract

A downloadable reminder application uses object recognition technology to recognize one or more objects appearing on a display screen of the mobile computing device and matches them to a set of known target objects. The downloadable reminder application process any associated trigger conditions for the target object. The downloadable reminder application then sets the future reminder for the target object to activate. The downloadable reminder application when it detects that a mobile computing device is satisfying the trigger condition for a target object for which it has set a future reminder on, then the downloadable reminder application in the mobile computing device fires the future reminder that was set for that target object and provides the alert to the user along with any triggerable contributions associated with that target object.

Claims

1. An apparatus, comprising

a downloadable reminder application scripted to have its application program interfaces and call routines to work with blocks of functionality in a mobile computing device and the downloadable application has a routine configured to communicate and cooperate with a reminder engine resident in a backend server system;

and

wherein the downloadable reminder application is scripted to set a future reminder on a target object, to be triggered based on local and temporal trigger conditions including

     1) time of day indicated by a clock function of the mobile computing device,

     2) date and time of day indicated by the clock function,

     3) an amount of time from an initial start time by referencing the clock function,

     4) a visual recognition of the target object being displayed on a display screen of the mobile computing device,

     5) a current geographic location of the mobile computing device indicated by a GPS module of the mobile computing device, and

     6) any combination of these five, wherein the downloadable reminder application is scripted to monitor the blocks of functionality in the mobile computing device to determine when the mobile computing device satisfies an above trigger condition for the target object and then the downloadable reminder application is scripted to notify a user that the future reminder on the target object is triggered via an alert

     1) appearing on a display screen of the mobile computing device,

     2) through an audile alert via speakers of the mobile computing device,

     3) through a vibrating of the mobile computing device, and

     4) any combination of the three, wherein the downloadable reminder application is implemented as software code, hardware logic circuits, and any combination of the two, and portions of the downloadable reminder application scripted in software code are stored in a non-transitory computing device readable medium in an executable format.

2. The apparatus of claim 1, wherein a visual recognition module of the downloadable reminder application is scripted to analyze frames of a captured video stream by a video capture block in the mobile computing device and identify any objects within each frame of the captured video stream to determine whether the target object is contained on that frame of the captured video stream to be displayed on the display screen.

3. The apparatus of claim 2, wherein the visual recognition module is scripted to perform recognition operations to recognize the target object, including comparing the objects within each frame of the captured video stream to known target objects stored in a local target object database stored in a memory of the mobile computing device, in order to determine when there is a close match between the objects in the frame and the known target objects, and where the local target object database in the mobile computing device is a smaller subset of known target objects, than the full set of all known target objects to the backend server system, which is stored in a target object database in the backend server system.

4. The apparatus of claim 2, wherein the visual recognition module is scripted to perform recognition operations, including object analysis for identifying distinguishing characteristics of an object and optical character recognition (OCR), in order to extract any distinctive features of the objects in the frame and then code the distinctive features of the object into a digital fingerprint of that object, which is then compared with a set of digital fingerprints of known target objects stored in a local target object database stored in a memory of the mobile computing device.

5. The apparatus of claim 1, wherein the downloadable reminder application is scripted to enhance certain objects appearing on the display screen with a high lighting, including objects appearing in still images, on web pages, in video streams, and any combination of the three, that 1) have been or 2) can be and 3) any combination of the two, set as a candidate target object that actionable future reminders can be set for, and the candidate target object that actionable future reminders can be set for is enhanced with the high lighting around the target object on the display screen, where the user 1) touches, 2) mouse clicks, or 3) otherwise selects the highlighted target to set the future reminder on that target object.

6. The apparatus of claim 1, wherein the downloadable reminder application is scripted to create a communication connection between itself and a reminder engine resident in the backend server system, where the reminder engine in the backend server system is configured to send one or more digital fingerprints of known target objects that have actionable trigger conditions in a reminder packet to the downloadable reminder application to update a local target object database of known target objects set for that specific instance of the downloadable reminder application, where the reminder engine is scripted to generate customized reminder packets for each instance of the downloadable reminder application and sends the customized reminder packets over a wide area network to be stored in the memory of the mobile computing device, and where a first instance of the downloadable reminder application has a different set of known target objects in its local target object database than a second instance of the downloadable reminder application.

7. The apparatus of claim 6, wherein the downloadable reminder application is scripted to regularly reference the local target database and actively request regular updates for the customized reminder packet rather than having the reminder engine pushing the updates of the customized reminder packet to each instance of the downloadable reminder application, and each known target object has both triggerable conditions to trigger the future reminder on that known target object and triggerable contributions associated with the known target object to cause an additional beneficial action beyond the alert when the future reminder on that known target object is triggered.

8. The apparatus of claim 1, wherein the downloadable reminder application is scripted to create a communication connection between itself and the reminder engine resident in the backend server system, where the downloadable reminder application is scripted to be able to flag any objects appearing on the display screen of the mobile computing device, including objects potentially coming from a video stream, and then send the flagged object to the backend server system through a communication module of the mobile computing device, and one or more recognition engines in the backend server system are configured to analyze the flagged object in an attempt to identify the flagged object and make a known target object.

9. The apparatus of claim 1, wherein the downloadable reminder application is scripted to function independently of the backend server system when a communication connection between them is interrupted by referring to a local target object database stored in the memory of the mobile device, performing its own target object identification with a visual recognition module, and storing currently unknown objects desired to be set as target objects, any candidate known target objects selected and flagged by the user to have a future reminder set on, as well as any commands that the user selects while the communication connection is interrupted in non-volatile portions of the mobile computing device, and then when the communication connection is re-established between them, the downloadable reminder application is scripted to retrieve the unknown objects, known target objects selected to have a future reminder set, and any commands and then transmit this information over a public wide area network to a reminder engine in the backend server system.

10. A computer-implemented method for enabling a user of a mobile computing device to set a future reminder based on visual recognition, the method comprising:

capturing any of a video or a still image associated with a target object, including at least one of an object, a service, a movie, a broadcast program, and any combination of these, using a built-in camera of a mobile computing device, where the mobile computing device includes a processor, a memory, a display screen, a built-in battery to power the mobile computing device, a built-in communication module to enable wireless communication, and a downloadable reminder application downloaded from a backend server system, the downloadable reminder application is configured to enable a user of the mobile computing device to select the target object and then set the future reminder on that target object, wherein the mobile computing device and the backend server system are connected to a network;

performing recognition operations to recognize the target object having an actionable future reminder associated with that target object, wherein the recognition operations are performed by at least comparing information related to the target object with previously stored information on a set of known target objects candidates;

setting a future reminder on the target object to trigger at a subsequent time based on a trigger condition including a location of the mobile computing device as determined by a built-in global positioning system (GPS) module or based on a time as determined by a built-in clock of the mobile computing device and any combination of the two; and

triggering the future reminder to generate an alert for the target object to the user of the mobile computing device when the trigger condition is satisfied, wherein the alert is displayed on a built-in display screen of the mobile computing device, where any portions of the computer-implemented method that are scripted in software are stored on one or more non transitory computer readable mediums in an executable format.

11. The computer-implemented method of claim 10, wherein the downloadable reminder application in the mobile computing device is configured to perform the visual recognition operation of the target object captured in the video or the still image, the setting of the future reminder for the target object, and determining when the trigger condition for the target object is satisfied, and wherein, while operating independently of the server computing device, the mobile computing device is configured to store information relating to the recognition operation, the setting of the reminder, and the triggering of the reminder locally until a time when the mobile computing device is synchronized with the backend server system.

12. The computer-implemented method of claim 10, wherein engines cooperating with databases in the backend server system are configured to present a user interface to a third party to set the trigger conditions for the target object, perform a recognition operation on the target object, and send reminder packets containing known target objects and their trigger conditions to the downloadable reminder application resident in the mobile computing device.

13. The computer-implemented method of claim 10, wherein the backend server system includes a database configured to store a backup copy of current future reminders set for each instance of the downloadable reminder application, and a history of past future reminders for each instance of the downloadable reminder application.

14. A non-transitory computing device readable medium, which stores instructions in an executable format, which when executed cause the computing device to perform the method of claim 10.

15. A system, comprising:

a reminder application database configured to store multiple instances of a downloadable reminder application that the backend server system is configured to serve a given instance of the downloadable reminder application to a mobile computing device, where each downloadable reminder application is configured to be resident in its own mobile computing device;

a backend reminder server system has a set of engines and databases, including the reminder application database, that are configured to exchange information with the multiple instances of the downloadable reminder application;

a target object database in the set of databases configured to store and maintain a complete master set of all known target objects in the backend server system;

a trigger condition database configured to store trigger conditions for each known target object; and

a reminder engine configured to generate customized reminder packets for each instance of the downloadable reminder application based on an entry in the set of databases that a future reminder is currently set for that target object with its current trigger conditions for that target object in that instance of the downloadable reminder application, where each instance of the downloadable reminder application is scripted to use its customized reminder packet to detect when the known target objects satisfy their trigger conditions.

16. The system of claim 15, further comprising:

one or more object recognition engine in the backend server system to identify an object to set a future reminder on, where both an instance of the downloadable application in the mobile computing device and an interface for third party'"'"'s presented by the backend server system are used as sources to supply objects to the object recognition engine in the backend server system to identify an object to create a target object with an actionable a future reminder, and thus, the downloadable reminder application is configured to enable a user of the mobile computing device to select an object and flag the object as a candidate to make a known target object, and likewise the third party interface is configured to solicit a third party administrator to identify an object to set a future reminder on and assign actionable trigger conditions for a future reminder for that object, and then the interface for third party'"'"'s is configured to send the object to the object recognition engine in the backend server system.

17. The system of claim 15, further comprising:

a local target object database stored in the memory of the mobile computing device and derived from the customized reminder packet;

a visual recognition module in the downloadable reminder application is scripted to use object recognition technology, where the visual recognition module of the downloadable reminder application is configured to use the object recognition technology to recognize one or more objects appearing on a display screen of the mobile computing device and match them to a set of known target objects stored in the local target database, where the downloadable reminder application in the mobile computing device is scripted to after recognizing and matching the target object appearing on the display screen, then the downloadable reminder application processes any associated trigger conditions for the target object and sets the future reminder for the target object up to activate, and where the downloadable reminder application creates a local target object database, and the local target object database is stored in the memory of the mobile computing device and is derived from the reminder packet from the reminder engine to form a small subset of known target objects as compared to an amount of known target objects stored in the target object database in the backend server system.

18. The system of claim 17, wherein when the downloadable reminder application sets up the future reminder for the target object, then the downloadable reminder application transmits a communication to the reminder engine in the backend server system, the reminder engine receives the flagged target object to set the reminder on, updates its databases, checks what the latest triggering conditions and triggerable contributions are for that target object, and sends an updated reminder packet over the public wide area network back to the downloadable reminder application.

19. The system of claim 18, where the downloadable reminder application when it detects that the mobile computing device is satisfying the trigger condition for a target object for which it has set a future reminder on, then the downloadable reminder application in the mobile computing device fires the future reminder that was set for that target object and provides the alert to the user along with any triggerable contributions associated with that target object.

20. The system of claim 18, further comprising:

an interface for third parties presented by the backend server system, where an administrator of the third party server interacts with the interface to register an object, including real world objects, services, broadcast programs, and movies, as a target object with the backend server system, and assigns trigger conditions with that target object, where object recognition engine of the backend server system process the registered object to determine any identifying characteristics of that object, and then the reminder engine stores the registered object, its identifying characteristics, and assigned trigger conditions into the target object database as a known target object.