Intuitive computing methods and systems

• US 9,916,519 B2
• Filed: 09/08/2016
• Issued: 03/13/2018
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F., Rhoads, Geoffrey B.
• Primary Examiner(s): MIZRAHI, DIANE D
• Application Number:
  US15/259,882
• Publication Number:
  US 20160379082A1
• Time in Patent Office: 551 Days
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/36 Input/output arrangements for on-board computers
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04N 23/00 : Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/027 : using movement velocity, acceleration information
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services

Abstract

In one particular aspect, a portable computing device (e.g., a tablet or smartphone) senses audio and/or image content from a user'"'"'s environment, and initiates one or more recognition agents (e.g., performing image watermark recognition, image recognition, object recognition, facial recognition, barcode recognition, optical character recognition, audio watermark recognition, speech recognition, speaker recognition, or music recognition). Resource allocation to a recognition agent can be varied based on (a) progress of the recognition agent to achieve its recognition goal, and (b) user interest data indicating user interest in the output of the recognition agent. A second candidate recognition agent can be evaluated for possible launch, based on a relevance score, and a cost score. In some embodiments, the device adapts its operation to changing context, by terminating a first recognition agent in favor of a second recognition agent, without express user instruction to do so. A great number of other features and arrangements are also detailed.

Claims

1. A method practiced by a battery-powered mobile wireless communications device equipped with a camera and microphone, the device having multiple recognition modes enabling recognition of multiple types of content, the method comprising the acts:

initiating one or more stages of a first recognition agent process to recognize audio or image content data captured by the mobile device, the first recognition agent process performing a recognition selected from a list consisting of;

image watermark recognition, image recognition, object recognition, facial recognition, barcode recognition, optical character recognition, audio watermark recognition, speech recognition, speaker recognition, or music recognition;

receiving detection state data indicating a state of the first recognition agent process in performing the selected recognition;

after initiating the first recognition agent process, receiving first user interest data indicating interest of the user in obtaining a result of the first recognition agent process; and

varying an allocation of processing resources to the first recognition agent process based on both said detection state data and on said user interest data;

the method thereby optimizing use of the device battery by allocating processing resources in response both to the state of the first recognition agent process in performing the selected recognition, and to user input received after the first recognition agent process has been initiated, indicating user interest in the result of the first recognition agent process.

2. The method of claim 1 that further includes:

determining a relevance score for a second recognition agent process that the mobile device may run, the relevance score being based on data including at least one data selected from a list consisting of;

location, sensor data available in a memory, context, expressed user intent, and user history;

determining a cost score for said second recognition agent process, the cost score being based on data including at least one data selected from a list consisting of;

memory usage, CPU usage, and communication bandwidth; and

based on at least (a) the detection state data for the first recognition agent process, (b) the relevance score of the second recognition agent process, and (c) the cost score for the second recognition agent process, terminating the first recognition agent process and initiating the second agent process;

wherein operation of the device adapts in response to changing context, by terminating the first recognition agent process in favor of the second recognition agent process, without express user instruction to do so.

3. The method of claim 1 that further includes:

determining a relevance score for each of plural candidate recognition agent processes that the mobile device may run, the relevance score for a first of said candidate recognition agent processes being based on data including at least one data selected from the list consisting of;

location, sensor data available in a memory, context, expressed user intent, and user history;

determining a cost score for each of said plural candidate recognition agent processes, the cost score for a first of said candidate recognition agent processes being based on data including at least one data selected from the list consisting of;

memory usage, CPU usage, and communication bandwidth; and

determining, using both the relevance and cost scores, a further recognition agent process to initiate.

4. The method of claim 1 in which the act of varying the allocation of processing resources to the first recognition agent process depends on a derivative of said detection state data.

5. The method of claim 1 in which the act of varying the allocation of processing resources to the first recognition agent process depends on a speed or acceleration at which the first recognition agent process is progressing in performing the selected recognition.

6. The method of claim 1, performed first and second times, wherein the first time the method is performed, the first recognition agent process is terminated when the detection state data has a first value, and the second time the method is performed, the first recognition agent process is not terminated when the detection state data has said first value, because the received first user interest data indicates a higher level of user interest the second time the method is performed, than the first time the method is performed.

7. The method of claim 1 in which the received first user interest data comprises data related to express user encouragement of the first recognition agent process.

8. The method of claim 1 in which the received first user interest data comprises data related to implied user encouragement of the first recognition agent process.

9. The method of claim 8 in which the implied user encouragement comprises a pose at which the user positions the mobile device.

10. The method of claim 9 in which the implied user encouragement comprises an orientation at which the user positions the mobile device.

11. The method of claim 1 in which the first user interest data comprises accelerometer data.

12. The method of claim 1 in which the first user interest data comprises data relating to a camera zoom function.

13. The method of claim 1 in which the first user interest data comprises user positioning of the device to place a subject of the first recognition agent process at a position remote from a center of a camera field of view.

14. The method of claim 1 that further includes:

initiating one or more stages of a second recognition agent process to recognize audio or image data captured by the mobile device, the second recognition agent process being different than the first recognition agent process, the second recognition agent process performing a second recognition selected from the list consisting of;

image watermark recognition, image recognition, object recognition, facial recognition, barcode recognition, optical character recognition, audio watermark recognition, speech recognition, speaker recognition, or music recognition;

receiving second detection state data indicating a state of the second recognition agent process in performing the selected second recognition;

after initiating the second recognition agent process, receiving second user interest data indicating interest of the user in obtaining a result of the second recognition agent process; and

varying an allocation of processing resources to the second recognition agent process based on both said second detection state data and on said second user interest data;

wherein the first and second recognition agent processes are performed concurrently.

15. The method of claim 1 in which the device includes a touch screen display and the method also includes:

initiating a discovery mode, in response to a request received from the user via a device input, in which the discovery mode includes processing camera-captured imagery to recognize an object depicted therein using said first recognition agent process;

presenting captured imagery, including video, in a first area of said display of said device, along with augmented reality graphics;

controlling a user interface to display user-selectable graphic icons in a second area of said display, wherein the user-selectable graphic icons include visible indicia to graphically represent content;

controlling the user interface to present a user-selectable graphic icon on the display to facilitate switching discovery modes from an image discovery mode to an audio discovery mode; and

wherein the audio discovery mode includes processing microphone-captured audio to determine identification therefrom, employing an audio recognition agent process.

16. A method practiced by a battery-powered mobile wireless communications device equipped with a processor, memory, wireless communication interface, camera and microphone, the device having multiple recognition modes enabling recognition of multiple types of content, the method comprising the acts:

initiating one or more stages of a first recognition agent process to recognize audio or image content data captured by the mobile device, the first recognition agent process performing a recognition selected from the list consisting of;

image watermark recognition, image recognition, object recognition, facial recognition, barcode recognition, optical character recognition, audio watermark recognition, speech recognition, speaker recognition, or music recognition;

receiving detection state data indicating a state of the first recognition agent process in performing the selected recognition;

determining a relevance score for a second recognition agent process that the mobile device may run, the relevance score being based on data including at least one data selected from a list consisting of;

location, sensor data available in the memory, context, expressed user intent, and user history;

determining a cost score for said second recognition agent process, the cost score being based on data including at least one data selected from a list consisting of;

memory usage, processor usage, and communication bandwidth; and

varying an allocation of processing resources to the first recognition agent process based on at least (a) the detection state data for the first recognition agent process, (b) the relevance score of the second recognition agent process, and (c) the cost score for the second recognition agent process.

17. The method of claim 16 in which:

said varying the allocation of processing resources to the first recognition agent process comprises terminating the first recognition agent process; and

the method further includes initiating the second recognition agent process;

wherein operation of the device adapts in response to changing context, by terminating the first recognition agent process in favor of the second recognition agent process, without express user instruction to do so.

18. The method of claim 16 that further includes:

after initiating the first recognition agent process, receiving first user interest data indicating interest of the user in obtaining a result of the first recognition agent process; and

varying the allocation of processing resources to the first recognition agent process based on at least (a) the detection state data for the first recognition agent process, (b) the relevance score of the second recognition agent process, (c) the cost score for the second recognition agent process, and (d) the first user interest data.

19. A non-transitory computer readable medium containing software instructions adapted to configure a battery-powered, camera- and microphone-equipped mobile wireless communications device to perform acts including:

initiating one or more stages of a first recognition agent process to recognize audio or image content data captured by the mobile device, the first recognition agent process performing a recognition selected from a list consisting of;

image watermark recognition, image recognition, object recognition, facial recognition, barcode recognition, optical character recognition, audio watermark recognition, speech recognition, speaker recognition, or music recognition;

receiving detection state data indicating a state of the first recognition agent process in performing the selected recognition;

after initiating the first recognition agent process, receiving first user interest data indicating interest of the user in obtaining a result of the first recognition agent process; and

varying an allocation of processing resources to the first recognition agent process based on both said detection state data and on said user interest data;

wherein said software instructions serve to optimize use of the device battery by allocating processing resources in response both to the state of the first recognition agent process in performing the selected recognition, and to user input received after the first recognition agent process has been initiated, indicating user interest in the result of the first recognition agent process.

20. The computer readable storage medium of claim 19 in which said software instructions are further adapted to configure the device to perform acts including:

determining a relevance score for a second recognition agent process that the mobile device may run, the relevance score being based on data including at least one data selected from a list consisting of;

location, sensor data available in a memory, context, expressed user intent, and user history;

determining a cost score for said second recognition agent process, the cost score being based on data including at least one data selected from a list consisting of;

memory usage, CPU usage, and communication bandwidth; and

based on at least (a) the detection state data for the first recognition agent process, (b) the relevance score of the second recognition agent process, and (c) the cost score for the second recognition agent process, terminating the first recognition agent process and initiating the second agent process;

wherein the software instructions adapt operation of the device in response to changing context, by terminating the first recognition agent process in favor of the second recognition agent process, without express user instruction to do so.

21. A battery-powered wireless system including:

plural sensors, including a camera and a microphone;

a wireless communications interface;

a battery;

one or more battery-powered processors; and

a memory including software instructions, the software instructions configuring the system to perform acts including;

initiating one or more stages of a first recognition agent process to recognize audio or image content data captured by one of said sensors, the first recognition agent process performing a recognition selected from a list consisting of;

image watermark recognition, image recognition, object recognition, facial recognition, barcode recognition, optical character recognition, audio watermark recognition, speech recognition, speaker recognition, or music recognition;

receiving detection state data indicating a state of the first recognition agent process in performing the selected recognition;

after initiating the first recognition agent process, receiving first user interest data indicating interest of the user in obtaining a result of the first recognition agent process; and

varying an allocation of processing resources to the first recognition agent process based on both said detection state data and on said user interest data;

wherein the system is configured to optimize use of the battery by allocating processing resources in response both to the state of the first recognition agent process in performing the selected recognition, and to user input received after the first recognition agent process has been initiated, indicating user interest in the result of the first recognition agent process.