INTUITIVE COMPUTING METHODS AND SYSTEMS

• US 20170289341A1
• Filed: 06/12/2017
• Published: 10/05/2017
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F., Rhoads, Geoffrey B., Davis, Bruce L., Shaw, Gilbert B.
• Granted Patent: US 10,785,365 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: H04M 1/72448 with means for adapting the functionality of the device according to specific conditions
• CPC Class Codes: H04M 1/72469 for operating the device by selecting functions from two or more displayed items, e.g. menus or icons
• CPC Class Codes: H04W 72/02 Selection of wireless resources by user or terminal

Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user'"'"'s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone'"'"'s camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user'"'"'s apparent interest in the task. In some arrangements, the phone is guided in various of its intuitive computing operations by user-spoken clues. A discovery session may be launched by the user speaking a cueing expression, which serves to switch the device from a lower activity state to a heightened alert state. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Claims

1-82. -82. (canceled)

83. A method employing a device equipped with a processor, a display, a camera and a microphone, the camera capturing imagery depicting plural items in a user'"'"'s physical environment, the method comprising the acts:

capturing first speech of the user, with the device microphone;

the device processor detecting that the captured first speech includes a cueing expression, and in response to detection of the cueing expression, the device switching from a lower activity state to a heightened alert state, in the heightened alert state the device performing functionality including;

capturing second user speech with the device microphone;

sending data corresponding to the second user speech to a recognition module, and receiving recognized second speech data in return, the recognized second user speech indicating one of said plural items depicted in the captured imagery as of particular user interest;

based on one or more descriptors included in the recognized second speech data, determining a first of said plural depicted items as being of likely user interest;

presenting a marking on the device display, at a location indicating said first item;

capturing third user speech with the device microphone, the captured third user speech being different than the second user speech;

sending data corresponding to the third user speech to the recognition module, and receiving recognized third speech data in return, the recognized third speech data again indicating one of said plural items as of particular user interest;

based on one or more descriptors included in the recognized third speech data, determining that a second, different one of said plural depicted items is of greater interest to the user than the first item;

moving said marking on the device display to a location indicating said second item; and

taking an action based on the second item, said action including presenting information related to the second item to the user;

wherein the device is not on heightened alert all the time, but is cued into activation from a lower activity state by the cueing expression, thereby bounding the device'"'"'s processing efforts, and the descriptors in the recognized second and third speech data iteratively guide the device in identifying which of the plural items in the user'"'"'s physical environment is of user interest, thereby further bounding the device'"'"'s processing efforts.

84. The method of claim 83 that includes sending data corresponding to the second item to a recognition engine, receiving second item recognition data in return, and taking an action based on the second item recognition data.

85. The method of claim 84 in which the second item is a face of a person, and the method includes sending data corresponding to said face to a facial recognition engine, receiving information about said person in return, and presenting the received information to the user.

86. The method of claim 83 that further includes receiving a user confirmation that the second item is of interest, prior to presenting information related to the second item to the user.

87. The method of claim 83 that further includes presenting information related to the second item to the user on the device display.

88. The method of claim 83 in which said device is battery-powered, wherein the method includes the device processor detecting the cueing expression using battery power.

89. A method employing a device equipped with a processor, a camera and a microphone, the camera capturing imagery depicting plural items in a user'"'"'s physical environment, the method comprising the acts:

capturing first speech of the user, with the device microphone;

detecting, with said device processor, that the captured first speech includes a cueing expression;

in response to detection of the cueing expression, switching the device from a lower activity state to a heightened alert state, in the heightened alert state the device performing functionality including;

capturing second speech of the user;

sending data from the device, said data including data corresponding to the second user speech and data corresponding to the captured imagery, and receiving data, including (a) recognized second speech data and (b) recognition-processed data about a subject depicted in the imagery, in return; and

taking an action based on said received data, including presenting information based on the recognition-processed data to the user;

wherein the device is not on heightened alert all the time, but is cued into activation from a lower activity state by the cueing expression, thereby bounding the device'"'"'s processing efforts, and wherein, in its heightened alert state, the device cooperates with a remote computer system to recognition-process imagery captured by the device camera.

90. The method of claim 89 in which said imagery depicts plural items in a user'"'"'s physical environment, and said functionality further includes:

based on one or more descriptors included in the recognized second speech data, determining a first of said plural depicted items as being of likely user interest;

presenting a marking on a display of the device, at a location indicating said first item;

capturing third user speech with the device microphone;

sending data corresponding to the third user speech from the device, and receiving recognized third speech data in return;

based on one or more descriptors included in the recognized third speech data, determining that a second, different one of said plural depicted items is of greater interest to the user than the first item;

moving said marking on the device display to a location indicating said second item;

receiving a user confirmation that the second item is of interest; and

sending data corresponding to the second item from the device for recognition processing;

wherein descriptors in the recognized second and third speech data iteratively guide the device in identifying which of the plural items in the user'"'"'s physical environment is of user interest, thereby further bounding the device'"'"'s processing efforts.

91. The method of claim 89 in which the cueing expression consists of two words.

92. The method of claim 89 in which said device is battery-powered, wherein the method includes the device processor detecting the cueing expression using battery power.

93. A tangible computer readable medium containing instructions to configure a device, equipped with a display, a camera and a microphone, to perform acts including:

capturing first speech of the user;

detecting that the captured first speech includes a cueing expression, and in response to detection of the cueing expression, switching the device from a lower activity state to a heightened alert state, in the heightened alert state the instructions configuring the device to perform functions including;

capturing second user speech;

sending data corresponding to the second user speech to a recognition module, and receiving recognized second speech data in return;

based on one or more descriptors included in the recognized second speech data, determining a first of said plural depicted items as being of likely user interest;

presenting a marking on the display, at a location indicating said first item;

capturing third user speech, the captured third user speech being different than the second user speech;

sending data corresponding to the third user speech to the recognition module, and receiving recognized third speech data in return;

based on one or more descriptors included in the recognized third speech data, determining that a second, different one of said plural depicted items is of greater interest to the user than the first item;

moving said marking on the display to a location indicating said second item; and

taking an action based on the second item, said action including presenting information related to the second item to the user;

wherein the device is not on heightened alert all the time, but is cued by said instructions into activation from a lower activity state by the cueing expression, thereby bounding the device'"'"'s processing efforts, and the instructions enable descriptors in the recognized second and third speech data to iteratively guide the device in identifying which of the plural items in the user'"'"'s physical environment is of user interest, thereby further bounding the device'"'"'s processing efforts.