INTUITIVE COMPUTING METHODS AND SYSTEMS

• US 20140337733A1
• Filed: 07/22/2014
• Published: 11/13/2014
• Est. Priority Date: 10/28/2009
• Status: Abandoned Application

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F., Rhoads, Geoffrey B.
• Application Number:
  US14/337,607
• Publication Number:
  US 20140337733A1
• Time in Patent Office: Days
• US Class Current: 715/718
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: H04M 1/72448 with means for adapting the functionality of the device according to specific conditions
• CPC Class Codes: H04M 1/72469 for operating the device by selecting functions from two or more displayed items, e.g. menus or icons
• CPC Class Codes: H04W 72/02 Selection of wireless resources by user or terminal

Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user'"'"'s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone'"'"'s camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user'"'"'s apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Claims

1-4. -4. (canceled)

5. A portable device including a processor, a memory, a microphone, a camera, a touch screen, and a wireless interface, the memory containing software instructions that cause the device to perform acts including:

displaying, on the touch screen, a user interface enabling a user to select between audio and image recognition modalities;

responding to user selection of the audio recognition modality by initiating fingerprint processing of audio stimulus information captured by the microphone, said processing leading to receipt of first data identifying a source of the captured audio stimulus;

presenting first information on the touch screen about the audio stimulus, based on the received first data;

responding to user selection of the image recognition modality by initiating processing of image stimulus information captured by the camera, said processing leading to receipt of second data identifying an object imaged by the camera;

presenting second information on the touch screen about the image stimulus, based on the received second data; and

displaying, on the touch screen, a graphical feature in the user interface that enables the user to review both earlier-presented first information about the audio stimulus and also earlier-presented second information about the image stimulus.

6. The portable device of claim 5 in which the user interface includes first and second elements, the first element providing information about the captured stimulus, the second element providing a map detailing where the stimulus was captured.

7. The portable device of claim 5 in which the captured audio stimulus comprises a person'"'"'s speech, and said fingerprint processing of the audio stimulus information leads to receipt of first data identifying said person.

8. The portable device of claim 5 in which the captured image stimulus information represents a barcode, and said processing of the image stimulus information leads to receipt of second data identifying an item marked by said barcode.

9. The portable device of claim 5 in which the software instructions cause the device to display, on the touch screen, a user interface including a button that, in response to repeated touching, causes the device to toggle between audio and image recognition modalities.

20. The device of claim 5 in which said acts include displaying, on the touch screen, a single history button in the user interface, selection of said single history button enabling the user to review both earlier-presented first information about the audio stimulus and also earlier-presented second information about the image stimulus.

33. The device of claim 9 in which said software instructions cause the device to respond to said repeated touching of the button by causing an indicia associated with said button to alternate between first and second appearances, said first and second appearances respectively corresponding to the image recognition modality and the audio recognition modality.

10. A seeing/hearing method comprising the acts:

capturing stimulus data from a user'"'"'s environment, the stimulus including both imagery and audio, said capturing employing sensors in a user-carried portable device;

processing data resulting from said capturing, said processing including performing recognition-processing on image data to yield data about an object depicted in the imagery, said processing also performing recognition-processing on audio data to yield data identifying a source of the audio, said recognition-processing actions operating in cyclical serial fashion or in parallel fashion to serve both seeing and hearing functions; and

presenting results based on said processing of the captured stimulus data to a user.

11. The method of claim 10 in which the recognition-processing of image data includes:

processing the image data with at least first, second and third different initial image processing operations, to thereby produce at least first, second and third different sets of processed data derived from the image data;

based at least in part on said processed data derived from the image data, launching a further data processing operation;

making an assessment concerning an output of the further data processing operation; and

allocating resources to said further data processing operation in an amount based, at least in part, on said assessment.

12. The method of claim 10 in which the audio comprises a person'"'"'s speech, and said recognition-processing of the audio data yields data identifying said person.

13. The method of claim 10 that further includes automatically selecting and undertaking processing operations responsive to the captured audio and imagery stimulus.

14. The method of claim 10 that further includes automatically selecting and undertaking processing operations responsive to the captured imagery stimulus.

15. The method of claim 10 that further includes automatically selecting and undertaking processing operations responsive to the captured audio stimulus.

16. The method of claim 10 that further includes storing audio data and image data resulting from said capturing in a common data structure, and recalling the stored data from said common data structure for said processing.

17-19. -19. (canceled)

21. A seeing/hearing method comprising the acts:

receiving stimulus data captured by microphone and camera sensors of a user'"'"'s portable device;

displaying, on a touch screen of said device, a user interface enabling the user to select between audio and image recognition modalities;

responding to user selection of the audio recognition modality by initiating fingerprint processing of audio stimulus information captured by the microphone, said processing leading to receipt of first data identifying a source of the captured audio stimulus;

presenting first information on the touch screen about the audio stimulus, based on the received first data;

responding to user selection of the image recognition modality by initiating processing of image stimulus information captured by the camera, said processing leading to receipt of second data identifying an object imaged by the camera;

presenting second information on the touch screen about the image stimulus, based on the received second data; and

displaying, on the touch screen, a graphical feature in the user interface that enables the user to review both earlier-presented first information about the audio stimulus and also earlier-presented second information about the image stimulus.

22. The method of claim 21 in which the user interface includes first and second elements, the first element providing information about the captured stimulus, the second element providing a map detailing where the stimulus was captured.

23. The method of claim 21 in which the captured audio stimulus comprises a person'"'"'s speech, and said fingerprint processing of the audio stimulus information leads to receipt of first data identifying said person.

24. The method of claim 21 in which the captured image stimulus information represents a barcode, and said processing of the image stimulus information leads to receipt of second data identifying an item marked by said barcode.

25. The method of claim 21 in which the user interface includes a button that, in response to repeated touching, causes the device to toggle between audio and image recognition modalities.

26. A portable device including a processor, a memory, a microphone, a camera, a touch screen, and a wireless interface, the memory containing software instructions that cause the device to perform acts including:

receiving stimulus data captured from a user'"'"'s environment by said camera and microphone, the stimulus including both imagery and audio;

processing data resulting from said capturing, said processing including performing recognition-processing on image data to yield data about an object depicted in the imagery, said processing also performing recognition-processing on audio data to yield data identifying a source of the audio, said recognition-processing actions operating in cyclical serial fashion or in parallel fashion to serve both seeing and hearing functions; and

presenting results based on said processing of the captured stimulus data on said touch screen.

27. The device of claim 26 in which said act of recognition-processing of image data includes:

processing the image data with at least first, second and third different initial image processing operations, to thereby produce at least first, second and third different sets of processed data derived from the image data;

based at least in part on said processed data derived from the image data, launching a further data processing operation;

making an assessment concerning an output of the further data processing operation; and

allocating resources to said further data processing operation in an amount based, at least in part, on said assessment.

28. The device of claim 26 in which said audio comprises a person'"'"'s speech, and said recognition-processing of the audio data yields data identifying said person.

29. The device of claim 26 in which said acts further include automatically selecting and undertaking processing operations responsive to the captured audio and imagery stimulus.

30. The device of claim 26 in which said acts further include automatically selecting and undertaking processing operations responsive to the captured imagery stimulus.

31. The device of claim 26 in which said acts further include automatically selecting and undertaking processing operations responsive to the captured audio stimulus.

32. The device of claim 26 in which said acts further include storing audio data and image data resulting from said capturing in a common data structure, and recalling the stored data from said common data structure for said processing.