Intuitive computing methods and systems

• US 9,118,771 B2
• Filed: 03/08/2013
• Issued: 08/25/2015
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F.
• Primary Examiner(s): MIZRAHI, DIANE D
• Application Number:
  US13/790,607
• Publication Number:
  US 20130324161A1
• Time in Patent Office: 900 Days
• Field of Search: 455/456.1, 705/14.4, 707/736
• US Class Current: 1/1
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/36 Input/output arrangements for on-board computers
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04N 23/00 : Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/027 : using movement velocity, acceleration information
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services

Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user'"'"'s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone'"'"'s camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user'"'"'s apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Claims

1. A cell phone device comprising a processor, a memory, a display, and a microphone, the memory containing programming instructions that configure the device to perform acts including:

capturing first and second items of audio content using said cell phone microphone;

presenting, simultaneously on said display, a first graphical indicia corresponding to said first item of audio content, and a second graphical indicia corresponding to said second item of audio content;

presenting a geographical map, with a plurality of third graphical indicia, on said display, at least one of the third graphical indicia corresponding to an item of audio content that was previously discovered, and being presented on the map at a position corresponding to a geographical location at which said previously discovered item of audio content was sampled, the map including plural roads; and

responding to user selection of one of said indicia by presenting additional information about an audio item corresponding thereto.

2. The device of claim 1 in which the instructions further configure the device to present, on said map, a further indicia indicating a current location of a user of the device.

3. The device of claim 2 in which the instructions further configure the device to provide audio information sampled by the microphone to a content recognition module, and to present resultant metadata on the display.

4. The device of claim 3 in which the resultant metadata indicates that the audio information corresponds to a birdsong.

5. The device of claim 3 in which the resultant metadata indicates that the audio information corresponds to road noise.

6. The device of claim 3 in which the resultant metadata indicates that the audio information corresponds to music.

7. The device of claim 3 in which the resultant metadata indicates that the audio information corresponds to speech.

8. The device of claim 3 in which the instructions configure the device to present the map on a first area of the display, and to present the resultant metadata on a second area of the display different than the first area, the first and second areas being separated by a straight border.

9. The device of claim 1 in which the third graphical indicia are time-filtered, so that only indicia corresponding to discoveries within a specified previous interval of time are presented on the display.

10. The device of claim 9 in which said instructions perform said time-filtering.

11. The device of claim 1 in which the instructions further configure the device to present a graphical control by which a user can change an interval of time for which third indicia are presented on the display, wherein the user can employ said control to present more or less third indicia on the display.

12. The device of claim 1 in which the instructions further configure the device to filter the third indicia by a user-set criterion, so that only third indicia corresponding to said criterion are presented.

13. The device of claim 1 in which the instructions further configure the device to respond to user input to zoom-in on the display, changing a scale at which the map is presented.

14. The device of claim 1 wherein the first graphical indicia is different than the second graphical indicia.

15. The device of claim 1 wherein at least one of said third graphical indicia corresponds to an item of visual content that was previously discovered.

16. The device of claim 1, wherein the first graphical indicia comprises an image of a person.

17. The device of claim 1 wherein said instructions configure the device to simultaneously present said map and said first and second graphical indicia.

18. The device of claim 1 wherein said instructions enable the device to perform music recognition, and at least one of speaker recognition or speech recognition.

19. The device of claim 1 wherein said instructions enable the device to perform music recognition, speaker recognition, and speech recognition.

20. The device of claim 1 wherein said instructions configure the device to perform visual object recognition, wherein said map includes a third indicium corresponding to a recognized visual object.

21. A non-transitory computer readable medium containing software instructions that enable a cell phone device programmed thereby to perform acts including:

presenting a geographical map and a plurality of first graphical indicia on a display of the cell phone, each of the first indicia representing a corresponding audio content recognition event, and being presented on the map at a position corresponding to a geographical location at which audio for the audio content recognition event was sampled;

responding to user operation of a graphical time-based filter control to present on said map only graphical indicia corresponding to audio content recognition events that took place within a particular previous interval of time; and

responding to user selection of one of said indicia by presenting additional information about the audio content recognition event.

22. A method comprising the acts:

presenting a map on a display of a device, the map depicting plural roads and including indicia corresponding to plural audio content recognition events, each indicium being presented on the map at a position corresponding to a geographical location at which audio for the audio content recognition event was sampled, the map being presented in a first part of a user interface;

responding to user selection of one of said indicia by presenting additional information about the corresponding audio content recognition event;

sampling ambient audio with a microphone of the device, and providing associated information to a content recognition module; and

presenting metadata received from said content recognition module on said display;

wherein one or more of said acts is performed by a hardware processor of a cell phone; and

wherein the audio content recognition event comprises (a) recognizing words spoken by a person whose speech was captured by said microphone;

or (b) recognizing said person.

23. The method of claim 22 that further includes presenting a pin indicium on the map to indicate the metadata received from the content recognition module.

24. The method of claim 22 wherein said providing comprises sending said associated information to a content recognition module at a processing system remote from the device.