Intuitive computing methods and systems

• US 8,121,618 B2
• Filed: 02/24/2010
• Issued: 02/21/2012
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rhoads, Geoffrey B., Rodriguez, Tony F., Shaw, Gilbert B.
• Primary Examiner(s): MIZRAHI, DIANE D
• Application Number:
  US12/712,176
• Publication Number:
  US 20110098056A1
• Time in Patent Office: 727 Days
• Field of Search: 701/202, 348/222.1, 705/14.1, 358/442, 345/156, 345/174, 715/810, 455/425, 455/456.3, 455/556.1, 455/456.1, 455/456.2
• US Class Current: 455/456.1
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/36 Input/output arrangements for on-board computers
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04N 23/00 : Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/027 : using movement velocity, acceleration information
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services

Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user'"'"'s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone'"'"'s camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user'"'"'s apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Claims

1. A method of operating a camera-equipped smart phone device that processes image data, the device being conveyed by a user, the method including the following acts:

performing an initial set of plural different image processing operations on the image data, using processing hardware in the smart phone device that is configured to perform such act; and

without express user command, invoking additional image processing operations based on one or more circumstances, said circumstance(s) including at least one circumstance not based on the image data, said additional image processing operations serving to extract meaning from the image data in connection with an inferred or anticipated user action.

2. The method of claim 1 that includes storing, or arranging for the storage of, data objects resulting from one or more of said image processing operations, and transmitting semantic assertions relating to said data objects to a remote linked data registry.

3. The method of claim 1 that includes discerning one or more visual features within a scene represented by the image data, and presenting visual baubles on a screen of the device at location(s) corresponding to said visual feature(s) in the scene.

4. The method of claim 3 wherein the baubles are non-rectangular in shape.

5. The method of claim 3 that includes sensing a user'"'"'s gesture on the device screen in relation to one or more baubles, and taking an action based thereon.

6. The method of claim 5 in which the action includes at least one of:

(a) allocating more or less processing resources to a function associated with a bauble, said function having been initiated prior to sensing the user'"'"'s gesture;

(b) curtailing a process associated with a bauble, and storing information related thereto so that a user preference or pattern of behavior can be discerned;

(c) at least temporarily curtailing a process associated with a bauble on the device, while continuing a related process in a remote processing system;

(d) editing an image to exclude one or more features;

(e) changing a projection of one or more features in image data presented on the device screen; and

(f) defining a social relationship between entities represented by plural baubles.

7. The method of claim 6 in which the action includes allocating more or less processing resources to a function associated with a bauble, said function having been initiated prior to sensing the user'"'"'s gesture.

8. The method of claim 6 in which the action includes curtailing a process associated with a bauble, and storing information related thereto so that a user preference or pattern of behavior can be discerned.

9. The method of claim 6 in which the action includes at least temporarily curtailing a process associated with a bauble on the device, while continuing a related process in a remote processing system.

10. The method of claim 6 in which the action includes editing an image to exclude one or more features.

11. The method of claim 6 in which the action includes changing a projection of one or more features in image data presented on the device screen.

12. The method of claim 6 in which the action includes defining a social relationship between entities represented by plural baubles.

13. The method of claim 3 that includes perspectively warping at least one of said presented baubles to correspond to a surface feature discerned in the scene.

14. The method of claim 3 that includes changing brightness, shape, or size of one of the presented baubles when one or more of the image processing operations makes progress towards a desired result, such as recognizing or identifying a feature in the scene.

15. The method of claim 14 that includes changing brightness of one of the presented baubles when one or more of the image processing operations makes progress towards a desired result, such as recognizing or identifying a feature in the scene.

16. The method of claim 14 that includes changing shape of one of the presented baubles when one or more of the image processing operations makes progress towards a desired result, such as recognizing or identifying a feature in the scene.

17. The method of claim 14 that includes changing size of one of the presented baubles when one or more of the image processing operations makes progress towards a desired result, such as recognizing or identifying a feature in the scene.

18. The method of claim 1 wherein the invoking act includes invoking additional image processing operations based on a circumstance including at least one of:

(a) location;

(b) time of day;

(c) proximity to one or more people;

(d) an output based on the initial set of image processing operations;

or

(e) a statistical model of user behavior.

19. The method of claim 18 wherein the invoking act includes invoking additional image processing operations based on a circumstance including location.

20. The method of claim 18 wherein the invoking act includes invoking additional image processing operations based on a circumstance including time of day.

21. The method of claim 18 wherein the invoking act includes invoking additional image processing operations based on a circumstance including proximity to one or more people.

22. The method of claim 18 wherein the invoking act includes invoking additional image processing operations based on a circumstance including an output based on the initial set of image processing operations.

23. The method of claim 18 wherein the invoking act includes invoking additional image processing operations based on a statistical model of user behavior.

24. The method of claim 1 that includes inferring, from data including results from one or more of said image processing operations, information about a type of interaction desired by the user, and invoking additional image processing operations based on such information.

25. The method of claim 1 that also includes sending data to a remote system so that the remote system can perform one or more of the same image processing operations as said device.

26. The method of claim 1 wherein the device acts autonomously to determine the value of a group of coins imaged by the device'"'"'s camera.

27. The method of claim 1 that includes selecting a first set of additional image processing operations to be performed, from a larger second set of possible image processing operations, based on data indicating one or more of:

(a) device resource usage;

(b) resource demands associated with different of the possible operations; and

(c) correspondence between different of the possible operations.

28. The method of claim 27 that includes selecting a first set of additional image processing operations to be performed, from a larger second set of possible image processing operations, based—

at least in part—

on data indicating device resource usage.

29. The method of claim 27 that includes selecting a first set of additional image processing operations to be performed, from a larger second set of possible image processing operations, based—

at least in part—

on data indicating resource demands associated with different of the possible operations.

30. The method of claim 27 that includes selecting a first set of additional image processing operations to be performed, from a larger second set of possible image processing operations, based—

at least in part—

on data indicating correspondence between different of the possible operations.

31. The method of claim 1 that includes discerning one or more visual features within a scene represented by the image data, and storing data related to each such feature in association with a corresponding identifier, wherein the identifier is based on at least two of the following:

(a) a session ID;

(b) an explicit object ID; and

(c) data derived from the feature, or derived from a related circumstance.

32. The method of claim 1 that includes using a non-image sensor system in the device to produce non-image information, and employing such information for at least one of the following:

(a) influencing selection of image processing operations; and

(b) disambiguating between two or more candidate conclusions about the image data;

wherein the non-image sensor system includes at least one of a geolocation system, an audio sensor, a temperature sensor, a magnetic field sensor, a motion sensor, or an olfactory sensor.

33. The method of claim 32 that includes employing the non-image information for influencing selection of image processing operations.

34. The method of claim 32 that includes employing the non-image information for disambiguating between two or more candidate conclusions about the image data.

35. The method of claim 32 wherein the non-image sensor comprises a geolocation system, and the method includes using the geolocation system to produce said non-image information.

36. The method of claim 32 wherein the non-image sensor comprises an audio sensor, and the method includes using the audio sensor to produce said non-image information.

37. The method of claim 32 wherein the non-image sensor comprises a temperature sensor, and the method includes using the temperature sensor to produce said non-image information.

38. The method of claim 32 wherein the non-image sensor comprises a magnetic field sensor, and the method includes using the magnetic field sensor to produce said non-image information.

39. The method of claim 32 wherein the non-image sensor comprises a motion sensor, and the method includes using the motion sensor to produce said non-image information.

40. The method of claim 32 wherein the non-image sensor comprises an olfactory sensor, and the method includes using the olfactory sensor to produce said non-image information.

41. The method of claim 1 that further includes transmitting at least certain of the image data, or data from one or more of the image processing operations, to a remote computer system, so the remote computer system can continue image processing earlier performed by the device, in an effort to glean information that the device—

in its processing—

did not discern.

42. A smart phone device including:

a wireless portion;

a camera portion; and

a processing portion coupled to the camera portion and the wireless portion;

the processing portion being configured to perform an initial set of plural different image processing operations on data from the camera portion and, without express user command, invoke additional image processing operations based on one or more circumstances;

wherein said circumstance(s) includes at least one circumstance not based on the image data; and

wherein the additional image processing operations serve to extract meaning from the image data in connection with an inferred or anticipated user action.