Intuitive computing methods and systems

• US 10,270,896 B2
• Filed: 02/05/2018
• Issued: 04/23/2019
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rhoads, Geoffrey B.
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US15/889,083
• Publication Number:
  US 20180248995A1
• Time in Patent Office: 442 Days
• Field of Search: 701517, 455411, 4554561
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/36 Input/output arrangements for on-board computers
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/027 : using movement velocity, acceleration information
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services

Abstract

A device senses audio, imagery, and/or other stimulus from a user'"'"'s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns device-based cognition of a scene viewed by the device'"'"'s camera. Tasks, which can be selected with the aid of context, are allocated increased or decreased resources based on data comprising (a) user input data indicating express or implied encouragement or discouragement of the task and/or (b) a detection state metric, representing a quantified likelihood that a goal sought by the task will be reached. A great number of other features and arrangements are also detailed.

Claims

1. An image processing method, executed by a hardware device, comprising the acts:

receiving imagery captured by a camera;

launching an operation, the operation employing the imagery as an input;

receiving first data, after the operation has been launched, said first data comprising (a) user input data indicating express or implied user encouragement or discouragement of said operation by the user, and/or (b) a detection state metric, representing a quantified likelihood that a goal sought by the operation will be reached; and

allocating increased or decreased resources to said operation based on said received first data, said allocation of increased resources being apart from increased resource usage due to any increasing base complexity of successive stages of the operation.

2. The method of claim 1 in which the received first data comprises user input data indicating user encouragement of said operation, and the method includes allocating increased resources to said operation based on said indication of encouragement.

3. The method of claim 1 in which the received first data indicates comprises user input data indicating user discouragement of said operation, and the method includes allocating reduced resources to said operation based on said indication of discouragement.

4. The method of claim 1 in which the received first data comprises a detection state metric representing a quantified likelihood that a goal sought by the operation will be reached, and the method includes allocating increased resources to the operation as a consequence of said detection state metric.

5. The method of claim 1 in which the received first data comprises a detection state metric representing a quantified likelihood that the goal sought by the operation will be reached, and the method includes allocating decreased resources to the operation as a consequence of said detection state metric.

6. The method of claim 1 that includes:

receiving user data indicating express or implied user encouragement or discouragement of the operation; and

receiving a detection state metric representing a quantified likelihood that the goal sought by the operation will be reached;

wherein the method includes allocating resources to said operation based on both (1) said received user data; and

(2) said received detection state metric.

7. The method of claim 1 that includes selecting said operation from among several operations that the system is able to apply to imagery, said selecting of the operation being based on context information rather than on express user instruction.

8. The method of claim 7 that includes selecting said operation, from the several operations the system is equipped to apply, based on results of baseline image processing operations applied to the received imagery.

9. The method of claim 7 that includes selecting said operation, from the several operations the system is equipped to apply, based on context information distinct from image data.

10. The method of claim 7 in which the context information comprises location information.

11. The method of claim 1 in which the context information comprises image information.

12. The method of claim 1 in which allocating increased resources to said operation comprises making GPU resources available instead of CPU resources, allocating more scratchpad memory for calculations, allowing access to faster-responding memory, allowing increased power consumption, or allowing access to a faster network connection.

13. The method of claim 1 in which the operation is a recognition operation to discern information about a subject depicted in the imagery.

14. The method of claim 1 in which:

the method is executed by a battery-powered, headworn hardware device; and

said receiving comprises receiving imagery captured by a camera of said battery-powered, headworn hardware device.

15. The method of claim 14 in which said operation comprises an operation for recognizing a subject depicted in said imagery captured by the camera of said battery-powered, headworn hardware device.

16. The method of claim 14 in which said operation comprises an operation for recognizing a face depicted in said imagery captured by the camera of said battery-powered, headworn hardware device.

17. The method of claim 14 in which said operation comprises an operation for recognizing a bar code or digital watermark depicted in said imagery captured by the camera of said battery-powered, headworn hardware device.

18. The method of claim 14 in which said operation comprises an operation for recognizing text depicted in said imagery captured by the camera of said battery-powered, headworn hardware device.

19. The method of claim 14 that includes allocating said increased or decreased resources to the operation based on a first- or higher-order derivative of said detection state metric.

20. The method of claim 14 that further includes said operation skipping one or more stages in a normal sequence of operational stages, based on said detection state metric.

21. The method of claim 14 in which different tasks associated with said operation are assigned to different cores of a multi-processor system.

22. The method of claim 14 in which said operation comprises a first recognition agent that is configured to recognize a first type of object, the method including:

allocating increased resources to said first recognition agent based on said first received data, and allocating decreased resources to a second recognition agent that is configured to recognize a second type of object;

wherein an increase in detection state metric associated with said first recognition agent serves as negative encouragement for the second recognition agent.

23. The method of claim 14 in which allocating increased resources to said operation comprises making GPU resources available instead of CPU resources, allocating more scratchpad memory for calculations, allowing access to faster-responding memory, allowing increased power consumption, or allowing access to a faster network connection.

24. The method of claim 1 that further includes generating said detection state metric as part of said method.

25. A non-transitory computer readable storage medium containing software instructions to configure a processor-equipped computer system to perform acts including:

receiving imagery captured by a camera;

launching an operation, the operation employing the imagery as an input;

receiving first data, after the operation has been launched, said first data comprising (a) user input data indicating express or implied user encouragement or discouragement of said operation by the user, and/or (b) a detection state metric, representing a quantified likelihood that a goal sought by the operation will be reached; and

allocating increased or decreased resources to said operation based on said received first data, said allocation of increased resources being apart from increased resource usage due to any increasing base complexity of successive stages of the operation.

26. The computer readable storage medium of claim 25 in which said instructions configure the system to allocate increased resources to said operation based on user input data indicating express or implied user encouragement of said operation by the user.

27. The computer readable storage medium of claim 25 in which said instructions configure the system to allocate decreased resources to said operation based on user input data indicating express or implied user discouragement of said operation by the user.

28. The computer readable storage medium of claim 25 in which said instructions configure the system to allocate increased resources to said operation based on said detection state metric.

29. The computer readable storage medium of claim 25 in which said instructions configure the system to allocate decreased resources to said operation based on said detection state metric.

30. A system comprising:

a processor;

a camera;

a memory including software instructions to configure the system to perform acts including launching an operation on imagery captured by the camera to discern information about a subject depicted in the imagery; and

receiving first data, after the operation has been launched, said first data comprising (a) user input data indicating express or implied user encouragement or discouragement of said operation by the user, and/or (b) a detection state metric, representing a quantified likelihood that a goal sought by the operation will be reached; and

means for allocating increased or decreased resources to said operation based on said received first data, said allocation of increased resources being apart from increased resource usage due to any increasing base complexity of successive stages of the operation.

31. The system of claim 30 in which said means comprises software instructions to make GPU resources available instead of CPU resources, allocate more scratchpad memory for calculations, allow access to faster-responding memory, allow increased power consumption, or allow access to a faster network connection.