Methods and systems for image or audio recognition processing

• US 9,271,133 B2
• Filed: 07/01/2014
• Issued: 02/23/2016
• Est. Priority Date: 08/17/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F.
• Primary Examiner(s): Jackson, Blane J
• Application Number:
  US14/321,404
• Publication Number:
  US 20150011194A1
• Time in Patent Office: 602 Days
• Field of Search: 455/66.1, 455/414.1, 455/556.1, 382/103, 382/115, 382/118, 382/124, 382/173, 382/181, 382/195, 382/276, 382/307
• US Class Current: 1/1
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services
• CPC Class Codes: H04N 1/00127 Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00328 with an apparatus processing optically-read information
• CPC Class Codes: H04N 1/00331 with an apparatus performing optical character recognition character recognition G06V30/10
• CPC Class Codes: H04N 1/00336 with an apparatus performing pattern recognition, e.g. of a face or a geographic feature image or video recognition or understanding of scenes G06V20/00
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/16 Communication-related supplementary services, e.g. call-transfer or call-hold
• CPC Class Codes: H04W 40/02 Communication route or path selection, e.g. power-based or shortest path routing
• CPC Class Codes: H04W 88/02 Terminal devices

Abstract

Many of the detailed technologies are useful in enabling a smart phone to respond to a user'"'"'s environment, e.g., so it can serve as an intuitive hearing and seeing device. A few of the detailed arrangements involve optimizing division of shared processing tasks between the phone and remote devices; using a phone GPU for exhaustive speculative execution and machine vision purposes (including facial recognition); novel device architectures involving abstraction layers that facilitate substitution of different local and remote services; interactions with private networks as they relate to audio/image processing; adapting the orders in which operations are executed, and the types of data that are exchanged with remote servers, in accordance with current context; reconfiguring networks based on sensed social affiliations among users and in accordance with predictive models of user behavior; etc. A great variety of other features and arrangements are also detailed.

Claims

1. In a distributed processing method that includes performing an operation on stimuli captured by a camera or microphone sensor of a user'"'"'s mobile device, by using a combined system that includes both processing hardware in the mobile device and processing hardware remote from the mobile device, an improvement wherein:

the operation comprises an image or audio recognition operation, and the method includes;

identifying, using a hardware processor, a set of component operations that should be executed by said combined system to perform said recognition operation; and

determining a sequence in which said component operations should be performed based on one or more circumstance or context factors selected from the group consisting of;

(a) information about mobile device power availability or usage;

(b) information about a needed function response time;

(c) information about a routing constraint;

(d) information about a state of hardware resources within the mobile device;

(e) information about mobile device connectivity;

(f) information about a geographical consideration;

(g) information about a pipeline stall risk;

(h) information about turnaround time or cost associated with the remote processor; and

(i) information about a user preference regarding remote processing;

and wherein, at a first time, said component operations are performed in a first sequence, and at a second time, said component operations are performed in a second, different, sequence, due to a difference in one or more of said factors between the first and second times.

2. The method of claim 1 in which said act of identifying a set of component operations is also based on one or more circumstance or context factors selected from said list, wherein at one time, a first set of component operations is identified to perform said recognition operation, and at another time, a second, different, set of component functions is identified to perform said recognition operation, due to a difference in one or more of said factors between said one and another times.

3. The method of claim 1 in which said act of determining a sequence is based on two or more of said factors.

4. The method of claim 1 in which said act of determining a sequence is based on one or more factors including information about a needed function response time.

5. The method of claim 1 in which said act of determining a sequence is based on one or more factors including information about a routing constraint.

6. The method of claim 5 in which the routing constraint is imposed by a provider of a local wireless network.

7. The method of claim 1 in which said act of determining a sequence is based on one or more factors including information about a state of hardware resources within the processing device.

8. The method of claim 1 in which said act of determining a sequence is based on one or more factors including information about mobile device connectivity.

9. The method of claim 1 in which said act of determining a sequence is based on one or more factors including information about a geographical consideration.

10. The method of claim 1 in which said act of determining a sequence is based on one or more factors including information about a pipeline stall risk.

11. The method of claim 10 that includes assessing the pipeline stall risk by reference to historical patterns, or based on information that completion of an operation requires further data of uncertain availability.

12. The method of claim 1 in which said act of determining a sequence is based on one or more factors including information about turnaround time or cost associated with the remote processor.

13. The method of claim 1 in which said act of determining a sequence is based on one or more factors including information about remote processor readiness, or information about remote processor cost.

14. The method of claim 1 in which said act of determining a sequence is based on one or more factors including information about a user preference regarding remote processing.

15. The method of claim 14 wherein the user preference comprises user preference about location of a remote service provider.

16. The method of claim 1 that includes determining that one component operation should be performed before another component operation, and as a consequence, performing said one component operation using processing hardware in the mobile device, and performing said another component operation using processing hardware remote from the mobile device.

17. The method of claim 1 that includes determining that one component operation should be performed before another component operation, and as a consequence, performing said one component operation using processing hardware remote from the mobile device, and performing said another component operation using processing hardware in the mobile device.

18. A mobile device comprising at least one processor, memory, camera, and microphone, the memory containing software instructions that configure the device to perform an image or audio recognition operation in conjunction with a cooperating remote processing device, the recognition operation comprising plural component operations—

one or more of which are performed by said at least one processor of the mobile device, and one or more of which are performed by the cooperating remote processing device, wherein said instructions in the mobile device memory include instructions for determining a sequence in which said component operations should be performed, based on one or more circumstance or context factors selected from the group consisting of;

(a) information about mobile device power availability or usage;

(b) information about a needed function response time;

(c) information about a routing constraint;

(d) information about a state of hardware resources within the mobile device;

(e) information about mobile device connectivity;

(f) information about a geographical consideration;

(g) information about a pipeline stall risk;

(h) information about turnaround time or cost associated with the remote device; and

(i) information about a user preference regarding remote processing;

and wherein, at a first time, said component operations are performed in a first sequence, and at a second time, said component operations are performed in a second, different, sequence, due to a difference in one or more of said factors between the first and second times.