METHODS AND SYSTEMS FOR IMAGE OR AUDIO RECOGNITION PROCESSING
First Claim
1. In a distributed processing method that includes performing an operation on stimuli captured by a camera or microphone sensor of a user'"'"'s mobile device, by using a combined system that includes both processing hardware in the mobile device and processing hardware remote from the mobile device, an improvement wherein:
- the operation comprises an image or audio recognition operation, and the method includes;
identifying, using a hardware processor, a set of component functions that should be executed by said combined system to perform said recognition operation; and
determining a sequence in which certain of said component functions should be performed based on one or more circumstance or context factors selected from the group consisting of;
(a) information about mobile device power availability or usage;
(b) information about a needed function response time;
(c) information about a routing constraint;
(d) information about a state of hardware resources within the mobile device;
(e) information about mobile device connectivity;
(f) information about a geographical consideration;
(g) information about a pipeline stall risk;
(h) information about turnaround time or cost associated with the remote processor; and
(i) information about a user preference regarding remote processing;
and wherein, at a first time, said certain component operations are performed in a first sequence, and at a second time, said certain component operations are performed in a second, different, sequence, due to a difference in one or more of said factors between the first and second times.
0 Assignments
0 Petitions
Accused Products
Abstract
Many of the detailed technologies are useful in enabling a smart phone to respond to a user'"'"'s environment, e.g., so it can serve as an intuitive hearing and seeing device. A few of the detailed arrangements involve optimizing division of shared processing tasks between the phone and remote devices; using a phone GPU for exhaustive speculative execution and machine vision purposes (including facial recognition); novel device architectures involving abstraction layers that facilitate substitution of different local and remote services; interactions with private networks as they relate to audio/image processing; adapting the orders in which operations are executed, and the types of data that are exchanged with remote servers, in accordance with current context; reconfiguring networks based on sensed social affiliations among users and in accordance with predictive models of user behavior; etc. A great variety of other features and arrangements are also detailed.
-
Citations
19 Claims
-
1. In a distributed processing method that includes performing an operation on stimuli captured by a camera or microphone sensor of a user'"'"'s mobile device, by using a combined system that includes both processing hardware in the mobile device and processing hardware remote from the mobile device, an improvement wherein:
-
the operation comprises an image or audio recognition operation, and the method includes; identifying, using a hardware processor, a set of component functions that should be executed by said combined system to perform said recognition operation; and determining a sequence in which certain of said component functions should be performed based on one or more circumstance or context factors selected from the group consisting of; (a) information about mobile device power availability or usage; (b) information about a needed function response time; (c) information about a routing constraint; (d) information about a state of hardware resources within the mobile device; (e) information about mobile device connectivity; (f) information about a geographical consideration; (g) information about a pipeline stall risk; (h) information about turnaround time or cost associated with the remote processor; and (i) information about a user preference regarding remote processing; and wherein, at a first time, said certain component operations are performed in a first sequence, and at a second time, said certain component operations are performed in a second, different, sequence, due to a difference in one or more of said factors between the first and second times. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
-
-
18. A mobile device comprising at least one processor, memory, camera, and microphone, the memory containing software instructions that configure the device to perform an image or audio recognition operation in conjunction with a cooperating remote processing device, the recognition operation comprising plural component functions—
- one or more of which are performed by said at least one processor of the mobile device, and one or more of which are performed by the cooperating remote processing device, wherein said instructions in the mobile device memory include instructions for determining a sequence in which certain of said component functions should be performed, based on one or more circumstance or context factors selected from the group consisting of;
(a) information about mobile device power availability or usage; (b) information about a needed function response time; (c) information about a routing constraint; (d) information about a state of hardware resources within the mobile device; (e) information about mobile device connectivity; (f) information about a geographical consideration; (g) information about a pipeline stall risk; (h) information about turnaround time or cost associated with the remote device; and (i) information about a user preference regarding remote processing; and wherein, at a first time, said certain component operations are performed in a first sequence, and at a second time, said certain component operations are performed in a second, different, sequence, due to a difference in one or more of said factors between the first and second times.
- one or more of which are performed by said at least one processor of the mobile device, and one or more of which are performed by the cooperating remote processing device, wherein said instructions in the mobile device memory include instructions for determining a sequence in which certain of said component functions should be performed, based on one or more circumstance or context factors selected from the group consisting of;
-
19. In a method that includes portable device processing of audio or image data captured by a sensor of said device, to extract information from said captured audio or image data, said processing involving sequential application of plural component tasks, an improvement comprising the acts:
-
before an actual set of one or more input conditions for a first of said component tasks becomes known, commencing execution of plural instances of said first task, in parallel, using a GPU of the portable device, each instance using an alternate set of input conditions, thereby producing plural alternate outputs for the first task; and selecting one of said plural outputs for use in connection with a second of said component tasks, once an actual set of input conditions for the first task becomes known.
-
Specification