Methods and systems for resource management on portable devices

• US 9,183,580 B2
• Filed: 10/21/2011
• Issued: 11/10/2015
• Est. Priority Date: 11/04/2010
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rhoads, Geoffrey B., Rodriguez, Tony F., Conwell, William Y.
• Primary Examiner(s): Rush, Eric
• Application Number:
  US13/278,949
• Publication Number:
  US 20120134548A1
• Time in Patent Office: 1,481 Days
• Field of Search: 381/56, 704/200, 704231-234, 704237-240, 704/246, 704/270, 704/270.1, 719312-318, 719/328, 719/329, 709212-216, 711147-151, 717/120, 717162-167, 712216-219, 712/225, 712/226, 382/100, 382/181, 382/325
• US Class Current: 1/1
• CPC Class Codes: G06F 8/61 Installation
• CPC Class Codes: G06F 9/3885 using a plurality of independent parallel functional units
• CPC Class Codes: G06F 9/445 Program loading or initiating bootstrapping G06F9/4401; security arrangements for program loading or initiating G06F21/57
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06Q 20/322 Aspects of commerce using mobile devices [M-devices]
• CPC Class Codes: G06Q 20/3276 using a pictured code, e.g. barcode or QR-code, being read by the M-device
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G10L 15/285 Memory allocation or algorithm optimisation to reduce hardware requirements
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: G10L 25/81 for discriminating voice from music
• CPC Class Codes: G10L 25/84 for discriminating voice from noise
• CPC Class Codes: H04L 2463/101 applying security measures for digital rights management data processing systems or methods, specially adapted for commerce, e.g. marketing, shopping, billing, auctions or e-commerce G06Q30/00
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

Methods and arrangements involving portable devices are disclosed. One arrangement enables a content creator to select software with which that content should be rendered—assuring continuity between artistic intention and delivery. Another arrangement utilizes the camera of a smartphone to identify nearby subjects, and take actions based thereon. Others rely on near field chip (RFID) identification of objects, or on identification of audio streams (e.g., music, voice). Some of the detailed technologies concern improvements to the user interfaces associated with such devices. Others involve use of these devices in connection with shopping, text entry, sign language interpretation, and vision-based discovery. Still other improvements are architectural in nature, e.g., relating to evidence-based state machines, and blackboard systems. Yet other technologies concern use of linked data in portable devices—some of which exploit GPU capabilities. Still other technologies concern computational photography. A great variety of other features and arrangements are also detailed.

Claims

1. A method employing a portable device that includes a processor, a microphone for capturing audio, a camera system for capturing frames of imagery, an image recognition module for responding to features recognized in frames of captured imagery, and a fingerprinting module for song recognition, the method comprising:

providing captured audio from the microphone to a speech classifier;

disabling the fingerprinting module for a first portion of the captured audio, when the speech classifier indicates said first portion is speech;

processing a second portion of the captured audio with the fingerprinting module, when the speech classifier indicates said second portion is non-speech;

monitoring utilization of the processor; and

based on said monitoring, causing the image recognition module to skip processing of one or more frames of image data while the fingerprinting module is processing said second portion of the captured audio.

2. The method of claim 1 that further includes:

publishing the captured audio to a blackboard data structure, through which components of the device communicate;

the speech classifier reading the captured audio from said blackboard data structure, and publishing output information to the blackboard data structure; and

the fingerprinting module reading the output information from the speech classifier from said blackboard data structure, and also reading the captured audio from said blackboard data structure, the fingerprinting module further publishing output information to said blackboard data structure.

3. The method of claim 2 that further includes:

plural application programs reading output information from the fingerprinting module from said blackboard data structure;

wherein an identity of music identified once through use of the fingerprinting module is provided to plural application programs on the device via the blackboard data structure.

4. The method of claim 1 wherein causing the image recognition module to skip processing of selected frames of image data while processing the second portion of the captured audio by the fingerprinting module comprises causing the image recognition module to skip processing of selected frames of image data without interrupting processing of the second portion of the captured audio at the fingerprinting module.

5. A portable device including a microphone for capturing audio, a camera system for capturing frames of imagery, an image recognition module for responding to features recognized in frames of captured imagery, a fingerprinting module for music recognition, a processor and a memory, the memory containing software instructions that configure the device to perform acts including:

providing captured audio from the microphone to a speech classifier;

disabling the fingerprinting module for a first portion of the captured audio, when the speech classifier indicates said first portion is speech;

processing a second portion of the captured audio with the fingerprinting module, when the speech classifier indicates said second portion is non-speech;

monitoring utilization of the processor; and

based on said monitoring, causing the image recognition module to skip processing of one or more frames of image data while the fingerprinting module is processing said second portion of the captured audio.

6. The portable device of claim 5 in which the fingerprinting module comprises part of a middleware architecture for object recognition, which is configured to take output information from the fingerprinting module and provide the output information to plural software applications running on the device such that the identity of music identified once through use of the fingerprinting module is provided to multiple applications on the device.

7. The portable device of claim 6 in which the middleware architecture further includes a blackboard data structure to which data from the microphone is publishable, and also to which the output information from the fingerprinting module and output information from the speech classifier are publishable, said output information from the fingerprinting module being readable from said blackboard data structure by said plural software applications, wherein said blackboard data structure is a shared data repository through which data communication between the microphone, the fingerprinting module, the speech classifier, and the applications, about microphone-sensed audio, occurs.

8. The portable device of claim 6 wherein the camera system is further configured to output frames of image data for processing by said device, wherein said software instructions implement a priority system configured to cause the device to skip processing of selected frames of image data while processing a stream of audio data by the fingerprinting module.

9. The portable device of claim 8 wherein said priority system is configured to cause the device to skip processing of selected frames of image data without interrupting processing of the stream of audio data by the fingerprinting module.

10. The portable device of claim 8 wherein the camera system is configured to output frames of image data for processing by said processor and wherein said priority system is configured to cause the device to skip processing of selected frames of image data while the stream of audio data is processed by the fingerprinting module.