Smartphone-based methods and systems

• US 10,181,339 B2
• Filed: 11/27/2017
• Issued: 01/15/2019
• Est. Priority Date: 11/04/2010
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F., Rogers, Eliot
• Primary Examiner(s): Tran, Dzung
• Application Number:
  US15/823,168
• Publication Number:
  US 20180174620A1
• Time in Patent Office: 414 Days
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06K 7/1443 locating of the code in an image
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06T 1/0021 Image watermarking
• CPC Class Codes: G06T 1/0064 Geometric transfor invariant watermarking, e.g. affine transform invariant
• CPC Class Codes: G06T 2201/0052 Embedding of the watermark in the frequency domain
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals
• CPC Class Codes: H04W 4/50 Service provisioning or reconfiguring
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 8/22 Processing or transfer of terminal data, e.g. status or physical capabilities
• CPC Class Codes: H05B 47/11 by determining the brightness or colour temperature of ambient light
• CPC Class Codes: H05B 47/12 by detecting audible sound
• CPC Class Codes: Y02B 20/40 Control techniques providing energy savings, e.g. smart controller or presence detection

Abstract

Arrangements involving portable devices (e.g., smartphones and tablet computers) are disclosed. One arrangement enables a content creator to select software with which that creator'"'"'s content should be rendered—assuring continuity between artistic intention and delivery. Another utilizes a device camera to identify nearby subjects, and take actions based thereon. Others rely on near field chip (RFID) identification of objects, or on identification of audio streams (e.g., music, voice). Some technologies concern improvements to the user interfaces associated with such devices. Others involve use of these devices in connection with shopping, text entry, sign language interpretation, and vision-based discovery. Still other improvements are architectural in nature, e.g., relating to evidence-based state machines, and blackboard systems. Yet other technologies concern use of linked data in portable devices—some of which exploit GPU capabilities. Still other technologies concern computational photography. A great variety of other features and arrangements are also detailed.

Claims

1. A method employing a mobile device equipped with processing circuitry, a microphone, and a camera, the method comprising the acts:

with the microphone, capturing audio;

with the camera, capturing a series of images while the microphone is capturing the audio;

digitizing the captured audio, yielding audio data samples;

digitizing the captured images, yielding image data frames;

in a first mode, (a) providing an uninterrupted stream of audio data samples to an audio recognition agent, and (b) providing an uninterrupted stream of image data frames to an image recognition agent;

in a second mode, (a) providing an uninterrupted stream of audio data samples to the audio recognition agent, and (b) providing an interrupted stream of image data frames to the image recognition agent; and

switching from the first mode to the second mode when available device resources decrease;

wherein processing of image data frames by the image recognition agent is deliberately throttled-back between the first and second modes, whereas processing of audio data by the audio recognition agent is maintained constant between the first and second modes.

2. The method of claim 1 in which the audio recognition agent comprises at least one of a watermark detector or a fingerprint-based audio recognizer.

3. The method of claim 1 in which the image recognition agent comprises at least one of a watermark detector or a barcode reader.

4. The method of claim 1 in which the mobile device is equipped with first and second image recognition agents, and the method includes applying a classifier to the image samples, and based on an output of said classifier:

(a) processing said samples by the first image recognition agent, (b) processing said samples by the second image recognition agent, or (c) processing said samples by both the first and second image recognition agents.

5. The method of claim 1 that further includes receiving data from plural different sensors, and switching to a third mode based on a particular combination of said data, said third mode including the device interrupting processing of said image samples.

6. The method of claim 1 that further includes receiving data from plural different sensors, and switching to a third mode based on a particular combination of said data, said third mode including the device interrupting processing of said audio samples.

7. The method of claim 1 that includes, in the second mode, generating an image quality metric, and setting a rate at which the device processes image data frames based on the quality metric.

8. The method of claim 7 in which:

in a first instance, when the image quality metric has a first value, the method includes setting a first rate at which the device processes image data frames; and

in a second instance, when the image quality metric again has said first value, setting a second rate, different from the first rate, at which the device processes image data frames;

wherein the device sets first and second different rates, despite the same first image quality metric, due to different system context of the device.

9. A non-transitory computer readable medium containing instructions for configuring a portable device, equipped with a processor and a microphone and a camera, to perform acts including:

with the microphone, capturing audio;

with the camera, capturing a series of images while the microphone is capturing the audio;

digitizing the captured audio, yielding audio data samples;

digitizing the captured images, yielding image data frames;

in a first mode, (a) providing an uninterrupted stream of audio data samples to an audio recognition agent, and (b) providing an uninterrupted stream of image data frames to an image recognition agent;

in a second mode, (a) providing an uninterrupted stream of audio data samples to the audio recognition agent, and (b) providing an interrupted stream of image data frames to the image recognition agent; and

switching from the first mode to the second mode when available device resources decrease;

wherein processing of image data frames by the image recognition agent is deliberately throttled-back between the first and second modes, whereas processing of audio data by the audio recognition agent is maintained constant between the first and second modes.

10. A mobile device comprising:

processing circuitry;

a memory;

a microphone;

a camera;

first means for recognizing audio content;

second means for recognizing image content;

configuration instructions, stored in the memory, that configure the device to perform acts including;

capturing a series of image frames while capturing a series of audio samples;

operating in a first mode that includes (a) providing an uninterrupted stream of audio data samples to the first means, and (b) providing an uninterrupted stream of image data frames to the second means;

operating in a second mode that includes (a) providing an uninterrupted stream of audio data samples to the first means, and (b) providing an interrupted stream of image data frames to the second means; and

switching from the first mode to the second mode when available device resources decrease;

wherein processing of image data frames by the second means is deliberately throttled-back between the first and second modes, whereas processing of audio data by the first means is maintained constant between the first and second modes.