Variable processing of both image and audio data based on processor utilization

• US 9,202,254 B2
• Filed: 07/18/2012
• Issued: 12/01/2015
• Est. Priority Date: 11/04/2010
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F., Alattar, Adnan M., Rogers, Eliot
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US13/552,337
• Publication Number:
  US 20130183952A1
• Time in Patent Office: 1,231 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06K 7/1443 locating of the code in an image
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06T 1/0021 Image watermarking
• CPC Class Codes: G06T 1/0064 Geometric transfor invariant watermarking, e.g. affine transform invariant
• CPC Class Codes: G06T 2201/0052 Embedding of the watermark in the frequency domain
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals
• CPC Class Codes: H04W 4/50 Service provisioning or reconfiguring
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 8/22 Processing or transfer of terminal data, e.g. status or physical capabilities
• CPC Class Codes: H05B 47/11 by determining the brightness or colour temperature of ambient light
• CPC Class Codes: H05B 47/12 by detecting audible sound
• CPC Class Codes: Y02B 20/40 Control techniques providing energy savings, e.g. smart controller or presence detection

Abstract

Arrangements involving portable devices (e.g., smartphones and tablet computers) are disclosed. One arrangement enables a content creator to select software with which that creator'"'"'s content should be rendered—assuring continuity between artistic intention and delivery. Another utilizes a device camera to identify nearby subjects, and take actions based thereon. Others rely on near field chip (RFID) identification of objects, or on identification of audio streams (e.g., music, voice). Some technologies concern improvements to the user interfaces associated with such devices. Others involve use of these devices in connection with shopping, text entry, sign language interpretation, and vision-based discovery. Still other improvements are architectural in nature, e.g., relating to evidence-based state machines, and blackboard systems. Yet other technologies concern use of linked data in portable devices—some of which exploit GPU capabilities. Still other technologies concern computational photography. A great variety of other features and arrangements are also detailed.

Claims

1. A method of adaptively controlling operation of a portable computing system based on computing system context, the portable computing system including a processor, an imaging sensor system that outputs frames of image information, and audio sensor system that outputs audio information, the method comprising the acts:

for a first interval, in which the processor has a first utilization, applying an image recognition operation to all of plural frames of image information output by the imaging sensor system;

for a second interval, in which the processor has a second utilization higher than the first utilization, and during which the imaging sensor system outputs N frames of image information, applying said image recognition operation to M of said frames, where 0<

M<

N;

for a third interval, in which the processor has a third utilization, applying an audio recognition operation to all of the audio information output from the audio sensor system; and

for a fourth interval, in which the processor has a fourth utilization higher than the third utilization, applying said audio recognition operation to none of the audio information output from the audio sensor system;

wherein the image recognition applied to frames of image information output by the imaging sensor system is throttled-back with an increase in processor utilization, whereas the audio recognition applied to audio information output by the audio sensor system is discontinued with an increase in processor utilization.

2. The method of claim 1 wherein the third of said intervals comprises several seconds of time.

3. The method of claim 1 that includes, during one interval in which the imaging sensor system outputs P frames of image information, applying an audio recognition operation to all of said audio information output from the audio sensor system, while applying said image recognition operation to Q of said frames, where 0<

Q<

P.

4. The method of claim 1 in which the portable computing system also includes a movement sensor, and the method further includes:

using data from the movement sensor to assess motion blur; and

not applying an image recognition operation to a frame of image information output by the imaging sensor system, based on said assessment of motion blur.

5. The method of claim 1 in which the portable computing system is equipped to perform plural different image recognition operations, and in which the method further includes the acts:

applying a classifier to inspect imagery provided from the imaging sensor system; and

applying a first of said plural different image recognition operations on a frame of imagery, and not applying a second of said plural different image recognitions on said frame of imagery, based on an output from said classifier;

wherein system performance is improved by excluding operation of said second image recognition operation.

6. The method of claim 5 wherein the second image recognition operation is a barcode recognition operation.

7. The method of claim 1 that includes performing portions of said image recognition operation using software written in assembly language.

8. The method of claim 1 that includes performing portions of said image recognition operation using software written using a NEON®

instruction set.

9. A smartphone including an imaging sensor system, an audio sensor system, a processor, and hardware that stores software instructions which configure the smartphone to perform operations including:

for a first interval, in which the processor has a first utilization, applying an image recognition operation to all of plural frames of image information output by the imaging sensor system;

for a second interval, in which the processor has a second utilization higher than the first utilization, and during which the imaging sensor system outputs N frames of image information, applying said image recognition operation to M of said frames, where 0<

M<

N;

for a third interval, in which the processor has a third utilization, applying an audio recognition operation to all of the audio information output from the audio sensor system; and

for a fourth interval, in which the processor has a fourth utilization higher than the third utilization, applying said audio recognition operation to none of the audio information output from the audio sensor system;

wherein the image recognition applied to frames of image information output by the imaging sensor system is throttled-back with an increase in processor utilization, whereas the audio recognition applied to audio information output by the audio sensor system is discontinued with an increase in processor utilization.

10. The smartphone of claim 9 in which said instructions configure the smartphone to apply, during one interval in which the imaging sensor system outputs P frames of image information, an audio recognition operation to all of said audio information output from the audio sensor system, while applying said image recognition operation to Q of said frames, where 0<

Q<

P.

11. The smartphone of claim 9 that further includes a movement sensor, wherein said instructions further configure the smartphone to assess motion blur using data from the movement sensor, and not apply an image recognition operation to a frame of image information output by the imaging sensor system, based on said assessment of motion blur.

12. The smartphone of claim 9 in which the smartphone is configured to perform plural different image recognition operations, and in which the instructions further configure the smartphone to:

apply a classifier to inspect imagery provided from the imaging sensor system; and

apply a first of said plural different image recognition operations on a frame of imagery, and not apply a second of said plural different image recognitions on said frame of imagery, based on an output from said classifier;

wherein system performance is improved by excluding operation of said second image recognition operation.

13. The smartphone of claim 12 wherein the second image recognition operation is a barcode recognition operation.

14. The smartphone of claim 9 wherein portions of said instructions that configure the smartphone to perform said image recognition operation comprise instructions written in assembly language.

15. The smartphone of claim 9 wherein portions of said instructions that configure the smartphone to perform said image recognition operation comprise instructions in a NEON®

instruction set.

16. A non-transitory computer readable medium containing instructions for configuring a portable device, equipped with a processor, an image sensor system, and an audio sensor system, to perform acts including:

for a first interval, in which the processor has a first utilization, applying an image recognition operation to all of plural frames of image information output by the imaging sensor system;

for a second interval, in which the processor has a second utilization higher than the first utilization, and during which the imaging sensor system outputs N frames of image information, applying said image recognition operation to M of said frames, where 0<

M<

N;

for a third interval, in which the processor has a third utilization, applying an audio recognition operation to all of the audio information output from the audio sensor system; and

for a fourth interval, in which the processor has a fourth utilization higher than the third utilization, applying said audio recognition operation to none of the audio information output from the audio sensor system;

wherein the image recognition applied to frames of image information output by the imaging sensor system is throttled-back with an increase in processor utilization, whereas the audio recognition applied to audio information output by the audio sensor system is discontinued with an increase in processor utilization.

17. The computer readable medium of claim 16 in which said acts include, during one interval in which the imaging sensor system outputs P frames of image information, applying an audio recognition operation to all of said audio information output from the audio sensor system, while applying said image recognition operation to Q of said frames, where 0<

Q<

P.

18. The computer readable medium of claim 16 in which said device is also equipped with a movement sensor, and said instructions configure said device to perform acts including:

assessing motion blur using data from the movement sensor; and

not applying an image recognition operation to a frame of image information output by the imaging sensor system, based on said assessment of motion blur.

19. The computer readable medium of claim 16 in which said instructions configure the smartphone to perform plural different image recognition operations, and in which the instructions further configure the smartphone to perform acts including:

applying a classifier to inspect imagery provided from the imaging sensor system; and

applying a first of said plural different image recognition operations on a frame of imagery, and not applying a second of said plural different image recognitions on said frame of imagery, based on an output from said classifier;

wherein performance of said device is improved by excluding operation of said second image recognition operation.

20. The computer readable medium of claim 19 wherein the second image recognition operation is a barcode recognition operation.

21. The computer readable medium of claim 16 wherein portions of said instructions that configure the device to perform said image recognition operation comprise instructions written in assembly language.

22. The computer readable medium of claim 16 wherein portions of said instructions that configure the smartphone to perform said image recognition operation comprise instructions in a NEON®

instruction set.