Methods and systems for determining image processing operations relevant to particular imagery

• US 9,256,806 B2
• Filed: 02/25/2014
• Issued: 02/09/2016
• Est. Priority Date: 03/19/2010
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Aller, Joshua V., Rodriguez, Tony F.
• Primary Examiner(s): PARK, SOO JIN
• Application Number:
  US14/189,236
• Publication Number:
  US 20140286542A1
• Time in Patent Office: 714 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06Q 30/0281 Customer communication at a business location, e.g. providing product or service information, consulting
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

Image data, such as from a mobile phone camera, is analyzed to determine a colorfulness metric (e.g., saturation) or a contrast metric (e.g., Weber contrast). This metric is then used in deciding which of, or in which order, plural different image recognition processes should be invoked in order to present responsive information to a user. A great number of other features and arrangements are also detailed.

Claims

1. A method of improving the functioning of a cell phone, the cell phone including a battery, a camera, and a processing system, the cell phone further being equipped to perform plural different types of image recognition operations including at least barcode recognition and digital watermark decoding on frames of imagery captured by the camera, the method comprising the acts:

performing a plurality of preliminary image processing operations on each of 15 or more frames of imagery captured per second, said performing requiring application of less than 10% of computational capabilities of the cell phone processing system, said preliminary image processing operations including analyzing each of said frames of imagery to determine at least one of;

a colorfulness metric, a color saturation metric, or a contrast metric, said preliminary image processing operations yielding intermediate data, including said determined metric, for each of said frames;

(a) based at least in part on said determined metric, performing a barcode recognition operation and not a digital watermark decoding operation on a first of said frames of imagery captured by the camera;

(b) based at least in part on said determined metric, performing a digital watermark decoding operation and not a barcode recognition operation on a second of said frames of imagery captured by the camera;

wherein said barcode recognition and watermark decoding operations are performed using said cell phone processing system, without off-loading to a processor remote from the cell phone; and

wherein a type of image recognition process performed on a frame of captured imagery, as well as a type of image recognition process not-performed, are based on said determined metric;

wherein the cell phone does not perform all of said plural types of image recognition operations on said first or second frames of captured imagery, but rather selects among said plural types of image recognition operations based on imagery characteristics, thereby conserving battery power and processing resources, despite performing said barcode recognition and watermark decoding operations using said cell phone processing system without reliance on a processor remote from the cell phone.

2. The method of claim 1 wherein selection of a particular image recognition operation to apply to a particular frame of captured imagery proceeds without input from a user, wherein the phone thereby seems to respond intuitively to content depicted in the captured imagery.

3. The method of claim 1 wherein said plural different types of image recognition operations additionally includes optical character recognition, wherein:

act (a) includes performing both said barcode recognition operation and said optical character recognition operation, and not said digital watermark decoding operation; and

act (b) includes performing said digital watermark decoding operation, and not said barcode recognition operation and not said optical character recognition operation.

4. The method of claim 1 wherein said plural different types of image recognition operations additionally includes facial recognition, wherein:

act (a) includes performing said barcode recognition operation, and not said digital watermark decoding operation and not said facial recognition operation; and

act (b) includes performing said digital watermark decoding operation and said facial recognition operation, and not said barcode recognition operation.

5. The method of claim 4 wherein said plural different types of image recognition operations additionally includes optical character recognition, wherein:

act (a) includes performing both said barcode recognition operation and said optical character recognition operation, and not said digital watermark decoding operation and not said facial recognition operation; and

act (b) includes performing both said digital watermark decoding operation and said facial recognition operation, and not said barcode recognition operation and not said optical character recognition operation.

6. The method of claim 1 in which said plurality of preliminary image processing operations includes analyzing to determine a colorfulness metric.

7. The method of claim 1 in which said plurality of preliminary image processing operations includes analyzing to determine a color saturation metric.

8. The method of claim 1 in which said plurality of preliminary image processing operations includes analyzing to determine a contrast metric.

9. The method of claim 1 in which said plurality of preliminary image processing operations includes converting red/green/blue signals of frames of captured image data into another representation in which color is represented separately than luminance.

10. The method of claim 1 in which the intermediate data indicates more than 50% of said first frame of captured image data has a color saturation of less than 50%.

11. The method of claim 1 in which the first intermediate data indicates more than 50% of said first frame of captured image data has a color saturation of less than 15%.

12. A cell phone including a battery, a camera, a memory containing software instructions, and a processing system, said instructions configuring the cell phone to define plural recognition agents, said agents including a barcode recognition agent that extracts barcode data from camera-captured imagery, and a watermark decoding agent that extracts watermark data from camera-captured imagery, said barcode recognition agent and watermark decoding agent not off-loading processing to a processor remote from the cell phone, said instructions in the memory further configuring the cell phone to perform acts including:

performing a plurality of preliminary image processing operations on each of 15 or more frames of imagery captured per second, said performing requiring application of less than 10% of computational capabilities of the cell phone processing system, said preliminary image processing operations including analyzing each of said frames of imagery to determine at least one of;

a colorfulness metric, a color saturation metric, or a contrast metric, said preliminary image processing operations yielding intermediate data, including said determined metric, for each of said frames;

based at least in part on said determined metric, selecting less than all of said plural recognition agents to invoke on one of said frames of imagery; and

invoking the selected recognition agent(s) to operate on said frame of imagery, and not invoking an unselected recognition agent to operate on said frame of imagery;

wherein the recognition agent(s) invoked to operate on said frame of imagery, as well as the recognition agent not invoked to operate on said frame of imagery, are identified based on said determined metric, wherein the cell phone does not invoke all of said plural recognition agents to operate on said frame of imagery, but rather selects among said plural recognition agents based on image characteristics, thereby conserving battery power and processing resources, despite performing said barcode recognition and watermark decoding operations using said cell phone processing system without reliance on a processor remote from the cell phone.

13. The cell phone of claim 12 wherein said plural recognition agents additionally include a facial recognition agent that extracts facial identification information from camera-captured imagery.

14. The cell phone of claim 13 wherein said plural recognition agents additionally include an optical character recognition agent that extracts text information from camera-captured imagery.

15. The cell phone of claim 12 wherein said plural recognition agents additionally include an optical character recognition agent that extracts text information from camera-captured imagery.

16. The cell phone of claim 12 wherein said selection of one or more of said plural recognition agents to apply to a particular frame of captured imagery proceeds without input from a user, wherein the phone thereby seems to respond intuitively to content depicted in the captured imagery.

17. The cell phone of claim 12 wherein said preliminary image processing operations include analyzing each of said frames of imagery to determine a colorfulness metric.

18. The cell phone of claim 12 wherein said preliminary image processing operations include analyzing each of said frames of imagery to determine a color saturation metric.

19. The cell phone of claim 12 wherein said preliminary image processing operations include analyzing each of said frames of imagery to determine a contrast metric.

20. The cell phone of claim 12 wherein said preliminary image processing operations include concerning red/green/blue signals of frames of captured image data into another representation in which color is represented separately than luminance.

22. The cell phone of claim 16 wherein selection of the recognition agent(s) to operate on said particular frame of imagery occurs without instruction from a user, wherein the phone thereby seems to respond intuitively to content depicted in the frame of captured imagery.

21. A cell phone including:

a battery;

a camera;

a memory containing software instructions;

a processing system;

means defining plural recognition agents, including a barcode recognition agent that extracts barcode data from camera-captured imagery, and a watermark decoding agent that extracts watermark data from camera-captured imagery, said barcode recognition agent and watermark decoding agent not off-loading processing to a processor remote from the cell phone; and

means for determining an image metric for each of at least 15 frames of imagery captured per second, said means requiring application of less than 10% of computational capabilities of the cell phone processing system;

wherein said software instructions configure the cell phone to perform acts including;

based on said image metric determined from a particular frame of imagery, selecting less than all of said plural recognition agents; and

invoking the selected recognition agent(s) to operate on said particular frame of imagery, and not invoking an unselected recognition agent to operate on said particular frame of imagery;

wherein the recognition agent(s) invoked to operate on said particular frame of imagery, as well as the recognition agent not invoked to operate on the particular frame of imagery, are identified based on said determined metric, wherein the cell phone does not invoke all of said plural recognition agents to operate on said particular frame of imagery, but rather selects among said plural recognition agents based on image characteristics, thereby conserving battery power and processing resources.