Methods and systems for content processing

• US 8,385,971 B2
• Filed: 06/12/2009
• Issued: 02/26/2013
• Est. Priority Date: 08/19/2008
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rhoads, Geoffrey B., Rodriguez, Tony F.
• Primary Examiner(s): WANG-HURST, KATHY W
• Application Number:
  US12/484,115
• Publication Number:
  US 20100048242A1
• Time in Patent Office: 1,355 Days
• Field of Search: 379/908, 348/207.99, 348/207.1, 348/207.11, 348/208.5, 348/208.99, 348/208.12, 348/208.13, 348/211.2, 348/254, 348/308, 348/376, 358/473, 358/479, 455/556.1, 455/556.2, 455/557, 455/550.1, 382/165, 382/162, 382/181, 382/276, 382/312, 382/280, 382/304
• US Class Current: 455/556.1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04M 1/0202 Portable telephone sets, e.g. cordless phones, mobile phones or bar type handsets constructional features of telephone transmitters or receivers, e.g. of speakers or microphones H04M1/03
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/2133 Recording or reproducing at a specific moment, e.g. time interval or time-lapse
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/3225 of data relating to an image, a page or a document
• CPC Class Codes: H04N 2201/3274 Storage or retrieval of prestored additional information
• CPC Class Codes: H04N 2201/3278 Transmission
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04W 4/50 : Service provisioning or reconfiguring

Abstract

Cell phones and other portable devices are equipped with a variety of technologies by which existing functionality can be improved, and new functionality can be provided. Some aspects relate to data driven imaging architectures, in which a cell phone'"'"'s image sensor is one in a chain of stages that successively act on packetized instructions/data, to capture and later process imagery. Other aspects relate to distribution of processing tasks between the device and remote resources (“the cloud”). Elemental image processing, such as filtering and edge detection—and even some simpler template matching operations—may be performed on the cell phone. Other operations are referred out to remote service providers. The remote service providers can be identified using techniques such as a reverse auction, though which they compete for processing tasks. Other aspects of the disclosed technologies relate to visual search capabilities, and determining appropriate actions responsive to different image inputs. Still others concern metadata generation, processing, and representation. Yet others relate to coping with fixed focus limitations of cell phone cameras, e.g., in reading digital watermark data. Still others concern user interface improvements. A great number of other features and arrangements are also detailed.

Claims

1. A portable device including a microphone and a radio transceiver, and additionally comprising:

an image sensor that senses imagery;

a human visual system processing module, selected from the group consisting of a white balance correction module, a gamma correction module, an edge enhancement module, and a lossy image compression module;

a machine vision processing module, selected from the group consisting of a fast Fourier transform module, an edge detection module, a pattern recognition module, a Fourier-Meltin transformer, a texture classifier, and a robust local feature detector, the machine vision processing module outputting processed data derived from the sensed imagery from the image sensor; and

an interconnection arrangement;

wherein the machine vision processing module employs processing circuitry integrated on a common substrate with the image sensor, and wherein the interconnection arrangement couples image data from the image sensor to the machine vision processing module without passing through the human visual system processing module;

the portable device thereby being configured so that the processed data output by the machine vision processing module is available for subsequent processing for the purpose of testing for characteristics indicating that the chance the image depicts a certain class of information is greater than a random chance, without such data having been impaired for such purpose by having been processed by the human visual system processing module.

2. The portable device of claim 1 in which the human visual processing module comprises a white balance correction module.

3. The portable device of claim 1 in which the human visual processing module comprises a gamma correction module.

4. The portable device of claim 1 in which the human visual processing module comprises an edge enhancement module.

5. The portable device of claim 1 in which the human visual processing module comprises a lossy image compression module.

6. The portable device of claim 1 in which the machine vision processing module comprises a fast Fourier transform module.

7. The portable device of claim 1 in which the machine vision processing module comprises an edge detection module.

8. The portable device of claim 1 in which the machine vision processing module comprises a pattern extraction module.

9. The portable device of claim 1 in which the machine vision processing module comprises a Fourier-Mellin transform.

10. The portable device of claim 1 in which the machine vision processing module comprises a texture classifier.

11. The portable device of claim 1 in which the machine vision processing module also comprises a color histogram module.

12. The portable device of claim 1 in which the machine vision processing module comprises a robust local feature detector.

14. The portable device of claim 12 in which the robust local feature detector comprises a scale invariant feature detector.

15. The portable device of claim 12 in which the robust local feature detector comprises a speeded-up robust feature detector.

13. A portable device including an image data sensor that senses imagery, a memory, a data processor, and a display, the data processor being adapted to process image data and defining two parallel processing chains:

a first processing chain for receiving the sensed imagery from the image data sensor and for producing first data to be rendered into perceptual form for use by human viewers, the first processing chain including at least a first processor selected from the group consisting of a de-mosaic processor and a lossy image compressor; and

a second processing chain for receiving the sensed imagery from the image data sensor and for producing second data to be subsequently analyzed by one or more machine-implemented algorithms for the purpose of testing for characteristics indicating that the chance the image depicts a certain class of information is greater than a random chance, the second processing chain including at least a second processor selected from the group consisting of a Fourier transform processor, a discrete cosine transform processor, and an eigenface processor;

wherein, as a consequence of the two parallel processing chains, said second data is provided for said subsequent analysis without having been impaired for said purpose by having been processed by the first processing chain.

16. The portable device of claim 13 in which the second processing chain comprises a Fourier transform processor.

17. The portable device of claim 13 in which the second processing chain comprises a discrete cosine transform processor.

18. The portable device of claim 13 in which the second processing chain comprises an eigenface processor.

19. The portable device of claim 13 in which the second processor is integrated on a common substrate with the image data sensor.

20. The portable device of claim 13 in which the second processing chain additionally comprises an edge detection module.

21. The portable device of claim 13 in which the second processing chain additionally comprises an a pattern recognition module.

22. The portable device of claim 13 in which the second processing chain additionally comprises a robust local feature detector.

23. The portable device of claim 13 in which the first processing chain additionally comprises a white balance correction module.

24. The portable device of claim 13 in which the second processing chain additionally comprises an edge enhancement module.

25. A portable device including an image data sensor that senses imagery, a memory, a data processor, and a display, the data processor being adapted to process image data and defining two parallel processing chains:

a first processing chain for receiving the sensed imagery from the image data sensor and for producing first data to be rendered into perceptual form for use by human viewers, the first processing chain including at least a first processor selected from the group consisting of a de-mosaic processor, a lossy image compressor, a white balance correction module, a gamma correction module, and an edge enhancement module; and

a second processing chain for receiving the sensed imagery from the image data sensor and for producing second data to be subsequently analyzed by one or more machine-implemented algorithms for the purpose of testing for characteristics indicating that the chance the image depicts a certain class of information is greater than a random chance, the second processing chain including at least a second processor selected from the group consisting of a Fourier transform processor, a discrete cosine transform processor, an eigenface processor, an edge detection module, a pattern recognition module, a Fourier-Mellin transformer, a texture classifier, and a robust local feature detector;

wherein, as a consequence of the two parallel processing chains, said second data is provided for said subsequent analysis without having been impaired for said purpose by having been processed by the first processing chain.