Methods and systems for content processing

• US 8,755,837 B2
• Filed: 02/22/2013
• Issued: 06/17/2014
• Est. Priority Date: 08/19/2008
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rhoads, Geoffrey B., Rodriguez, Tony F.
• Primary Examiner(s): WANG-HURST, KATHY W
• Application Number:
  US13/774,056
• Publication Number:
  US 20130273968A1
• Time in Patent Office: 480 Days
• Field of Search: 379/908, 348/207.99, 348/207.1, 348/207.11, 348/208.5, 348/208.99, 348/208.12, 348/208.13, 348/211.2, 348/254, 348/308, 348/376, 358/473, 358/479, 455/556.1, 455/556.2, 455/557, 455/550.1, 382/165, 382/162, 382/181, 382/276, 382/312, 382/280, 382/304
• US Class Current: 455/556.1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04M 1/0202 Portable telephone sets, e.g. cordless phones, mobile phones or bar type handsets constructional features of telephone transmitters or receivers, e.g. of speakers or microphones H04M1/03
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/2133 Recording or reproducing at a specific moment, e.g. time interval or time-lapse
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/3225 of data relating to an image, a page or a document
• CPC Class Codes: H04N 2201/3274 Storage or retrieval of prestored additional information
• CPC Class Codes: H04N 2201/3278 Transmission
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04W 4/50 : Service provisioning or reconfiguring

Abstract

Cell phones and other portable devices are equipped with a variety of technologies by which existing functionality can be improved, and new functionality can be provided. Some aspects relate to data driven imaging architectures, in which a cell phone'"'"'s image sensor is one in a chain of stages that successively act on packetized instructions/data, to capture and later process imagery. Other aspects relate to distribution of processing tasks between the device and remote resources (“the cloud”). Elemental image processing, such as filtering and edge detection—and even some simpler template matching operations—may be performed on the cell phone. Other operations are referred out to remote service providers. The remote service providers can be identified using techniques such as a reverse auction, though which they compete for processing tasks. Other aspects of the disclosed technologies relate to visual search capabilities, and determining appropriate actions responsive to different image inputs. Still others concern metadata generation, processing, and representation. Yet others relate to coping with fixed focus limitations of cell phone cameras, e.g., in reading digital watermark data. Still others concern user interface improvements. A great number of other features and arrangements are also detailed.

Claims

1. A cell phone including a microphone and a cellular radio transceiver, and additionally comprising:

an image sensor;

a human visual system processing module, selected from the group consisting of a white balance correction module, a gamma correction module, an edge enhancement module, and a lossy image compression module;

a machine vision processing module, selected from the group consisting of an edge detection module, a pattern extraction module, a Fourier-Mellin transformer, a texture classifier, a SIFT processor, and a SURF processor; and

an interconnection arrangement;

wherein the machine vision processing module employs processing circuitry integrated on a common substrate with the image sensor, and wherein the interconnection arrangement couples image data from the image sensor to the machine vision processing module without passing through the human visual system processing module.

2. A cell phone including an image data sensor, a memory, a data processor, and a screen, the data processor defining two parallel processing chains:

a first processing chain for producing data to be rendered into perceptual form for use by human viewers, the first processing chain including at least one of a de-mosaic processor and a lossy image compressor; and

a second processing chain that performs processing as part of one or more machine-implemented object-recognition algorithms, the second processing chain including at least one of a Fourier transform processor, a discrete cosine transform processor, and an eigenface processor;

wherein said parallel arrangement couples the first processing chain to the image data sensor without passing through the second processing chain, and said parallel arrangement couples the second processing chain to the image data sensor without passing through the first processing chain.

3. A portable device including a microphone and a radio transceiver, and additionally comprising:

an image sensor that senses imagery;

a human visual system processing module, selected from the group consisting of a white balance correction module, a gamma correction module, an edge enhancement module, and a lossy image compression module;

a machine vision processing module, selected from the group consisting of a Fourier transform module, an edge detection module, a pattern recognition module, a Fourier-Mellin transformer, a texture classifier, and a robust local feature detector, the machine vision processing module outputting processed data derived from the sensed imagery from the image sensor; and

an interconnection arrangement;

wherein the machine vision processing module employs processing circuitry integrated on a common substrate with the image sensor;

wherein the interconnection arrangement couples image data from the image sensor to the machine vision processing module without passing through the human visual system processing module; and

wherein the interconnection arrangement couples image data from the image sensor to the human visual system processing module without passing through the machine vision processing module.

4. The portable device of claim 3 that further comprises a processor, a memory, and instructions stored in the memory that configure the device to perform an object recognition operation on data output by the machine vision processing module, wherein said object recognition operation is performed on data that has not been processed by the human visual system processing module.

5. A portable device including an image data sensor that senses imagery, a memory, a data processor, and a display, the data processor being adapted to process image data and defining two parallel processing chains:

a first processing chain for receiving the sensed imagery from the image data sensor and for producing first data to be rendered into perceptual form for use by human viewers, the first processing chain including at least a first processor selected from the group consisting of a de-mosaic processor and a lossy image compressor; and

a second processing chain for receiving the sensed imagery from the image data sensor and for producing second data to subsequently be analyzed by one or more machine-implemented object recognition algorithms, the second processing chain including at least a second processor selected from the group consisting of a Fourier transform processor, a discrete cosine transform processor, and an eigenface processor;

wherein, as a consequence of the two parallel processing chains, said second data is provided for said subsequent analysis without having been impaired for said purpose by having been processed by the first processing chain.

6. The portable device of claim 5 in which the robust local feature detector comprises a scale invariant feature detector.

7. The portable device of claim 5 in which the robust local feature detector comprises a speeded-up robust feature detector.

8. The portable device of claim 5 in which the second processing chain comprises a Fourier transform processor.

9. The portable device of claim 5 in which the second processing chain comprises a discrete cosine transform processor.

10. The portable device of claim 5 in which the second processing chain comprises an eigenface processor.

11. The portable device of claim 5 in which the second processor is integrated on a common substrate with the image data sensor.

12. The portable device of claim 5 in which the second processing chain additionally comprises an edge detection module.

13. The portable device of claim 5 in which the second processing chain additionally comprises an a pattern recognition module.

14. The portable device of claim 5 in which the second processing chain additionally comprises a robust local feature detector.

15. The portable device of claim 5 in which the first processing chain additionally comprises a white balance correction module.

16. The portable device of claim 5 in which the second processing chain additionally comprises an edge enhancement module.

17. The portable device of claim 5 in which the data processor is further adapted to analyze the second data using a machine-implemented object recognition algorithm, wherein said algorithm is applied to data that has not been processed by the first processing chain.

18. A portable device including an image data sensor that senses imagery, a memory, a data processor, and a display, the data processor being adapted to process image data and defining two parallel processing chains:

a first processing chain for receiving the sensed imagery from the image data sensor and for producing first data to be rendered into perceptual form for use by human viewers, the first processing chain including at least a first processor selected from the group consisting of a de-mosaic processor, a lossy image compressor, a white balance correction module, a gamma correction module, and an edge enhancement module; and

a second processing chain for receiving the sensed imagery from the image data sensor and for producing second data to subsequently be analyzed by one or more machine-implemented object-recognition algorithms, the second processing chain including at least a second processor selected from the group consisting of a Fourier transform processor, a discrete cosine transform processor, an eigenface processor, an edge detection module, a pattern recognition module, a Fourier-Mellin transformer, a texture classifier, and a robust local feature detector;

wherein, as a consequence of the two parallel processing chains, said second data is provided for said subsequent analysis without having been impaired for said purpose by having been processed by the first processing chain.

19. The portable device of claim 18 in which the data processor is further adapted to analyze the second data using a machine-implemented object recognition algorithm, wherein said algorithm is applied to data that has not been processed by the first processing chain.

25. The portable device of claim 18 that includes a pattern recognition module to perform said one or more object-recognition algorithms, said pattern recognition module comprising a barcode decoding module, a digital watermark decoding module, or an image fingerprinting module;

wherein;

said first data is provided for presentation on the display; and

said second data is provided for use by the pattern recognition module.

26. The portable device of claim 25 in which the first processing chain comprises a white balance correction module.

27. The portable device of claim 25 in which the first processing chain comprises a gamma correction module.

28. The portable device of claim 25 in which the first processing chain comprises an edge enhancement module.

29. The portable device of claim 25 in which the first processing chain comprises a lossy image compression module.

30. The portable device of claim 25 in which the second processing chain comprises a Fourier transform module.

31. The portable device of claim 25 in which the second processing chain comprises an edge detection module.

32. The portable device of claim 25 in which the second processing chain comprises a pattern extraction module.

33. The portable device of claim 25 in which the second processing chain comprises a Fourier-Mellin transform module.

34. The portable device of claim 25 in which the second processing chain comprises a texture classifier.

35. The portable device of claim 25 in which the second processing chain also comprises a color histogram module.

36. The portable device of claim 25 in which the second processing chain comprises a robust local feature detector.

20. A cell phone including a microphone and a cellular radio transceiver, and additionally comprising:

an image sensor;

a human visual system module, selected from the group consisting of a white balance correction module, a gamma correction module, an edge enhancement module, and a lossy image compression module;

a pattern recognition module selected from the list consisting of a barcode decoding module, a digital watermark decoding module, and an image fingerprinting module;

said pattern recognition module employing a processing module drawn from the list consisting of;

an edge detection module, a pattern extraction module, a Fourier-Mellin transformer, a texture classifier, a SIFT processor, and a SURF processor; and

an interconnection arrangement;

wherein the interconnection arrangement couples image data from the image sensor to said processing module without passing through the human visual system module.

21. The cell phone of claim 20, wherein said processing module employs processing circuitry integrated on a common substrate with the image sensor.

22. A cell phone including:

an image data sensor;

a memory;

a screen;

a pattern recognition module selected from the list consisting of a barcode decoding module, a digital watermark decoding module, and an image fingerprinting module;

a data processor including first and second processing portions in a parallel arrangement, the first processing portion producing data to be rendered into perceptual form for use by human viewers, the second processing portion producing data to be processed by the pattern recognition module;

said first processing portion including at least one of a de-mosaic processor and a lossy image compressor; and

said second processing portion including at least one of a Fourier transform processor, a discrete cosine transform processor, and an eigenface processor;

wherein said parallel arrangement couples the first processing portion to the image data sensor without passing through the second processing portion, and said parallel arrangement couples the second processing portion to the image data sensor without passing through the first processing portion.

23. An apparatus comprising:

an image sensor;

first and second processing circuits;

the first processing circuit being coupled to receive image data from the image sensor without passing through the second processing circuit;

the second processing circuit being coupled to receive image data from the image sensor without passing through the first processing circuit;

the first processing circuit being configured to perform lossy image compression;

the second processing circuit being configured to perform one or more of;

edge detection, pattern extraction, Fourier-Mellin transformation, texture classification, or detection of scale- and rotation-invariant feature points;

wherein the image sensor, and the first and second processing circuits, are integrated on a common semiconductor substrate.

24. The apparatus of claim 23 that further includes a microphone, a microprocessor, and a cellular radio transceiver, interconnected to comprise a cell phone.