Mobile phone using dedicated and programmable processors for pipelined image processing, and method thereof

• US 8,855,712 B2
• Filed: 03/15/2013
• Issued: 10/07/2014
• Est. Priority Date: 08/19/2008
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Lord, John D., Rhoads, Geoffrey B., Rodriguez, Tony F.
• Primary Examiner(s): Maung, Nay A
• Assistant Examiner(s): Rivero, Alejandro
• Application Number:
  US13/836,166
• Publication Number:
  US 20130217440A1
• Time in Patent Office: 571 Days
• Field of Search: None
• US Class Current: 455/556.1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04M 1/0202 Portable telephone sets, e.g. cordless phones, mobile phones or bar type handsets constructional features of telephone transmitters or receivers, e.g. of speakers or microphones H04M1/03
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/2133 Recording or reproducing at a specific moment, e.g. time interval or time-lapse
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/3225 of data relating to an image, a page or a document
• CPC Class Codes: H04N 2201/3274 Storage or retrieval of prestored additional information
• CPC Class Codes: H04N 2201/3278 Transmission
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04W 4/50 : Service provisioning or reconfiguring

Abstract

Cell phones and other portable devices are equipped with a variety of technologies by which existing functionality is improved, and new functionality is provided. Some aspects relate to imaging architectures, in which a cell phone'"'"'s image sensor is one in a chain of stages that successively act on instructions/data, to capture and later process imagery. Other aspects relate to distribution of processing tasks between the device and remote resources (“the cloud”). Elemental image processing, such as filtering and edge detection—and even some simpler template matching operations—may be performed on the cell phone. Other operations are referred out to remote service providers. The remote service providers can be identified using techniques such as a reverse auction, through which they compete for processing tasks. Other aspects of the disclosed technologies relate to visual search capabilities, and determining appropriate actions responsive to different image inputs. Still others concern metadata generation, processing, and representation. A great number of other features and arrangements are also detailed.

Claims

1. A cell phone comprising:

an image sensor portion that outputs frames of image data;

a processing system coupled to the image sensor portion, the processing system including plural hardware processing units, including;

a first processing unit comprising first circuitry; and

a second processing unit different than the first processing unit, the second processing unit comprising second circuitry different than the first circuitry;

the processing system including an interconnection arrangement that conveys data from an output of the first processing unit to an input of the second processing unit;

a memory containing instructions that cause the processing system to configure a pipelined arrangement in which one of said processing units processes one frame of image data while another of said processing units processes a different frame of image data;

a setup module that sends information specifying operations to be performed on a first frame of data by the first and second processing units, before the pipelined arrangement begins processing said first frame of data; and

a radio interface coupled to the processing system to enable wireless transmission of one or more frames of processed image data from the cell phone;

wherein said instructions and information;

cause the first processing unit to process a first frame of data at a first time, applying a first function;

cause the same first processing unit to process said same first frame of data at a second, later time, applying a second function different than the first function; and

cause the second processing unit to process said same first frame of data between said first and second times, the second processing unit applying a third function different than the first and second functions.

2. The cell phone of claim 1 in which said second processing unit is a dedicated purpose processor whose function is fixed rather than programmable.

3. The cell phone of claim 1 in which said interconnection arrangement permits a frame of data output by the first processing unit to be input to the second processing unit for further processing, and also permits a frame of data output by the second processing unit to be input to the first processing unit for further processing.

4. The cell phone of claim 1 in which one of said plural hardware processing units is arranged to receive both image data, and instructions about processing the received image data, from a packet data structure.

5. The cell phone of claim 1 in which said image sensor outputs a first frame of data captured with a first exposure time, and a second frame of data captured with a second exposure time, and the instructions cause the processing system to process said first and second frames to yield a high dynamic range image.

6. The cell phone of claim 1 that further includes a pipe manager module configured to negotiate with other services in the cell phone to access wireless channel bandwidth for transmitting the processed image data, said pipe manager module both specifying an expected size of the image data to be transmitted, and also specifying when the image data is expected to be ready for transmitting.

7. The cell phone of claim 6 in which the pipe manager module is further configured to send information enabling a remote processing service to prepare for its processing of the transmitted image data by retrieving facial recognition features that may be used in said processing by said remote processing service.

8. The cell phone of claim 1 in which said instructions further cause the cell phone to:

wirelessly transmit a frame of processed image data to a remote processing system;

wirelessly receive data produced by the remote processing system from its processing of said transmitted frame; and

apply further image processing to said received data using one of said plural processing units of said pipelined arrangement, said further image processing serving to process the received data for object recognition, rather than for presentation to a human viewer.

9. The cell phone of claim 8 in which said further image processing comprises an operation including:

edge-detection, Discrete Cosine Transform (DCT), wavelet transform, Haar transform, Hough transform, Fourier-Mellin transform, filtering, facial recognition, computation of eigenvalues, extraction of shape feature data, extraction of color feature data, extraction of texture feature data, barcode decoding, watermark decoding, object segmentation, pattern recognition, age detection, gender detection, emotion classification, orientation determination, compression, decompression, log-polar mapping, convolution, interpolation, decimation, anti-aliasing, correlation, square-root, squaring, array multiplication, perspective transformation, or butterfly operation.

10. The cell phone of claim 1 in which the first processing unit comprises a graphics processing unit (GPU) employing parallelism.

11. A cell phone comprising:

an image sensor portion that outputs frames of image data;

a processing system coupled to the image sensor portion, the processing system including plural hardware processing units, including a first processing unit comprising first circuitry; and

a second processing unit comprising second circuitry different than the first circuitry;

a memory containing instructions that cause the processing system to configure a pipelined arrangement in which one of said processing units processes one frame of image data while another of said processing units processes a different frame of image data; and

a radio interface coupled to the processing system that enables wireless transmission of one or more frames of processed image data from the cell phone;

wherein said processing units are coupled to an interconnection arrangement that permits a frame of data output to an output of the first processing unit to be input to an input of the second processing unit for further processing, and also permits a frame of data output to an output of the second processing unit to be input to an input of the first processing unit for further processing; and

wherein said instructions;

cause the first processing unit to process a first frame of data at a first time, applying a first function;

cause the same first processing unit to process said same first frame of data at a second, later time, applying a second function different than the first function; and

cause the second processing unit to process said same first frame of data between said first and second times, the second processing unit applying a third function different than the first and second functions.

12. The cell phone of claim 11 in which said instructions further cause the cell phone to:

wirelessly transmit a frame of processed image data to a remote processing system;

wirelessly receive data produced by the remote processing system from its processing of said transmitted frame; and

apply further image processing to said received data using one of said plural processing units of said pipelined arrangement, said further image processing serving to process the received data for object recognition, rather than for presentation to a human viewer.

13. The cell phone of claim 12 in which said further image processing comprises an operation including:

edge-detection, Discrete Cosine Transform (DCT), wavelet transform, Haar transform, Hough transform, Fourier-Mellin transform, filtering, facial recognition, computation of eigenvalues, extraction of shape feature data, extraction of color feature data, extraction of texture feature data, barcode decoding, watermark decoding, object segmentation, pattern recognition, age detection, gender detection, emotion classification, orientation determination, compression, decompression, log-polar mapping, convolution, interpolation, decimation, anti-aliasing, correlation, square-root, squaring, array multiplication, perspective transformation, or butterfly operation.

14. The cell phone of claim 11 that further includes a pipe manager module configured to negotiate with other services in the cell phone to access wireless channel bandwidth for transmitting processed image data, said pipe manager module both specifying an expected size of image data to be transmitted, and also specifying when image data is expected to be ready for transmitting.

15. The cell phone of claim 14 in which the pipe manager module is further configured to send information enabling a remote processing service to prepare for its processing of the transmitted image data by retrieving facial recognition features that may be used in said processing by said remote processing service.

16. The cell phone of claim 11 in which the first processing unit comprises a graphics processing unit (GPU) employing parallelism.

17. A method comprising:

capturing first and second frames of image data, and providing each frame of image data to an image processing system of a cell phone;

the image processing system including plural different processing units, each comprising hardware circuitry, said processing units operating in a pipelined arrangement in which data from an output of one of said processing units is conveyed to an input of another of said processing units, and in which one of said processing units processes one frame of image data while another of said processing units processes a different frame of image data, said pipelined image processing system also performing acts including;

at a first time, processing one of said frames of image data with a dedicated purpose one of said processing units, whose function is fixed, while also processing the other of said frames of image data with a programmable one of said processing units whose function has been configured to perform a first function;

at a second, later, time, again processing said other frame of image data with said programmable processing unit, said programmable processing unit this second time having been configured to perform a second function different than the first function; and

between said first and second times, processing said other frame of image data with a processing unit different than said programmable processing unit, which applies a function different than said first and second functions;

wherein said programmable processing unit in said pipelined image processing system is used first and second times to perform first and second different functions on a single frame of image data, with said different processing unit processing said single frame of data between said first and second times.

18. The method of claim 17 that includes:

capturing a first frame of image data with a first exposure time;

capturing a second frame of image data with a second, different exposure time; and

processing the first and second frames of image data to yield a high dynamic range image.

19. The method of claim 17 that further includes wirelessly transmitting processed imagery resulting from one of said processing acts, from said cell phone.

20. The method of claim 17 that further includes:

wirelessly transmitting processed image data resulting from one of said processing acts, from said cell phone to a remote processing service;

wirelessly receiving data produced by said remote processing service from its processing of said transmitted image data; and

further processing said received data using one of said plural processing units of the pipelined arrangement, said further image processing serving to process the received data for object recognition, rather than for presentation to a human viewer.

21. The method of claim 20 that includes negotiating with other services in the cell phone to access wireless channel bandwidth for transmitting the processed image data, said negotiating occurring before the processed image data is ready for transmitting, said negotiating including specifying an expected size of the image data to be transmitted, and specifying when the image data is expected to be ready for transmitting.

22. The method of claim 21 that further includes sending information that enables the remote processing service to prepare for its processing of the transmitted image data by retrieving facial recognition features that may be used in said processing by the remote processing service.

23. The method of claim 20 that further includes receiving information indicating that there will be a delay in receiving the data produced by the remote processing service, and alerting a user regarding the delay.

24. The method of claim 20 in which said further image processing includes performing an operation including:

edge-detection, Discrete Cosine Transform (DCT), wavelet transform, Haar transform, Hough transform, Fourier-Mellin transform, filtering, facial recognition, computation of eigenvalues, extraction of shape feature data, extraction of color feature data, extraction of texture feature data, barcode decoding, watermark decoding, object segmentation, pattern recognition, age detection, gender detection, emotion classification, orientation determination, compression, decompression, log-polar mapping, convolution, interpolation, decimation, anti-aliasing, correlation, square-root, squaring, array multiplication, perspective transformation, or butterfly operation.

25. The method of claim 17 that further includes conveying both image data and processing unit instructions together, between processing units, in the form of a packet data structure.

26. The method of claim 17 that further includes processing said another frame of image data with a third processing unit between said first and second times of processing of said another frame of image data by the programmable processing unit.

27. The method of claim 17 that further includes the programmable processing unit receiving both a frame of image data, and instructions about processing the received frame of image data, from a packet data structure.

28. The method of claim 17 in which said programmable processing unit comprises a graphics processing unit (GPU) employing parallelism.

29. An apparatus including:

a processing system adapted to process received image frames of data, the processing system including plural hardware processing units configured in a pipelined arrangement, so that one of the processing units processes one frame of data while another of said processing units processes a different frame of data, the processing system including;

a first processing unit comprising first circuitry;

a second processing unit comprising second circuitry different than the first circuitry; and

an interconnection arrangement that conveys signals between outputs and inputs of the processing units in said pipelined arrangement;

a setup module that is adapted to send setup information to the pipelined arrangement, the setup information specifying operations to be performed by the first and second processing units on the frames of data;

wherein said setup information;

causes the first processing unit to process a first frame of data at a first time, applying a first function;

causes the same first processing unit to process said same first frame of data at a second, later time, applying a second function different than the first function; and

causes the second processing unit to process said same first frame of data between said first and second times, the second processing unit applying a third function different than the first and second functions; and

wherein the setup module sends the setup information specifying operations to be performed on a first frame of data by the first and second processing units, before the pipelined arrangement begins processing said first frame of data.

30. The apparatus of claim 29 in which the setup module is further adapted to send setup instructions for use by a camera, to establish particular parameters to be used in capturing a frame of data for processing by the apparatus.

31. The apparatus of claim 29 in which the setup module is adapted to send said setup information to the pipelined arrangement on a frame-by-frame basis, for the first, and for immediately succeeding second and third, frames of imagery, wherein said setup information specifies a first set of operations for the first frame, a second set of operations for the second frame, and a third set of operations for the third frame, wherein each of said sets of operations is different.