Image distortion correction in scaling circuit

• US 9,148,571 B2
• Filed: 09/14/2012
• Issued: 09/29/2015
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tripathi, Brijesh
• Primary Examiner(s): HANNETT, JAMES M
• Application Number:
  US13/617,428
• Publication Number:
  US 20140078324A1
• Time in Patent Office: 1,110 Days
• Field of Search: 348/208.1
• US Class Current: 1/1
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/6842 by controlling the scanning position, e.g. windowing
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

Techniques relating to correction of image distortion caused by movement of a camera unit during image capture. In one embodiment, an apparatus may include a camera unit and a scaling circuit. The apparatus may be configured to calculate a shift value for a line of an image captured by the camera unit, where the shift value is indicative of an amount of movement of the camera unit during at least a portion of capture of the image. The scaling circuit may be configured to operate on the line starting at a line position that is based on the calculated shift value. The calculated shift value may be based on movement information generated by a motion sensor. The scaling circuit may include a digital differential analyzer and one or more multi-tap polyphase filters. The line position may be specified as a fractional pixel value.

Claims

1. A method, comprising:

storing in a buffer, by an apparatus having a camera unit, an image captured by the camera unit, wherein the storing includes storing lines of pixels of the image in entries of the buffer;

measuring, by the apparatus, an amount of movement of the camera unit during capture of the image;

calculating, by the apparatus, a first number of pixels of the image, wherein the first number of pixels corresponds to the amount of movement of the camera;

determining a starting pixel for a stored line in the image, wherein the starting pixel is offset from an end pixel of the line by a number of pixels that is determined based on the first number of pixels and the number of lines of pixels in the image; and

scaling, by a scaling circuit in the apparatus, the line of pixels, wherein the scaling the line of pixels includes scaling the line of pixels beginning at the starting pixel.

2. The method of claim 1, wherein the measuring an amount of movement comprises measuring amount of movement by a motion sensor during capture of the image.

3. The method of claim 2, wherein the motion sensor comprises an accelerometer and a gyroscope.

4. The method of claim 1, further comprising the scaling circuit discarding one or more unused pixels preceding the starting pixel.

5. The method of claim 1, wherein the scaling includes using a scaling factor that is selected, based on the amount of movement of the camera unit during capture of the image, to generate a scaled line of pixels having a same size as the line of pixels.

6. An apparatus, comprising:

buffer circuitry;

a camera unit configured to capture an image; and

a scaling circuit;

wherein the apparatus is configured to;

store lines of pixels of a captured image in the buffer circuitry;

determine a first number of pixels of the image that corresponds to an amount of movement of the camera during at least a portion of capture of the image;

calculate a shift value for a stored line of pixels in the image, wherein the calculated shift value is based on the first number of pixels and the number of lines of pixels in the image; and

wherein the scaling circuit is configured to scale the line starting at a line position that is based on the calculated shift value, wherein the scaling uses a scaling factor greater than one.

7. The apparatus of claim 6, further comprising a motion sensor configured to determine the amount of movement of the camera.

8. The apparatus of claim 6, wherein the camera unit is configured to capture the image by scanning different lines of pixels at different points in time.

9. The apparatus of claim 8, wherein the camera unit includes complementary metal-oxide-semiconductor sensors.

10. The apparatus of claim 6, where the line position is specified as a fractional pixel position.

11. A method, comprising:

an apparatus having a camera unit capturing an image that has distortion resulting from the camera unit moving during capture of the image;

the apparatus correcting the distortion, wherein the correcting comprises;

determining a total number of pixels corresponding to movement of the camera unit during capture of the image;

calculating shift values for lines of pixels in the image, wherein the shift values are calculated based on the total number of pixels and a number of lines of pixels in the image; and

processing lines of pixels in the captured image, using a scaling circuit, starting with respective line positions that are based on the calculated shift value corresponding to that line;

wherein the processing uses a scaling factor for each line that is determined for the image based on the total number of pixels and the number of pixels in each line of pixels such that the processing produces an output line of pixels that has the same number of pixels as the captured line of pixels.

12. The method of claim 11, wherein the shift values correspond to an average shift value over capture of the image.

13. The method of claim 11, wherein the scaling factor is determined based on the calculated shift value to compensate for unused pixels preceding the respective line positions in order to retain a particular image size.

14. The method of claim 11, further comprising reading the image from a location in a memory before the processing and storing the image to the same location in the memory after the processing.

15. An apparatus, comprising:

buffer circuitry

a camera unit configured to capture an image, wherein the apparatus is configured to store lines of pixels of the captured image using the buffer circuitry;

a motion sensor unit configured to generate motion information during capture of the image;

a processing element configured to determine a number of pixels of the image corresponding to an amount of motion indicated by the motion information and calculate an initial scale location for a line of pixels of the image based on the number of pixels and a number of lines of pixels in the image; and

at least one scaling circuit configured to operate on the line of pixels starting at the initial scale location, wherein the operating generates a scaled line of pixels using a scaling factor that is dynamically determined for the image based on the motion information and the number of pixels in the line to generate an output line of pixels having a same size as the stored line of pixels.

16. The apparatus of claim 15, wherein the scaling circuit is included in a memory scaler rotator that comprises a plurality of configuration registers and is configured to:

read the image from a memory;

rotate the image based on values in one or more rotation configuration registers;

scale the image based on values in on one or more scale configuration registers and one or more initial scale location registers; and

write the image to the memory.

17. The apparatus of claim 16, wherein the processing element is configured to:

write to the one or more rotation configuration registers to indicate that no rotation should take place.

18. The apparatus of claim 15, wherein the apparatus further comprises a horizontal scaling unit and a vertical scaling unit;

wherein the line of pixels is a horizontal line of pixels;

wherein the horizontal scaling unit is configured to operate on the line of pixels starting at a line position that is based on the initial scale location; and

wherein the vertical scaling unit is configured to operate on a vertical line of pixels starting at a line position that is based on another initial scale location calculated by the processing element for the vertical line of pixels.

19. The apparatus of claim 15;

wherein the motion sensor unit comprises an accelerometer and a gyroscope;

wherein the apparatus further comprises a graphics processing unit; and

wherein the camera unit comprises an image signal processor.

20. An apparatus, comprising:

a camera unit configured to capture an image that has distortion resulting from movement of the camera unit during image capture; and

a scaling circuit;

wherein the apparatus is configured to calculate shift values for lines of pixels in the image, wherein the shift values are determined by dividing a total number of pixels corresponding to an amount of movement of the camera during image capture by the number of lines of pixels in the image; and

wherein the scaling circuit is configured to process lines of pixels in the captured image at respective line positions that are based on the calculated shift value for that line, including determining pixel attributes for a pixel in one of the processed lines based on multiple pixels in the line being processed.

21. The apparatus of claim 20, wherein the scaling circuit comprises a digital differential analyzer to process pixels of the lines of pixels.

22. The apparatus of claim 20, wherein the scaling circuit comprises one or more multiple-tap polyphase filters for processing pixels of the lines of pixels.

23. The apparatus of claim 20, wherein the apparatus is configured to linearly interpolate based on a plurality of movement measurements to calculate the shift values.