Motion compensation for image sensor with a block based analog-to-digital converter

• US 10,075,640 B2
• Filed: 12/31/2015
• Issued: 09/11/2018
• Est. Priority Date: 12/31/2015
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Wong, Ping Wah
• Primary Examiner(s): Hannett, James
• Application Number:
  US14/985,601
• Publication Number:
  US 20170195574A1
• Time in Patent Office: 985 Days
• Field of Search: None
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/70 SSIS architectures; Circuits associated therewith
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/78 Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
• CPC Class Codes: H04N 9/01 Circuitry for demodulating colour component signals modulated spatially by colour striped filters by phase separation
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

An electronic device including a motion sensor and motion correction circuitry. The motion sensor detects motion of the electronic device and outputs motion information based on the detected motion. The motion correction circuitry corrects raw domain image data, received from an image sensor that has a block-based analog-to-digital converter architecture, based on the motion information. The corrected image data is output in a same raw domain image data format as the uncorrected raw domain image data.

Claims

1. An electronic device comprising:

a motion sensor configured to

detect motion of the electronic device, and

output motion information based on the motion that is detected; and

motion correction circuitry that is separate from the motion sensor, the motion correction circuitry configured to

receive raw domain image data from an image sensor including a block-based analog-to-digital converter architecture and image sensing elements, the raw domain image data being generated by the block-based analog-to-digital converter architecture based on analog pixel signals that are generated from an exposure of the image sensing elements to incident light, and

correct the raw domain image data based on the motion information,

wherein detecting the motion of the electronic device is synchronous with the exposure of the image sensing elements to the incident light.

2. The electronic device of claim 1,

wherein the motion correction circuitry is configured to correct pixel values of the raw domain image data by:

determining uncorrected locations, in a two-dimensional space, of the pixel values based on a data format of the raw domain image data, the uncorrected locations forming a regular grid,

determining a corrected location, in the two-dimensional space, for each of the pixel values based on the pixel values'"'"' respective uncorrected locations and the motion information,

generating a corrected pixel value for each node of the regular grid based on the pixel values and their respective corrected locations, and

outputting corrected pixel values as a motion-corrected frame in the data format of the raw domain image data.

3. The electronic device of claim 2,

wherein the motion correction circuitry is configured to generate a given one of the corrected pixel values by performing an interpolation utilizing at least one of the pixel values and a corresponding corrected location.

4. The electronic device of claim 2,

wherein the motion correction circuitry is configured to generate a given one of the corrected pixel values by performing bi-linear interpolation utilizing four of the pixel values and their respective corrected locations.

5. An electronic device comprising:

a motion sensor configured to detect motion of the electronic device and output motion information based on the detected motion; and

motion correction circuitry configured to correct raw domain image data, received from an image sensor that has a block-based analog-to-digital converter architecture, based on the motion information,

wherein the motion correction circuitry is configured to correct pixel values of the raw domain image data by;

determining uncorrected locations, in a two-dimensional space, of the pixel values based on a data format of the raw domain image data, the uncorrected locations forming a regular grid,

determining a corrected location, in the two-dimensional space, for each of the pixel values based on the pixel values'"'"' respective uncorrected locations and the motion information,

generating a corrected pixel value for each node of the regular grid based on the pixel values and their respective corrected locations, and

outputting corrected pixel values as a motion-corrected frame in the data format of the raw domain image data,

wherein the motion correction circuitry is configured to generate a given one of the corrected pixel values by performing bi-linear interpolation utilizing four of the pixel values and their respective corrected locations, and

wherein each of the pixel values in the raw domain image data corresponds to one of a plurality of colors according to a color pattern associated with the data format of the raw domain image data,

each of the corrected pixel values in the motion-corrected frame corresponds to one of the plurality of colors according to the color pattern, and

the four of the pixel values that are utilized by the motion correction circuitry in the bi-linear interpolation for the given one of the corrected pixel values correspond to a same one of the plurality of colors that the given one of the corrected pixel values corresponds to.

6. An electronic device comprising:

a motion sensor configured to detect motion of the electronic device and output motion information based on the detected motion; and

motion correction circuitry configured to correct raw domain image data, received from an image sensor that has a block-based analog-to-digital converter architecture, based on the motion information,

wherein the motion correction circuitry is configured to correct pixel values of the raw domain image data by;

determining uncorrected locations, in a two-dimensional space, of the pixel values based on a data format of the raw domain image data, the uncorrected locations forming a regular grid,

determining a corrected location, in the two-dimensional space, for each of the pixel values based on the pixel values'"'"' respective uncorrected locations and the motion information,

generating a corrected pixel value for each node of the regular grid based on the pixel values and their respective corrected locations, and

outputting corrected pixel values as a motion-corrected frame in the data format of the raw domain image data, and

wherein the block-based analog-to-digital converter architecture includes having a plurality of blocks that each include N light sensing elements that share an analog-to-digital converter, N>

1, the N light sensing elements in each block being exposed sequentially in N exposure phases per frame period,

each of the pixel values corresponds to one of the exposure phases,

the motion sensor is configured to output the motion information as motion vectors that each correspond to one of the exposure phases,

the motion correction circuitry is configured to determine the corrected location of each of the pixel values based on the one of the motion vectors that corresponds to the one of the exposure phases to which the respective one of the pixel values corresponds.

7. The electronic device of claim 6,

wherein N=4,

each of the pixel values in the raw domain image data corresponds to one of four colors according to a color pattern associated with the data format of the raw domain image data,

each of the corrected pixel values in the motion-corrected frame corresponds to one of the four colors according to the color pattern, and

each of the phases corresponds to one of the four colors.

8. The electronic device of claim 7,

wherein the motion correction circuitry is configured to generate a given one of the corrected pixel values by performing bi-linear interpolation utilizing four of the pixel values and their respective corrected locations, the four of the pixel values corresponding to a same one of the four colors that the given one of the corrected pixel values corresponds to.

9. The electronic device of claim 6,

wherein each of the pixel values in the raw domain image data corresponds to one of a plurality of colors according to a color pattern associated with the data format of the raw domain image data,

each of the corrected pixel values in the motion-corrected frame corresponds to one of the plurality of colors according to the color pattern, and

each of the phases corresponds to one of the plurality of colors

the motion correction circuitry is configured to generate each of the corrected pixel values by performing interpolations, each of the interpolations utilizing only ones of the pixel values that correspond to a same one of the plurality of colors that the one of the corrected pixel values that is being generated corresponds to.

10. An image sensing device, comprising:

an image sensor including

image sensing elements configured to generate analog pixel signals from an exposure to incident light, and

a block-based analog-to-digital converter architecture configured to generate raw domain image data based on the analog pixel signals;

a motion sensor that is separate from the image sensor, the motion sensor configured to

detect motion of the image sensing device, and

output motion information based on the motion that is detected; and

motion correction circuitry configured to

receive the raw domain image data from the image sensor, and

correct the raw domain image data based on the motion information,

wherein detecting the motion of the image sensing device is synchronous with the exposure of the image sensing elements to the incident light.

11. The image sensing device of claim 10,

wherein the motion correction circuitry is configured to correct pixel values of the raw domain image data by:

determining uncorrected locations, in a two-dimensional space, of the pixel values based on a data format of the raw domain image data, the uncorrected locations forming a regular grid,

determining a corrected location, in the two-dimensional space, for each of the pixel values based on the pixel values'"'"' respective uncorrected locations and the motion information,

generating a corrected pixel value for each node of the regular grid based on the pixel values and their respective corrected locations, and

outputting corrected pixel values as a motion-corrected frame in the data format of the raw domain image data.

12. The image sensing device of claim 11,

wherein the motion correction circuitry is configured to generate a given one of the corrected pixel values by performing an interpolation utilizing at least one of the pixel values and its corrected location.

13. The image sensing device of claim 11,

wherein the motion correction circuitry is configured to generate a given one of the corrected pixel values by performing bi-linear interpolation utilizing four of the pixel values and their respective corrected locations.

19. The image sensing device of claim 10,

wherein the image sensor is part of a same circuit package as at least one of the motion sensor or the motion correction circuitry.

20. The image sensing device of claim 10,

wherein the image sensor is part of a different circuit package than at least one of the motion sensor or the motion correction circuitry.

21. An electronic apparatus comprising:

the imaging sensing device of claim 10, and

an image signal processor configured to execute predetermined signal processing on motion-corrected raw domain image data output by the motion correction circuitry.

22. The electronic apparatus of claim 21,

wherein the image signal processor is part of a same circuit package as at least one of the image sensor, the motion sensor, or the motion correction circuitry.

23. The image sensing device of claim 10, further comprising:

an image signal processor,

wherein the image signal processor is part of a different circuit package than at least one of the image sensor, the motion sensor, or the motion correction circuitry.

14. An image sensing device, comprising:

an image sensor that has a block-based analog-to-digital converter architecture;

a motion sensor configured to detect motion of the image sensing device and output motion information based on the detected motion; and

motion correction circuitry configured to correct raw domain image data, received from the image sensor, based on the motion information,

wherein the motion correction circuitry is configured to correct pixel values of the raw domain image data by;

determining uncorrected locations, in a two-dimensional space, of the pixel values based on a data format of the raw domain image data, the uncorrected locations forming a regular grid,

determining a corrected location, in the two-dimensional space, for each of the pixel values based on the pixel values'"'"' respective uncorrected locations and the motion information,

generating a corrected pixel value for each node of the regular grid based on the pixel values and their respective corrected locations, and

outputting corrected pixel values as a motion-corrected frame in the data format of the raw domain image data,

wherein the motion correction circuitry is configured to generate a given one of the corrected pixel values by performing bi-linear interpolation utilizing four of the pixel values and their respective corrected locations, and

wherein each of the pixel values in the raw domain image data corresponds to one of a plurality of colors according to a color pattern associated with the data format of the raw domain image data,

each of the corrected pixel values in the motion-corrected frame corresponds to one of the plurality of colors according to the color pattern, and

the four of the pixel values that are utilized by the motion correction circuitry in the bi-linear interpolation for the given one of the corrected pixel values correspond to a same one of the plurality of colors that the given one of the corrected pixel values corresponds to.

15. An image sensing device, comprising:

an image sensor that has a block-based analog-to-digital converter architecture;

a motion sensor configured to detect motion of the image sensing device and output motion information based on the detected motion; and

motion correction circuitry configured to correct raw domain image data, received from the image sensor, based on the motion information,

wherein the motion correction circuitry is configured to correct pixel values of the raw domain image data by;

determining uncorrected locations, in a two-dimensional space, of the pixel values based on a data format of the raw domain image data, the uncorrected locations forming a regular grid,

determining a corrected location, in the two-dimensional space, for each of the pixel values based on the pixel values'"'"' respective uncorrected locations and the motion information,

generating a corrected pixel value for each node of the regular grid based on the pixel values and their respective corrected locations, and

outputting corrected pixel values as a motion-corrected frame in the data format of the raw domain image data, and

wherein the block-based analog-to-digital converter architecture includes having a plurality of blocks that each include N light sensing elements that share an analog-to-digital converter, N>

1, the N light sensing elements in each block being exposed sequentially in N exposure phases per frame period,

each of the pixel values corresponds to one of the exposure phases,

the motion sensor is configured to output the motion information as motion vectors that each correspond to one of the exposure phases,

the motion correction circuitry is configured to determine the corrected location of each of the pixel values based on the one of the motion vectors that corresponds to the one of the exposure phases to which the respective one of the pixel values corresponds.

16. The image sensing device of claim 15,

wherein N=4,

each of the pixel values in the raw domain image data corresponds to one of four colors according to a color pattern associated with the data format of the raw domain image data,

each of the corrected pixel values in the motion-corrected frame corresponds to one of the four colors according to the color pattern, and

each of the phases corresponds to one of the four colors.

17. The image sensing device of claim 16,

wherein the motion correction circuitry is configured to generate a given one of the corrected pixel values by performing bi-linear interpolation utilizing four of the pixel values and their respective corrected locations, the four of the pixel values corresponding to a same one of the four colors that the given one of the corrected pixel values corresponds to.

18. The image sensing device of claim 15,

wherein each of the pixel values in the raw domain image data corresponds to one of a plurality of colors according to a color pattern associated with the data format of the raw domain image data,

each of the corrected pixel values in the motion-corrected frame corresponds to one of the plurality of colors according to the color pattern, and

each of the phases corresponds to one of the plurality of colors

the motion correction circuitry is configured to generate each of the corrected pixel values by performing interpolations, each of the interpolations utilizing only ones of the pixel values that correspond to a same one of the plurality of colors that the one of the corrected pixel values that is being generated corresponds to.

24. A method of controlling a motion correction device, the method comprising:

detecting, with a motion sensor, motion of the motion correction device;

generating motion information based on the motion that is detected by the motion sensor;

receiving, with a motion correction circuitry, raw domain image data from an image sensor including a block-based analog-to-digital converter architecture and image sensing elements, the raw domain image data being generated by the block-based analog-to-digital converter architecture based on analog pixel signals that are generated from an exposure of the image sensing elements to incident light; and

correcting the raw domain image data based on the motion information,

wherein detecting the motion of the motion correction device is synchronous with the exposure of the image sensing elements to the incident light.

25. The method of claim 24, further comprising:

causing the motion correction device to correct pixel values of the raw domain image data by;

determining uncorrected locations, in a two-dimensional space, of the pixel values based on a data format of the raw domain image data, the uncorrected locations forming a regular grid,

determining a corrected location, in the two-dimensional space, for each of the pixel values based on the pixel values'"'"' respective uncorrected locations and the motion information, and

generating a corrected pixel value for each node of the regular grid based on the pixel values and their respective corrected locations.

26. The method of claim 25, further comprising:

outputting the corrected pixel values as a motion-corrected frame in the data format of the raw domain image data.

27. A non-transitory computer readable medium storing program code executable by a processor of a motion correction device to cause the processor to perform a set of operations comprising:

controlling a motion sensor to detect motion of the motion correction device;

generating motion information based on the motion that is detected by the motion sensor;

receiving raw domain image data from an image sensor that is separate from the motion sensor, the image sensor including a block-based analog-to-digital converter architecture and image sensing elements, the raw domain image data being generated by the block-based analog-to-digital converter architecture based on analog pixel signals that are generated from an exposure of the image sensing elements to incident light; and

correcting the raw domain image data based on the motion information,

wherein the motion of the motion correction device that is detected by the motion sensor is synchronous with the exposure of the image sensing elements to the incident light.

28. The non-transitory computer readable medium of claim 27, wherein the set of operations further includes

causing the motion correction device to correct pixel values of the raw domain image data by:

determining uncorrected locations, in a two-dimensional space, of the pixel values based on a data format of the raw domain image data, the uncorrected locations forming a regular grid,

determining a corrected location, in the two-dimensional space, for each of the pixel values based on the pixel values'"'"' respective uncorrected locations and the motion information, and

generating a corrected pixel value for each node of the regular grid based on the pixel values and their respective corrected locations.

29. The non-transitory computer readable medium of claim 28, wherein the set of operations further includes

outputting the corrected pixel values as a motion-corrected frame in the data format of the raw domain image data.