Electronic image stabilization of a captured image

• US 10,477,107 B2
• Filed: 01/27/2017
• Issued: 11/12/2019
• Est. Priority Date: 01/27/2017
• Status: Active Grant

• Current Assignee: Invensense Incorporated (TDK Corporation)
• Original Assignee: Invensense Incorporated (TDK Corporation)
• Inventors: Keal, William Kerry
• Primary Examiner(s): Khan, Usman A
• Application Number:
  US15/418,488
• Publication Number:
  US 20180220073A1
• Time in Patent Office: 1,019 Days
• Field of Search: 3482085
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

In a method of electronic image stabilization, a processor buffers image data into a memory buffer, the image data being obtained by the processor from an image sensor disposed in an electronic device. The processor obtains motion data from a motion sensor disposed in the electronic device, wherein the motion data corresponds with a time of capture of the image data. The processor analyzes the motion data to determine a stabilization correction to apply to the image data. The processor applies the determined stabilization correction to the image data to achieve stabilized image data. The determined stabilization correction is applied, and the stabilized image data is achieved, by the processor without requiring a transfer of the image data from the memory buffer to a graphics processing unit. The stabilized image data is output.

Claims

1. A method of electronic image stabilization comprising:

buffering, by a processor into a memory buffer, image data obtained by said processor from an image sensor disposed in an electronic device;

obtaining, by said processor, motion data from a motion sensor disposed in said electronic device, said motion sensor for measuring motion of said electronic device, wherein said motion data is indicative of said motion of said electronic device and corresponds with a time of capture of said image data;

analyzing, by said processor, said motion data to determine a stabilization correction to apply to said image data, said analyzing comprising;

analyzing displacement of portions of said image data relative to other portions of said image data, wherein said displacement is based on said motion data; and

determining said stabilization correction to correct for said displacement;

applying, by said processor, said determined stabilization correction to said image data from said memory buffer to achieve stabilized image data, wherein said determined stabilization correction is applied and said stabilized image data is achieved by said processor without requiring a transfer of said image data from said memory buffer to a graphics processing unit; and

outputting said stabilized image data.

2. The method as recited in claim 1, further comprising:

controlling, by said processor, an optical image stabilization system of said image sensor based on said motion data.

3. The method as recited in claim 1, wherein said applying, by said processor, said determined stabilization correction to said image data to achieve stabilized image data comprises:

reading, by said processor, said image data from said memory buffer into a second memory buffer in a data order specified by said determined stabilized correction, such that said reading creates said stabilized image data.

4. The method as recited in claim 3, wherein said outputting said stabilized image data comprises:

outputting said stabilized image data from said second memory buffer after a complete stabilized image frame has been buffered in said second memory buffer.

5. The method as recited in claim 1, wherein said buffering, by a processor into a memory buffer, image data obtained by said processor from an image sensor disposed in an electronic device comprises:

said processor changing a registry address associated with said image sensor to a second registry address that is associated with said processor.

6. The method as recited in claim 1, wherein said outputting said stabilized image data comprises:

outputting said stabilized image data in a format that mimics a characteristic of said image data.

7. The method as recited in claim 6, wherein said outputting said stabilized image data in a format identical to that of said image data comprises:

outputting said stabilized image data at a frame rate of the obtained image data.

8. The method as recited in claim 6, wherein said outputting said stabilized image data in a format identical to that of said image data comprises:

outputting said stabilized image data at an aspect ratio of said obtained image data.

9. The method as recited in claim 6, wherein said outputting said stabilized image data in a format identical to that of said image data comprises:

outputting said stabilized data with a delay, wherein an amount of said delay is based on said determined stabilization correction.

10. A stand-alone electronic image stabilizer comprising:

a memory; and

a processor coupled with said memory and configured to;

buffer, into a first portion of said memory, image data obtained by said processor from an image sensor disposed in an electronic device;

obtain motion data from a motion sensor disposed in said electronic device, said motion sensor for measuring motion of said image sensor, wherein said motion data is indicative of said motion of said image sensor and corresponds with a time of capture of said image data;

analyze said motion data to determine a stabilization correction to apply to said image data, wherein said motion data identifies displacement of portions of said image data relative to other portions of said image data and wherein said stabilization correction is based on said motion data to correct for said displacement;

apply said determined stabilization correction to said image data from said first portion of said memory to achieve stabilized image data, wherein said stabilization correction is applied and said stabilized image data is achieved by said processor without requiring a transfer of said image data from said memory to a graphics processing unit; and

output said stabilized image data.

11. The stand-alone electronic image stabilizer of claim 10, wherein said image data represents a sub-portion of an image frame.

12. The stand-alone electronic image stabilizer of claim 10, wherein the size of said first portion of said memory is adapted to at least one of the size of said image data and stabilization settings utilized by said stabilization correction.

13. The stand-alone electronic image stabilizer of claim 10, wherein said processor obtains said image data by selectively reading from one of said image sensor and a camera buffer, wherein said selective reading is based on said motion data.

14. The stand-alone electronic image stabilizer of claim 10, wherein said stabilization correction comprises at least one of:

cropping said image data, transforming said image data, resizing said image data, and filtering said image data.

15. The stand-alone electronic image stabilizer of claim 10, wherein a stabilized image data frame rate is identical to a frame rate of said image data.

16. The stand-alone electronic image stabilizer of claim 15, wherein a delay is applied to said output of said stabilized image data, wherein said delay is linked to said stabilization correction applied to said image data to achieve said stabilized image data.

17. The stand-alone electronic image stabilizer of claim 10, wherein said processor, said memory, and said motion sensor are disposed in a single integrated circuit with one another.

18. The stand-alone electronic image stabilizer of claim 10, where said processor is further configured to access a registry associated with said image sensor.

19. The stand-alone electronic image stabilizer of claim 10, where said processor is further configured to change a registry address associated with said image sensor to a second registry address associated with said stand-alone electronic image stabilizer.

20. A mobile electronic device, said mobile electronic device comprising:

a host processor;

a gyroscope or accelerometer;

an image sensor coupled with said host processor; and

a stand-alone electronic image stabilizer coupled with said image sensor, said host processor, and said gyroscope or accelerometer, said stand-alone electronic image stabilizer comprising;

a memory; and

a processor coupled with said memory and said image sensor and configured to;

buffer, into a first portion of said memory, image data obtained by said processor from said image sensor;

obtain motion data from said gyroscope or accelerometer, wherein said motion data is indicative of motion of said image sensor and corresponds with a time of capture of said image data;

analyze said motion data to determine a stabilization correction to apply to said image data, wherein said motion data identifies displacement of portions of said image data relative to other portions of said image data and wherein said stabilization correction is based on said motion data to correct for said displacement;

apply said determined stabilization correction to said image data from said first portion of said memory to achieve stabilized image data, wherein said determined stabilization correction is applied and said stabilized image data is achieved by said processor without requiring a transfer of said image data from said memory to a graphics processing unit; and

output said stabilized image data.

21. The mobile electronic device of claim 20, wherein said image data represents a sub-portion of an image frame.

22. The mobile electronic device of claim 20, wherein the size of said first portion of said memory is adapted to at least one of the size of said image data and stabilization settings utilized by said stabilization correction.

23. The mobile electronic device of claim 20, wherein said processor obtains said image data by selectively reading from one of said image sensor and a camera buffer, and wherein said selective reading is based on said motion data.

24. The mobile electronic device of claim 20, wherein said stabilization correction comprises a transformation of said image data.

25. The mobile electronic device of claim 20, wherein said stabilization correction comprises one or a combination of stabilization corrections selected from the list of stabilization corrections consisting of:

cropping said image data, resizing said image data, and filtering said image data.

26. The mobile electronic device of claim 20, wherein a stabilized image data frame rate is identical to a frame rate of said image data.

27. The mobile electronic device of claim 26, wherein a delay is applied to said output of said stabilized image data.

28. The mobile electronic device of claim 27, wherein said delay is linked to a type of said stabilization correction applied to said image data to achieve said stabilized image data.

29. The mobile electronic device of claim 20, wherein said stabilized image data is written to an output buffer associated with said image sensor, wherein said output buffer is external to said stand-alone electronic image stabilizer.