USING ACCELEROMETER INFORMATION FOR DETERMINING ORIENTATION OF PICTURES AND VIDEO IMAGES

• US 20110228112A1
• Filed: 03/22/2010
• Published: 09/22/2011
• Est. Priority Date: 03/22/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Baradei, Mostafa, Refaat, Mahmoud, El-Saban, Motaz, El Arabawy, Ahmed, Kaheel, Ayman
• Granted Patent: US 9,124,804 B2
• Time in Patent Office: Days
• US Class Current: 348/208.4
• CPC Class Codes: H04N 1/00127 Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3254 Orientation, e.g. landscape or portrait; Location or order of the image data, e.g. in memory
• CPC Class Codes: H04N 2201/3277 The additional information being stored in the same storage device as the image data
• CPC Class Codes: H04N 2201/3278 Transmission
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure

Abstract

A computing device, such as a mobile device, can capture pictures or video images using a digital camera and obtain associated orientation information using an accelerometer. The orientation information can be used to adjust one or more of the captured pictures or video images to compensate for rotation in one or more planes of rotation. The orientation information can be saved along with the captured pictures or video images. The orientation information can also be transmitted or streamed along with the captured pictures or video images. Image matching operations can be performed using pictures or video images that have been adjusted using orientation information.

Claims

1. A method, implemented at least in part by a computing device, for capturing and processing video data, the method comprising:

for each of a plurality of video images;

obtaining, by the computing device, orientation information of the computing device at a time of capturing the video image, wherein the computing device comprises a three-axis accelerometer, and wherein the orientation information is obtained from the three-axis accelerometer;

adjusting, by the computing device, the captured video image based on the orientation information; and

outputting, by the computing device, for display the adjusted captured video image.

2. The method of claim 1 wherein the orientation information comprises:

a first value representing acceleration along an x-axis of the computing device;

a second value representing acceleration along a y-axis of the computing device; and

a third value representing acceleration along a z-axis of the computing device.

3. The method of claim 1 wherein the adjusting the captured video image based on the orientation information comprises compensating for a difference between the obtained orientation information for the captured video image and a reference orientation.

4. The method of claim 1 wherein the adjusting the captured video image based on the orientation information comprises:

calculating a rotation angle in a first plane using the orientation information for the captured video image;

determining a difference in rotation angle between the calculated rotation angle in the first plane and a reference rotation angle in the first plane; and

adjusting the captured video image to compensate for the difference in rotation angle.

5. The method of claim 1 wherein the adjusting the captured video image based on the orientation information comprises:

calculating a rotation angle in a first plane using the orientation information for the captured video image;

calculating a rotation angle in a second plane using the orientation information for the captured video image, wherein the first plane is different from the second plane;

determining a difference in rotation angle between the calculated rotation angle in the first plane and a reference rotation angle in the first plane;

determining a difference in rotation angle between the calculated rotation angle in the second plane and a reference rotation angle in the second plane; and

adjusting the captured video image to compensate for the difference in rotation angle in the first plane and the difference in rotation angle in the second plane.

6. The method of claim 1 wherein, for each of the plurality of video images, the obtaining, adjusting, and displaying are performed in real-time during recording of the plurality of video images by the computing device.

7. The method of claim 1 further comprising:

for each of the plurality of video images;

storing, by the computing device, the adjusted captured video image.

8. The method of claim 7 further comprising:

for each of the plurality of video images;

storing, by the computing device, the orientation information associated with the stored adjusted captured video image, wherein the orientation information is stored as plural values representing acceleration along axes of the computing device or as plural rotation angles for the computing device.

9. A mobile device for capturing and processing video data, the mobile device comprising:

an accelerometer;

storage;

a digital camera; and

a processing unit, wherein the processing is configured for performing operations comprising;

for each of a plurality of video images;

obtaining, by the processing unit from the accelerometer, orientation information of the mobile device that indicates orientation of the mobile device at a time of capturing the video image;

storing, by the processing unit in the storage, the captured video image and the orientation information, wherein the orientation information is associated with its respective captured video image.

10. The mobile device of claim 9 wherein the plurality of captured video images are stored in a first data file, and wherein the orientation information is stored in a second data file different than the first data file.

11. The mobile device of claim 9 wherein the accelerometer is a three-axis accelerometer, wherein the plurality of captured video images and the orientation information are stored in the same data file, and wherein the orientation information is stored as plural values representing acceleration along axes of the computing device or as plural rotation angles for the computing device.

12. The mobile device of claim 9 further comprising:

for each of the plurality of video images;

streaming the obtained orientation information and the captured video image from the mobile device to a network server.

13. The mobile device of claim 9 further comprising:

for each of the plurality of video images;

adjusting, by the mobile device, the captured video image based on the orientation information; and

streaming the adjusted captured video image from the mobile device to a network server.

14. The mobile device of claim 9 wherein the accelerometer is a three-axis accelerometer, and wherein the orientation information comprises:

a first value representing acceleration along an x-axis of the mobile device;

a second value representing acceleration along a y-axis of the mobile device; and

a third value representing acceleration along a z-axis of the mobile device.

15. A method, implemented at least in part by a computing device, for matching image data, the method comprising:

obtaining image data and associated orientation information for a first captured image, wherein the orientation information for the first captured image comprises three-axis accelerometer information obtained for a first time of capture of the first image by a digital camera;

obtaining image data and associated orientation information for a second captured image, wherein the orientation information for the second captured image comprises three-axis accelerometer information obtained for a second time of capture of the second image by the digital camera;

adjusting orientation of at least one of the first and second captured images, using the orientation information associated with the first and second captured images, to compensate for differences in orientation of the digital camera along at least a first plane of rotation; and

matching one or more corresponding interest points in the first captured image and the second captured image as adjusted to compensate for differences in orientation of the digital camera along the at least first plane of rotation;

wherein the first and second captured images are one of first and second still pictures, and first and second video images.

16. The method of claim 15 further comprising:

adjusting orientation of at least one of the first and second captured images, using the orientation information associated with the first and second captured images, to compensate for differences in orientation of the digital camera along a second different plane of rotation.

17. The method of claim 15 wherein the matching the one or more corresponding interest points is performed using relatively simple pixel comparison operations.

18. The method of claim 15 further comprising:

stitching the first and second captured images, as adjusted to compensate for differences in orientation of the digital camera along the at least first plane of rotation, to form a single stitched image, wherein the stitching is performed, at least in part, based on the matched one or more corresponding interest points.

19. The method of claim 15 wherein the orientation information for the first and second captured images comprises:

a first value representing acceleration along an x-axis;

a second value representing acceleration along a y-axis; and

a third value representing acceleration along a z-axis.

20. The method of claim 15 further comprising:

outputting a result of the matching, comprising an indication of the matched one or more corresponding interest points.