Using accelerometer information for determining orientation of pictures and video images

• US 9,124,804 B2
• Filed: 03/22/2010
• Issued: 09/01/2015
• Est. Priority Date: 03/22/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kaheel, Ayman, El-Saban, Motaz, Refaat, Mahmoud, El Arabawy, Ahmed, El Baradei, Mostafa
• Primary Examiner(s): VU, NGOC YEN T
• Application Number:
  US12/729,177
• Publication Number:
  US 20110228112A1
• Time in Patent Office: 1,989 Days
• Field of Search: 348/208.2, 348/222.1, 348/231.3
• US Class Current: 1/1
• CPC Class Codes: H04N 1/00127 Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3254 Orientation, e.g. landscape or portrait; Location or order of the image data, e.g. in memory
• CPC Class Codes: H04N 2201/3277 The additional information being stored in the same storage device as the image data
• CPC Class Codes: H04N 2201/3278 Transmission
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure

Abstract

A computing device, such as a mobile device, can capture pictures or video images using a digital camera and obtain associated orientation information using an accelerometer. The orientation information can be used to adjust one or more of the captured pictures or video images to compensate for rotation in one or more planes of rotation. The orientation information can be saved along with the captured pictures or video images. The orientation information can also be transmitted or streamed along with the captured pictures or video images. Image matching operations can be performed using pictures or video images that have been adjusted using orientation information.

Claims

1. A method, implemented at least in part by a computing device, for capturing and processing video data, the method comprising:

for each of a plurality of video images;

obtaining, by the computing device, orientation information of the computing device at a time of capturing the video image, wherein the computing device comprises a three-axis accelerometer, and wherein the orientation information is obtained from the three-axis accelerometer;

adjusting, by the computing device, the captured video image based on the orientation information, wherein the adjusting the captured video image based on the orientation information comprises;

obtaining a reference orientation that is an orientation of an initial video image of the video data; and

adjusting the video image in real-time to return the video image to the reference orientation; and

outputting, by the computing device, for display the adjusted captured video image;

wherein the obtaining, adjusting, and outputting are performed in real-time during recording of the plurality of video images by the computing device.

2. The method of claim 1 wherein the orientation information comprises:

a first value representing acceleration along an x-axis of the computing device;

a second value representing acceleration along a y-axis of the computing device; and

a third value representing acceleration along a z-axis of the computing device.

3. The method of claim 1 wherein the adjusting the captured video image based on the orientation information comprises compensating for a difference between the obtained orientation information for the captured video image and a reference orientation.

4. The method of claim 1 wherein the adjusting the captured video image based on the orientation information comprises:

calculating a rotation angle in a first plane using the orientation information for the captured video image;

determining a difference in rotation angle between the calculated rotation angle in the first plane and a reference rotation angle in the first plane; and

adjusting the captured video image to compensate for the difference in rotation angle.

5. The method of claim 1 wherein the adjusting the captured video image based on the orientation information comprises:

calculating a rotation angle in a first plane using the orientation information for the captured video image;

calculating a rotation angle in a second plane using the orientation information for the captured video image, wherein the first plane is different from the second plane;

determining a difference in rotation angle between the calculated rotation angle in the first plane and a reference rotation angle in the first plane;

determining a difference in rotation angle between the calculated rotation angle in the second plane and a reference rotation angle in the second plane; and

adjusting the captured video image to compensate for the difference in rotation angle in the first plane and the difference in rotation angle in the second plane.

6. The method of claim 1 further comprising:

for each of the plurality of video images;

storing, by the computing device, the adjusted captured video image.

7. The method of claim 6 further comprising:

for each of the plurality of video images;

storing, by the computing device, the orientation information associated with the stored adjusted captured video image, wherein the orientation information is stored as plural values representing acceleration along axes of the computing device or as plural rotation angles for the computing device.

8. The method of claim 1 wherein the reference orientation includes an arbitrary roll value, wherein the plurality of video images are captured subsequent to the initial video image, and wherein the adjusting the video image, for each of the plurality of subsequent video images, comprises:

determining a difference between the arbitrary roll value of the reference orientation and a roll value of the subsequent video image; and

adjusting the subsequent video image in real-time, based on the difference, to match the arbitrary roll value of the reference orientation.

9. A mobile device for capturing and processing video data, the mobile device comprising:

an accelerometer;

storage;

a digital camera; and

a processing unit, wherein the processing is configured for performing operations comprising;

for each of a plurality of video images;

obtaining, by the processing unit from the accelerometer, orientation information of the mobile device that indicates orientation of the mobile device at a time of capturing the video image;

adjusting the captured video image in real-time based on the orientation information, wherein the adjusting the captured video image based on the orientation information comprises;

obtaining a reference orientation that is an orientation of an initial video image of the video data; and

adjusting the video image in real-time to return the video image to the reference orientation;

storing, by the processing unit in the storage, the adjusted captured video image and the orientation information, wherein the orientation information is associated with its respective captured video image;

streaming the obtained orientation information and the adjusted captured video image from the mobile device to a network server.

10. The mobile device of claim 9 wherein the plurality of captured video images are stored in a first data file, and wherein the orientation information is stored in a second data file different than the first data file.

11. The mobile device of claim 9 wherein the accelerometer is a three-axis accelerometer, wherein the plurality of captured video images and the orientation information are stored in the same data file, and wherein the orientation information is stored as plural values representing acceleration along axes of the computing device or as plural rotation angles for the computing device.

12. The mobile device of claim 9 wherein the accelerometer is a three-axis accelerometer, and wherein the orientation information comprises:

a first value representing acceleration along an x-axis of the mobile device;

a second value representing acceleration along a y-axis of the mobile device; and

a third value representing acceleration along a z-axis of the mobile device.

13. One or more computer-readable storage media, wherein the one or more computer-readable storage media are one or more of volatile memory, non-volatile memory, optical storage media, and magnetic storage media, storing computer-executable instructions for causing a computing device to perform a method for capturing and processing video data, the method comprising:

for each of a plurality of video images;

obtaining orientation information of the computing device at a time of capturing the video image, the computing device comprising a three-axis accelerometer, and the orientation information being obtained from the three-axis accelerometer, wherein the obtained orientation information is a reference orientation of an initial video image of the video data;

adjusting the captured video image based on the orientation information to return the captured video image to the reference orientation of the initial video image, the adjusting comprising;

calculating a rotation angle in a first plane using the orientation information for the captured video image;

calculating a rotation angle in a second plane using the orientation information for the captured video image, wherein the first plane is different from the second plane;

determining a difference in rotation angle between the calculated rotation angle in the first plane and a reference rotation angle in the first plane;

determining a difference in rotation angle between the calculated rotation angle in the second plane and a reference rotation angle in the second plane; and

applying an affine transformation to the captured video image using the difference in rotation angle in the first plane and the difference in rotation angle in the second plane; and

outputting for display the transformed captured video image;

the obtaining, adjusting, and outputting being performed in real-time during recording of the plurality of video images by the computing device.

14. The one or more computer-readable storage media of claim 13 wherein the orientation information comprises:

a first value representing acceleration along an x-axis of the computing device;

a second value representing acceleration along a y-axis of the computing device; and

a third value representing acceleration along a z-axis of the computing device.

15. The one or more computer-readable storage media of claim 13 wherein the applying an affine transformation comprises compensating for a difference between the obtained orientation information for the captured video image and the reference orientation.