Creating composite images based on image capture device poses corresponding to captured images

• US 20060072176A1
• Filed: 09/29/2004
• Published: 04/06/2006
• Est. Priority Date: 09/29/2004
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Silverstein, D. Amnon, Deng, Yining
• Granted Patent: US 9,049,396 B2
• Time in Patent Office: Days
• US Class Current: 358/540
• CPC Class Codes: G06T 3/08 Projecting images onto non-planar surfaces, e.g. geodetic screens
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 1/3876 Recombination of partial images to recreate the original image

Abstract

An exemplary method for creating a composite image includes the steps of capturing a plurality of images with an image capture device which includes a sensor, determining an image capture device pose corresponding to each of the images based on data from the sensor, transforming each of the images based on its corresponding pose, projecting the transformed images to a common projection space, and combining the projected images to form a composite image.

Claims

1. A method for creating a composite image based on a plurality of images captured at different poses, comprising:

capturing a plurality of images with an image capture device, said image capture device including a sensor;

determining an image capture device pose corresponding to each of said images based on data from said sensor;

transforming each of said images based on its corresponding pose;

projecting said transformed images to a common projection space; and

combining said projected images to form a composite image.

2. The method of claim 1, wherein said common projection space is a sphere.

3. The method of claim 1, wherein said common projection space is a cylinder.

4. The method of claim 1, further comprising projecting said composite image from said common projection space into a desired viewing plane.

5. The method of claim 4, wherein said transforming and projecting occur in a single step.

6. The method of claim 1, wherein said image capture device pose includes angular orientations of said pose.

7. The method of claim 6, wherein said angular orientations include one or more of roll, pitch and yaw.

8. The method of claim 1, wherein said transforming said image includes rotating said image in the opposite direction of said roll.

9. The method of claim 1, wherein said transforming said image includes rotating said image in the opposite direction of said pitch.

10. The method of claim 1, wherein said combining said projected images to form a composite image includes making translational adjustments for improved matching of corresponding features of overlapping images.

11. The method of claim 1, wherein said sensor includes an accelerometer for measuring the gravitational acceleration vector acting on the image capture device, and said image capture device pose is determined by resolving said gravitational acceleration vector into its components in an image capture device-fixed coordinate system.

12. The method of claim 11, wherein said sensor includes a gyroscopic sensor for measuring a change of acceleration of the image capture device.

13. The method of claim 1, further comprising:

capturing a second plurality of images with a second image capture device, said second image capture device being connected to said image capture device;

determining a pose of said second image capture device based on said pose of said image capture device at each instant when one of said second plurality of images is captured;

transforming each of said second plurality of images based on its corresponding pose; and

projecting said transformed second plurality of images to a common projection space.

14. The method of claim 13, further comprising:

combining said projected second plurality of images to form a second composite image.

15. The method of claim 13, further comprising:

combining said projected second plurality of images with said projected images to form said composite image.

16. A system for creating a composite image based on a plurality of images captured at different poses, comprising:

a lens and imaging array configured to capture a plurality of digital images;

a sensor configured to obtain a plurality of poses of said lens and image array, each said pose corresponding to a respective one of said images;

a computer-readable medium containing logic instructions for using said pose for each said image to transform each of said images;

a computer-readable medium containing logic instructions for projecting said transformed images to a common projection space; and

a computer-readable medium containing logic instructions for combining said projected images to form a composite image.

17. The system of claim 16, further comprising a computer-readable medium containing logic instructions for projecting said composite image from said common projection space into a desired viewing plane.

18. The system of claim 16, wherein said pose includes angular orientations.

19. The system of claim 16, wherein said sensor includes an accelerometer for measuring the gravitational acceleration vector acting on the image capture device, and said pose is determined by resolving said gravitational acceleration vector into its components in said fixed coordinate system.

20. An optical imaging array for capturing a plurality of images at different poses, said plurality of images being combinable into a composite image, comprising:

an optical imaging array configured to capture a plurality of digital images, each image being representable in a fixed coordinate system;

a sensor configured to obtain a plurality of poses of said optical imaging array, each said pose corresponding to a respective one of said images;

an interface for outputting said pose to an image manipulation system configured to transform each said image based on each said corresponding pose, project said transformed images to a common projection space, and combine said projected images to form a composite image.

21. The image capture device of claim 20 where said pose measurement includes angular orientations of said pose.

22. The image capture device of claim 20 where said sensor includes an accelerometer for measuring the gravitational acceleration vector acting on the image capture device, and said pose measurement is determined by resolving said gravitational acceleration vector into its components in said image capture device-fixed coordinate system.

23. The system of claim 20, further comprising:

a second optical imaging array coupled to said optical imaging array, said second optical imaging array being configured to capture a second plurality of digital images; and

a microprocessor being configured to determine a pose of said second image capture device based on said pose of said image capture device at each instant when one of said second plurality of images is captured.

24. The system of claim 23, further comprising:

an interface outputting said pose of said second image capture device to an image manipulation system configured to transform each said second plurality of images based on each said corresponding pose, project said transformed second plurality of images to a common projection space, and combine said projected second plurality of images to form a second composite image.

25. The system of claim 23, further comprising:

an interface outputting said pose of said second image capture device to an image manipulation system configured to transform each said second plurality of images based on each said corresponding pose, project said transformed second plurality of images to a common projection space, and combine said projected second plurality of images to said projected images to form said composite image.

26. A computer-implemented application for creating a composite image based on a plurality of images captured at different poses, comprising:

computer-implemented logic instructions that, when executed on a processor, receive a plurality of digital images captured by an image capture device at different poses, and a plurality of corresponding poses for said images measured by a sensor attached to said image capture device;

computer-implemented logic instructions that, when executed on a processor, transform said images and project said images to a common projection space; and

computer-implemented logic instructions that, when executed on a processor, combine said projected images into a composite image.

27. The application of claim 26 further comprising computer-implemented logic instructions that, when executed on a processor, project said composite image from said common projection space into a desired viewing plane.

28. The application of claim 26 where said sensor includes an accelerometer for measuring the gravitational acceleration vector acting on the image capture device, and said pose is determined by resolving said gravitational acceleration vector into its components in a image capture device-fixed coordinate system.

29. A method for generating a composite image, comprising:

determining multi-dimensional angular orientations of an image capture device from a sensor attached thereto, said angular orientations corresponding to an image captured by said image capture device;

transforming said image based on said angular orientations;

projecting said transformed image to a projection space; and

enabling stitching of said transformed image with one or more other images to generate a composite image.

30. The method of claim 29 where said projecting includes a first projection of said image and said other images to a curved projection space and a second projection of said images to a substantially flat viewing plane for display.

31. The method of claim 29 where said determining said angular orientations from a sensor includes measuring the gravitational acceleration vector acting on the image capture device, and resolving said vector into its components along the axes of said image capture device.

32. A system for creating a composite image, comprising:

an image capture device for capturing an image;

one or more sensors configured to obtain the magnitude and direction of an instantaneous acceleration force acting on said image capture device upon acquiring said image; and

a microcontroller configured to;

receive acceleration data from said sensor;

determine an angular position of said image capture device based on said acceleration data; and

transform said image based on said angular position of said image capture device to enable stitching of said image and one or more other images to create a composite image.

33. A system for transforming a plurality of images captured at different poses into a composite image, comprising:

means for capturing a plurality of digital images by an image capture device, each image being representable in a fixed coordinate system;

means for obtaining a plurality of poses of said image capture device, each said pose corresponding to a respective one of said images;

means for using said pose for each said image to transform each of said images;

means for projecting said transformed images to a common projection space; and

means for combining said projected images to form a composite image.