NAVIGATION ASSISTED MOSAIC PHOTOGRAPHY

• US 20090059018A1
• Filed: 09/05/2007
• Published: 03/05/2009
• Est. Priority Date: 09/05/2007
• Status: Abandoned Application

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Micron Technology, Inc.
• Inventors: Brosnan, Michael John
• Application Number:
  US11/850,135
• Publication Number:
  US 20090059018A1
• Time in Patent Office: Days
• US Class Current: 348/218.100
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

An imaging apparatus for producing a mosaic image of a scene, including: an imager operable to capture a plurality of images of the scene; a motion sensor coupled to the imager; a transformation processor electrically coupled to the imager and the motion sensor and a mosaic processor electrically coupled to the transformation processor. The motion sensor is adapted to determine a pitch parameter and a yaw parameter of the imager associated with each captured image of the scene. The transformation processor is adapted to transform each captured image into a mosaic coordinate system using the associated pitch parameter and yaw parameter of the imager. The mosaic processor is adapted to produce a mosaic image of the scene from the transformed images.

Claims

1. A method for producing a mosaic image of a scene from a plurality of images of the scene captured by an imager, comprising the steps of:

storing the plurality of images of the scene, each image including a plurality of pixels;

concurrently with the storage of each image, storing an associated pitch parameter of the imager and an associated yaw parameter of the imager;

transforming each stored image by assigning each pixel of the stored image a set of coordinates in a mosaic coordinate system using the associated pitch parameter and yaw parameter of the imager; and

for each set of coordinates in the mosaic coordinate system, performing one of;

selecting one assigned pixel of the plurality of transformed images as a mosaic pixel having that set of mosaic coordinates;

or

blending all assigned pixels of the plurality of transformed images to determine the mosaic pixel having that set of mosaic coordinates.

2. A method according to claim 1, further comprising the step of:

storing the plurality of mosaic pixels as the mosaic image.

3. A method according to claim 1, further comprising, concurrently with the storage of each image, the steps of:

storing two motion sensor images of a section of the scene;

determining a motion vector for the section of the scene using the two motion sensor images; and

determining the associated pitch parameter and yaw parameter of the imager using the motion vector.

4. A method according to claim 3, wherein the motion vector for the section of the scene is determined by correlating the two motion sensor images.

5. A method according to claim 3, wherein the motion vector for the section of the scene is determined by:

subtracting one motion sensor image from the other motion sensor image to generate a motion sensor difference image;

correlating one of the two motion sensor images to the motion sensor difference image.

6. A method according to claim 1, further comprising, concurrently with the storage of each image, determining the associated pitch parameter and yaw parameter of the imager using a gyroscopic sensor in the imager.

7. A method according to claim 1, wherein each image is transformed into the mosaic coordinate system using:

the associated pitch parameter of the imager to determine a Y-axis translation of the image; and

the associated yaw parameter of the imager to determine an X-axis translation of the image.

8. A method according to claim 1, wherein:

transforming each image into the mosaic coordinate system includes undistorting each image using lens parameters of the imager associated with that image; and

the lens parameters include focal length and field of view.

9. A method according to claim 1,

further comprising, concurrently with the storage of each image, storing an associated roll parameter of the imager;

wherein each image is transformed into the mosaic coordinate system using the associated pitch parameter, yaw parameter, and roll parameter of the imager.

10. A method according to claim 9, further comprising, concurrently with the storage of each image, the steps of:

storing two motion sensor images of a first section of the scene;

storing two motion sensor images of a second section of the scene that differs from the first section of the scene;

determining a first motion vector for the first section of the scene using the two motion sensor images of the first section of the scene;

determining a second motion vector for the second section of the scene using the two motion sensor images of the second section of the scene; and

determining the associated pitch parameter, roll parameter, and yaw parameter of the imager using the first motion vector for the first section of the scene and second motion vector for the second section of the scene.

11. A method according to claim 9, further comprising, concurrently with the storage of each image, the steps of:

storing two motion sensor images of each of a plurality of sections of the scene, each section of the scene differing from other sections in the plurality of sections of the scene;

determining a motion vector for each section of the scene using the two motion sensor images of that section of the scene; and

determining the associated pitch parameter, roll parameter, and yaw parameter of the imager using the plurality of motion vectors for the plurality of sections of the scene.

12. A method according to claim 9, further comprising, concurrently with the storage of each image, determining the associated pitch parameter, yaw parameter, and roll parameter of the imager using a gyroscopic sensor in the imager.

13. A method according to claim 9, wherein each image is transformed to the mosaic coordinate system using:

the associated roll parameter of the imager to determine;

a rotation of the image; and

an X-axis and a Y-axis of the image;

the associated pitch parameter of the imager to determine a Y-axis translation of the image; and

the associated yaw parameter of the imager to determine an X-axis translation of the image.

14. A method according to claim 1, wherein:

the plurality of images of the scene have a hierarchy;

for each set of coordinates in the mosaic coordinate system, among all of the assigned pixels, the assigned pixel of the image that is highest in the hierarchy is selected as the mosaic pixel having that set of mosaic coordinates.

15. A method according to claim 1, wherein, for each set of coordinates in the mosaic coordinate system, among all of the assigned pixels, the assigned pixel that is closest to a center of its transformed image is selected as the mosaic pixel having that set of mosaic coordinates.

16. A method according to claim 1, wherein, for each set of coordinates in the mosaic coordinate system, the assigned pixels of the plurality of transformed images are blended to determine the mosaic pixel having that set of mosaic coordinates by averaging pixel values of the assigned pixels.

17. A method according to claim 16, further comprising, cropping the plurality of mosaic pixels to form a contiguous rectangular of mosaic pixels as the mosaic image.

18. A method for producing a mosaic image of a scene from a sequence of images of the scene captured by imager, comprising the steps of:

storing a starting image of the sequence of images as the mosaic image, including a plurality of pixels assigned to sets of coordinates in a mosaic coordinate system; and

for each remaining image of the sequence of images;

storing the image of the scene;

storing a pitch parameter and a yaw parameter of the imager that are associated with the image, the pitch parameter and the yaw parameter measured relative to a position of the imager associated with the starting image;

transforming the image by assigning each pixel of the image a set of coordinates in the mosaic coordinate system using the associated pitch parameter and yaw parameter of the imager;

updating the mosaic image by combining the mosaic image with the transformed image; and

storing the updated mosaic image.

19. A method according to claim 18, further comprising the step of:

storing the updated mosaic image.

20. A method according to claim 18, wherein the sequence of images of the scene includes a sequence of video frames.

21. A method according to claim 18,

further comprising, for each remaining image of the sequence of images, storing an associated roll parameter of the imager, the associated roll parameter measured relative to the position of the imager associated with the starting image;

wherein the image is transformed to the mosaic coordinate system using the associated pitch parameter, yaw parameter, and roll parameter of the imager.

22. A method according to claim 18, wherein updating the mosaic image includes:

for every set of mosaic coordinates that includes a previously assigned pixel in the mosaic image, selecting the previously assigned pixel of the mosaic image as the mosaic pixel having that set of mosaic coordinates; and

for every set of mosaic coordinates that includes an assigned pixel in the transformed image and does not include a previously assigned pixel in the mosaic image, selecting the assigned pixel of the transformed image as the mosaic pixel having that set of mosaic coordinates.

23. A method according to claim 18, wherein updating the mosaic image includes:

for every set of mosaic coordinates that includes an assigned pixel in the transformed image, selecting the assigned pixel of the transformed image as the mosaic pixel having that set of mosaic coordinates; and

for every set of mosaic coordinates that includes a previously assigned pixel in the mosaic image and does not include an assigned pixel in the transformed image, selecting the previously assigned pixel of the mosaic image as the mosaic pixel having that set of mosaic coordinates.

24. A method according to claim 18, wherein updating the mosaic image includes:

for every set of mosaic coordinates that includes an assigned pixel in the transformed image and does not include a previously assigned pixel in the mosaic image, selecting the assigned pixel of the transformed image as the mosaic pixel having that set of mosaic coordinates;

for every set of mosaic coordinates that includes a previously assigned pixel in the mosaic image and does not include an assigned pixel in the transformed image, selecting the previously assigned pixel of the mosaic image as the mosaic pixel having that set of mosaic coordinates; and

for every set of mosaic coordinates that includes a previously assigned pixel in the mosaic image and an assigned pixel in the transformed image, blending the previously assigned pixel of the mosaic image with the assigned pixel in the transformed image to determine the mosaic pixel having that set of mosaic coordinates.

25. An imaging apparatus for producing a mosaic image of a scene, comprising:

an imager operable to capture a plurality of images of the scene;

a motion sensor coupled to the imager, the motion sensor adapted to determine a pitch parameter and a yaw parameter of the imager associated with each captured image of the scene;

a transformation processor electrically coupled to the imager and the motion sensor, the transformation processor adapted to transform each captured image into a mosaic coordinate system using the associated pitch parameter and yaw parameter of the imager; and

a mosaic processor electrically coupled to the transformation processor, the mosaic processor adapted to produce the mosaic image of the scene from the plurality of transformed images.

26. An imaging apparatus according to claim 25, wherein the imager includes at least one of:

a still camera;

or a video camera.

27. An imaging apparatus according to claim 25, wherein the motion sensor includes at least one of:

an optical motion sensor;

or a gyroscopic motion sensor.

28. An imaging apparatus according to claim 25, wherein the motion sensor includes:

a plurality of sensor devices each operable to capture one or more sensor images;

a lenslet array comprised of a plurality of lenses, each lens is positioned in an imaging path of a respective sensor device such that the one or more sensor images captured by one of the sensor devices in the plurality of sensor devices differs from the sensor images captured by the other sensor devices; and

a correlation processor electrically coupled to the plurality of sensor devices, the correlation processor adapted to determine the associated pitch parameter and yaw parameter of the imager using the one or more sensor images of each sensor device.

29. An imaging apparatus according to claim 25, wherein:

the motion sensor is further adapted to determine a roll parameter of the imager associated with each captured image of the scene; and

the transformation processor is further adapted to transform each captured image into a mosaic coordinate system using the associated pitch parameter, yaw parameter, and roll parameter of the imager.

30. An imaging apparatus according to claim 25, wherein the transformation processor includes at least one of:

an application specific integrated circuit (ASIC);

special purpose processor circuitry;

or

a general purpose processor programmed to transform each captured image into the mosaic coordinate system using the associated pitch parameter and yaw parameter of the imager.

31. An imaging apparatus according to claim 25, wherein the mosaic processor includes at least one of:

an application specific integrated circuit (ASIC);

special purpose processor circuitry;

or

a general purpose processor programmed to produce the mosaic image of the scene from the plurality of transformed images.

32. An imaging apparatus according to claim 25, further comprising a cropping processor electrically coupled to the mosaic processor, the cropping processor adapted to crop the mosaic image produced by the mosaic processor to form a contiguous rectangular of mosaic pixels as the mosaic image of the scene.

33. An imaging apparatus according to claim 25, further comprising image memory electrically coupled to the mosaic processor to store the mosaic image of the scene.

34. An imaging apparatus according to claim 25, further comprising an image display electrically coupled to the mosaic processor to display the mosaic image of the scene.

35. An imaging apparatus according to claim 25, wherein the imaging apparatus is integrated into one of:

a camcorder;

a digital camera;

a portable computer;

or a cell phone.