IN CAMERA IMPLEMENTATION OF SELECTING AND STITCHING FRAMES FOR PANORAMIC IMAGERY

• US 20130038680A1
• Filed: 01/26/2012
• Published: 02/14/2013
• Est. Priority Date: 04/06/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Ayelet Mashiah
• Inventors: MASHIAH, AYELET
• Granted Patent: US 8,947,502 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen

Abstract

A system, method, and computer program product for selecting qualifying frames from an image sequence for use in subsequent stitching into a composite panoramic image are disclosed. Incoming frames from any source may be cropped and downscaled prior to evaluation against qualifying criteria relating to image overlap and local motion. Qualifying images are saved and/or output. The resulting panoramic image generally uses fewer qualifying images and appears smoother and has fewer artifacts than those of the prior art. The qualifying criterion for image overlap is a predetermined overlap margin or percentage between a current image and a previous image from the sequence. The qualifying criterion for image motion includes a maximum amount of local motion, often due to passing objects. The embodiments may process incoming images in real time or from stored sequences. Problems may trigger user warnings.

Claims

1. A method, comprising:

for at least some chosen frames of an image sequence;

evaluating frame overlap constraints;

evaluating frame motion constraints;

selectively saving a current frame as being a qualifying frame if the frame overlap constraints and the frame local motion constraints are met; and

outputting the qualifying frames.

2. The method of claim 1 wherein the method operates in real time.

3. The method of claim 1 wherein all the frames of an image sequence are processed in order.

4. The method of claim 1 wherein the image sequence is processed during image sequence capture.

5. The method of claim 1 wherein the image sequence is processed after image sequence capture.

6. The method of claim 1 wherein the image sequence comprises a set of frames captured in succession.

7. The method of claim 1 wherein the image sequence is one of an uncompressed video sequence and a compressed video sequence.

8. The method of claim 1 wherein the image sequence is acquired by a digital camera during a pan of a target scene.

9. The method of claim 1, further comprising providing a count of frames required for production of a composite panoramic image.

10. The method of claim 1 wherein the frame overlap constraints comprise a specified directional overlap percentage with respect to a previous qualifying frame.

11. The method of claim 1 wherein the frame motion constraints comprise a maximum allowable local motion within a frame overlap region.

12. The method of claim 1 wherein the frame motion constraints are evaluated by:

performing motion estimation of a current frame with respect to a previous chosen frame;

dividing overlapping frame regions between the current frame and the previous chosen frame into blocks for subsequent analysis;

summing the number of pixels in each block having a difference in pixel illumination value for the current frame and the previous chosen frame exceeding a predefined threshold; and

selectively determining that large local motion violates the frame motion constraints according to the percentage of blocks having pixel sums indicating high illumination value differences.

13. The method of claim 1 wherein a first frame in the image sequence is saved and outputted as a qualifying frame by default.

14. The method of claim 1 further comprising retaining qualifying frames for further use as reference frames during further processing of the image sequence.

15. The method of claim 1 further comprising stitching the outputted frames into a composite panoramic image for at least one of output and storage.

16. The method of claim 1 further comprising downscaling the frames to be processed.

17. The method of claim 1 further comprising cropping the frames to be processed.

18. The method of claim 1 further comprising warning a user if an error condition occurs, including at least one of:

the frame motion constraints are violated, the frame overlap constraints are violated, excess camera tilt occurs in any orientation, and insufficient qualifying frames are saved.

19. The method of claim 18 wherein the frame motion constraints are violated by at least one of:

excess local motion, excess panning speed, insufficient panning speed, and excess hand motion drift in a perpendicular direction to a sweep direction.

20. The method of claim 1, wherein frame overlap constraints for two particular frames of the image sequence are evaluated against a predefined overlap margin, and frame motion constraints are evaluated for the two particular frames and at least one additional frame of the image sequence within the predefined overlap margin.

21. A camera that, for at least some chosen frames of an image sequence:

evaluates frame overlap constraints;

evaluates frame motion constraints;

selectively saves a current frame as being a qualifying frame if the frame overlap constraints and the frame local motion constraints are met; and

outputs the qualifying frames.

22. A computer program product comprising a machine-readable medium tangibly embodying non-transitory program instructions thereon that, when executed by the machine, cause the machine to, for at least some chosen frames of an image sequence:

evaluate frame overlap constraints;

evaluate frame motion constraints;

selectively save a current frame as being a qualifying frame if the frame overlap constraints and the frame local motion constraints are met; and

output the qualifying frames.

23. An integrated circuit, comprising:

for at least some chosen frames of an image sequence;

a first circuit element for evaluating frame overlap constraints;

a second circuit element for evaluating frame motion constraints;

a third circuit element for selectively saving a current frame as being a qualifying frame if the frame overlap constraints and the frame local motion constraints are met; and

a fourth circuit element for outputting the qualifying frames.

24. A system, comprising:

for at least some chosen frames of an image sequence;

means for evaluating frame overlap constraints;

means for evaluating frame motion constraints;

means for selectively saving a current frame as being a qualifying frame if the frame overlap constraints and the frame local motion constraints are met; and

means for outputting the qualifying frames.

25. A method for processing video frames for generation of a panoramic image, the method comprising:

capturing a sequence of video frames;

selecting a first video frame from the sequence of video frames;

storing the first video frame in a first memory location;

selecting a second video frame from the sequence of video frames that succeeds the first video frame and that is within a predefined overlap margin with said first video frame;

storing the second video frame in a second memory location;

estimating local motion between the first selected video frame and the second video frame;

selecting a third video frame from the sequence of video frames that succeeds the second video frame and that is within the predefined overlap margin with said first video frame;

estimating local motion between the third selected video frame and the first video frame;

storing the selected third video frame in the second memory location if the local motion estimated for the selected third video frame is smaller than the local motion of the selected second video frame;

repeating selection of the third video frame until the predefined overlap margin is fully consumed;

stitching the image stored in the first memory location and the image stored in the second memory location into a panoramic image having a minimal distortion caused by overlap margin and local motion therebetween; and

storing the minimally distorted panoramic image in the first memory location for output.