Rectified stereoscopic 3D panoramic picture

• US 9,602,708 B2
• Filed: 10/23/2012
• Issued: 03/21/2017
• Est. Priority Date: 11/15/2011
• Status: Active Grant

• Current Assignee: Telefonaktiebolaget LM Ericsson
• Original Assignee: Optis Circuit Technology, LLC (Brevet Capital)
• Inventors: Drouot, Antoine
• Primary Examiner(s): BAILEY, FREDERICK D
• Application Number:
  US14/356,698
• Publication Number:
  US 20140327745A1
• Time in Patent Office: 1,610 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 9/76 for mixing of colour signals H04N9/75 takes precedence

Abstract

A method of generating first and second panoramic pictures based on a series of frames acquired by a frame acquisition device is provided. While the device is panned in a main direction, a global motion value reflecting a displacement in the main direction between consecutive frames and a perspective transform reflecting a perspective change between the frames are computed. For each frame, a first area of the frame is determined for the first panoramic picture and a second area of the frame, distinct from the first area, is determined for the second panoramic picture. An adapted first area is determined by applying an inverse of the global transform to the first area of the current frame, the adapted first area being included into the first panoramic picture and the second area of the current frame being included into the second panoramic picture.

Claims

1. A method of generating at least a first panoramic picture and a second panoramic picture based on a series of frames acquired by one frame acquisition device while the device is panned in a main direction, the method comprising:

computing via the device, a global motion value and a perspective transform for each pair of consecutive frames of the series, said global motion value reflecting a displacement in the main direction between the frames of the pair of frames and said perspective transform reflecting a perspective change between the frames of the pair of frames;

for each frame of the series, determining via the device, a first area of the frame for the first panoramic picture and a second area of the frame, distinct from the first area, for the second panoramic picture; and

when a current frame of the series of frames is acquired;

determining via the device, a first frame among the already acquired frames of the series of frames, which has a second area that shares at least a common area with the first area of the current frame,

calculating via the device, a global transform based on the perspective transforms of the pairs of consecutive frames that have been acquired between the first frame and the current frame, and

determining via the device, an adapted first area by applying an inverse of the global transform to the first area of the current frame, said adapted first area being included into the first panoramic picture and the second area of the current frame being included into the second panoramic picture,

wherein the first and second areas are respectively a first strip and a second strip that are extending in a direction perpendicular to the main direction, centers of said first and second strips being located at constant positions in the frames and being separated by a constant distance from one frame to the other,

wherein the method further comprises

before determining the first frame, comparing a sum of the global motion values of the pairs of consecutive frames of the series of frames with said constant distance,

wherein the step of determining the first frame is performed only if the sum of the global motion values is greater than the constant distance.

2. The method of claim 1, wherein

if the sum of the global motion values is lower than the constant distance, then the first strip of the current frame is included into the first panoramic picture and the second strip of the current frame is included into the second panoramic picture, and

the method further comprises acquiring at least one new frame, said new frame being added to the series to form a new series of frames, and calculating a new global motion value between the current frame and the new frame to repeat the step of comparing a sum of the global motion values for each pair of consecutive frames of the new series with the constant distance.

3. The method according to claim 1, wherein a width of the first and second strips of a given frame is proportional to the global motion value of the pair of frames comprised of said given frame and the previous frame acquired before the given frame.

4. The method according to claim 1, wherein the first frame is determined as being a frame of index p for which an absolute value of following expression is minimized:

(Σ

<sub>iε

[p+1;

k]</sub>GMV_i)−

d

where i is an index for each one of the already acquired frames, i varying between p+1 and k, with k being an index of the current frame, GMV_i is the global motion value of the pair of frames comprised of the frames of indices i−

1 and l, respectively, d is the constant distance, and p is between 0 that is an index of a first acquired frame, and k that is the index of the current frame.

5. The method according to claim 1, wherein the perspective transforms are homographies and the global transform is a composition of the perspective transforms of the pairs of consecutive frames that have been acquired between the first frame and the current frame.

6. A non-transitory computer readable medium having stored thereon computer program instructions loadable into a computing device, which when the program instructions loaded and executed performs a method according to claim 1.

7. A device for generating at least a first panoramic picture and a second panoramic picture based on a series of frames, said device comprising:

an input operative to acquire said series of frames while said device is panned in a main direction; and

a computer coupled to the input and being operative to;

compute a global motion value and a perspective transform for each pair of consecutive frames of the series, said global motion value reflecting a displacement in the main direction between the frames of the pair of frames and said perspective transform reflecting a perspective change between the frames of the pair of frames;

determine, for each frame of the series, a first area of the frame for the first panoramic picture and a second area of the frame, distinct from the first area, for the second panoramic picture;

determine, upon acquisition of a current frame of the series of frames, a first frame among the already acquired frames of the series of frames, which has a second area that shares at least a common area with the first area of the current frame;

calculate a global transform based on the perspective transforms of the pairs of consecutive frames that have been acquired between the first frame and the current frame; and

determine an adapted first area by applying an inverse of the global transform to the first area of the current frame, said adapted first area being included into the first panoramic picture and the second area of the current frame being included into the second panoramic picture,

wherein the first and second areas are respectively a first strip and a second strip that are extending in a direction perpendicular to the main direction, centers of said first and second strips being located at constant positions in the frames and being separated by a constant distance from one frame to the other,

wherein the computer is further operative to

compare a sum of the global motion values of the pairs of consecutive frames of the series of frames with said constant distance, wherein the computer determines the first frame if the sum of the global motion values is greater than the constant distance.

8. The device according to claim 7, wherein if the sum of the global motion values is lower than the constant distance, the computer includes the first strip of the current frame into the first panoramic picture and the second strip of the current frame into the second panoramic picture, and wherein the device is arranged to acquire at least one new frame, said new frame being added to the series to form a new series of frames, and to calculate a new global motion value between the current frame and the new frame to repeat the step of comparing a sum of the global motion values for each pair of consecutive frames of the new series with the constant distance.

9. The device according to claim 7, wherein the device is arranged to determine a width of the first and second strips of a given frame, which is proportional to the global motion value of the pair of frames comprised of said given frame and the previous frame acquired before the given frame.

10. The device according to claim 7, wherein the computer determines the first frame as being a frame of index p for which an absolute value of following expression is minimized:

(Σ

<sub>iε

[p+1;

k]</sub>GMV_i)−

d

where i is an index for each one of the already acquired frames, i varying between p+1 and k, with k being an index of the current frame, GMVi is the global motion value of the pair of frames comprised of the frames of indices i−

1 and l, respectively, d is the constant distance, and p is between 1 that is an index of a first acquired frame, and k that is the index of the current frame.

11. The device according to claim 7, wherein the perspective transforms are homographies and the computer calculates the global transform as a composition of the perspective transforms of the pairs of consecutive frames that have been acquired between the first frame and the current frame.