Methods and apparatuses for determining positions of multi-directional image capture apparatuses

• US 10,565,803 B2
• Filed: 04/09/2018
• Issued: 02/18/2020
• Est. Priority Date: 04/11/2017
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Wang, Tinghuai, You, Yu, Fan, Lixin
• Primary Examiner(s): Hasan, Mainul
• Application Number:
  US15/948,012
• Publication Number:
  US 20180293805A1
• Time in Patent Office: 680 Days
• CPC Class Codes: G01B 11/245 using a plurality of fixed, simultaneously operating transducers G01B11/2408 - G01B11/2425, G01B11/255 take precedence
• CPC Class Codes: G01B 21/22 for measuring angles or tapers; for testing the alignment of axes
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

A method comprising performing image re-projection on each of a plurality of first images of a scene, thereby to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera; processing the plurality of second images based on a previously generated virtual three dimensional model of the scene, thereby to generate respective positions of the virtual cameras associated with the second images; and determining a position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras.

Claims

1. A method comprising:

in response to a movement of a first multi-directional image capture apparatus to a changed position;

performing image re-projection on a plurality of first images of a scene to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of the first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera of a plurality of virtual cameras;

processing the plurality of re-projected second images based on a previously generated virtual three dimensional model of the scene to generate respective positions of the virtual cameras associated with the second images; and

determining a new position of the first multi-directional image capture apparatus based on both the movement of the first multi-directional image capture apparatus and on one or more of the generated respective positions of the virtual cameras, wherein the new position corresponds to the changed position of the first multi-directional image capture apparatus;

wherein determining the new position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras comprises;

determining inliers in the generated respective positions of the virtual cameras; and

determining the new position of the first multi-directional image capture apparatus based on the inliers;

wherein the new position of the first multi-directional image capture apparatus is determined as a position vector of the first multi-directional image capture apparatus in a reference coordinate system, wherein the position vector of the first multi-directional image capture apparatus in the reference coordinate system is determined by taking a difference between a position vector of at least one camera and a position vector of the at least one camera relative to the first multi-directional image capture apparatus.

2. The method of claim 1, further comprising updating the previously generated virtual three dimensional model with the determined new position of the first multi-directional image capture apparatus.

3. The method of claim 1, wherein the previously generated virtual three dimensional model comprises information describing features of the scene.

4. The method of claim 1, wherein the previously generated three dimensional model is generated based at least in part on images previously captured by the first multi-directional image capture apparatus;

or images previously captured by a set of one or more multi-directional image capture apparatuses which does not include the first multi-directional image capture apparatus.

5. The method of claim 1, wherein processing the plurality of re-projected second images based on the previously generated virtual three dimensional model comprises:

determining point correspondences between the re-projected second images and the previously generated virtual three dimensional model; and

performing a Direct Linear Transform based on the determined point correspondences to determine positions of the virtual cameras.

6. The method of claim 1, wherein determining the new position of the first multi-directional image capture apparatus based on the generated respective positions of the virtual cameras comprises:

determining a position of a camera of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras; and

determining the new position of the first multi-directional image capture apparatus based on the determined position of the camera.

7. The method of claim 1, wherein determining the new position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras further comprises:

selecting a virtual camera from among the plurality of virtual cameras based on a smallest re-projection error associated therewith; and

determining the new position of the first multi-directional image capture apparatus based only on the position of the virtual camera selected based on the smallest re-projection error associated therewith.

8. The method of claim 1, wherein the processing of the plurality of re-projected second images further generates respective orientations of the virtual cameras, and the method further comprises:

determining an orientation of the first multi-directional image capture apparatus based on one or more of the generated respective orientations of the virtual cameras.

17. The method of claim 1, wherein the changed position of the first multi-directional image capture apparatus corresponds to an addition of the first multi-directional image capture apparatus to an arrangement of one or more other multi-directional image capture apparatuses.

9. An apparatus comprising:

at least one processor; and

at least one memory including computer program code, which when executed by the at least one processor, causes the apparatus to;

in response to a movement of a first multi-directional image capture apparatus to a changed position;

perform image re-projection on a plurality of first images of a scene to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of the first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera of a plurality of virtual cameras;

process the plurality of re-projected second images based on a previously generated virtual three dimensional model of the scene to generate respective positions of the virtual cameras associated with the second images; and

determine a new position of the first multi-directional image capture apparatus based on both the movement of the first multi-directional image capture apparatus and on one or more of the generated respective positions of the virtual cameras, wherein the new position corresponds to the changed position of the first multi-directional image capture apparatus;

wherein to determine the new position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras, the apparatus is further caused to;

determine inliers in the generated respective positions of the virtual cameras; and

determine the new position of the first multi-directional image capture apparatus based on the inliers;

wherein the new position of the first multi-directional image capture apparatus is determined as a position vector of the first multi-directional image capture apparatus in a reference coordinate system, wherein the position vector of the first multi-directional image capture apparatus in the reference coordinate system is determined by taking a difference between a position vector of at least one camera and a position vector of the at least one camera relative to the first multi-directional image capture apparatus.

10. The apparatus of claim 9, wherein the computer program code, which when executed by the at least one processor, further causes the apparatus to:

update the previously generated virtual three dimensional model with the determined new position of the first multi-directional image capture apparatus.

11. The apparatus of claim 9, wherein the previously generated virtual three dimensional model comprises information describing features of the scene.

12. The apparatus of claim 9, wherein the previously generated three dimensional model is generated based at least in part on images previously captured by the first multi-directional image capture apparatus;

or

images previously captured by a set of one or more multi-directional image capture apparatuses which does not include the first multi-directional image capture apparatus.

13. The apparatus of claim 9, wherein to process the plurality of second images based on the virtual three dimensional model, the apparatus is further caused to:

determine point correspondences between the re-projected second images and the previously generated virtual three dimensional model; and

perform a Direct Linear Transform based on the determined point correspondences to determine positions of the virtual cameras.

14. The apparatus of claim 9, wherein to determine the new position of the first multi-directional image capture apparatus based on the generated positions of the virtual cameras, the apparatus is further caused to:

determine a position of a camera of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras; and

determine the new position of the first multi-directional image capture apparatus based on the determined position of the camera.

15. The apparatus of claim 9, wherein to determine the new position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras, the apparatus is further caused to:

select a virtual camera from among the plurality of virtual cameras based on a smallest re-projection error associated therewith; and

determine the new position of the multi-directional image capture apparatus based only on the position of the virtual camera selected based on the smallest re-projection error associated therewith.

16. A non-transitory computer-readable medium having computer-readable code stored thereon, the computer readable code, when executed by at least one processor, causes an apparatus to:

in response to a movement of a first multi-directional image capture apparatus to a changed position;

performing image re-projection on a plurality of first images of a scene to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of the first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera of a plurality of virtual cameras;

processing the plurality of re-projected second images based on a previously generated virtual three dimensional model of the scene to generate respective positions of the virtual cameras associated with the second images; and

determining a new position of the first multi-directional image capture apparatus based on both the movement of the first multi-directional image capture apparatus and on one or more of the generated respective positions of the virtual cameras, wherein the new position corresponds to the changed position of the first multi-directional image capture apparatus;

wherein determining the new position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras comprises;

determining inliers in the generated respective positions of the virtual cameras; and

determining the new position of the first multi-directional image capture apparatus based on the inliers;

wherein the new position of the first multi-directional image capture apparatus is determined as a position vector of the first multi-directional image capture apparatus in a reference coordinate system, wherein the position vector of the first multi-directional image capture apparatus in the reference coordinate system is determined by taking a difference between a position vector of at least one camera and a position vector of the at least one camera relative to the first multi-directional image capture apparatus.