METHODS AND APPARATUSES FOR DETERMINING POSITIONS OF MULTI-DIRECTIONAL IMAGE CAPTURE APPARATUSES

• US 20180293805A1
• Filed: 04/09/2018
• Published: 10/11/2018
• Est. Priority Date: 04/11/2017
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: WANG, Tinghuai, YOU, Yu, FAN, Lixin
• Granted Patent: US 10,565,803 B2
• Time in Patent Office: Days
• CPC Class Codes: G01B 11/245 using a plurality of fixed, simultaneously operating transducers G01B11/2408 - G01B11/2425, G01B11/255 take precedence
• CPC Class Codes: G01B 21/22 for measuring angles or tapers; for testing the alignment of axes
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

A method comprising performing image re-projection on each of a plurality of first images of a scene, thereby to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera; processing the plurality of second images based on a previously generated virtual three dimensional model of the scene, thereby to generate respective positions of the virtual cameras associated with the second images; and determining a position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras.

Claims

1. A method comprising:

performing image re-projection on a plurality of first images of a scene to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera;

processing the plurality of re-projected second images based on a previously generated virtual three dimensional model of the scene to generate respective positions of the virtual cameras associated with the second images; and

determining a position of the first multi-directional image capture apparatus based on one or more of the generated respective positions of the virtual cameras.

2. The method of claim 1, further comprising updating the previously generated virtual three dimensional model with the determined position of the first multi-directional image capture apparatus.

3. The method of claim 1, wherein the previously generated virtual three dimensional model comprises information describing features of the scene.

4. The method of claim 1, wherein the previously generated three dimensional model is generated based at least in part on images previously captured by the first multi-directional image capture apparatus;

or images previously captured by a set of one or more multi-directional image capture apparatuses which does not include the first multi-directional image capture apparatus.

5. The method of claim 1, wherein processing the plurality of re-projected second images based on the previously generated virtual three dimensional model comprises:

determining point correspondences between the re-projected second images and the previously generated virtual three dimensional model; and

performing a Direct Linear Transform based on the determined point correspondences to determine positions of the virtual cameras.

6. The method of claim 1, wherein determining the position of the first multi-directional image capture apparatus based on the generated respective positions of the virtual cameras comprises:

determining a position of a camera of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras; and

determining the position of the first multi-directional image capture apparatus based on the determined position of the camera.

7. The method of claim 1, wherein determining the position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras comprises:

determining inliers in the generated respective positions of the virtual cameras; and

determining the position of the first multi-directional image capture apparatus based on the inliers.

8. The method of claim 1, wherein determining the position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras further comprises:

determining a virtual camera based on re-projection error associated therewith; and

determining the position of the first multi-directional image capture apparatus based only on the position of the virtual camera determined based on the re-projection error associated therewith.

9. The method of claim 1, wherein the processing of the plurality of re-projected second images further generates respective orientations of the virtual cameras, and the method further comprises:

determining an orientation of the first multi-directional image capture apparatus based on one or more of the generated respective orientations of the virtual cameras.

10. Apparatus comprising:

at least one processor; and

at least one memory including computer program code, which when executed by the at least one processor, causes the apparatus to;

perform image re-projection on a plurality of first images of a scene to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera;

process the plurality of re-projected second images based on a previously generated virtual three dimensional model of the scene to generate respective positions of the virtual cameras associated with the second images; and

determine a position of the first multi-directional image capture apparatus based on one or more of the generated respective positions of the virtual cameras.

11. The apparatus of claim 10, wherein the computer program code, which when executed by the at least one processor, further causes the apparatus to:

update the previously generated virtual three dimensional model with the determined position of the first multi-directional image capture apparatus.

12. The apparatus of claim 10, wherein the previously generated virtual three dimensional model comprises information describing features of the scene.

13. The apparatus of claim 10, wherein the previously generated three dimensional model is generated based at least in part on images previously captured by the first multi-directional image capture apparatus;

or

images previously captured by a set of one or more multi-directional image capture apparatuses which does not include the first multi-directional image capture apparatus.

14. The apparatus of claim 10, wherein to process the plurality of second images based on the virtual three dimensional model, the apparatus is further caused to:

determine point correspondences between the re-projected second images and the previously generated virtual three dimensional model; and

perform a Direct Linear Transform based on the determined point correspondences to determine positions of the virtual cameras.

15. The apparatus of claim 10, wherein to determine the position of the first multi-directional image capture apparatus based on the generated positions of the virtual cameras, the apparatus is further caused to:

determine a position of a camera of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras; and

determine the position of the first multi-directional image capture apparatus based on the determined position of the camera.

16. The apparatus of claim 10, wherein to determine the position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras, the apparatus is further caused to:

determine inliers in the generated respective positions of the virtual cameras; and

determine the position of the first multi-directional image capture apparatus based on the inliers.

17. The apparatus of claim 10, wherein to determine the position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras, the apparatus is further caused to:

determine a virtual camera based on re-projection error associated therewith; and

determine the position of the multi-directional image capture apparatus based only on the position of the virtual camera determined based on the re-projection error associated therewith.

18. The apparatus of claim 10, wherein the processing of the plurality of second images further generates respective orientations of the virtual cameras, and the computer program code, which when executed by the at least one processor, further causes the apparatus to:

determine an orientation of the first multi-directional image capture apparatus based on one or more of the generated respective orientations of the virtual cameras.

19. The apparatus of claim 18, wherein the position of the multi-directional image capture apparatus is determined based on the one or more generated positions of the virtual cameras and the one or more generated orientations of the virtual cameras.

20. A computer-readable medium having computer-readable code stored thereon, the computer readable code, when executed by at least one processor, causes an apparatus to:

performing image re-projection on a plurality of first images of a scene to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera;

processing the plurality of re-projected second images based on a previously generated virtual three dimensional model of the scene to generate respective positions of the virtual cameras associated with the second images; and

determining a position of the first multi-directional image capture apparatus based on one or more of the generated respective positions of the virtual cameras.