Stereoscopic Imaging Using Mobile Computing Devices Having Front-Facing And Rear-Facing Cameras

• US 20170324941A1
• Filed: 04/26/2017
• Published: 11/09/2017
• Est. Priority Date: 05/04/2016
• Status: Active Grant

• Current Assignee: InsideMaps, Inc.
• Original Assignee: InsideMaps, Inc.
• Inventors: Birkler, Paul Joergen
• Granted Patent: US 10,554,896 B2
• Time in Patent Office: Days
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Stereoscopic imaging is performed using a mobile computing device having front- and rear-facing cameras and mounted on a rotational mechanism. The cameras and rotational mechanism are controlled, e.g., by a downloaded application, to capture images of a room or other environment using both the front- and rear-facing cameras at different pan angle and tilt angle combinations. First and second images of a portion of the room or other environment, captured using the front- and rear-facing cameras, are selected for inclusion in a stereo image pair. Obtained images and corresponding metadata are transferred to a remote system. A structure from motion pipeline is used to generate a three-dimensional model of the room or other environment. Data that enables the mobile computing device to display a three-dimensional model of the room or other environment is received from the remote system and used to display the three-dimensional model.

Claims

1. For use with a mobile computing device, a method for obtaining stereo image pairs that can be used for generating three-dimensional models of a room or other environment or producing virtual reality (VR) content, wherein the mobile computing device includes a front-facing camera and a back-facing camera and is mounted on a rotational mechanism such that the front- and rear-facing cameras of the mobile computing device are each offset from a center or rotation of the rotational mechanism, the method comprising:

controlling the cameras of the mobile computing device and controlling the rotational mechanism to capture images of the room or other environment using both the front- and rear-facing cameras at each of a plurality of different pan angle and tilt angle combinations, such that images corresponding to different tilt angles are obtained by each of the front- and rear-facing cameras of the mobile computing device at each of a plurality of different pan angles.

2. The method of claim 1, further comprising:

selecting for inclusion in a stereo image pair a first image of a portion of the room or other environment captured using the front-facing camera and a second image of the portion of the room or other environment captured using the rear-facing camera, wherein the first and second images of each stereo image pair are captured at different times relative to one another.

3. The method of claim 1, further comprising:

using a structure from motion pipeline to generate a three-dimensional model of the room or other environment for which the images are captured; and

displaying the three-dimensional model of the room or other environment.

4. The method of claim 3, wherein the structure from motion pipeline includes a bundle adjustment stage, and wherein the method further comprises specifying constraints for the bundle adjustment stage based on one or more of the following:

geometrical configuration information associated with the mobile computing device;

internal configuration information associated with the mobile computing device;

or

geometrical configuration information associated with the rotational mechanism.

5. The method of claim 1, further comprising:

transferring the obtained images and corresponding metadata from the mobile computing device to a remote system that is configured to generate three-dimensional models of rooms or other environments based on images;

receiving, from the remote system, data that enables the mobile computing device to display a three-dimensional model of the room or other environment to a user; and

displaying the three-dimensional model of the room or other environment, the displaying performed using the data received from the remote system.

6. The method of claim 5, wherein the metadata corresponding to each of the obtained images comprises one or more of the following:

a pan angle and a tilt angle of the mobile computing device when the image was obtained;

a pan angle and a tilt angle of the rotational mechanism when the image was obtained;

a time that the image was obtained;

accelerometer sensor data when the image was obtained;

magnetometer sensor data when the image was obtained;

gyroscope sensor data when the image was obtained.

7. The method of claim 6, wherein metadata that is transferred from the mobile computing device to the remote system also includes data that specifies which two obtained images are included in each stereo image pair.

8. The method of claim 1, wherein a plurality of images are captured at each pan angle and tilt angle combination, with each of the images captured at a same pan angle and tilt angle combination using a different exposure setting.

9. The method of claim 1, wherein the mobile computing device comprises a smartphone or a tablet computing device, and wherein the controlling of the cameras and the controlling the rotational mechanism are performed by an application downloaded to the smartphone or tablet computing device.

10. The method of claim 1, wherein:

multiple images are obtained by each of the front- and rear-facing cameras of the mobile computing device at each of a plurality of different pan angles; and

at each of the plurality of different pan angles, images corresponding to different tilt angles are obtained by each of the front- and rear-facing cameras of the mobile computing device.

11. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors of a mobile computing device to perform a method for obtaining stereo image pairs that can be used for generating three-dimensional models of a room or other environment or producing virtual reality (VR) content, wherein the mobile computing device includes a front-facing camera and a back-facing camera and is mounted on a rotational mechanism such that the front- and rear-facing cameras of the mobile computing device are each offset from a center or rotation of the rotational mechanism, the method comprising:

controlling the cameras of the mobile computing device and controlling the rotational mechanism to capture images of the room or other environment using both the front- and rear-facing cameras at each of a plurality of different pan angle and tilt angle combinations.

12. The one or more processor readable storage devices of claim 11, wherein the method further comprises:

selecting for inclusion in a stereo image pair a first image of a portion of the room or other environment captured using the front-facing camera and a second image of the portion of the room or other environment captured using the rear-facing camera;

wherein the first and second images of each stereo image pair are captured at different times relative to one another.

13. The one or more processor readable storage devices of claim 11, wherein the method further comprises:

using a structure from motion pipeline to generate a three-dimensional model of the room or other environment for which the images are captured;

wherein the structure from motion pipeline includes a bundle adjustment stage, and wherein the method further comprises specifying constraints for the bundle adjustment stage based on one or more of the following;

(a) geometrical configuration information associated with the mobile computing device;

(b) internal configuration information associated with the mobile computing device;

or (c) geometrical configuration information associated with the rotational mechanism; and

displaying the three-dimensional model of the room or other environment.

14. The one or more processor readable storage devices of claim 11, wherein the method further comprises:

transferring the obtained images and corresponding metadata from the mobile computing device to a remote system that is configured to generate three-dimensional models of rooms or other environments based on images;

receiving, from the remote system, data that enables the mobile computing device to display a three-dimensional model of the room or other environment to a user; and

displaying the three-dimensional model of the room or other environment, the displaying performed using the data received from the remote system.

15. The one or more processor readable storage devices of claim 14, wherein the metadata corresponding to each of the obtained images comprises one or more of the following:

a pan angle and a tilt angle of the mobile computing device when the image was obtained;

a pan angle and a tilt angle of the rotational mechanism when the image was obtained;

a time that the image was obtained;

accelerometer sensor data when the image was obtained;

magnetometer sensor data when the image was obtained;

or

gyroscope sensor data when the image was obtained.

16. The one or more processor readable storage devices of claim 15, wherein metadata that is transferred from the mobile computing device to the remote system also includes data that specifies which two obtained images are included in each stereo image pair.

17. A mobile computing device, comprising:

a display;

a front-facing camera;

a back-facing camera; and

one or more processors configured to control the cameras of the mobile computing device and control a rotational mechanism to which the mobile computing device is mounted to capture images of a room or other environment using both the front- and rear-facing cameras at each of a plurality of different pan angle and tilt angle combinations, such that images corresponding to different tilt angles are obtained by each of the front- and rear-facing cameras of the mobile computing device at each of a plurality of different pan angles.

18. The mobile computing device of claim 17, wherein the one or more processors are also configured to:

select for inclusion in a stereo image pair a first image of a portion of the room or other environment captured using the front-facing camera and a second image of the portion of the room or other environment captured using the rear-facing camera, wherein the first and second images of each stereo image pair are captured at different times relative to one another;

use stereo image pairs to generate a three-dimensional model of the room or other environment for which the images are captured; and

cause the three-dimensional model of the room or other environment to be displayed on the display.

19. The mobile computing device of claim 17, wherein the one or more processor are also configured to:

transfer the obtained images and corresponding metadata from the mobile computing device to a remote system that is configured to generate 3D models of rooms or other environments based on images; and

receive, from the remote system, data that enables the mobile computing device to display a three-dimensional model of the room or other environment to a user.

20. The mobile computing device of claim 19, wherein the one or more processor are also configured to cause the three-dimensional model of the room or other environment to be displayed on the display using the data received from the remote system.