Mobile device configured to compute 3D models based on motion sensor data

• US 9,639,959 B2
• Filed: 11/09/2012
• Issued: 05/02/2017
• Est. Priority Date: 01/26/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Qi, Yingyong, Yang, Ruiduo, Sundaresan, Sairam
• Primary Examiner(s): Huang, Frank
• Application Number:
  US13/673,681
• Publication Number:
  US 20130194392A1
• Time in Patent Office: 1,635 Days
• Field of Search: 348 50, 37524026, 3752401
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/207 using a single 2D image sensor

Abstract

A particular method includes determining, based on data received from at least one motion sensor, a movement of a mobile device from a first position to a second position. The method also includes computing a three-dimensional (3D) model of an object based on a first image of the object corresponding to a first view of the object from the first position of the mobile device, a second image of the object corresponding to a second view of the object from the second position of the mobile device, and the movement of the mobile device.

Claims

1. A method comprising:

determining, based on data output by at least one motion sensor of a mobile device, a movement of the mobile device from a reference position to a second position; and

computing, using a bundle adjustment technique, at least a first portion of a three-dimensional (3D) model of an object based on a first image corresponding to a first view of the object from the reference position, a second image corresponding to a second view of the object from the second position, and the movement of the mobile device, wherein the bundle adjustment technique includes estimating a second projection matrix associated with the second image based at least on a first projection matrix associated with the first image and a relative pose matrix based on the movement of the mobile device from the reference position to the second position.

2. The method of claim 1, further comprising:

capturing, at a camera of the mobile device, the first image while the mobile device is at the reference position;

capturing, at the camera, the second image of the object while the mobile device is at the second position; and

displaying an image corresponding to the 3D model, wherein the first image and the second image are video frames associated with a stored video file.

3. The method of claim 1, wherein the first image and the second image are video frames associated with a video frame sequence, and further comprising accessing the video frames prior to storing the video frame sequence as a video file.

4. The method of claim 1, further comprising computing a second portion of the 3D model of the object based on a second movement of the mobile device from the second position to a third position, and wherein an image corresponding to the 3D model is displayed at the mobile device while the mobile device is moved from the second position to the third position.

5. The method of claim 1, wherein the data output by the at least one motion sensor indicates rotational motion associated with the mobile device, translational motion associated with the mobile device, or a combination thereof, and wherein the movement of the mobile device includes a rotation of the mobile device, a translation of the mobile device, or a combination thereof.

6. The method of claim 1, further comprising computing a second portion of the 3D model, wherein computing the second portion of the 3D model comprises:

projecting into a 3D space first rays from the reference position through pixels of the first image corresponding to the object;

projecting into the 3D space second rays from the second position through pixels of the second image corresponding to the object;

identifying intersections of the first rays and the second rays;

selecting a shortest intersection of the intersections; and

generating the second portion of the 3D model using the shortest intersection.

7. The method of claim 1, wherein computing the first portion of the 3D model of the object further comprises determining feature points of the object based on a function of a first estimated reprojected image point, a second estimated reprojected image point, and the data output by the at least one motion sensor, and wherein computing the first portion of the 3D model is based on 3D spatial positions associated with the feature points.

8. The method of claim 1, wherein computing the first portion of the 3D model of the object further comprises:

determining a first estimated reprojected image point of the object based on a product of a first estimate of the reference position and one or more feature points of the object;

determining a second estimated reprojected image point of the object based on a product of a second estimate of the second position and the one or more feature points of the object;

determining a square of a first Euclidean distance between the first estimated reprojected image point and coordinates of the object in the first image;

determining a square of a second Euclidean distance between the second estimated reprojected image point and coordinates of the object in the second image; and

modifying an estimate of the 3D model to minimize a sum of the square of the first Euclidean distance and the square of the second Euclidean distance,

wherein the second estimate of the second position is generated based on the first estimate of the reference position and the data output by the at least one motion sensor.

9. The method of claim 1, wherein the bundle adjustment technique is performed to determine a maximum likelihood solution of:

20. The method of claim 1, wherein the second position is distinct from the reference position.

21. The method of claim 1, further comprising:

determining the first projection matrix;

determining the relative pose matrix; and

estimating the second projection matrix based at least on the first projection matrix and the relative pose matrix.

22. The method of claim 1, wherein the first image and the second image are still images, and wherein the first image and the second image are captured in response to user input.

24. The method of claim 1, wherein the bundle adjustment technique is performed without performing a least squares calculation.

25. The method of claim 1, further comprising:

identifying intersections of first rays and second rays, the first rays projected into a 3D space from the reference position through pixels of the first image, and the second rays projected into the 3D space from the second position through pixels of the second image; and

generating a second portion of the 3D model using a shortest intersection of the intersections.

10. An apparatus comprising:

a camera configured to capture a first image of an object while at a reference position and to capture a second image of the object while at a second position;

at least one motion sensor; and

a processor configured to;

determine, based on data output by the at least one motion sensor, a movement of the camera from the reference position to the second position; and

compute, using a bundle adjustment technique, a first portion of a three-dimensional (3D) model of the object based on the first image, the second image, and the movement of the camera, wherein the bundle adjustment technique includes estimating a second projection matrix associated with the second image based at least on a first projection matrix associated with the first image and a relative pose matrix based on the movement of the camera from the reference position to the second position.

11. The apparatus of claim 10, wherein the processor is integrated into a mobile device, the camera, a multimedia player, an entertainment unit, a navigation device, a personal digital assistant (PDA), a portable computer, or any combination thereof.

12. The apparatus of claim 10, wherein the data output by the at least one motion sensor includes rotational information associated with the camera, translational information associated with the camera, or a combination thereof, and wherein the movement of the camera includes a rotation of the camera, a translation of the camera, or a combination thereof.

13. The apparatus of claim 10, wherein the processor is further configured to determine feature points of the object based on a function of a first estimated reprojected image point, a second estimated reprojected image point, and the data output by the at least one motion sensor.

14. The apparatus of claim 10, wherein the processor is further configured to:

determine a first estimated reprojected image point of the object based on a product of a first estimate of the reference position and one or more feature points of the object;

determine a second estimated reprojected image point of the object based on a product of a second estimate of the second position and the one or more feature points of the object;

determine a square of a first Euclidean distance between the first estimated reprojected image point and coordinates of the object in the first image;

determine a square of a second Euclidean distance between the second estimated reprojected image point and coordinates of the object in the second image; and

modify an estimate of the first portion of the 3D model to minimize a sum of the square of the first Euclidean distance and the square of the second Euclidean distance,

wherein the second estimate of the second position is generated based on the first estimate of the reference position and the data output by the at least one motion sensor.

15. A method comprising:

capturing, at a camera of a mobile device, a first image of an object while the mobile device is at a reference position;

capturing a second image of the object while the mobile device is at a second position;

determining, based on data received from at least one motion sensor, a movement of a mobile device from the reference position to the second position; and

computing, using a bundle adjustment technique, a first portion of a three-dimensional (3D) model of an object based on the first image, based further on the second image, and based further on the movement of the mobile device, wherein the bundle adjustment technique includes estimating a second projection matrix associated with the second image based at least on a first projection matrix associated with the first image and a relative pose matrix based on the movement of the mobile device from the reference position to the second position.

23. The method of claim 15, wherein the 3D model is computed concurrently with capturing at least one of the first image and the second image.

16. An apparatus comprising:

means for determining, based on data received from at least one means for sensing motion, a movement of a mobile device from a reference position to a second position; and

means for computing a portion of a three-dimensional (3D) model of an object based on a first image of the object corresponding to a first view of the object from the reference position, based further on a second image of the object corresponding to a second view of the object from the second position, based further on the movement of the mobile device, and based further on an estimate of a second projection matrix associated with the second image based at least on a first projection matrix associated with the first image and a relative pose matrix based on the movement of the mobile device from the reference position to the second position.

26. The apparatus of claim 16, wherein the means for determining and the means for computing are integrated into the mobile device, and wherein the mobile device comprises a communication device, a camera, a multimedia player, an entertainment unit, a navigation device, a personal digital assistant (PDA), a portable computer, or any combination thereof.

17. A computer-readable non-transitory medium comprising instructions that, when executed by a processor, cause the processor to:

determine, based on data received from at least one motion sensor, a movement of a mobile device from a reference position to a second position; and

compute, using a bundle adjustment technique, a first portion of a three-dimensional (3D) model of an object based on a first image of the object corresponding to a first view of the object from the reference position of the mobile device, based further on a second image of the object corresponding to a second view of the object from the second position of the mobile device, and based further on the movement of the mobile device, wherein the bundle adjustment technique includes estimating a second projection matrix associated with the second image based at least on a first projection matrix associated with the first image and a relative pose matrix based on the movement of the mobile device from the reference position to the second position.

18. The non-transitory computer-readable medium of claim 17, further comprising instructions executable by the processor to determine feature points of the object based on a function of a first estimated reprojected image point, a second estimated reprojected image point, and the data.

19. The non-transitory computer-readable medium of claim 17, further comprising instructions executable by the processor to generate one or more silhouettes associated with the object, wherein a second portion of the 3D model of the object is based on the one or more silhouettes.