Methods for facilitating computer vision application initialization

• US 9,667,873 B2
• Filed: 05/02/2014
• Issued: 05/30/2017
• Est. Priority Date: 05/02/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Ramachandran, Mahesh, Mulloni, Alessandro, Reitmayr, Gerhard, Wagner, Daniel
• Primary Examiner(s): Dang, Hung
• Assistant Examiner(s): Wendmagegn, Girumsew
• Application Number:
  US14/268,784
• Publication Number:
  US 20140327792A1
• Time in Patent Office: 1,124 Days
• Field of Search: 3482118
• CPC Class Codes: G01C 25/005 initial alignment, calibration or starting-up of inertial devices
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

Embodiments disclosed pertain to systems, method s and apparatus for the initialization of Computer Vision (CV) applications on user devices (UDs) comprising a camera and a display. In some embodiments, an optimal camera trajectory for initialization of a Computer Vision (CV) application may be determined based on an initial camera pose and an estimated pivot distance. For example, the initial camera pose may be estimated based on a first image captured by the camera. Further, the display may be updated in real-time with an indication of a desired movement direction for the camera. In some embodiments, the indication of desired movement direction may be based, in part, on a current camera pose and the optimal trajectory, where the current camera pose may be estimated based on a current image captured by the camera.

Claims

1. A method on a user device (UD) comprising a camera and a display, the method comprising:

determining a trajectory of the camera for targetless initialization of a Computer Vision (CV) application based, in part, on an initial camera pose and an estimated pivot distance between the camera and a pivot position, wherein the initial camera pose is estimated based on a targetless first image captured by the camera; and

updating the display in real-time with an indication of a desired movement direction for the camera, the indication of the desired movement direction being based, in part, on a current camera pose and the trajectory of the camera, wherein the current camera pose is estimated based on a current image captured by the camera.

2. The method of claim 1, wherein the desired movement direction for the camera is obtained based in part, on a comparison of a current trajectory of the camera with the trajectory of the camera, the current trajectory of the camera being determined from a sequence of prior camera poses.

3. The method of claim 1, further comprising:

updating the display in real-time upon a movement of the camera with one or more additional indications of desired movement directions for the camera, wherein each additional indication of desired movement direction is based, in part, on a corresponding additional camera pose associated with the movement of the camera.

4. The method of claim 1, wherein the pivot distance is determined as at least one of:

an arbitrary positive value;

an average distance of a subset of observed feature points in the first image;

a value above a threshold.

5. The method of claim 1, further comprising:

computing a measure of quality of a current trajectory of the camera, the current trajectory of the camera being determined based, in part, on the current camera pose and a sequence of prior camera poses, and wherein the measure of quality is based on a comparison of the current trajectory of the camera with the trajectory of the camera; and

displaying an indication of the measure of quality of the current trajectory of the camera.

6. The method of claim 1, wherein determining the trajectory of the camera, comprises:

selecting the first image from one or more camera captured images based, in part, on a number of feature points in the one or more camera captured images.

7. The method of claim 6, further comprising:

providing, upon selection of the first image, an indication of suitability of the first image for initialization by displaying a first virtual object;

or

providing an indication of unsuitability for initialization of the one or more captured images by displaying a second virtual object.

8. The method of claim 1, wherein the indication of the desired movement direction is provided using at least one of:

a Screen-aligned 2-Dimensional (2D) guidance scheme comprising 2D screen aligned UI elements to provide the indication of the desired movement direction;

or

a Screen-aligned 3-Dimensional (3D) guidance scheme comprising 3D screen aligned UI elements to provide the indication of the desired movement direction;

or

a World aligned 3D guidance scheme, comprising 3D UI elements aligned to real world objects to provide the indication of the desired movement direction;

or

a World aligned 2.5-Dimensional (2.5-D) guidance scheme, comprising UI elements displayed in 2.5-D, the UI elements aligned to real world objects to provide the indication of the desired movement direction.

9. The method of claim 1, wherein the indication of the desired movement direction is provided for both sideward translational and pivot initialization motion sequences.

10. The method of claim 1, wherein the indication of the desired movement direction comprises an indication of past movement and an indication of a preferred future movement direction.

11. The method of claim 1, wherein the indication of the desired movement direction also indicates rotational movement of the camera.

12. The method of claim 1, further comprising:

providing at least one of an auditory feedback or a tactile feedback to a user in conjunction with the indication of the desired movement direction.

13. A user device (UD) comprising:

an image sensor to capture a sequence of images;

a display, and

a processor coupled to the image sensor and the display, wherein the processor is configured to;

determine a trajectory of the image sensor for targetless initialization of a Computer Vision (CV) application based on an initial image sensor pose and an estimated pivot distance between the image sensor and a pivot position, wherein the initial image sensor pose is estimated based on a targetless first image captured by the image sensor, and

update the display in real-time with an indication of a desired movement direction for the image sensor, the indication of the desired movement direction being based, in part, on a current image sensor pose and the trajectory of the image sensor, wherein the current image sensor pose is estimated based on a current image captured by the image sensor.

14. The UD of claim 13, wherein to obtain the desired movement direction for the image sensor, the processor is configured to:

compare the trajectory of the image sensor with a current trajectory of the image sensor, the current trajectory of the image sensor being determined from a sequence of prior image sensor poses.

15. The UD of claim 13, the processor is further configured to:

update the display in real-time upon a movement of the image sensor with a one or more additional indications of desired movement directions for the image sensor, wherein each additional indication of desired movement direction is based, in part, on a corresponding additional image sensor pose associated with the movement of the image sensor.

16. The UD of claim 13, wherein the processor is configured to:

determine the pivot distance as at least one of;

an arbitrary positive value;

an average distance of observed feature points in the first image;

a value above a threshold.

17. The UD of claim 13, wherein the processor is further configured to:

compute a measure of quality of a current trajectory of the image sensor, the current trajectory of the image sensor being determined based, in part, on the current image sensor pose and a sequence of prior image sensor poses, and wherein the measure of quality is based on a comparison of the current trajectory of the image sensor with the trajectory of the image sensor; and

display an indication of the measure of quality of the current trajectory of the image sensor.

18. The UD of claim 13, wherein to determine a trajectory of the image sensor, the processor is configured to:

select the first image from one or more image sensor captured images based, in part, on a number of feature points in the one or more image sensor captured images.

19. The UD of claim 18, wherein the processor is further configured to:

provide, upon selection of the first image, an indication of suitability of the first image for initialization by displaying a first virtual object;

or

provide an indication of unsuitability of the one or more captured images for initialization by displaying a second virtual object.

20. The UD of claim 13, wherein the indication of the desired movement direction is provided using at least one of:

a Screen-aligned 2-Dimensional (2D) guidance scheme comprising 2D screen aligned UI elements to provide the indication of the desired movement direction;

or

a Screen-aligned 3-Dimensional (3D) guidance scheme comprising 3D screen aligned UI elements to provide the indication of the desired movement direction;

or

a World aligned 3D guidance scheme, comprising 3D UI elements aligned to real world objects to provide the indication of the desired movement direction;

or

a World aligned 2.5-Dimensional (2.5-D) guidance scheme, comprising UI elements displayed in 2.5-D, the UI elements aligned to real world objects to provide the indication of the desired movement direction.

21. The UD of claim 13, wherein the indication of the desired movement direction is provided for both sideward translational and pivot initialization motion sequences.

22. The UD of claim 13, wherein the indication of the desired movement direction comprises an indication of past movement and an indication of a preferred future movement direction.

23. The UD of claim 13, wherein the indication of the desired movement direction also indicates rotational movement of the UD.

24. The UD of claim 13, wherein the processor is further configured to:

cause the provision of at least one of an auditory feedback or a tactile feedback to a user in conjunction with the indication of the desired movement direction.

27. The apparatus of claim 21, wherein the indication of the desired movement direction is provided using at least one of:

a Screen-aligned 2-Dimensional (2D) guidance scheme comprising 2D screen aligned UI elements to provide the indication of the desired movement direction;

or

a Screen-aligned 3-Dimensional (3D) guidance scheme comprising 3D screen aligned UI elements to provide the indication of the desired movement direction;

or

a World aligned 3D guidance scheme, comprising 3D UI elements aligned to real world objects to provide the indication of the desired movement direction;

or

a World aligned 2.5-Dimensional (2.5-D) guidance scheme, comprising UI elements displayed in 2.5-D, the UI elements aligned to real world objects to provide the indication of the desired movement direction.

25. An apparatus comprising:

image capture means, the image capture means to capture a sequence of images;

display means;

means for determining a trajectory of the image capture means for targetless initialization of a Computer Vision (CV) application based on an initial pose of the image capture means and an estimated pivot distance between the camera and a pivot position, wherein the initial pose of the image capture means is estimated based on a targetless first image captured by the image capture means; and

means for updating the display means in real-time with an indication of a desired movement direction for the image capture means, the indication of desired movement direction being based, in part, on a current pose of the image capture means and the trajectory of the image capture means, wherein the current pose of the image capture means is estimated based on a current image captured by the image capture means.

26. The apparatus of claim 25, wherein means for updating the display means comprises:

means for obtaining the desired movement direction for the image sensor, wherein the means for obtaining the desired movement direction for the image sensor comprises;

means for comparing the trajectory of the image capture means with a current trajectory of the image capture means, the current trajectory of the image capture means being determined from a sequence of prior image capture means poses.

28. A non-transitory computer-readable medium comprising instructions, which, when executed by a processor, perform a method on a user device (UD) comprising a camera and a display, the method comprising:

determining a trajectory of a camera for targetless initialization of a Computer Vision (CV) application based on an initial camera pose and an estimated pivot distance between the camera and a pivot position, wherein the initial camera pose is estimated based on a targetless first image captured by the camera; and

updating the display in real-time with an indication of a desired movement direction for the camera, the indication of the desired movement direction being based, in part, on a current camera pose and the trajectory of the camera, wherein the current camera pose is estimated based on a current image captured by the camera.

29. The computer-readable medium of claim 28, wherein the desired movement direction for the camera is obtained based in part, on a comparison of a current trajectory of the camera with the trajectory of the camera, the current trajectory of the camera being determined from a sequence of prior camera poses.

30. The computer-readable medium of claim 28, wherein the indication of the desired movement direction is provided using at least one of:

a Screen-aligned 2-Dimensional (2D) guidance scheme comprising 2D screen aligned UI elements to provide the indication of the desired movement direction;

or

a Screen-aligned 3-Dimensional (3D) guidance scheme comprising 3D screen aligned UI elements to provide the indication of the desired movement direction;

or

a World aligned 3D guidance scheme, comprising 3D UI elements aligned to real world objects to provide the indication of the desired movement direction;

or

a World aligned 2.5-Dimensional (2.5-D) guidance scheme, comprising UI elements displayed in 2.5-D, the UI elements aligned to real world objects to provide the indication of the desired movement direction.