Method of determining a position and orientation of a device associated with a capturing device for capturing at least one image

• US 9,990,726 B2
• Filed: 09/27/2012
• Issued: 06/05/2018
• Est. Priority Date: 09/27/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Langer, Marion, Meier, Peter, BenHimane, Selim
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Uhl, Lindsay
• Application Number:
  US14/430,170
• Publication Number:
  US 20150235367A1
• Time in Patent Office: 2,077 Days
• Field of Search: None
• CPC Class Codes: G01S 19/53 Determining attitude
• CPC Class Codes: G01S 5/0247 Determining attitude
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A method of determining a position and orientation of a device is provided. The position and orientation of the device is determined based on multiple degrees of freedom (DoF) and the device is associated with a capturing device for capturing at least one image is provided. The method includes: capturing at least one image of a real object with the capturing device, and providing a coordinate system in relation to the object; providing an estimation of intrinsic parameters of the capturing device; providing pose data to compute first and second DoFs in the coordinate system, with each DoF having a confidence degree; determining an initial pose of the device; performing a pose estimation process, and calculating in the pose estimation process an estimation of the DoFs having a second confidence degree; and determining a position and orientation of the device.

Claims

1. A method of determining a position and orientation of a capturing device, the method comprising:

receiving an image depicting a real object from the capturing device;

determining a coordinate system in relation to the real object;

determining a first value of a first degree of freedom of the capturing device in the coordinate system based on first pose data, wherein the first value is associated with a first confidence degree;

determining a second value of a second degree of freedom of the capturing device in the coordinate system based on second pose data, wherein the second value is associated with a second confidence degree different from the first confidence degree;

determining an initial pose of the capturing device with respect to the real object based on the first value of the first degree of freedom and the second value of the second degree of freedom;

determining an updated value of the second degree of freedom of the capturing device based on the second confidence degree, the initial pose and the image; and

determining a position and an orientation of the capturing device in the coordinate system based on the first value of the first degree of freedom and the updated value of the second degree of freedom.

2. The method according to claim 1, wherein the first pose data includes first sensor data from a first sensor and the second pose data includes second sensor data from a second sensor.

3. The method according to claim 2, wherein the first confidence degree is associated with the first sensor and the second confidence degree is associated with the second sensor.

4. The method according to claim 2, wherein the first sensor includes an accelerometer, a gyroscope, a global position sensor, an inertial sensor, a magnetometer, an odometer, a mechanical sensor, a rotary encoder, a tracking sensor, a measuring arm, a laser tracker, or a combination thereof.

5. The method according to claim 1, wherein the first pose data indicates a gravity vector of the capturing device.

6. The method according to claim 5, wherein the first pose data is provided from a gravity sensor or combination of sensors indicative of the gravity vector.

7. The method according to claim 1, further comprising:

projecting a digital representation of the real object into the image based on the initial pose; and

determining the updated value of the second degree of freedom based on the digital representation of the real object.

8. The method according to claim 7, wherein determining the updated value includes:

matching a first feature of the real object depicted in the image with a second feature of the digital representation to identify a feature correspondence; and

determining the updated value based on the feature correspondence.

9. The method according to claim 1, wherein the first confidence degree is greater than the second confidence degree.

10. The method according to claim 1, further including receiving user input that comprises interaction with the capturing device, the image, a digital representation of the real object, or a projection of the digital representation of the real object into the image.

11. The method according to claim 10, wherein the interaction includes at least one of:

moving the capturing device with respect to the real object or inputting a command to translate, scale or rotate the digital representation.

12. The method according to claim 10, wherein the interaction includes performing a selection, segmentation, or drawing on a view of the real object in the image.

13. The method according to claim 10, wherein the second value of the second degree of freedom is determined based further on the user input.

14. The method according to claim 10, wherein the interaction includes initiating a transformation of the projection of the digital representation into a second projection of the digital representation, wherein the second value of the second confidence degree is determined based further on the transformation.

15. The method according to claim 8, wherein the feature correspondence has approximately the same orientation as the projection of the digital representation.

16. The method according to claim 8, wherein at least one of search direction or search range for determining the feature correspondence is constrained by the first value of the first degree of freedom based on the first confidence degree.

17. The method according to claim 1, further comprising determining a score indicating a level of success of an object localization by:

projecting a digital representation of the real object into the image based on the determined position and orientation of the capturing device; and

comparing an orientation of the digital representation to an orientation of the real object in the image.

18. The method according to claim 1, further comprising initializing an image-to-image tracking process based on the determined position and orientation of the capturing device.

19. The method according to claim 18, wherein the image-to-image tracking process is configured to disregard the second degree of freedom based on the second confidence degree.

20. The method according to claim 18, wherein the image-to-image tracking process includes detecting correspondences between another image and a projected digital representation of the real object in the image.

22. The method according to claim 11, wherein the interaction includes moving, pinching, or rotation gestures detected by a touch screen or a gesture recognition system.

23. The method of claim 1, wherein the updated value of the second degree of freedom is determined in response to the second confidence degree satisfying a confidence threshold.

24. The method of claim 1, further comprising determining an intrinsic parameter of the capturing device, wherein the updated value of the second degree of freedom is determined based further on the intrinsic parameter of the capturing device.

21. A non-transitory computer readable medium comprising software code executable to perform a method of determining a position and orientation of a device, the method comprising:

receiving an image depicting a real object from a capturing device;

determining a coordinate system in relation to the real object;

determining a first value of a first degree of freedom of the capturing device in the coordinate system based on first pose data, wherein the first value is associated with a first confidence degree;

determining a second value of a second degree of freedom of the capturing device in the coordinate system based on second pose data, wherein the second value is associated with a second confidence degree different from the first confidence degree;

determining an initial pose of the capturing device with respect to the real object based on the first value of the first degree of freedom and the second value of the second degree of freedom;

determining an updated value of the second degree of freedom of the capturing device based on the second confidence degree, the initial pose and the image; and

determining a position and an orientation of the capturing device in the coordinate system based on the first value of the first degree of freedom and the updated value of the second degree of freedom.

25. An apparatus comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including;

receiving an image depicting a real object from a capturing device;

determining a coordinate system in relation to the real object;

determining a first value of a first degree of freedom of the capturing device in the coordinate system based on first pose data, wherein the first value is associated with a first confidence degree;

determining a second value of a second degree of freedom of the capturing device in the coordinate system based on second pose data, wherein the second value is associated with a second confidence degree different from the first confidence degree;

determining an initial pose of the capturing device with respect to the real object based on the first value of the first degree of freedom and the second value of the second degree of freedom;

determining an updated value of the second degree of freedom of the capturing device based on the second confidence degree, the initial pose and the image; and

determining a position and an orientation of the capturing device in the coordinate system based on the first value of the first degree of freedom and the updated value of the second degree of freedom.