SYSTEMS AND METHODS FOR EFFICIENT 3D TRACKING OF WEAKLY TEXTURED PLANAR SURFACES FOR AUGMENTED REALITY APPLICATIONS

• US 20140037137A1
• Filed: 08/03/2012
• Published: 02/06/2014
• Est. Priority Date: 08/03/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Hagbi, Netanel, BROADDUS, CHRISTOPHER, Grygorenko, Andriy, Bergig, Oriel Y.
• Granted Patent: US 9,147,251 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/248 involving reference images or patches

Abstract

The present system provides an on the fly simple to complex 6DOF registration approach using the direct method. On the fly means it does not require training time, a user points a phone/camera to a planar surface and can start tracking it instantly. Simple to complex means the system performs registration in multiple levels of complexity from 2DOF to 6DOF. By increasing the complexity model the system enables more surfaces to be tracked and for surfaces that are tracked the system can avoid local minima solution providing a more robust and accurate 6DOF tracking. Even surfaces that are very weak in features can be tracked in 6DOF and virtual content can be registered to them. The system enables playing Augmented Reality games on low-end devices such as mobile phones on almost any surface in the real world.

Claims

1. A method for tracking comprising:

Acquiring a first image from an image capture device;

defining the image as a reference image;

Acquiring a second image from the image capture device;

Performing a first position determining step to determine the change in position of the camera at a first level between the first image and the second image;

Performing a second position determining step to determine the change in position of the camera at a second level between the first image and the second image.

2. The method of claim 1 wherein the step of defining the image as a reference image comprises defining one or more reference patches on the image.

3. The method of claim 2 wherein the first level is four degrees of freedom (DOF).

4. The method of claim 2 wherein the second level is six degrees of freedom (DOF).

5. The method of claim 1 wherein the image capture device comprises a mobile device.

6. The method of claim 5 wherein the mobile device comprises a smart-phone.

7. The method of claim 1 further including the step of overlaying an augmented reality (AR) layer on top of the second image and displaying the combined image to the user.

8. The method of claim 2 wherein the first positioning step comprises defining a translation range within a threshold boundary, and checking all X and Y translations in the translation range.

9. The method of claim 8 wherein the second positioning step comprises using the results of the first positioning step as a seed for a convergence step.

10. The method of claim 1 wherein the first and second positioning steps are seeded using rotation and acceleration information.

11. The method of claim 10 wherein the rotation and acceleration information is obtained from a gyroscope and an accelerometer.

12. A method of tracking comprising:

Acquiring a first image from an image capture device;

defining the first image as a reference image;

Acquiring a second image from the image capture device

Applying a translation registration operation to the second image;

Applying a similarity registration operation to the second image;

Applying a projective registration operation to the second image.

13. The method of claim 12 wherein the step of defining the image as a reference image comprises defining one or more reference patches on the image.

14. The method of claim 13 further comprising decomposing a six degree of freedom (DOF) position and orientation of the image capture device.

15. The method of claim 13 wherein results from the translation registration step are used in subsequent steps.

16. The method of claim 13 wherein results from the similarity registration step are used in the projective registration step.

17. The method of claim 13 wherein the image capture device comprises a mobile device.

18. The method of claim 17 wherein the mobile device comprises a smart-phone.

19. The method of claim 12 further including the step of overlaying an augmented reality (AR) layer on top of the second image and displaying the combined image to the user.

20. The method of claim 12 wherein the first and second positioning steps are seeded using rotation and acceleration information.

21. The method of claim 20 wherein the rotation and acceleration information is obtained from a gyroscope and an accelerometer.

22. A method of tracking comprising:

Obtaining an image from a device;

Performing a static analysis of the image

Performing tracking and dynamic analysis when static analysis provides a good target;

Performing higher degree of tracking when dynamic analysis provides a good target.

23. The method of claim 22 further including the step of drawing a camera frame when static analysis does not provide a good target.

24. The method of claim 22 further including the step of drawing 2D content and a camera frame when the dynamic analysis does not provide a good target.

25. The method of claim 22 wherein the higher degree of tracking is 6 degrees of freedom (6DOF).

26. The method of claim 22 wherein the device is a mobile device.

27. The method of claim 26 wherein the mobile device is a smart-phone.

28. A method of tracking comprising:

Acquiring a first image from an image capture device;

Defining the first image as a reference image;

Acquiring a second image from the image capture device

Applying a dominant motion operation to the second image;

Applying a refinement operation to the second image;

Applying a similarity registration operation to the second image;

Applying a projective registration operation to the second image.

29. The method of claim 28 wherein the step of defining the image as a reference image comprises defining one or more reference patches on the image.

30. The method of claim 29 further comprising decomposing a six degree of freedom (DOF) position and rotation of the image capture device.

31. The method of claim 29 wherein results from the dominant motion step are used in subsequent steps.

32. The method of claim 29 wherein results from the refinement step are used in the projective convergence step.

33. The method of claim 29 wherein results from the similarity convergence step are used in subsequent steps.

34. The method of claim 29 wherein results from the projective registration step are used in subsequent steps.

35. The method of claim 29 wherein the image capture device comprises a mobile device.

36. The method of claim 35 wherein the mobile device comprises a smart-phone.

37. The method of claim 28 further including the step of overlaying an augmented reality (AR) layer on top of the second image and displaying the combined image to the user.

38. The method of claim 28 wherein the first and second positioning steps are seeded using rotation and acceleration information.

39. The method of claim 38 wherein the rotation and acceleration information is obtained from a gyroscope and an accelerometer.