SPATIAL INTERACTION IN AUGMENTED REALITY

• US 20160140763A1
• Filed: 09/18/2015
• Published: 05/19/2016
• Est. Priority Date: 11/14/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: SCHMALSTIEG, Dieter, SEICHTER, Hartmut
• Granted Patent: US 9,911,235 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

A method for spatial interaction in Augmented Reality (AR) includes displaying an AR scene that includes an image of a real-world scene, a virtual target object, and a virtual cursor. A position of the virtual cursor is provided according to a first coordinate system within the AR scene. A user device tracks a pose of the user device relative to a user hand according to a second coordinate system. The second coordinate system is mapped to the first coordinate system to control movements of the virtual cursor. In a first mapping mode, virtual cursor movement is controlled to change a distance between the virtual cursor and the virtual target object. In a second mapping mode, virtual cursor movement is controlled to manipulate the virtual target object. User input is detected to control which of the first mapping mode or the second mapping mode is used.

Claims

1. A method, comprising:

acquiring, by a user device, an image of a real-world scene;

displaying, by the user device, an augmented reality (AR) scene that includes the image of the real-world scene, a virtual target object, and a virtual cursor, wherein a position of the virtual cursor is provided according to a first coordinate system within the AR scene;

tracking a pose of the user device relative to a user hand according to a second coordinate system that defines a relationship between the user device and the user hand;

mapping the second coordinate system to the first coordinate system to control movement of the virtual cursor in the AR scene in response to movements of the user hand, wherein mapping the second coordinate system to the first coordinate system includes a first mapping mode and a second mapping mode, wherein the first mapping mode is configured to control movement of the virtual cursor to change a distance between the virtual cursor and the virtual target object in the AR scene, and wherein the second mapping mode is configured to control movement of virtual cursor to manipulate the virtual target object within the AR scene; and

detecting, at the user device, a user input to control which of the first mapping mode and the second mapping mode is used to control movement of the virtual cursor in the AR scene.

2. The method of claim 1, wherein detecting the user input comprises detecting a finger posture of the user hand.

3. The method of claim 2, wherein detecting the finger posture comprises detecting a finger point of the user hand to manipulate the virtual target object in the second mapping mode.

4. The method of claim 1, wherein detecting the user input comprises detecting a hand gesture of the user hand.

5. The method of claim 4, wherein detecting the hand gesture of the user hand comprises detecting an open hand gesture of the user hand to change the distance between the virtual cursor and the virtual target object in the AR scene in the first mapping mode.

6. The method of claim 5, further comprising:

detecting a first posture of the user hand; and

switching from the first mapping mode to the second mapping mode in response to detecting the first posture of the user hand.

7. The method of claim 5, wherein detecting the hand gesture of the user hand comprises detecting a pinching gesture by the user hand to change the distance between the virtual cursor and the virtual target object in the AR scene in the first mapping mode.

8. The method of claim 1, wherein mapping the second coordinate system to the first coordinate system according to the first mapping mode comprises:

determining translational and rotational movement of the user hand relative to the user device;

determining translational and rotational movement of the user device relative to the virtual target object; and

combining the translational and rotational movement of the user hand with the translational and rotational movements of the user device to provide translational and rotational movement of the virtual cursor within the AR scene.

9. The method of claim 8, wherein mapping the second coordinate system to the first coordinate system according to the second mapping mode comprises:

determining translational and rotational movement of the user hand relative to the user device; and

mapping the translational and rotational movement of the user hand to a translational and rotational movement of the virtual cursor within the AR scene independent of any translational and rotational movement of the user device relative to the virtual target object.

10. The method of claim 1, wherein virtual cursor is a virtual hand displayed in the AR scene.

11. The method of claim 1, wherein tracking the pose of the user device relative to the user hand comprises detecting a translational movement of the user hand for a first distance, and wherein the first mapping mode includes normalizing the translational movement of the user hand to a translational movement of the virtual cursor a second distance in the AR scene, wherein the second distance is greater than the first distance.

12. The method of claim 1, wherein tracking the pose of the user device relative to the user hand comprises detecting a translational movement of the user hand for a first distance, and wherein the second mapping mode includes a one-to-one mapping of the second coordinate system to the first coordinate system for a translational movement of the virtual cursor for the first distance in the AR scene.

13. An apparatus, comprising:

means for acquiring, by a user device, an image of a real-world scene;

means for displaying, on the user device, an augmented reality (AR) scene that includes the image of the real-world scene, a virtual target object, and a virtual cursor, wherein a position of the virtual cursor is provided according to a first coordinate system within the AR scene;

means for tracking a pose of the user device relative to a user hand according to a second coordinate system that defines a relationship between the user device and the user hand;

means for mapping the second coordinate system to the first coordinate system to control movement of the virtual cursor in the AR scene in response to movements of the user hand, wherein the means for mapping the second coordinate system to the first coordinate system includes a first mapping mode and a second mapping mode, wherein the first mapping mode is configured to control movement of the virtual cursor to change a distance between the virtual cursor and the virtual target object in the AR scene, and wherein the second mapping mode is configured to control movement of virtual cursor to manipulate the virtual target object within the AR scene; and

means for detecting, at the user device, a user input to control which of the first mapping mode and the second mapping mode is used to control movement of the virtual cursor in the AR scene.

14. The apparatus of claim 13, wherein detecting the user input comprises detecting at least one of a finger posture of the user hand or a hand gesture of the user hand.

15. The apparatus of claim 13, wherein the means for tracking the pose of the user device relative to the user hand comprises means for detecting a translational movement of the user hand for a first distance, and wherein the first mapping mode includes normalizing the translational movement of the user hand to a translational movement of the virtual cursor a second distance in the AR scene, wherein the second distance is greater than the first distance.

16. The apparatus of claim 13, wherein the means for tracking the pose of the user device relative to the user hand comprises means for detecting a translational movement of the user hand for a first distance, and wherein the second mapping mode includes a one-to-one mapping of the second coordinate system to the first coordinate system for a translational movement of the virtual cursor for the first distance in the AR scene.

17. A user device, comprising:

a camera configured to capture an image of a real-world scene;

a display configured to display an augmented reality (AR) scene that includes the image of the real-world scene, a virtual target object, and a virtual cursor, wherein a position of the virtual cursor is provided according to a first coordinate system within the AR scene;

memory adapted to store program code; and

a processing unit coupled to the memory to access and execute instructions included in the program code to direct the user device to;

track a pose of the user device relative to a user hand according to a second coordinate system that defines a relationship between the user device and the user hand;

map the second coordinate system to the first coordinate system to control movement of the virtual cursor in the AR scene in response to movements of the user hand, wherein mapping the second coordinate system to the first coordinate system includes a first mapping mode and a second mapping mode, wherein the first mapping mode is configured to control movement of the virtual cursor to change a distance between the virtual cursor and the virtual target object in the AR scene, and wherein the second mapping mode is configured to control movement of virtual cursor to manipulate the virtual target object within the AR scene; and

detect a user input to control which of the first mapping mode and the second mapping mode is used to control movement of the virtual cursor in the AR scene.

18. The user device of claim 17, wherein the instructions to detect the user input comprises instructions to detect a finger posture of the user hand.

19. The user device of claim 18, wherein the instructions to detect the finger posture comprises instructions to detect a finger point of the user hand to manipulate the virtual target object in the second mapping mode.

20. The user device of claim 17, wherein the instructions to detect the user input comprises instructions to detect a hand gesture of the user hand.

21. The user device of claim 20, wherein the instructions to detect the hand gesture of the user hand comprises instructions to detect an open hand gesture of the user hand to change the distance between the virtual cursor and the virtual target object in the AR scene in the first mapping mode.

22. The user device of claim 21, further comprising instructions to direct the user device to:

detect a first posture of the user hand; and

switch from the first mapping mode to the second mapping mode in response to detecting the first posture of the user hand.

23. The user device of claim 20, wherein the instructions to detect the hand gesture of the user hand comprises instructions to detect a pinching gesture by the user hand to change the distance between the virtual cursor and the virtual target object in the AR scene in the first mapping mode.

24. The user device of claim 17, wherein the instructions to track the pose of the user device relative to the user hand comprises instructions to detect a translational movement of the user hand for a first distance, and wherein the first mapping mode includes normalizing the translational movement of the user hand to a translational movement of the virtual cursor a second distance in the AR scene, wherein the second distance is greater than the first distance.

25. The user device of claim 17, wherein the instructions to track the pose of the user device relative to the user hand comprises instructions to detect a translational movement of the user hand for a first distance, and wherein the second mapping mode includes a one-to-one mapping of the second coordinate system to the first coordinate system for a translational movement of the virtual cursor for the first distance in the AR scene.

26. A non-transitory computer-readable medium including program code stored thereon, the program code comprising instructions which when executed cause a user device to:

acquire an image of a real-world scene;

display an augmented reality (AR) scene that includes the image of the real-world scene, a virtual target object, and a virtual cursor, wherein a position of the virtual cursor is provided according to a first coordinate system within the AR scene;

track a pose of the user device relative to a user hand according to a second coordinate system that defines a relationship between the user device and the user hand;

map the second coordinate system to the first coordinate system to control movement of the virtual cursor in the AR scene in response to movements of the user hand, wherein mapping the second coordinate system to the first coordinate system includes a first mapping mode and a second mapping mode, wherein the first mapping mode is configured to control movement of the virtual cursor to change a distance between the virtual cursor and the virtual target object in the AR scene, and wherein the second mapping mode is configured to control movement of virtual cursor to manipulate the virtual target object within the AR scene; and

detect a user input to control which of the first mapping mode and the second mapping mode is used to control movement of the virtual cursor in the AR scene.

27. The non-transitory computer-readable medium of claim 26, wherein the instructions to detect the user input comprises instructions to detect at least one of a finger posture of the user hand or a hand gesture of the user hand.

28. The non-transitory computer-readable medium of claim 26, further comprising instructions to direct the user device to:

detect a first posture of the user hand; and

switch from the first mapping mode to the second mapping mode in response to detecting the first posture of the user hand.

29. The non-transitory computer-readable medium of claim 26, wherein the instructions to track the pose of the user device relative to the user hand comprises instructions to detect a translational movement of the user hand for a first distance, and wherein the first mapping mode includes normalizing the translational movement of the user hand to a translational movement of the virtual cursor a second distance in the AR scene, wherein the second distance is greater than the first distance.

30. The non-transitory computer-readable medium of claim 26, wherein the instructions to track the pose of the user device relative to the user hand comprises instructions to detect a translational movement of the user hand for a first distance, and wherein the second mapping mode includes a one-to-one mapping of the second coordinate system to the first coordinate system for a translational movement of the virtual cursor for the first distance in the AR scene.