Method and apparatus for controlling augmented reality

• US 9,349,218 B2
• Filed: 03/15/2013
• Issued: 05/24/2016
• Est. Priority Date: 07/26/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Keating, Virginia Walker, Gervautz, Michael, Nielsen, Per O.
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US13/815,820
• Publication Number:
  US 20140028712A1
• Time in Patent Office: 1,166 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: H04N 21/4126 : The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/42208 : Display device provided on the remote control
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4852 : for modifying audio parameters, e.g. switching between mono and stereo
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Method and apparatus for controlling an augmented reality interface are disclosed. In one embodiment, a method for use with an augmented reality enabled device (ARD) comprises receiving image data for tracking a plurality of objects, identifying an object to be selected from the plurality of objects, determining whether the object has been selected based at least in part on a set of selection criteria, and causing an augmentation to be rendered with the object if it is determined that the object has been selected.

Claims

1. A method for use with an augmented reality enabled device (ARD), comprising:

receiving image data for tracking a plurality of objects;

identifying an object to be selected from the plurality of objects;

determining whether the object has been selected based at least in part on a set of selection criteria, wherein the set of criteria is based at least in part on user'"'"'s capability in handling the ARD and wherein the user'"'"'s capability includes developmental state and health condition of the user, wherein determining whether the object has been selected includes determining an adjustable degree of stillness of the ARD with respect to the object to be selected, an adjustable time of hold of the ARD with respect to the object to be selected, and an adjustable tolerance of temporary loss of target by the ARD with respect to the object to be selected, based at least in part on the development state and health condition of the user; and

causing an augmentation to be rendered with the object if it is determined that the object has been selected.

2. The method of claim 1, wherein the tracking comprises 3-dimensional tracking comprising:

determining relative poses of the plurality of objects with respect to the ARD; and

updating states of the plurality of objects using the relative poses of the plurality of objects with respect to the ARD, wherein the states of the plurality of objects include relational information of the plurality of objects.

3. The method of claim 2, wherein the determining relative poses of the plurality of objects with respect to the ARD comprises:

detecting poses of the plurality of objects with respect to previously received image data of the plurality of objects.

4. The method of claim 2, wherein the relational information comprises:

relationships among the plurality of objects in a game;

relationships among the plurality of objects in a story;

relationships between the plurality of objects with respect to a background;

or some combination thereof.

5. The method of claim 1, wherein identifying an object to be selected comprises:

analyzing the plurality of objects using the image data; and

identifying the object being closest to a center of an image represented by the image data.

6. The method of claim 1, wherein identifying an object to be selected further comprises:

causing the plurality of objects in the image data to be rendered on a display of the ARD; and

identifying the object that covers most of center pixels of the display of the ARD.

7. The method of claim 1, wherein determining whether the object has been selected further comprises:

measuring relative velocity between the object and the ARD; and

measuring amount of time the object is stationary with respect to the ARD.

8. The method of claim 1, wherein determining whether the object has been selected further comprises:

receiving an audio command; and

measuring amplitude of the audio command.

9. The method of claim 1, wherein the set of selection criteria comprises:

the object being in view of the ARD for a predetermined period of time;

the object being in view of the ARD and a relative velocity between the object and the ARD being below a predetermined threshold;

the object being in view of the ARD and amplitude of an audio command exceeding a predetermined value;

or some combination thereof.

10. The method of claim 1, wherein causing the augmentation to be rendered comprises:

presenting interactions between the object and the ARD in a display of the ARD.

11. The method of claim 1, wherein the identifying comprises determining that the object has been zoomed in on.

12. The method of claim 1, wherein the determining comprises:

causing an indicator to be presented, wherein the indicator indicating that the object has been identified; and

determining whether the object is deselected in response to presentation of the indicator.

13. The method of claim 12, wherein determining whether the object is deselected comprises:

determining that the object has been selected if a position of the object with respect to the ARD is maintained for a threshold amount of time after presentation of the indicator.

14. The method of claim 12, wherein the indicator comprises:

an icon;

a vibration of the object on a display of the ARD;

a vibration of the ARD when the object is centered on the display of the ARD;

a representation of an elapsed time;

or some combination thereof.

15. The method of claim 1, further comprising:

causing an image of a scene to be displayed on a display of the ARD, the scene including one or more objects;

causing the image to be augmented with an indicator identifying an object of to be selected from the plurality of objects; and

determining whether the object has been deselected subsequent to the image being augmented with the indicator.

16. An augmented reality enabled device (ARD), comprising:

a camera configured to receiving image data for tracking a plurality of objects;

a control unit including processing logic, the control unit comprising;

a processor configured to identify an object to be selected from the plurality of objects;

a tracking unit configured to determine whether the object has been selected based at least in part on a set of selection criteria, wherein the set of criteria is based at least in part on user'"'"'s capability in handling the ARD and wherein the user'"'"'s capability includes developmental state and health condition of the user, wherein the tracking unit is further configured to determine an adjustable degree of stillness of the ARD with respect to the object to be selected, an adjustable time of hold of the ARD with respect to the object to be selected, and an adjustable tolerance of temporary loss of target by the ARD with respect to the object to be selected, based at least in part on the development state and health condition of the user; and

an augmented reality user interface unit configured to cause an augmentation to be rendered with the object if it is determined that the object has been selected.

17. The augmented reality enabled device of claim 16, wherein the tracking unit further comprises 3-dimensional tracking configured to:

determine relative poses of the plurality of objects with respect to the ARD; and

update states of the plurality of objects using the relative poses of the plurality of objects with respect to the ARD, wherein the states of the plurality of objects include relational information of the plurality of objects.

18. The augmented reality enabled device of claim 17, wherein the tracking unit is further configured to:

detect poses of the plurality of objects with respect to previously received image data of the plurality of objects.

19. The augmented reality enabled device of claim 17, wherein the relational information comprises:

relationships among the plurality of objects in a game;

relationships among the plurality of objects in a story;

relationships between the plurality of objects with respect to a background;

or some combination thereof.

20. The augmented reality enabled device of claim 16, wherein the processor is further configured to:

analyze the plurality of objects using the image data; and

identify the object being closest to a center of an image represented by the image data.

21. The augmented reality enabled device of claim 16, wherein the processor is further configured to:

cause the plurality of objects in the image data to be rendered on a display of the ARD; and

identify the object that covers most of center pixels of the display of the ARD.

22. The augmented reality enabled device of claim 16, wherein the tracking unit is further configured to:

measure relative velocity between the object and the ARD; and

measure amount of time the object is stationary with respect to the ARD.

23. The augmented reality enabled device of claim 16, wherein the control unit is further configured to:

receive an audio command; and

measure amplitude of the audio command.

24. The augmented reality enabled device of claim 16, wherein the set of selection criteria comprises:

the object being in view of the ARD for a predetermined period of time;

the object being in view of the ARD and a relative velocity between the object and the ARD being below a predetermined threshold;

the object being in view of the ARD and amplitude of an audio command exceeding a predetermined value;

or some combination thereof.

25. The augmented reality enabled device of claim 16, wherein the augmented reality user interface unit is further configured to:

present interactions between the object and the ARD in a display of the ARD.

26. The augmented reality enabled device of claim 16, wherein the processor is further configured to determine that the object has been zoomed in on.

27. The augmented reality enabled device of claim 16, wherein the processor is further configured to:

cause an indicator to be presented, wherein the indicator indicating that the object has been identified; and

determine whether the object is deselected in response to presentation of the indicator.

28. The augmented reality enabled device of claim 27, wherein the processor is further configured to:

determine that the object has been selected if a position of the object with respect to the ARD is maintained for a threshold amount of time after presentation of the indicator.

29. The augmented reality enabled device of claim 27, wherein the indicator comprises:

an icon;

a vibration of the object on a display of the ARD;

a vibration of the ARD when the object is centered on the display of the ARD;

a representation of an elapsed time;

or some combination thereof.

30. The augmented reality enabled device of claim 16, wherein the augmented reality user interface unit is further configured to:

cause an image of a scene to be displayed on a display of the ARD, the scene including one or more objects;

cause the image to be augmented with an indicator identifying an object of to be selected from the plurality of objects; and

determine whether the object has been deselected subsequent to the image being augmented with the indicator.

31. A non-transitory medium storing instructions for execution by one or more computer systems, the instructions comprising:

instructions for receiving image data for tracking a plurality of objects;

instructions for identifying an object to be selected from the plurality of objects;

instructions for determining whether the object has been selected based at least in part on a set of selection criteria, wherein the set of criteria is based at least in part on user'"'"'s capability in handling the ARD and wherein the user'"'"'s capability includes developmental state and health condition of the user, wherein the instructions for determining whether the object has been selected includes instructions for determining an adjustable degree of stillness of the ARD with respect to the object to be selected, an adjustable time of hold of the ARD with respect to the object to be selected, and an adjustable tolerance of temporary loss of target by the ARD with respect to the object to be selected, based at least in part on the development state and health condition of the user; and

instructions for causing an augmentation to be rendered with the object if it is determined that the object has been selected.

32. An apparatus, comprising:

means for receiving image data for tracking a plurality of objects;

means for identifying an object to be selected from the plurality of objects;

means for determining whether the object has been selected based at least in part on a set of selection criteria, wherein the set of criteria is based at least in part on user'"'"'s capability in handling the ARD and wherein the user'"'"'s capability includes developmental state and health condition of the user, wherein the means for determining whether the object has been selected includes means for determining an adjustable degree of stillness of the ARD with respect to the object to be selected, an adjustable time of hold of the ARD with respect to the object to be selected, and an adjustable tolerance of temporary loss of target by the ARD with respect to the object to be selected, based at least in part on the development state and health condition of the user; and

means for causing an augmentation to be rendered with the object if it is determined that the object has been selected.

33. The apparatus of claim 32, wherein the tracking comprises 3-dimensional tracking comprising:

means for determining relative poses of the plurality of objects with respect to the ARD; and

means for updating states of the plurality of objects using the relative poses of the plurality of objects with respect to the ARD, wherein the states of the plurality of objects include relational information of the plurality of objects.

34. The apparatus of claim 32, wherein means for identifying an object to be selected comprises:

means for analyzing the plurality of objects using the image data; and

means for identifying the object being closest to a center of an image represented by the image data.

35. The apparatus of claim 32, wherein means for determining whether the object has been selected further comprises:

means for measuring relative velocity between the object and the ARD; and

means for measuring amount of time the object is stationary with respect to the ARD.

36. The apparatus of claim 32, wherein instructions for determining whether the object has been selected further comprises:

means for receiving an audio command; and

means for measuring amplitude of the audio command.

37. The apparatus of claim 32, wherein the set of selection criteria comprises:

the object being in view of the ARD for a predetermined period of time;

the object being in view of the ARD and a relative velocity between the object and the ARD being below a predetermined threshold;

the object being in view of the ARD and amplitude of an audio command exceeding a predetermined value;

or some combination thereof.

38. The apparatus of claim 32, wherein means for causing the augmentation to be rendered comprises:

means for presenting interactions between the object and the ARD in a display of the ARD.