Method and apparatus for controlling augmented reality

• US 20140028712A1
• Filed: 03/15/2013
• Published: 01/30/2014
• Est. Priority Date: 07/26/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Keating, Virginia Walker, Gervautz, Michael, Nielsen, Per O.
• Granted Patent: US 9,349,218 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: H04N 21/4126 : The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/42208 : Display device provided on the remote control
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4852 : for modifying audio parameters, e.g. switching between mono and stereo
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Method and apparatus for controlling an augmented reality interface are disclosed. In one embodiment, a method for use with an augmented reality enabled device (ARD) comprises receiving image data for tracking a plurality of objects, identifying an object to be selected from the plurality of objects, determining whether the object has been selected based at least in part on a set of selection criteria, and causing an augmentation to be rendered with the object if it is determined that the object has been selected.

Claims

1. A method for use with an augmented reality enabled device (ARD), comprising:

receiving image data for tracking a plurality of objects;

identifying an object to be selected from the plurality of objects;

determining whether the object has been selected based at least in part on a set of selection criteria; and

causing an augmentation to be rendered with the object if it is determined that the object has been selected.

2. The method of claim 1, wherein the tracking comprises 3-dimensional tracking comprising:

determining relative poses of the plurality of objects with respect to the ARD; and

updating states of the plurality of objects using the relative poses of the plurality of objects with respect to the ARD, wherein the states of the plurality of objects include relational information of the plurality of objects.

3. The method of claim 2, wherein the determining relative poses of the plurality of objects with respect to the ARD comprises:

detecting poses of the plurality of objects with respect to previously received image data of the plurality of objects.

4. The method of claim 2, wherein the relational information comprises at least one of:

relationships among the plurality of objects in a game;

relationships among the plurality of objects in a story; and

relationships between the plurality of objects with respect to a background.

5. The method of claim 1, wherein identifying an object to be selected comprises:

analyzing the plurality of objects using the image data; and

identifying the object being closest to a center of an image represented by the image data.

6. The method of claim 1, wherein identifying an object to be selected further comprises:

causing the plurality of objects in the image data to be rendered on a display of the ARD; and

identifying the object that covers most of center pixels of the display of the ARD.

7. The method of claim 1, wherein determining whether the object has been selected further comprises:

measuring relative velocity between the object and the ARD; and

measuring amount of time the object is stationary with respect to the ARD.

8. The method of claim 1, wherein determining whether the object has been selected further comprises:

receiving an audio command; and

measuring amplitude or the audio command.

9. The method of claim 1, wherein the set of selection criteria comprises at least one of:

the object being in view of the ARD for a predetermined period of time;

the object being in view of the ARD and a relative velocity between the object and the ARD being below a predetermined threshold; and

the object being in view of the ARD and amplitude of an audio command exceeding a predetermined value.

10. The method of claim 1, wherein causing the augmentation to be rendered comprises:

presenting interactions between the object and the ARD in a display of the ARD.

11. The method of claim 1, wherein the identifying comprises determining that the object has been zoomed in on.

12. The method of claim 1, wherein the determining comprises:

causing an indicator to be presented, wherein the indicator indicating that the object has been identified; and

determining whether the object is deselected in response to presentation of the indicator.

13. The method of claim 12, wherein determining whether the object is deselected comprises:

determining that the object has been selected if a position of the object with respect to the ARD is maintained for a threshold amount of time after presentation of the indicator.

14. The method of claim 12, wherein the indicator comprises at least one of:

an icon;

a vibration of the object on a display of the ARD;

a vibration of the ARD when the object is centered on the display of the ARD; and

a representation of an elapsed time.

15. The method of claim 1, further comprising:

causing an image of a scene to be displayed on a display of the ARD, the scene including one or more objects;

causing the image to be augmented with an indicator identifying an object of to be selected from the plurality of objects; and

determining whether the object has been deselected subsequent to the image being augmented with the indicator.

16. An augmented reality enabled device (ARD), comprising:

a control unit including processing logic, the processing logic comprising;

logic configured to receiving image data for tracking a plurality of objects;

logic configured to identify an object to be selected from the plurality of objects;

logic configured to determine whether the object has been selected based at least in part on a set of selection criteria; and

logic configured to cause an augmentation to be rendered with the object if it is determined that the object has been selected.

17. The augmented reality enabled device of claim 16, wherein tracking comprises 3-dimensional tracking comprising:

logic configured to determine relative poses of the plurality of objects with respect to the ARD; and

logic configured to update states of the plurality of objects using the relative poses of the plurality of objects with respect to the ARD, wherein the states of the plurality of objects include relational information of the plurality of objects.

18. The augmented reality enabled device of claim 17, wherein the logic configured to determine relative poses of the plurality of objects with respect to the ARD comprises:

logic configured to detect poses of the plurality of Objects with respect to previously received image data of the plurality of objects.

19. The augmented reality enabled device of claim 17, wherein the relational information comprises at least one of:

relationships among the plurality of objects in a game;

relationships among the plurality of objects in a story; and

relationships between the plurality of objects with respect to a background.

20. The augmented reality enabled device of claim 16, wherein logic configured to identify an object to be selected comprises:

logic configured to analyze the plurality of objects using the image data; and

logic configured to identify the object being closest to a center of an image represented by the image data.

21. The augmented reality enabled device of claim 16, wherein logic configured to identify an object to be selected further comprises:

logic configured to cause the plurality of objects in the image data to be rendered on a display of the ARD; and

logic configured to identify the object that covers most of center pixels of the display of the ARD.

22. The augmented reality enabled device of claim 16, wherein logic configured to determine whether the object has been selected further comprises:

logic configured to measure relative velocity between the object and the ARD; and

logic configured to measure amount of time the object is stationary with respect to the ARD.

23. The augmented reality enabled device of claim 16, wherein logic configured to determine whether the object has been selected further comprises:

logic configured to receive an audio command; and

logic configured to measure amplitude of the audio command.

24. The augmented reality enabled device of claim 16, wherein the set of selection criteria comprises at least one of:

the object being in view of the ARD for a predetermined period of time;

the object being in view of the ARD and a relative velocity between the object and the ARD being below a predetermined threshold; and

the object being in view of the ARD and amplitude of an audio command exceeding a predetermined value.

25. The augmented reality enabled device of claim 16, wherein logic configured to cause the augmentation to be rendered comprises:

logic configured to present interactions between the object and the ARD in a display of the ARD.

26. The augmented reality enabled device of claim 16, wherein the logic configured to identify comprises logic configured to determine that the object has been zoomed in on.

27. The augmented reality enabled device of claim 16, wherein the logic configured to determine comprises:

logic configured to cause an indicator to be presented, wherein the indicator indicating that the object has been identified; and

logic configured to determine whether the object is deselected in response to presentation of the indicator.

28. The augmented reality enabled device of claim 27, wherein logic configured to determine whether the object is deselected comprises:

logic configured to determine that the object has been selected if a position of the object with respect to the ARD is maintained for a threshold amount of time after presentation of the indicator.

29. The augmented reality enabled device of claim 27, wherein the indicator comprises at least one of:

an icon;

a vibration of the object on a display of the ARD;

a vibration of the ARD when the object is centered on the display of the ARD; and

a representation of an elapsed time.

30. The augmented reality enabled device of claim 16, further comprising:

logic configured to cause an image of a scene to be displayed on a display of the ARD, the scene including one or more objects;

logic configured to cause the image to be augmented with an indicator identifying an object of to be selected from the plurality of objects; and

logic configured to determine whether the object has been deselected subsequent to the image being augmented with the indicator.

31. A non-transitory medium storing instructions for execution by one or more computer systems, the instructions comprising:

instructions for receiving image data for tracking a plurality of objects;

instructions for identifying an object to be selected from the plurality of objects;

instructions for determining whether the object has been selected based at least in part on a set of selection criteria; and

instructions for causing an augmentation to be rendered with the object if it is determined that the object has been selected.

32. An apparatus, comprising:

means for receiving image data for tracking a plurality of objects;

means for identifying an object to be selected from the plurality of objects;

means for determining whether the object has been selected based at least in part on a set of selection criteria; and

means for causing an augmentation to be rendered with the object if it is determined that the object has been selected.

33. The apparatus of claim 32, wherein the tracking comprises 3-dimensional tracking comprising:

means for determining relative poses of the plurality of objects with respect to the ARD; and

means for updating states of the plurality of objects using the relative poses of the plurality of objects with respect to the ARD, wherein the states of the plurality of objects include relational information of the plurality of objects.

34. The apparatus of claim 32, wherein means for identifying an object to be selected comprises:

means for analyzing the plurality of objects using the image data; and

means for identifying the object being closest to a center of an image represented by the image data.

35. The apparatus of claim 32, wherein means for determining whether the object has been selected further comprises:

means for measuring relative velocity between the object and the ARD; and

means for measuring amount of time the object is stationary with respect to the ARD.

36. The apparatus of claim 32, wherein instructions for determining whether the object has been selected further comprises:

means for receiving an audio command; and

means for measuring amplitude of the audio command.

37. The apparatus of claim 32, wherein the set of selection criteria comprises at least one of:

the object being in view of the ARD for a predetermined period of time;

the object being in view of the ARD and a relative velocity between the object and the ARD being below a predetermined threshold; and

the object being in view of the ARD and amplitude of an audio command exceeding a predetermined value.

38. The apparatus of claim 32, wherein means for causing the augmentation to be rendered comprises:

means for presenting interactions between the object and the ARD in a display of the ARD.