Interactions of tangible and augmented reality objects

• US 9,361,730 B2
• Filed: 03/15/2013
• Issued: 06/07/2016
• Est. Priority Date: 07/26/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Keating, Virginia Walker, Gervautz, Michael, Nielsen, Per O.
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US13/844,589
• Publication Number:
  US 20140028713A1
• Time in Patent Office: 1,180 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: H04N 21/4126 : The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/42208 : Display device provided on the remote control
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4852 : for modifying audio parameters, e.g. switching between mono and stereo
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Method, computer program product, and apparatus for providing interactions of tangible and augmented reality objects are disclosed. In one embodiment, a method for use with an augmented reality enabled device (ARD) comprises performing 3-dimensional tracking of one or more objects based at least in part on captured images of the one or more objects, detecting a state change of at least one object of the one or more objects based at least in part on the captured images, and causing an augmentation to be rendered in response to the state change of the at least one object, where a type of the augmentation is based at least in part on the state change of the at least one object.

Claims

1. A method for use with an augmented reality enabled device (ARD), comprising:

performing 3-dimensional tracking of one or more objects, by a control unit supported by one or more cameras and/or sensors of the ARD, based at least in part on captured images of the one or more objects;

detecting a state change of at least one object of the one or more objects based at least in part on the captured images using the control unit, wherein states of the plurality of objects include relational information of the plurality of objects; and

causing an augmentation to be rendered by the control unit in response to the state change of the at least one object, wherein a type of the augmentation is based at least in part on the state change of the at least one object, and wherein the augmentation comprises an interactive audio prompt from the at least one object via a speaker of the ARD device.

2. The method of claim 1, wherein the one or more objects comprise a plurality of objects, and wherein performing 3-dimensional tracking comprises:

determining relative poses of the plurality of objects with respect to the ARD; and

updating states of the plurality of objects using the relative poses of the plurality of objects.

3. The method of claim 2, wherein determining relative poses comprises:

detecting poses of the plurality of objects with respect to a previously captured image of the plurality of objects.

4. The method of claim 2, wherein determining relative poses comprises:

detecting a new object in an image; and

updating the plurality of objects to include the new object.

5. The method of claim 2, wherein the relational information comprises:

relationships among the plurality of objects in a game;

relationships among the plurality of objects in a story;

relationships between the plurality of objects with respect to a background;

or some combination thereof.

6. The method of claim 1, wherein detecting a state change of at least one object comprises:

detecting a position change of the at least one object relative to the ARD;

detecting an orientation change of the at least one object relative to the ARD;

detecting a position change between at least two objects;

detecting change of constellations of a plurality of objects;

or some combination thereof.

7. The method of claim 6, wherein detecting the position change of the at least one object relative to the ARD comprises:

detecting the at least one object has been translated relative to the ARD;

detecting the at least one object has reached a first predetermined distance to the ARD;

detecting the at least one object has gone through a first predefined gesture motion relative to the ARD;

or some combination thereof.

8. The method of claim 6, wherein detecting the orientation change of the at least one object relative to the ARD comprises:

detecting the at least one object has been rotated relative to the ARD.

9. The method of claim 6, wherein detecting the position change between at least two objects comprises:

detecting the at least two objects have been moved relative to each other;

detecting the at least two objects have reached a second predetermined distance to each other;

detecting one of the at least two objects has gone through a second predefined gesture motion relative to the other;

or some combination thereof.

10. The method of claim 1, wherein the type of the augmentation comprises an interaction between a plurality of objects in a display of the ARD.

11. The method of claim 1, wherein the type of the augmentation comprises an interaction between the at least one object and the ARD in a display of the ARD.

12. The method of claim 1, wherein the type of the augmentation comprises a depiction of one or more elements having a focus that is based at least in part on the state change of the at least one object.

13. The method of claim 1, wherein the state change of the at least one object comprises:

change in geometrical relationship between the ARD and the at least one object;

change in geometrical relationship between the at least one object and another object in a field of view of a camera transmitting the captured images;

or some combination thereof.

14. The method of claim 1, wherein the state change comprises a change in a feature of the at least one object.

15. The method of claim 1, wherein the at least one object comprises a plurality of sub-objects linked together, wherein the state change comprises a change of at least one sub-object of the plurality of sub-objects.

16. The method of claim 15, wherein the at least one object comprises a car and the plurality of sub-objects comprise at least a door, hood, and trunk, wherein the method further comprises:

upon detecting the door of the a car being opened, presenting augmentations relating to entering the car;

upon detecting the hood of the car being opened, presenting augmentations relating to checking engine of the car;

upon detecting the hood of the car being opened, presenting augmentations relating to fixing engine of the car; and

upon detecting the trunk of the car being opened, presenting augmentations relating to loading the car.

17. The method of claim 1, wherein the augmentation comprises:

an interaction based at least in part on view of the at least one object from the ARD;

an interaction based at least in part on relative movements between the at least one object and the ARD;

an interaction based at least in part on a distance between the at least one object and the ARD;

an interaction based at least in part on a gesture from the at least one object;

or some combination thereof.

18. The method of claim 1, wherein the augmentation comprises:

an interaction based at least in part on object-on-object in a plurality of objects;

an interaction based at least in part on object-besides-object in the plurality of objects;

an interaction based at least in part on object-above-object in the plurality of objects;

an interaction based at least in part on object-in-object in the plurality of objects;

an interaction based at least in part on object-behind-object in the plurality of objects;

or some combination thereof.

19. The method of claim 1, wherein detecting a state change of at least one object comprises:

detecting zooming in on the at least one object with respect to the ARD;

detecting zooming out of the at least one object with respect to the ARD;

or some combination thereof.

20. The method of claim 1, wherein:

the state change comprises a change in a distance of the at least one object to the ARD, and wherein the augmentation comprises a character associated with the at least one object speaking words that are indicative of the distance;

or

the state change comprises the at least one object being rotated to face another object, and wherein the augmentation comprises a character associated with the at least one object speaking words to a character associated with the another object in response to the at least one object being rotated to face another object.

21. An augmented reality enabled device (ARD), comprising:

a control unit including processing logic, the processing logic comprising;

logic configured to perform 3-dimensional tracking of one or more objects based at least in part on captured images of the one or more objects;

logic configured to detect a state change of at least one object of the one or more objects based at least in part on the captured images, wherein states of the plurality of objects include relational information of the plurality of objects; and

logic configured to cause an augmentation to be rendered in response to the state change of the at least one object, wherein a type of the augmentation is based at least in part on the state change of the at least one object, and wherein the augmentation comprises an interactive audio prompt from the at least one object via a speaker of the ARD device.

22. The augmented reality enabled device of claim 21, wherein the one or more objects comprise a plurality of objects, and wherein logic configured to perform 3-dimensional tracking comprises:

logic configured to determine relative poses of the plurality of objects with respect to the ARD; and

logic configured to update states of the plurality of objects using the relative poses of the plurality of objects.

23. The augmented reality enabled device of claim 22, wherein logic configured to determine relative poses comprises:

logic configured to detect poses of the plurality of objects with respect to a previously captured image of the plurality of objects.

24. The augmented reality enabled device of claim 22, wherein logic configured to determine relative poses comprises:

logic configured to detect a new object in an image; and

logic configured to update the plurality of objects to include the new object.

25. The augmented reality enabled device of claim 22, wherein the relational information comprises:

relationships among the plurality of objects in a game;

relationships among the plurality of objects in a story;

relationships between the plurality of objects with respect to a background;

or some combination thereof.

26. The augmented reality enabled device of claim 21, wherein logic configured to detect a state change of at least one object comprises:

logic configured to detect a position change of the at least one object relative to the ARD;

logic configured to detect an orientation change of the at least one object relative to the ARD;

logic configured to detect a position change between at least two objects;

logic configured to detect change of constellations of a plurality of objects;

or some combination thereof.

27. The augmented reality enabled device of claim 26, wherein logic configured to detect the position change of the at least one object relative to the ARD comprises:

logic configured to detect the at least one object has been translated relative to the ARD;

logic configured to detect the at least one object has reached a first predetermined distance to the ARD;

logic configured to detect the at least one object has gone through a first predefined gesture motion relative to the ARD;

or some combination thereof.

28. The augmented reality enabled device of claim 26, wherein logic configured to detect the orientation change of the at least one object relative to the ARD comprises:

logic configured to detect the at least one object has been rotated relative to the ARD.

29. The augmented reality enabled device of claim 26, wherein logic configured to detect the position change between at least two objects comprises:

logic configured to detect the at least two objects have been moved relative to each other;

logic configured to detect the at least two objects have reached a second predetermined distance to each other;

logic configured to detect one of the at least two objects has gone through a second predefined gesture motion relative to the other;

or some combination thereof.

30. The augmented reality enabled device of claim 21, wherein the type of the augmentation comprises an interaction between a plurality of objects in a display of the ARD.

31. The augmented reality enabled device of claim 21, wherein the type of the augmentation comprises an interaction between the at least one object and the ARD in a display of the ARD.

32. The augmented reality enabled device of claim 21, wherein the type of the augmentation comprises a depiction of one or more elements having a focus that is based at least in part on the state change of the at least one object.

33. The augmented reality enabled device of claim 21, wherein the state change of the at least one object comprises:

change in geometrical relationship between the ARD and the at least one object;

change in geometrical relationship between the at least one object and another object in a field of view of a camera transmitting the captured images;

or some combination thereof.

34. The augmented reality enabled device of claim 21, wherein the state change comprises a change in a feature of the at least one object.

35. The augmented reality enabled device of claim 21, wherein the at least one object comprises a plurality of sub-objects linked together, wherein the state change comprises a change of at least one sub-object of the plurality of sub-objects.

36. The augmented reality enabled device of claim 21, wherein logic configured to detect a state change of at least one object comprises:

logic configured to detect zooming in on the at least one object with respect to the ARD;

logic configured to detecting zoom out of the at least one object with respect to the ARD;

or some combination thereof.

37. The augmented reality enabled device of claim 21, wherein:

the state change comprises a change in a distance of the at least one object to the ARD, and wherein the augmentation comprises a character associated with the at least one object speaking words that are indicative of the distance;

or

the state change comprises the at least one object being rotated to face another object, and wherein the augmentation comprises a character associated with the at least one object speaking words to a character associated with the another object in response to the at least one object being rotated to face another object.

38. A non-transitory medium storing instructions for execution by one or more computer systems, the instructions comprising:

instructions for performing 3-dimensional tracking of one or more objects based at least in part on captured images of the one or more objects;

instructions for detecting a state change of at least one object of the one or more objects based at least in part on the captured images, wherein states of the plurality of objects include relational information of the plurality of objects; and

instructions for causing an augmentation to be rendered in response to the state change of the at least one object, wherein a type of the augmentation is based at least in part on the state change of the at least one object, and wherein the augmentation comprises an interactive audio prompt from the at least one object via a speaker of the ARD device.

39. An apparatus, comprising:

means for performing 3-dimensional tracking of one or more objects based at least in part on captured images of the one or more objects;

means for detecting a state change of at least one object of the one or more objects based at least in part on the captured images, wherein states of the plurality of objects include relational information of the plurality of objects; and

means for causing an augmentation to be rendered in response to the state change of the at least one object, wherein a type of the augmentation is based at least in part on the state change of the at least one object, and wherein the augmentation comprises an interactive audio prompt from the at least one object via a speaker of the ARD device.

40. The apparatus of claim 39, wherein the one or more objects comprise a plurality of objects, and wherein means for performing 3-dimensional tracking comprises:

means for determining relative poses of the plurality of objects with respect to the ARD; and

means for updating states of the plurality of objects using the relative poses of the plurality of objects.

41. The apparatus of claim 40, wherein means for determining relative poses comprises:

means for detecting poses of the plurality of objects with respect to a previously captured image of the plurality of objects.

42. The apparatus of claim 40, wherein determining relative poses comprises:

means for detecting a new object in an image; and

means for updating the plurality of objects to include the new object.

43. The apparatus of claim 39, wherein means for detecting a state change of at least one object comprises:

means for detecting a position change of the at least one object relative to the ARD;

means for detecting an orientation change of the at least one object relative to the ARD;

means for detecting a position change between at least two objects;

means for detecting change of constellations of a plurality of objects;

or some combination thereof.

44. The apparatus of claim 43, wherein means for detecting the position change of the at least one object relative to the ARD comprises:

means for detecting the at least one object has been translated relative to the ARD;

means for detecting the at least one object has reached a first predetermined distance to the ARD;

means for detecting the at least one object has gone through a first predefined gesture motion relative to the ARD;

or some combination thereof.

45. The apparatus of claim 43, wherein means for detecting the orientation change of the at least one object relative to the ARD comprises:

means for detecting the at least one object has been rotated relative to the ARD.

46. The apparatus of claim 43, wherein means for detecting the position change between at least two objects comprises:

means for detecting the at least two objects have been moved relative to each other;

means for detecting the at least two objects have reached a second predetermined distance to each other;

means for detecting one of the at least two objects has gone through a second predefined gesture motion relative to the other;

or some combination thereof.

47. The apparatus of claim 39, wherein means for detecting a state change of at least one object comprises:

means for detecting zooming in on the at least one object with respect to the ARD;

means for detecting zooming out of the at least one object with respect to the ARD;

or some combination thereof.