Maintaining continuity of augmentations

• US 9,087,403 B2
• Filed: 03/15/2013
• Issued: 07/21/2015
• Est. Priority Date: 07/26/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Keating, Virginia Walker, Gervautz, Michael, Nielsen, Per O.
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US13/844,756
• Publication Number:
  US 20140028714A1
• Time in Patent Office: 858 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: H04N 21/4126 : The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/42208 : Display device provided on the remote control
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4852 : for modifying audio parameters, e.g. switching between mono and stereo
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Methods and apparatuses for maintaining continuity of augmentations are disclosed. In one embodiment, a method for use with an augmented reality enabled device (ARD) comprises tracking a plurality of objects and a background based at least in part on visual information derived from an image, maintaining states of the plurality of objects based at least in part on information other than the visual information, and providing data for rendering augmentation in response to the states of the plurality of objects.

Claims

1. A method for use with an augmented reality enabled device (ARD), comprising:

tracking a plurality of objects and a background based at least in part on visual information derived from an image;

maintaining states of the plurality of objects, wherein maintaining states of the plurality of objects comprises maintaining states of a first set of the plurality of objects in view of the ARD and maintaining states of a second set of the plurality of objects out of view of the ARD;

providing data for rendering augmentation in response to the states of the plurality of objects;

tracking at least one object in the second set of the plurality of objects out of view of the ARD;

determining the at least one object in the second set of the plurality of objects no longer exists; and

rendering at least one of a fading out transition and an ambient sound in a position of the at least one object in the second set of the plurality of objects.

2. The method of claim 1, wherein the tracking comprises performing 3-dimensional tracking comprising:

determining relative poses of the plurality of objects with respect to the ARD; and

updating states of the plurality of objects using the relative poses, wherein the states of the plurality of objects include relational information of the plurality of objects.

3. The method of claim 2, wherein determining relative poses comprises:

detecting a new object in the image; and

updating the plurality of objects to include the new object.

4. The method of claim 1, wherein the tracking comprises tracking at least one object of the plurality of objects using the visual information when the at least one object is within a field of view of the ARD, and

wherein the maintaining comprises maintaining the state of the at least one object using the information other than the visual information when the at least one object is out of the field of view.

5. The method of claim 1, wherein maintaining states of a second set of the plurality of objects out of view of the ARD comprises:

tracking offsets of the second set of the plurality of objects with respect to the first set of the plurality of objects in view of the ARD; and

determining positions of the second set of the plurality of objects using the offsets.

6. The method of claim 1, wherein maintaining states of a second set of the plurality of objects out of view of the ARD further comprises:

tracking relative movement of the ARD with respect to the second set of the plurality of objects out of view of the ARD; and

determining positions of the second set of the plurality of objects using position and relative movement of the ARD.

7. The method of claim 6, wherein tracking relative movement of the ARD is based at least in part on:

visual odometry;

dead reckoning with accelerometer;

dead reckoning with gyroscope;

or some combination thereof.

8. The method of claim 1, wherein maintaining states of a second set of the plurality of objects out of view of the ARD further comprises:

receiving information related to wireless signals for determining relative positions of the plurality of objects; and

updating positions of the second set of the plurality of objects using the information received.

9. The method of claim 8, wherein the wireless signals are received by the ARD from an RFID tag attached to at least one object in the second set of the plurality of objects.

10. The method of claim 8, wherein the wireless signals comprise at least one of near field communication signals and Bluetooth signals.

11. The method of claim 8, wherein the background comprises a mat including one or more sensors configured to detect the relative positions of the plurality of objects, and wherein the information is indicative of the relative positions of the plurality of objects detected by the one or more sensors.

12. The method of claim 8, wherein the information is received at a processor or chip integrated into the ARD based on the wireless signals being received at the ARD.

13. The method of claim 1, further comprising:

tracking at least one object in the second set of the plurality of objects out of view of the ARD;

determining the at least one object in the second set of the plurality of objects still exists; and

rendering at least one of sound and graphics in a position of the at least one object in the second set of the plurality of objects.

14. The method of claim 1, further comprising:

ceasing to track a first object in the second set when the ARD is panned to a location where the first object is expected to be located and it is determined that the first object is not present at the location; and

ceasing an audio augmentation associated with the first object.

15. The method of claim 1, further comprising:

ceasing to track a first object in the second set when a new scene is detected; and

ceasing an audio augmentation associated with the first object.

16. The method of claim 1, wherein rendering augmentation comprises:

rendering sound and graphics in a position when an indication of confidence of the states of the plurality of objects meets a first predetermined value;

rendering sound in the position when the indication of confidence of the states of the plurality of objects meets a second predetermined value;

rendering an ambient sound in the position when the indication of confidence of the states of the plurality of objects meets a third predetermined value;

rendering a fading out transition in the position when the indication of confidence of the states of the plurality of objects meets a fourth predetermined value;

or some combination thereof.

17. The method of claim 1, wherein the plurality of objects are game pieces and the background is a game board.

18. The method of claim 1, wherein the states of the plurality of objects comprise:

relational information of the plurality of objects with respect to each other;

relational information of the plurality of objects with respect to the background;

geometrical relationships of the plurality of objects with respect to each other;

geometrical relationships of the plurality of objects with respect to the background;

or some combination thereof.

19. The method of claim 1, further comprising:

tracking the plurality of objects and the background with multiple augmented reality enabled devices (ARDs);

maintaining states of the plurality of objects across the multiple ARDs; and

providing data for rendering augmentations in the multiple ARDs in response to the states of the plurality of objects.

20. The method of claim 1, wherein the background comprises:

a mat;

a wall;

or some combination thereof.

21. An augmented reality enabled device (ARD), comprising:

a control unit including processing logic, the processing logic comprising;

logic configured to track a plurality of objects and a background based at least in part on visual information derived from an image;

logic configured to maintain states of the plurality of objects, wherein logic configured to maintain states of the plurality of objects comprises logic configured to maintain states of a first set of the plurality of objects in view of the ARD and logic configured to maintain states of a second set of the plurality of objects out of view of the ARD;

logic configured to provide data for rendering augmentation in response to the states of the plurality of objects;

logic configured to track at least one object in the second set of the plurality of objects out of view of the ARD;

logic configured to determine the at least one object in the second set of the plurality of objects no longer exists; and

logic configured to render at least one of a fading out transition and an ambient sound in a position of the at least one object in the second set of the plurality of objects.

22. The augmented reality enabled device of claim 21, wherein the logic configured to track comprises performing 3-dimensional tracking comprising:

logic configured to determine relative poses of the plurality of objects with respect to the ARD; and

logic configured to update states of the plurality of objects using the relative poses, wherein the states of the plurality of objects include relational information of the plurality of objects.

23. The augmented reality enabled device of claim 22, wherein logic configured to determine relative poses comprises:

logic configured to detect poses of the plurality of objects with respect to a previously captured image of the plurality of objects.

24. The augmented reality enabled device of claim 22, wherein logic configured to determine relative poses comprises:

logic configured to detect a new object in the image; and

logic configured to update the plurality of objects to include the new object.

25. The augmented reality enabled device of claim 21, wherein logic configured to maintain states of a second set of the plurality of objects out of view of the ARD comprises:

logic configured to track offsets of the second set of the plurality of objects with respect to the first set of the plurality of objects in view of the ARD; and

logic configured to determine positions of the second set of the plurality of objects using the offsets.

26. The augmented reality enabled device of claim 21, wherein logic configured to maintain states of a second set of the plurality of objects out of view of the ARD further comprises:

logic configured to track relative movement of the ARD with respect to the second set of the plurality of objects out of view of the ARD; and

logic configured to determine positions of the second set of the plurality of objects using position and relative movement of the ARD.

27. The augmented reality enabled device of claim 26, wherein logic configured to track relative movement of the ARD is based at least in part on:

visual odometry;

dead reckoning with accelerometer;

dead reckoning with gyroscope;

or some combination thereof.

28. The augmented reality enabled device of claim 21, wherein logic configured to maintain states of a second set of the plurality of objects out of view of the ARD further comprises:

logic configured to receive information related to wireless signals for determining relative positions of the plurality of objects; and

logic configured to update positions of the second set of the plurality of objects using the information received.

29. The augmented reality enabled device of claim 28, wherein the wireless signals are received by the ARD from an RFID tag attached to at least one object in the second set of the plurality of objects.

30. The augmented reality enabled device of claim 28, wherein the wireless signals comprise at least one of near field communication signals and Bluetooth signals.

31. The augmented reality enabled device of claim 28, wherein the background comprises a mat including one or more sensors configured to detect the relative positions of the plurality of objects, and wherein the information is indicative of the relative positions of the plurality of objects detected by the one or more sensors.

32. The augmented reality enabled device of claim 28, wherein the information is received at a processor or chip integrated into the ARD based on the wireless signals being received at the ARD.

33. The augmented reality enabled device of claim 21, further comprising:

logic configured to track at least one object in the second set of the plurality of objects out of view of the ARD;

logic configured to determine the at least one object in the second set of the plurality of objects still exists; and

logic configured to render at least one of sound and graphics in a position of the at least one object in the second set of the plurality of objects.

34. The augmented reality enabled device of claim 21, further comprising:

logic configured to cease to track a first object in the second set when the ARD is panned to a location where the first object is expected to be located and it is determined that the first object is not present at the location; and

logic configured to cease an audio augmentation associated with the first object.

35. The augmented reality enabled device of claim 21, further comprising:

logic configured to cease to track a first object in the second set when a new scene is detected; and

logic configured to cease an audio augmentation associated with the first object.

36. The augmented reality enabled device of claim 21, wherein logic configured to render augmentation comprises:

logic configured to render sound and graphics in a position when an indication of confidence of the states of the plurality of objects meets a first predetermined value;

logic configured to render sound in the position when the indication of confidence of the states of the plurality of objects meets a second predetermined value;

logic configured to render an ambient sound in the position when the indication of confidence of the states of the plurality of objects meets a third predetermined value;

logic configured to render a fading out transition in the position when the indication of confidence of the states of the plurality of objects meets a fourth predetermined value;

or some combination thereof.

37. The augmented reality enabled device of claim 21, wherein the plurality of objects are game pieces and the background is a game board.

38. The augmented reality enabled device of claim 21, wherein the states of the plurality of objects comprise:

relational information of the plurality of objects with respect to each other;

relational information of the plurality of objects with respect to the background;

geometrical relationships of the plurality of objects with respect to each other;

geometrical relationships of the plurality of objects with respect to the background;

or some combination thereof.

39. The augmented reality enabled device of claim 21, further comprising:

logic configured to track the plurality of objects and the background with multiple augmented reality enabled devices (ARDs);

logic configured to maintain states of the plurality of objects across the multiple ARDs; and

logic configured to provide data for rendering augmentations in the multiple ARDs in response to the states of the plurality of objects.

40. The augmented reality enabled device of claim 21, wherein the background comprises:

a mat;

a wall;

or some combination thereof.

41. A non-transitory medium storing instructions for execution by one or more computer systems, the instructions comprising:

instructions for tracking a plurality of objects and a background based at least in part on visual information derived from an image;

instructions for maintaining states of the plurality of objects, wherein the instructions for maintaining states of the plurality of objects comprises instructions for maintaining states of a first set of the plurality of objects in view of the ARD and instructions for maintaining states of a second set of the plurality of objects out of view of the ARD;

instructions for providing data for rendering augmentation in response to the states of the plurality of objects;

instructions for tracking at least one object in the second set of the plurality of objects out of view of the ARD;

instructions for determining the at least one object in the second set of the plurality of objects no longer exists; and

instructions for rendering at least one of a fading out transition and an ambient sound in a position of the at least one object in the second set of the plurality of objects.

42. A system, comprising:

means for tracking a plurality of objects and a background based at least in part on visual information derived from an image;

means for maintaining states of the plurality of objects, wherein the means for maintaining states of the plurality of objects comprises means for maintaining states of a first set of the plurality of objects in view of the ARD and means for maintaining states of a second set of the plurality of objects out of view of the ARD;

means for providing data for rendering augmentation in response to the states of the plurality of objects;

means for tracking at least one object in the second set of the plurality of objects out of view of the ARD;

means for determining the at least one object in the second set of the plurality of objects no longer exists; and

means for rendering at least one of a fading out transition and an ambient sound in a position of the at least one object in the second set of the plurality of objects.