ANCHORING VIRTUAL IMAGES TO REAL WORLD SURFACES IN AUGMENTED REALITY SYSTEMS

• US 20120249741A1
• Filed: 03/29/2012
• Published: 10/04/2012
• Est. Priority Date: 03/29/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: MACIOCCI, Giuliano, EVERITT, Andrew J., MABBUTT, Paul, BERRY, David T.
• Granted Patent: US 9,384,594 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 9/3173 wherein the projection device is specially adapted for enhanced portability

Abstract

A head mounted device provides an immersive virtual or augmented reality experience for viewing data and enabling collaboration among multiple users. Rendering images in a virtual or augmented reality system may include capturing an image and spatial data with a body mounted camera and sensor array, receiving an input indicating a first anchor surface, calculating parameters with respect to the body mounted camera and displaying a virtual object such that the virtual object appears anchored to the selected first anchor surface. Further operations may include receiving a second input indicating a second anchor surface within the captured image that is different from the first anchor surface, calculating parameters with respect to the second anchor surface and displaying the virtual object such that the virtual object appears anchored to the selected second anchor surface and moved from the first anchor surface.

Claims

1. A method for rendering virtual images in an augmented reality system, comprising:

capturing an image with a body mounted camera;

capturing spatial data with a body mounted sensor array;

recognizing an object within the captured image;

receiving a first user input indicating a first anchor surface in the captured image that corresponds to a first surface located in the image;

calculating parameters including distance and orientation with respect to the body mounted camera that correspond to the first anchor surface;

displaying a virtual object so the virtual object appears anchored to the selected first anchor surface;

receiving a second input indicating a second anchor surface within the captured image that is different from the first anchor surface, wherein the second anchor surface corresponds to a second surface located in the image and the first and second surfaces are different;

calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the second anchor surface; and

displaying the virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface.

2. The method of claim 1, further comprising continuously updating the display of the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface as the user turns his/her head and moves with respect to the selected first or second anchor surface.

3. The method of claim 1, wherein displaying the virtual object comprises displaying the virtual object on a head mounted display, the method further comprising outputting video images as if appearing on the head mounted display anchored to the selected first or second anchor surface.

4. The method of claim 1, wherein displaying the generated virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface comprises:

generating the virtual object as a rectangular shaped virtual object that resembles a flat screen display; and

displaying the generated virtual object on a semitransparent display with the virtual object superimposed on a real world surface visible through the semitransparent display.

5. The method of claim 1, wherein the first or second anchor surface comprises a wall in a room, and wherein the virtual object remains anchored on the wall in the room as the user moves about in the room.

6. The method of claim 1, wherein displaying the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface comprises generating the virtual object so that it appears to be contacting the first or second anchor surface and so that the first or second anchor surface appear to be a rigid point of support for the anchored virtual object.

7. The method of claim 1, wherein:

displaying the generated virtual object so the virtual object appears anchored to the selected anchor surface comprises generating the virtual object so that it appears to be spaced apart from the first or second anchor surface but hovering in a fixed position and spaced relative to the first or second anchor surface.

8. The method of claim 3, further comprising transmitting data comprising the captured spatial data, the generated virtual object and the image to a second head mounted display.

9. The method of claim 1, further comprising:

receiving audio from the user utilizing a microphone;

applying a detection algorithm to the received audio to detect a predetermined audio signal; and

implementing the predefined user input when the predetermined audio signal is detected.

10. The method of claim 9, wherein applying a detection algorithm to the received audio to detect a predetermined audio signal comprises applying a detection algorithm to the received audio to detect a predetermined audio signal spoken by the user corresponding to a predefined user input.

11. The method of claim 9, wherein the predefined user input corresponding to the detected predetermined audio signal is one of the first input, the second input or a confirmation of one of the first and second inputs.

12. The method of claim 1, further comprising:

capturing the image with the body mounted camera that is mounted and disposed together with a head mounted display; and

displaying the virtual object on the head mounted display.

13. The method of claim 12, wherein the body mounted camera and head mounted display comprise a body mounted device, the method further comprising:

establishing a wireless data link between the body mounted device and a second body mounted device;

receiving audio data from the second body mounted device via the wireless data link; and

using the received audio data to output audio to the user.

14. The method of claim 13, further comprising:

receiving video data via the wireless data link; and

using the received video data to output video images on the head mounted display received from the second body mounted device,

wherein the video displays an anchored virtual object.

15. The method of claim 14, wherein receiving video data via the wireless data link comprises receiving one or more of a three dimensional model data for a virtual object, three dimensional data for an anchoring surface, three dimensional data for an anchoring point, three dimensional depth data, and three dimensional distance data.

16. The method of claim 1, further comprising:

receiving a third input indicating a standby location for the virtual object, wherein the standby location is different from the first and the second anchor surfaces;

calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the standby location; and

displaying the virtual object at the standby location.

17. The method of claim 16, wherein the standby location is displayed as free floating on a display, wherein the virtual object appears on the display as movable and unconnected to surfaces in the image as the user turns his/her head and moves.

18. The method of claim 1, wherein:

capturing an image with a body mounted camera comprises capturing video data by a full-color video camera; and

capturing spatial data with a body mounted sensor array comprises capturing spatial data utilizing a distance sensor to determine distances to objects in the image.

19. The method of claim 18, wherein capturing spatial data with a body mounted sensor array comprises capturing distance information of objects in the image using an infrared laser and an infrared sensor.

20. The method of claim 18, further comprising building a three dimensional map based on the captured video data and the calculated spatial data.

21. The method of claim 1, further comprising:

tracking movements of the user utilizing the body mounted camera;

applying a detection algorithm to the tracked user movements to detect a predetermined gesture; and

executing a command corresponding to the predetermined gesture when the predetermined gesture is detected.

22. The method of claim 21, wherein the detected predetermined gesture is selected from the group consisting of an “

OK”

sign, a fist, an open hand, pointing with one finger, pointing with two fingers, pointing with three or four fingers, an outstretched hand, a hand rotation, a wave, a movement of one or more fingers, a movement of a body part or a foot, and any combination thereof.

23. The method of claim 21, wherein the predetermined gesture is retrieved from a gesture dictionary.

24. The method of claim 23, wherein the gesture dictionary defines one or more of:

a poke gesture, a pat gesture, or a tap gesture for target selection or toggling;

a push gesture for guiding a screen object across a screen;

a flick gesture for imparting momentum to the screen object;

a turn gesture for rotating the screen object;

a grab gesture or a pull gesture for zoom operations;

a two hand move gesture for panning operations;

a drawing hands apart gesture for resizing the screen object; and

a swipe gesture for horizontal scrolling or shuttling through media.

25. The method of claim 21, wherein:

receiving the first input indicating a first anchor surface in the captured image comprises detecting a first predetermined gesture made by the wearer corresponding to a command to designate a surface within the captured image as an anchor surface and applying the command; and

receiving the second input to anchor the virtual object in the image comprises detecting a second predetermined gesture made by the wearer corresponding to a command to move the anchor surface to another surface in the captured image and applying the command.

26. The method of claim 25, wherein either of the first or second detected predetermined gestures comprises the user pointing to a surface in the captured image.

27. The method of claim 25, further comprising:

continuing tracking movements of the user after the first input is received;

applying a detection algorithm to the tracked movements to detect a third predetermined gesture corresponding to a user input confirming the first input; and

anchoring the virtual object to the selected anchor surface in response to the third predetermined gesture.

28. A computing device, comprising:

means for capturing an image from a body mounted camera;

means for capturing spatial data from a body mounted sensor array;

means for recognizing an object within the captured image;

means for receiving a first user input indicating a first anchor surface in a captured image, wherein the first anchor surface corresponds to a first surface located in the image;

means for calculating parameters including distance and orientation with respect to the body mounted camera that correspond to the first anchor surface;

means for displaying a virtual object so the virtual object appears anchored to the

selected first anchor surface;

means for receiving a second input indicating a second anchor surface within the captured image that is different from the first anchor surface, wherein the second anchor surface corresponds to a second surface located in the image and the first and second surfaces are different;

means for calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the second anchor surface; and

means for displaying the virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface.

29. The computing device of claim 28, further comprising means for continuously updating the display of the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface as the user turns his/her head and moves with respect to the selected first or second anchor surface.

30. The computing device of claim 28, wherein means for displaying the virtual object comprises means for displaying the virtual object on a head mounted display, the computing device further comprising:

means for outputting video images as if appearing on the head mounted display anchored to the selected first or second anchor surface.

31. The computing device of claim 28, wherein means for displaying the generated virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface comprises:

means for generating the virtual object as a rectangular shaped virtual object that resembles a flat screen display; and

means for displaying the generated virtual object on a semitransparent display with the virtual object superimposed on a real world surface visible through the semitransparent display.

32. The computing device of claim 28, further comprising:

means for selecting the first or second anchor surface to include a wall in a room; and

means for displaying the virtual object such that the virtual object remains anchored on the wall in the room as the user moves about in the room.

33. The computing device of claim 28, wherein means for displaying the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface comprises means for generating the virtual object so that it appears to be contacting the first or second anchor surface and so that the first or second anchor surface appear to be a rigid point of support for the anchored virtual object.

34. The computing device of claim 28, wherein means for displaying the generated virtual object so the virtual object appears anchored to the selected anchor surface comprises:

means for generating the virtual object so that it appears to be spaced apart from the first or second anchor surface but hovering in a fixed position and spaced relative to the first or second anchor surface.

35. The computing device of claim 30, further comprising:

means for transmitting data comprising the captured spatial data, the generated virtual object and the image to a second head mounted display.

36. The computing device of claim 28, further comprising:

means for receiving audio from the user utilizing a microphone;

means for applying a detection algorithm to the received audio to detect a predetermined audio signal; and

means for implementing the predefined user input when the predetermined audio signal is detected.

37. The computing device of claim 36, wherein means for applying a detection algorithm to the received audio to detect a predetermined audio signal comprises means for applying a detection algorithm to the received audio to detect a predetermined audio signal spoken by the user corresponding to a predefined user input.

38. The computing device of claim 36, wherein means for applying a detection algorithm to the received audio to detect a predetermined audio signal comprises means for detecting a predetermined audio signal that is one of the first input, the second input or a confirmation of one of the first and second inputs.

39. The computing device of claim 28, further comprising:

means for capturing the image with the body mounted camera that is mounted and disposed together with a head mounted display; and

means for displaying the virtual object on the head mounted display.

40. The computing device of claim 39, further comprising:

means for establishing a wireless data link between a body mounted device and another body mounted device;

means for receiving audio data from the second body mounted device via the wireless data link; and

means for using the received audio data to output audio to the user.

41. The computing device of claim 40, further comprising:

means for receiving video data via the wireless data link; and

means for using the received video data to output video images on the head mounted display received from the second body mounted device, wherein the video displays an anchored virtual object.

42. The computing device of claim 41, wherein means for receiving video data via the wireless data link comprises means for receiving one or more of a three dimensional model data for a virtual object, three dimensional data for an anchoring surface, three dimensional data for an anchoring point, three dimensional depth data, and three dimensional distance data.

43. The computing device of claim 28, further comprising:

means for receiving a third input indicating a standby location for the virtual object, wherein the standby location is different from the first and the second anchor surfaces;

means for calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the standby location; and

means for displaying the virtual object at the standby location.

44. The computing device of claim 43, further comprising means for displaying the standby location as free floating on a display such that the virtual object appears on the display as movable and unconnected to surfaces in the image as the user turns his/her head and moves.

45. The computing device of claim 28, wherein:

means for capturing an image with a body mounted camera mounted on a head of a user comprises means for capturing video data by a full-color video camera; and

means for capturing spatial data with a body mounted sensor array comprises means for capturing spatial data utilizing a distance sensor to determine distances to objects in the image.

46. The computing device of claim 45, wherein means for capturing spatial data with a body mounted sensor array comprises means for capturing distance information of objects in the image using an infrared laser and an infrared sensor.

47. The computing device of claim 45, further comprising means for building a three dimensional map based on the captured video data and the calculated spatial data.

48. The computing device of claim 28, further comprising:

means for tracking movements of the user utilizing the body mounted camera;

means for applying a detection algorithm to the tracked user movements to detect a predetermined gesture; and

means for executing a command corresponding to the predetermined gesture when the predetermined gesture is detected.

49. The computing device of claim 48, further comprising means for selecting the detected predetermined gesture from the group consisting of an “

OK”

sign, a fist, an open hand, pointing with one finger, pointing with two fingers, pointing with three or four fingers, an outstretched hand, a hand rotation, a wave, a movement of one or more fingers, a movement of a body part or a foot, and any combination thereof.

50. The computing device of claim 48, further comprising means for retrieving the predetermined gesture from a gesture dictionary.

51. The computing device of claim 50, wherein means for retrieving the predetermined gesture from a gesture dictionary comprises means for retrieving the predetermined gesture from a gesture dictionary that defines one or more of:

a poke gesture, a pat gesture, or a tap gesture for target selection or toggling;

a push gesture for guiding a screen object across a screen;

a flick gesture for imparting momentum to the screen object;

a turn gesture for rotating the screen object;

a grab gesture or a pull gesture for zoom operations;

a two hand move gesture for panning operations;

a drawing hands apart gesture for resizing the screen object; and

a swipe gesture for horizontal scrolling or shuttling through media.

52. The computing device of claim 48, wherein:

means for receiving the first input indicating a first anchor surface in the captured image comprises means for detecting a first predetermined gesture made by the wearer corresponding to a command to designate a surface within the captured image as an anchor surface and applying the command; and

means for receiving the second input to anchor the virtual object in the image comprises means for detecting a second predetermined gesture made by the wearer corresponding to a command to move the anchor surface to another surface in the captured image and applying the command.

53. The computing device of claim 52, further comprising means for selecting either of the first or second detected predetermined gestures from the user pointing to a surface in the captured image.

54. The computing device of claim 52, further comprising:

means for continuing tracking movements of the user after the first input is received;

means for applying a detection algorithm to the tracked movements to detect a third predetermined gesture corresponding to a user input confirming the first input; and

means for anchoring the virtual object to the selected anchor surface in response to the third predetermined gesture.

55. A computing device, comprising:

a transceiver;

a memory; and

a processor coupled to the transceiver and the memory, wherein the processor is configured with processor-executable instructions to perform operations comprising;

capturing an image with a body mounted camera;

capturing spatial data with a body mounted sensor array;

recognizing an object within the captured image;

receiving a first user input indicating a first anchor surface in a captured image, wherein the first anchor surface corresponds to a first surface located in the image;

calculating parameters including distance and orientation with respect to the body mounted camera that correspond to the first anchor surface;

displaying a virtual object so the virtual object appears anchored to the selected first anchor surface;

receiving a second input indicating a second anchor surface within the captured image that is different from the first anchor surface, wherein the second anchor surface corresponds to a second surface located in the image and the first and second surfaces are different;

calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the second anchor surface; and

displaying the virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface.

56. The computing device of claim 55, wherein the processor is configured with processor-executable instructions to perform operations further comprising continuously updating the display of the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface as the user turns his/her head and moves with respect to the selected first or second anchor surface.

57. The computing device of claim 55, wherein the processor is configured with processor-executable instructions such that displaying the virtual object comprises displaying the virtual object on a head mounted display, and wherein the processor is configured with processor-executable instructions to perform operations further comprising:

outputting video images as if appearing on the head mounted display anchored to the selected first or second anchor surface.

58. The computing device of claim 55, wherein the processor is configured with processor-executable instructions such that displaying the generated virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface comprises:

generating the virtual object as a rectangular shaped virtual object that resembles a flat screen display; and

displaying the generated virtual object on a semitransparent display with the virtual object superimposed on a real world surface visible through the semitransparent display.

59. The computing device of claim 55, wherein the processor is configured with processor-executable instructions such that the first or second anchor surface are selected to be a wall in a room and the virtual object is displayed such that the virtual object remains anchored on the wall in the room as the user moves about in the room.

60. The computing device of claim 55, wherein the processor is configured with processor-executable instructions such that displaying the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface comprises generating the virtual object so that it appears to be contacting the first or second anchor surface and so that the first or second anchor surface appear to be a rigid point of support for the anchored virtual object.

61. The computing device of claim 55, wherein the processor is configured with processor-executable instructions such that displaying the generated virtual object so the virtual object appears anchored to the selected anchor surface comprises generating the virtual object so that it appears to be spaced apart from the first or second anchor surface but hovering in a fixed position and spaced relative to the first or second anchor surface.

62. The computing device of claim 57, wherein the processor is configured with processor-executable instructions to perform operations further comprising transmitting data comprising the captured spatial data, the generated virtual object and the image to a second head mounted display.

63. The computing device of claim 55, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

receiving audio from the user utilizing a microphone;

applying a detection algorithm to the received audio to detect a predetermined audio signal; and

implementing the predefined user input when the predetermined audio signal is detected.

64. The computing device of claim 63, wherein the processor is configured with processor-executable instructions such that applying a detection algorithm to the received audio to detect a predetermined audio signal comprises applying a detection algorithm to the received audio to detect a predetermined audio signal spoken by the user corresponding to a predefined user input.

65. The computing device of claim 63, wherein the processor is configured with processor-executable instructions such that the predefined user input corresponding to the detected predetermined audio signal is one of the first input, the second input or a confirmation of one of the first and second inputs.

66. The computing device of claim 55, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

capturing the image with the body mounted camera that is mounted and disposed together with a head mounted display; and

displaying the virtual object on the head mounted display.

67. The computing device of claim 66, wherein the processor is configured with processor-executable instructions such that the body mounted camera and head mounted display comprise a body mounted device, and wherein the processor is configured with processor-executable instructions to perform operations further comprising:

establishing a wireless data link between the body mounted device and another body mounted device;

receiving audio data from the second body mounted device via the wireless data link; and

using the received audio data to output audio to the user.

68. The computing device of claim 67, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

receiving video data via the wireless data link; and

using the received video data to output video images on the head mounted display received from the second body mounted device, wherein the video displays an anchored virtual object.

69. The computing device of claim 68, wherein the processor is configured with processor-executable instructions such that receiving video data via the wireless data link comprises receiving one or more of a three dimensional model data for a virtual object, three dimensional data for an anchoring surface, three dimensional data for an anchoring point, three dimensional depth data, and three dimensional distance data.

70. The computing device of claim 55, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

receiving a third input indicating a standby location for the virtual object, wherein the standby location is different from the first and the second anchor surfaces;

calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the standby location; and

displaying the virtual object at the standby location.

71. The computing device of claim 70, wherein the processor is configured with processor-executable instructions such that the standby location is displayed as free floating on a display, wherein the virtual object appears on the display as movable and unconnected to surfaces in the image as the user turns his/her head and moves.

72. The computing device of claim 55, wherein the processor is configured with processor-executable instructions such that:

capturing an image with a body mounted camera mounted on a head of a user comprises capturing video data by a full-color video camera; and

capturing spatial data with a body mounted sensor array comprises capturing spatial data utilizing a distance sensor to determine distances to objects in the image.

73. The computing device of claim 72, wherein the processor is configured with processor-executable instructions such that capturing spatial data with a body mounted sensor array comprises capturing distance information of objects in the image using an infrared laser and an infrared sensor.

74. The computing device of claim 72, wherein the processor is configured with processor-executable instructions to perform operations further comprising building a three dimensional map based on the captured video data and the calculated spatial data.

75. The computing device of claim 55, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

tracking movements of the user utilizing the body mounted camera;

applying a detection algorithm to the tracked user movements to detect a predetermined gesture; and

executing a command corresponding to the predetermined gesture when the predetermined gesture is detected.

76. The computing device of claim 75, wherein the processor is configured with processor-executable instructions such that the detected predetermined gesture is selected from the group consisting of an “

OK”

sign, a fist, an open hand, pointing with one finger, pointing with two fingers, pointing with three or four fingers, an outstretched hand, a hand rotation, a wave, a movement of one or more fingers, a movement of a body part or a foot, and any combination thereof.

77. The computing device of claim 75, wherein the processor is configured with processor-executable instructions such that applying a detection algorithm to the tracked user movements to detect a predetermined gesture comprises retrieving the predetermined gesture from a gesture dictionary.

78. The computing device of claim 77, wherein the processor is configured with processor-executable instructions such that retrieving the predetermined gesture from a gesture dictionary comprises retrieving the predetermined gesture from a gesture dictionary that defines one or more of:

a poke gesture, a pat gesture, or a tap gesture for target selection or toggling;

a push gesture for guiding a screen object across a screen;

a flick gesture for imparting momentum to the screen object;

a turn gesture for rotating the screen object;

a grab gesture or a pull gesture for zoom operations;

a two hand move gesture for panning operations;

a drawing hands apart gesture for resizing the screen object; and

a swipe gesture for horizontal scrolling or shuttling through media.

79. The computing device of claim 75, wherein the processor is configured with processor-executable instructions such that:

receiving the first input indicating a first anchor surface in the captured image comprises detecting a first predetermined gesture made by the wearer corresponding to a command to designate a surface within the captured image as an anchor surface and applying the command; and

receiving the second input to anchor the virtual object in the image comprises detecting a second predetermined gesture made by the wearer corresponding to a command to move the anchor surface to another surface in the captured image and applying the command.

80. The computing device of claim 79, wherein the processor is configured with processor-executable instructions such that either of the first or second detected predetermined gestures comprises the user pointing to a surface in the captured image.

81. The computing device of claim 79, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

continuing tracking movements of the user after the first input is received;

applying a detection algorithm to the tracked movements to detect a third predetermined gesture corresponding to a user input confirming the first input; and

anchoring the virtual object to the selected anchor surface in response to the third predetermined gesture.

82. A non-transitory computer readable storage medium having stored thereon processor-executable software instructions configured to cause a processor to perform operations for rendering virtual images in an augmented reality system, the operations comprising:

capturing an image with a body mounted camera;

capturing spatial data with a body mounted sensor array;

recognizing an object within the captured image;

receiving a first user input indicating a first anchor surface in a captured image, wherein the first anchor surface corresponds to a first surface located in the image;

calculating parameters including distance and orientation with respect to the body mounted camera that correspond to the first anchor surface;

displaying a virtual object so the virtual object appears anchored to the selected first anchor surface;

receiving a second input indicating a second anchor surface within the captured image that is different from the first anchor surface, wherein the second anchor surface corresponds to a second surface located in the image and the first and second surfaces are different;

calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the second anchor surface; and

displaying the virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface.

83. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising continuously updating the display of the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface as the user turns his/her head and moves with respect to the selected first or second anchor surface.

84. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that displaying the virtual object comprises displaying the virtual object on a head mounted display, and wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

outputting video images as if appearing on the head mounted display anchored to the selected first or second anchor surface.

85. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that displaying the generated virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface comprises:

generating the virtual object as a rectangular shaped virtual object that resembles a flat screen display; and

displaying the generated virtual object on a semitransparent display with the virtual object superimposed on a real world surface visible through the semitransparent display.

86. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that the first or second anchor surface are selected to be a wall in a room and the virtual object is displayed such that the virtual object remains anchored on the wall in the room as the user moves about in the room.

87. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that displaying the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface comprises generating the virtual object so that it appears to be contacting the first or second anchor surface and so that the first or second anchor surface appear to be a rigid point of support for the anchored virtual object.

88. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that displaying the generated virtual object so the virtual object appears anchored to the selected anchor surface comprises generating the virtual object so that it appears to be spaced apart from the first or second anchor surface but hovering in a fixed position and spaced relative to the first or second anchor surface.

89. The non-transitory computer readable storage medium of claim 84, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising transmitting data comprising the captured spatial data, the generated virtual object and the image to a second head mounted display.

90. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

receiving audio from the user utilizing a microphone;

applying a detection algorithm to the received audio to detect a predetermined audio signal; and

implementing the predefined user input when the predetermined audio signal is detected.

91. The non-transitory computer readable storage medium of claim 90, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that applying a detection algorithm to the received audio to detect a predetermined audio signal comprises applying a detection algorithm to the received audio to detect a predetermined audio signal spoken by the user corresponding to a predefined user input.

92. The non-transitory computer readable storage medium of claim 90, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that the predefined user input corresponding to the detected predetermined audio signal is one of the first input, the second input or a confirmation of one of the first and second inputs.

93. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

capturing the image with the body mounted camera that is mounted and disposed together with a head mounted display; and

displaying the virtual object on the head mounted display.

94. The non-transitory computer readable storage medium of claim 93, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that the body mounted camera and head mounted display comprise a body mounted device, and wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

establishing a wireless data link between the body mounted device and another body mounted device;

receiving audio data from the second body mounted device via the wireless data link; and

using the received audio data to output audio to the user.

95. The non-transitory computer readable storage medium of claim 94, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

receiving video data via the wireless data link; and

using the received video data to output video images on the head mounted display received from the second body mounted device, wherein the video displays an anchored virtual object.

96. The non-transitory computer readable storage medium of claim 95, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that receiving video data via the wireless data link comprises receiving one or more of a three dimensional model data for a virtual object, three dimensional data for an anchoring surface, three dimensional data for an anchoring point, three dimensional depth data, and three dimensional distance data.

97. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

receiving a third input indicating a standby location for the virtual object, wherein the standby location is different from the first and the second anchor surfaces;

calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the standby location; and

displaying the virtual object at the standby location.

98. The non-transitory computer readable storage medium of claim 97, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that the standby location is displayed as free floating on a display, wherein the virtual object appears on the display as movable and unconnected to surfaces in the image as the user turns his/her head and moves.

99. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that:

capturing an image with a body mounted camera mounted on a head of a user comprises capturing video data by a full-color video camera; and

capturing spatial data with a body mounted sensor array comprises capturing spatial data utilizing a distance sensor to determine distances to objects in the image.

100. The non-transitory computer readable storage medium of claim 99, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that capturing spatial data with a body mounted sensor array comprises capturing distance information of objects in the image using an infrared laser and an infrared sensor.

101. The non-transitory computer readable storage medium of claim 99, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising building a three dimensional map based on the captured video data and the calculated spatial data.

102. The non-transitory computer readable storage medium of claim 82, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

tracking movements of the user utilizing the body mounted camera;

applying a detection algorithm to the tracked user movements to detect a predetermined gesture; and

executing a command corresponding to the predetermined gesture when the predetermined gesture is detected.

103. The non-transitory computer readable storage medium of claim 102, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that the detected predetermined gesture is selected from the group consisting of an “

OK”

sign, a fist, an open hand, pointing with one finger, pointing with two fingers, pointing with three or four fingers, an outstretched hand, a hand rotation, a wave, a movement of one or more fingers, a movement of a body part or a foot, and any combination thereof.

104. The non-transitory computer readable storage medium of claim 102, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that applying a detection algorithm to the tracked user movements to detect a predetermined gesture comprises retrieving the predetermined gesture from a gesture dictionary.

105. The non-transitory computer readable storage medium of claim 104, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that retrieving the predetermined gesture from a gesture dictionary comprises retrieving the predetermined gesture from a gesture dictionary that defines one or more of:

a poke gesture, a pat gesture, or a tap gesture for target selection or toggling;

a push gesture for guiding a screen object across a screen;

a flick gesture for imparting momentum to the screen object;

a turn gesture for rotating the screen object;

a grab gesture or a pull gesture for zoom operations;

a two hand move gesture for panning operations;

a drawing hands apart gesture for resizing the screen object; and

a swipe gesture for horizontal scrolling or shuttling through media.

106. The non-transitory computer readable storage medium of claim 102, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that:

receiving the first input indicating a first anchor surface in the captured image comprises detecting a first predetermined gesture made by the wearer corresponding to a command to designate a surface within the captured image as an anchor surface and applying the command; and

receiving the second input to anchor the virtual object in the image comprises detecting a second predetermined gesture made by the wearer corresponding to a command to move the anchor surface to another surface in the captured image and applying the command.

107. The non-transitory computer readable storage medium of claim 106, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that either of the first or second detected predetermined gestures comprises the user pointing to a surface in the captured image.

108. The non-transitory computer readable storage medium of claim 106, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

continuing tracking movements of the user after the first input is received;

applying a detection algorithm to the tracked movements to detect a third predetermined gesture corresponding to a user input confirming the first input; and

anchoring the virtual object to the selected anchor surface in response to the third predetermined gesture.