Touch Free Interface for Augmented Reality Systems

• US 20140361988A1
• Filed: 09/19/2012
• Published: 12/11/2014
• Est. Priority Date: 09/19/2011
• Status: Abandoned Application

• Current Assignee: eyeSight Mobile Technologies Ltd.
• Original Assignee: eyeSight Mobile Technologies Ltd.
• Inventors: Katz, Itay, Shenfeld, Amnon
• Application Number:
  US14/345,592
• Publication Number:
  US 20140361988A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes

Abstract

A method and system for augmented reality. Images of a real world scene are obtained from one or more from one or more image sensors. An orientation and/or location of the image sensors is obtained from one or more state sensors. A real world object at which a predefined pointing object is performing a predefined gesture in the images of the real world scene is identified and data associated with the identified object is displayed on a viewing device. The invention also provides a computer program comprising computer program code means for performing all the steps of the method of the invention when said program is run on a computer.

Claims

1-56. -56. (canceled)

57. An augmented reality device, comprising:

at least one processor configured to;

receive, from an image sensor, image information associated with a scene;

output, to a display, visual data to enable a user to view both the visual data and at least a portion of the scene; and

detect, in the image information, a predefined gesture performed by the user;

correlate the predefined gesture with the visual data; and

alter the displayed visual data based on the correlation.

58. The augmented reality device of claim 57, wherein the visual data includes at least one of information associated with one or more objects in the scene, images associated with the scene, and one or more distances associated with the scene.

59. The augmented reality device of claim 57, wherein, to correlate the predefined gesture with the visual data, the at least one processor is configured to:

determine a reference location in three dimensional space of at leas portion of a hand of the user; and

determine in at least one of the visual data and the image information, data associated with the reference location.

60. The augmented reality device of claim 59, wherein, to alter the displayed visual data, the at least one processor is configured to change the visual data as a function of the data associated with the reference location.

61. The augmented reality device of claim 57, wherein, the predefined gesture is a hand gesture.

62. The augmented reality device of claim 61, wherein the hand gesture includes a pointing finger.

63. The augmented reality device of claim 57, wherein the at least one processor is further configured to output at least one of a message and a command based on the correlation.

64. The augmented reality device of claim 57, wherein the predefined gesture is performed by a predefined object, and further wherein the at least one processor is further configured to:

calibrate one or more parameters associated with the predefined object; and

detect the predefined gesture using the one or more parameters.

65. The augmented reality device of claim 57, wherein the at least one processor is further configured to enable a user to choose one or more filters that screen what data is selected as the visual data.

66. The augmented reality device of claim 57, wherein the at least one processor is further configured to:

detect at least one object pointed to by the user; and

determine the visual data based on the detected object.

67. The augmented reality device of claim 57, wherein the display is a head-mounted display.

68. The augmented reality device of claim 67, wherein the at least one processor is further configured to:

determine an orientation of the head-mounted display;

detect, using the determined orientation, at least one object pointed to by the user; and

determine the visual data based on the detected object.

69. The augmented reality device of claim 67, wherein the at least one processor is further configured to:

determine an orientation of the head-mounted display;

determine at least one fingertip location in the image information;

detect, using the determined orientation and the at least one fingertip location, at least one object pointed to by the user; and

determine the visual data based on the detected object.

70. The augmented reality device of claim 57, wherein the predefined gesture is performed by a predefined object, and further wherein, to output the visual data to the display, the at least one processor is further configured to prevent data from being displayed at one or more locations associated with the predefined object.

71. The augmented reality device of claim 57, wherein the at least one processor is further configured to:

stop display of the visual data based on a head motion of the user; and

resume display of the visual data when the head motion is below a predefined motion speed.

72. The augmented reality device of claim 57, wherein the visual data provides an indication that information related to an object in the scene is available.

73. The augmented reality device of claim 57, wherein to output the visual data to the display, the at least one processor is further configured to superimpose the visual data on at least a portion of the user'"'"'s view of the scene.

74. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations including:

receiving, from an image sensor, image information associated with a scene;

outputting, to a display, visual data to enable a user to view both the visual data and at least a portion of the scene; and

detecting, in the image information, a predefined gesture performed by the user;

correlating the predefined gesture with the visual data; and

altering the displayed visual data based on the correlation.

75. The non-transitory computer-readable medium of claim 74, wherein the visual data includes at least one of information associated with one or more objects in the scene, images associated with the scene, and one or more distances associated with the scene.

76. The non-transitory compute readable medium of claim 74, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:

determining a reference location in three dimensional space of at least a portion of a hand of the user; and

determining, in at least one of the visual data and the image information, data associated with the reference location.

77. The non-transitory computer-readable medium of claim 74, wherein the predefined gesture is performed by a predefined object, and further wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:

calibrating one or more parameters associated with the predefined object; and

detecting the predefined gesture using the one or more parameters.

78. The non-transitory computer-readable medium of claim 74, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform an operation including enabling a user to choose one or more filters that screen what data is selected as the visual data.

79. The non-transitory computer-readable medium of claim 74, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:

detecting at least one object pointed to by the user; and

determining the visual data based on the detected object.

80. The non-transitory computer-readable medium of claim 74, wherein the display is a head-mounted display.

81. The non-transitory computer-readable medium of claim 80, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:

determining an orientation of the head-mounted display;

detecting, using the determined orientation, at least one object pointed to by the user; and

determining the visual data based on the detected object.

82. The non-transitory computer-readable medium of claim 80, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:

determining an orientation of the head-mounted display;

determining at least one fingertip location in the image information;

detecting, using the determined orientation and the at least one fingertip location, at least one object pointed to by the user; and

determining the visual data based on the detected object.

83. The non-transitory computer-readable medium of claim 74, wherein the predefined gesture is performed by a predefined object, and further wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform an operation including preventing data from being displayed at one or more locations associated with the predefined object.

84. The non-transitory computer-readable medium of claim 74, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations including:

stopping display of the visual data based on a head motion of the user; and

resuming display of the visual data when the head motion is below a predefined motion speed.

85. The non-transitory computer-readable medium of claim 74, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform an operation including superimposing the visual data on at least a portion of the user'"'"'s view of the scene.

86. A augmented reality method, comprising:

receiving, from an image sensor, image information associated with a scene;

outputting, to a display, visual data to enable a user to view both the visual data and at least a portion of the scene; and

detecting, in the image information, a predefined gesture performed by the user;

correlating the predefined gesture with the visual data; and

altering the displayed visual data based on the correlation.