Touch free interface for augmented reality systems

• US 10,401,967 B2
• Filed: 05/02/2016
• Issued: 09/03/2019
• Est. Priority Date: 09/19/2011
• Status: Active Grant

• Current Assignee: eyeSight Mobile Technologies Ltd.
• Original Assignee: eyeSight Mobile Technologies Ltd.
• Inventors: Katz, Itay, Shenfeld, Amnon
• Primary Examiner(s): Chowdhury, Afroza
• Application Number:
  US15/144,209
• Publication Number:
  US 20160320855A1
• Time in Patent Office: 1,219 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes

Abstract

A method and system for augmented reality. Images of a real world scene are obtained from one or more from one or more image sensors. An orientation and/or location of the image sensors is obtained item one or more state sensors. A real world object at which a predefined pointing object is performing a predefined gesture in the images of the real world scene is identified and data associated with the identified object is displayed on a viewing device. The invention also provides a computer program comprising computer program code means for performing all the steps of the method of the invention when said program is run on a computer.

Claims

1. An augmented reality device, comprising:

at least one processor configured to;

receive, from an image sensor, video frames of image information associated with a real world scene;

detect, in the image information, a predefined hand gesture performed by a user;

designate an area of selected image information associated with a real-world object at a first location in the video frames, wherein the selected image information is associated with the real world scene that does not include the user'"'"'s hand, the designated area being associated with the detected predefined hand gesture;

copy the designated area of the selected image information to a second location in a field of view of the user; and

track the selected image information associated with the real world scene to present a real-time view of the real-world object in the selected information in the second location.

2. The augmented reality device of claim 1, wherein the predefined hand gesture includes drawing a contour of the designated area.

3. The augmented reality device of claim 1, wherein the at least one processor is further configured to resize the designated area.

4. The augmented reality device of claim 3, wherein the second predefined hand gesture comprises separating two fingers, or bringing two fingers closer together.

5. The augmented reality device of claim 1, wherein the at least one processor is further configured to zoom in or out in association with a second predefined hand gesture detected in the image information.

6. The augmented reality device of claim 3, wherein the at least one processor is further configured to zoom in or out in association with the resize of the designated area.

7. The augmented reality device of claim 1, wherein the at least one processor is further configured to capture a frame from the video frames of image information in association with a second predefined hand gesture detected in the image information.

8. The augmented reality device of claim 1, wherein the second location is defined by the user.

9. The augmented reality device of claim 8, wherein the second location is defined by the user performing a dragging gesture associated with the designated area.

10. The augmented reality device of claim 1, wherein the at least one processor is further configured to employ a tracker on the selected image information to present the real-time view of the real-world object in the selected information in a resized area in the field of view of the user.

11. A method of operating an augmented reality device, the method comprising:

receiving, by at least one processor from an image sensor, video frames of image information associated with a real world scene;

detecting, by the at least one processor in the image information, a predefined hand gesture performed by a user;

designating, by the at least one processor, an area of selected image information associated with a real-world object at a first location in the video frames, wherein the selected image information is associated with the real world scene that does not include the user'"'"'s hand, the designated area being associated with the detected predefined hand gesture;

copy the designated area of the selected image information to a second location in a field of view of the user; and

track the selected image information associated with the real world scene to present a real-time view of the real-world object in the selected information in the second location.

12. The method of claim 11, wherein the predefined hand gesture includes drawing a contour of the designated area.

13. The method of claim 11, wherein the at least one processor is further configured to resize the designated area.

14. The method of claim 13, wherein the second predefined hand gesture comprises separating two fingers, or bringing two fingers closer together.

15. The method of claim 11, further comprising zooming in or out in association with a second predefined hand gesture detected in the image information.

16. The method of claim 11, further comprising capturing a frame from the image information in association with a second predefined hand gesture detected in the image information.

17. The method of claim 11, further comprising employing a tracker on the selected area to present the real-time view of the real-world object in the selected information in a resized area in the field of view of the user.

18. A non-transitory computer-readable medium storing instructions that, when executed, cause at least one processor to perform a method of operating an augmented reality device, the method comprising:

receiving, from an image sensor, video frames of image information associated with a real world scene;

detecting, in the image information, a predefined hand gesture performed by a user;

designating an area of selected image information associated with a real-world object at a first location in the video frames, wherein the selected image information is associated with the real world scene that does not include the user'"'"'s hand, the designated area being associated with the detected predefined hand gesture;

copy the designated area of the selected image information to a second location in a field of view of the user; and

tracking the selected image information associated with the real world scene to present a real-time view of the real-world object in the selected information in the second location.