Method for interacting with an object displayed on data eyeglasses

• US 9,910,506 B2
• Filed: 10/23/2015
• Issued: 03/06/2018
• Est. Priority Date: 04/25/2013
• Status: Active Grant

• Current Assignee: Bayerische Motoren Werke AG
• Original Assignee: Bayerische Motoren Werke AG
• Inventors: Spiessl, Wolfgang, Lauber, Felix, Pfannstiel, Jochen
• Primary Examiner(s): Marinelli, Patrick F
• Application Number:
  US14/921,087
• Publication Number:
  US 20160041624A1
• Time in Patent Office: 865 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

The invention relates to a method for interacting with an object that is displayed to a user by smart glasses, which includes a display. The method includes displaying the object for the user using the display, detecting that the user closes a first eye and keeps the first eye closed during a predetermined period of time using a first camera, recording a hand of the user using a second camera determining that the user performs an input action during the predetermined period of time, wherein the input action includes the hand assuming an attitude, and a position from a perspective of a second eye of the user with respect to the object, that meet a predetermined condition, and performing an action with respect to the object, wherein the action is associated with the input action in advance.

Claims

1. A method for interacting with a first object being displayed to a user via smart glasses, the smart glasses comprising a display, the method comprising acts of:

displaying the first object and a second object in a first virtual image for the user using the display of the smart glasses;

detecting that the user closes a first eye and keeps the first eye closed during a predetermined period of time using a first camera configured to record the first eye of the user;

recording a first hand of the user using a second camera;

determining a hand-eye-object vector associated with the second object;

determining that at least a portion of the second object is covering at least a portion of the first hand based at least on the determined hand-eye object vector and displaying the first and second objects in a second virtual image such that the portion of the second object covering the first hand is not displayed;

determining that the user performs a first input action for the first object in the second virtual image during the predetermined period of time, wherein the first input action includes the first hand of the user assuming an attitude, and a position from a perspective of a second eye of the user with respect to the first object, that meet a predetermined condition; and

carrying out a first action with respect to the first object, wherein the first action is associated with the first input action in advance, and wherein the hand-eye-object vector indicates a positional relationship among a second eye of the user, the first hand, and the second object.

2. The method according to claim 1, wherein the predetermined condition includes one or more of the following:

an attitude of the first hand in which a ring or a circle is formed with one or more fingers, and the first hand is positioned such that the first object is enclosed by the ring or circle from the perspective of the second eye;

an attitude of the first hand in which a thumb and an index finger are extended to form or symbolize an angle of approximately 90 degrees, and the first hand is positioned such that the first object is at least partially located in an imaginary rectangle from the perspective of the second eye, wherein the imaginary rectangle has two short edge lengths corresponding to a length of the thumb and two long edge lengths corresponding to a length of the index finger, and a position of the imaginary rectangle from the perspective of the second eye is such that two edges correspond to the thumb and index finger;

an attitude of the first hand in which the thumb and the index finger, together, form approximately a U shape, and the first hand is positioned such that the first object is at least partially contained in the U shape from the perspective of the second eye; and

an attitude of the first hand in which only the index finger is extended and the hand is positioned such that the index finger touches the first object, or at least points in a direction of the first object, from the perspective of the second eye.

3. The method according to claim 1, further comprising the act of:

recording a second hand of the user using the second camera;

wherein the first input action further includes the second hand of the user assuming an attitude and, a position from the perspective of the second eye of the user with respect to the first object, that meet the predetermined condition, wherein the predetermined condition further includes the attitude and the position for both the first hand and the second hand.

4. The method according to claim 3, wherein the predetermined condition further includes:

an attitude of the first hand and an attitude of the second hand in which, for each attitude, a respective thumb and a respective index finger, together, form approximately a U shape, and the first hand and the second hand are each positioned such that the first object is at least partially contained in the respective U shape from the perspective of the second eye, wherein the attitude and the position of the first hand and the second hand are such that openings of the respective U shapes point towards one another.

5. The method according to claim 1, the act of determining that the user performs the first input action further comprising:

determining a spatial position of the first hand with respect to a position of the display of the smart glasses; and

determining a position of the first object within the display,

wherein a spatial position of the second eye with respect to the display of the smart glasses is known.

6. The method according to claim 1, the act of determining that the user performs the first input action further comprising:

determining a spatial position of the first hand with respect to the second eye of the user;

determining a position of the first object within the display;

determining optical paths or vectors from points of the first hand to the second eye of the user;

determining a position of an intersection of the optical paths or the vectors using the display of the smart glasses; and

determining a position of the intersection with respect to the displayed first object using the display;

wherein a spatial position of the second eye with respect to the display of the smart glasses is known.

7. The method according to claim 5, wherein the smart glasses includes the second camera and a direction of recording of the second camera is in a direction of viewing of the user looking straight ahead and the second camera is configured to determine one or more distances of at least some recorded picture elements in the recording.

8. The method according to claim 1, wherein the smart glasses includes the first camera.

9. The method according to claim 1, further comprising the acts of:

determining that the user performs a second input action sequentially after the first input action during the predetermined period of time wherein the second input action involves the first hand of the user abandoning the attitude associated with the first input action, or the first hand of the user assuming a predetermined attitude; and

carrying out a second action with respect to the first object, wherein the second action is associated with the second input action in advance.

12. The method according to claim 2, wherein the ring or the circle is formed with the thumb and the index finger of the user.

13. The method according to claim 6, wherein the smart glasses includes the second camera and a direction of recording of the second camera is in a direction of viewing of the user looking straight ahead and the second camera is configured to determine one or more distances of at least some recorded picture elements in the recording.

10. A device comprising:

a display;

a first camera;

a second camera; and

at least one processor executing stored program instructions to;

display a first object and a second object in a first virtual image for a user using the display;

detect that the user closes a first eye and keeps the first eye closed during a predetermined period of time using the first camera configured to record the first eye of the user;

record a hand of the user using the second camera;

determine a hand-eye-object vector associated with the second object;

determine that at least a portion of the second object is covering at least a portion of the first hand based at least on the determined hand-eye object vector and displaying the first and second objects in a second virtual image such that the portion of the second object covering the first hand is not displayed;

determine that the user performs an input action for the first object in the second virtual image during the predetermined period of time, wherein the input action includes the hand of the user assuming an attitude, and a position from a perspective of a second eye of the user with respect to the first object, that meet a predetermined condition; and

carry out an action with respect to the first object, wherein the action is associated with the input action in advance, and wherein the hand-eye-object vector indicates a positional relationship among a second eye of the user, the first hand, and the second object.

14. The device according to claim 10, wherein the device is smart glasses.

11. A non-transitory computer readable medium storing program instructions, the program instructions when executed by at least one processor performs a method comprising acts of:

displaying a first object and a second object in a first virtual image for a user using a display

detecting that the user closes a first eye and keeps the first eye closed during a predetermined period of time using a first camera configured to record the first eye of the user;

record a hand of the user using a second camera;

determining a hand-eye-object vector associated with the second object;

determining that at least a portion of the second object is covering at least a portion of the first hand based at least on the determined hand-eye object vector and displaying the first and second objects in a second virtual image such that the portion of the second object covering the first hand is not displayed;

determining that the user performs a first input action for the first object in the second virtual image during the predetermined period of time, wherein the first input action includes the first hand of the user assuming an attitude, and a position from a perspective of a second eye of the user with respect to the first object, that meet a predetermined condition; and

carrying out a first action with respect to the first object, wherein the first action is associated with the first input action in advance, and wherein the hand-eye-object vector indicates a positional relationship among a second eye of the user, the first hand, and the second object.

15. The non-transitory computer readable medium according to claim 11, wherein the computer program is executed on smart glasses.