SYSTEMS AND METHODS FOR A GAZE AND GESTURE INTERFACE
First Claim
1. A method for a person wearing a head frame having a first camera aimed at an eye of the person to interact with a 3D object displayed on a display by gazing at the 3D object with the eye and by making a gesture with a body part, comprising:
- sensing an image of the eye, an image of the display and an image of the gesture with at least two cameras, one of the at least two cameras being mounted in the head frame adapted to be pointed at the display and the other of the at least two cameras being the first camera;
transmitting the image of the eye, the image of the gesture and the image of the display to a processor;
the processor determining a viewing direction of the eye and a location of the head frame relative to the display from the images and then determining the 3D object the person is gazing at;
the processor recognizing the gesture from the image of the gesture out of a plurality of gestures; and
the processor further processing the 3D object based on the gaze or the gesture or the gaze and the gesture.
3 Assignments
0 Petitions
Accused Products
Abstract
A system and methods for activating and interacting by a user with at least a 3D object displayed on a 3D computer display by at least the user'"'"'s gestures which may be combined with a user'"'"'s gaze at the 3D computer display. In a first instance the 3D object is a 3D CAD object. In a second instance the 3D object is a radial menu. A user'"'"'s gaze is captured by ahead frame containing at least an endo camera and an exo camera worn by a user. A user'"'"'s gesture is captured by a camera and is recognized from a plurality of gestures. User'"'"'s gestures are captured by a sensor and are calibrated to the 3D computer display.
407 Citations
20 Claims
-
1. A method for a person wearing a head frame having a first camera aimed at an eye of the person to interact with a 3D object displayed on a display by gazing at the 3D object with the eye and by making a gesture with a body part, comprising:
-
sensing an image of the eye, an image of the display and an image of the gesture with at least two cameras, one of the at least two cameras being mounted in the head frame adapted to be pointed at the display and the other of the at least two cameras being the first camera; transmitting the image of the eye, the image of the gesture and the image of the display to a processor; the processor determining a viewing direction of the eye and a location of the head frame relative to the display from the images and then determining the 3D object the person is gazing at; the processor recognizing the gesture from the image of the gesture out of a plurality of gestures; and the processor further processing the 3D object based on the gaze or the gesture or the gaze and the gesture. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16)
-
-
17. A system wherein a person interacts with one or more or a plurality of 3D objects through a gaze with a first eye and through a gesture by a body part, comprising:
-
a computer display that displays the plurality of 3D objects; a head frame containing a first camera adapted to point at the first eye of the person wearing the head frame and a second camera adapted to point to an area of the computer display and to capture the gesture; a processor, enabled to execute instructions to perform the steps; receiving data transmitted by the first and second cameras; processing the received data to determine a 3D object in the plurality of objects where the gaze is directed at; processing the received data to recognize the gesture from a plurality of gestures; and further processing the 3D object based on the gaze and gesture. - View Dependent Claims (18, 19)
-
-
20. A device with which a person interacts with a 3D object displayed on a 3D computer display through a gaze from a first eye and a gaze from a second eye and through a gesture by a body part of the person, comprising:
-
a frame adapted to be worn by the person; a first camera mounted in the frame adapted to point at the first eye to capture the first gaze, a second camera mounted in the frame adapted to point at the second eye to capture the second gaze, a third camera mounted in the frame adapted to point at the 3D computer display and to capture the gesture, a first and a second glass mounted in the frame such that the first eye looks through the first glass and the second eye looks through the second glass, the first and second glasses acting as 3D viewing shutters; and a transmitter to transmit data generated by the cameras.
-
Specification