Gesture-based human machine interface

• US 9,569,010 B2
• Filed: 05/31/2011
• Issued: 02/14/2017
• Est. Priority Date: 06/09/2010
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Campillo, David Esteban, Jimenez, David Luis Scarlatti, Ortiz, Nicolas Pena, Tapia, Pablo Soriano
• Primary Examiner(s): Thompson, James A
• Application Number:
  US13/149,462
• Publication Number:
  US 20110304650A1
• Time in Patent Office: 2,086 Days
• Field of Search: 345/661
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A gesture-based human machine interface, for example, a user interface for controlling a program executing on a computer, and related method are provided. Gestures of the user are monitored and a response is provided that is based upon the detected gestures. An object is used to point to information displayed on a screen. The information displayed on the screen is modified in response to a determination of the position on the screen to which the object is pointing and in response to the distance of the object from the screen.

Claims

1. A method of using a computer system through a gesture-based human machine interface, the method comprising:

pointing to a position on a screen of the computer system using an object to point, wherein pointing includes maintaining a distance between the object and the screen, the distance being a positive value, and wherein the screen comprises outer corners and a center;

capturing images of a space in front of the screen with at least four cameras configured to attach at the outer corners of the screen and point toward the center of the screen, wherein capturing images with the at least four cameras allows a three-dimensional map of the space in front of the screens to be constructed, the space including the object;

analyzing the images using a processor to identify the object, to determine the position of the object in three-dimensions, to determine the position on the screen to which the object is pointing, to determine the distance between the object and the screen, and to determine speeds of movement of the object; and

modifying information displayed on the screen in response to the determination of both the position on the screen to which the object is pointing and the distance between the object and the screen;

wherein modifying the information displayed on the screen comprises magnifying the information displayed on a portion of the screen with a magnification that is dependent upon the determined distance between the object and the screen, wherein a zoom button is positioned at the position on the screen to which the object is pointing;

wherein the determined distance is between a maximum value and a minimum value, wherein the magnification scaling varies exponentially as the determined distance varies between the maximum value and the minimum value; and

wherein a gradual speed of movement of the object towards a menu option presented on the screen is interpreted as a single click and a rapid speed of movement of the object towards the menu option presented on the screen is interpreted as a double click.

2. The method of claim 1, wherein determining the position on the screen to which the object is pointing comprises tracking movement of the object.

3. The method of claim 1, wherein determining the position on the screen to which the object is pointing comprises determining an extension of the object toward the screen.

4. The method of claim 1 further comprising:

pointing to positions on the screen of the computer system using two objects to point;

analyzing the images using the processor to identify the two objects, to determine screen positions relative to the two objects and the at least four cameras, the screen being a reference plane for determining the positions on the screen to which the two objects are pointing, and to determine distances of the two objects from the screen; and

modifying the information displayed on the screen in response to the determination of both the positions on the screen to which the two objects are pointing and the distances of the two objects from the screen.

5. The method of claim 1, wherein the object is an extended finger of a user'"'"'s hand.

6. The method of claim 1, wherein to adjust the magnification, the object is moved towards or away from the screen in a z-direction and moving the object towards the screen results in a greater magnification;

wherein an x, y position of the object determines an area on the screen that is magnified, and a z-position of the object determines the magnification;

wherein upper and lower values for the z-position are set to correspond to upper and lower magnification factors;

wherein magnification is set to 1 where the object is at least a certain distance, about 30 cm, away from the screen;

wherein a minimum separation from the screen, about 5 cm, is set as being the maximum magnification such that if the object comes closer than the minimum separation to the screen, the magnification no longer increases; and

wherein the magnification changes with an exponential relationship.

7. A method of using a computer system through a gesture-based human machine interface, the method comprising:

pointing to a position on a screen of the computer system using an object to point, wherein pointing includes maintaining a distance between the object and the screen, the distance being a positive value, and wherein the screen comprises outer corners and a center;

capturing images of a space in front of the screen with at least four cameras configured to attach at the outer corners of the screen and point toward the center of the screen, wherein capturing images with the at least four cameras allows a three-dimensional map of the space in front of the screens to be constructed, the space including the object;

analyzing the images using a processor to identify the object, to determine the position of the object in three-dimensions, to determine the position on the screen to which the object is pointing, and to determine the distance between the object and the screen;

modifying information displayed on the screen in response to the determination of both the position on the screen to which the object is pointing and the distance between the object and the screen;

analyzing the images using the processor to determine speeds of movement of the object; and

modifying the information displayed on the screen differently according to different speeds of movement of the object;

wherein a gradual speed of movement of the object towards a menu option presented on the screen is interpreted as a single click and a rapid speed of movement of the object towards menu option presented on the screen is interpreted as a double click;

wherein when movement of the object is a slow lateral movement causes an object displayed on the screen to be moved around within the screen;

wherein when movement of the object is a rapid movement causes the object displayed on the screen to be removed from the screen; and

wherein modifying the information displayed on the screen further comprises magnifying the information displayed on a portion of the screen to which the object is pointing with a magnification that is dependent upon the determined distance between the object and the screen, wherein the determined distance is between a maximum value and a minimum value, and wherein the magnification scaling varies exponentially as the determined distance varies between the maximum value and the minimum value.

8. The method of claim 7 further comprising:

analyzing the images using the processor to determine a speed of movement of the object away from the screen; and

modifying the information displayed on the screen in response to the determination of the speed of movement of the object towards or away from the screen;

wherein lateral movement is used such that different speeds give different results;

wherein the slow lateral movement is a slow movement left to right moves the object displayed on the screen from a central position to a position on a right hand edge of the screen; and

wherein the rapid movement is a rapid movement left to right that causes the object displayed on the screen to fly off the right hand edge of the screen.

9. The method of claim 7 further comprising:

analyzing the images using the processor to determine a speed of movement of the object laterally in front of the screen; and

modifying the information displayed on the screen in response to the determination of the speed of movement of the object.

10. An apparatus comprising:

a screen operable to display information, wherein the screen comprises outer corners and a center;

at least four cameras configured to attach at the outer corners of the screen and point toward the center of the screen to capture images of a space in front of the screen, wherein capturing images with the at least four cameras allows a three-dimensional map of the space in front of the screens to be constructed, and

a processor configured to;

receive the images,

analyze the images to identify an object pointing at a position on the screen, to determine a screen position relative to the object and the at least four cameras, the screen being a reference plane for determining the position on the screen to which the object is pointing, to determine a distance of the object from the screen, the distance being a positive value, and to determine speeds of movement of the object, and

modify the information displayed on the screen in response to the determination of both the position on the screen to which the object is pointing and the distance of the object from the screen;

wherein modifying the information displayed on the screen comprises magnifying the information displayed on a portion of the screen with a magnification that is dependent upon the determined distance between the object and the screen, wherein a zoom button is positioned at the position on the screen to which the object is pointing;

wherein the determined distance is between a maximum value and a minimum value, wherein the magnification scaling varies exponentially as the determined distance varies between the maximum value and the minimum value; and

wherein a gradual speed of movement of the object towards a menu option presented on the screen is interpreted as a single click and a rapid speed of movement of the object towards the menu option presented on the screen is interpreted as a double click.

11. The apparatus of claim 10, wherein the processor is configured to analyze the images to track movement of the object to determine the position on the screen to which the object is pointing.

12. The apparatus of claim 10, wherein the processor is configured to analyze the images to determine an extension of the object toward the screen to determine the position on the screen to which the object is pointing.

13. The apparatus of claim 10, wherein the processor is configured to magnify the information displayed on a portion of the screen to which the object is pointing with a magnification that is dependent upon the determined distance of the object from the screen.

14. The apparatus of claim 10, wherein the object is an extended finger of a user'"'"'s hand.

15. An apparatus comprising:

a screen operable to display information, wherein the screen comprises outer corners and a center;

at least four cameras configured to attach at the outer corners of the screen and point toward the center of the screen to capture images of a space in front of the screen, wherein capturing images with the at least four cameras allows a three-dimensional map of the space in front of the screens to be constructed, and

a processor configured to;

receive the images,

analyze the images to identify an object pointing at a position on the screen, to determine a screen position relative to the object and the at least four cameras, the screen being a reference plane for determining the position on the screen to which the object is pointing, and to determine a distance of the object from the screen, the distance being a positive value, and

modify the information displayed on the screen in response to the determination of both the position on the screen to which the object is pointing and the distance of the object from the screen, and determine a speed of movement of the object and to modify the information displayed on the screen in response to the determination of the speed of movement, wherein a gradual speed of movement of the object towards a menu option presented on the screen is interpreted as a single click and a rapid speed of movement of the object towards the menu option presented on the screen is interpreted as a double click;

wherein modifying the information displayed on the screen comprises magnifying the information displayed on a portion of the screen to which the object is pointing with a magnification that is dependent upon the determined distance between the object and the screen, wherein the determined distance is between a maximum value and a minimum value, and wherein the magnification scaling varies exponentially as the determined distance varies between the maximum value and the minimum value.

16. A gesture-based human interface of a computer system, the interface being a computer program product stored on a non-transitory storage medium in the computer system, and further comprising:

a first computer program product module configured to enable a screen operable to display information, wherein the screen comprises outer corners and a center;

a second computer program product module configured to instruct at least four cameras configured to attach at the outer corners of the screen and point toward the center of the screen to capture images of a space in front of the screen; and

a third computer program product module configured to control a processor configured to;

receive the images,

analyze the images to identify an object pointing at a position on the screen, to determine a screen position relative to the object and the at least four cameras, the screen being a reference plane for determining the position on the screen to which the object is pointing and for determining a distance of the object from the screen, the distance being a positive value, and to determine a speed of movement of the object, and

modify the information displayed on the screen in response to the determination of the position on the screen to which the object is pointing, the distance of the object from the screen, and the speed of movement;

wherein modifying the information displayed on the screen comprises magnifying the information displayed on a portion of the screen to which the object is pointing with a magnification that is dependent upon the determined distance between the object and the screen, wherein the determined distance is between a maximum value and a minimum value, and wherein the magnification scaling varies exponentially as the determined distance varies between the maximum value and the minimum value; and

wherein a gradual speed of movement of the object towards a menu option presented on the screen is interpreted as a single click and a rapid speed of movement of the object towards the menu option presented on the screen is interpreted as a double click.

17. The interface of claim 16, wherein the processor is configured to magnify the information displayed on a portion of the screen to which the object is pointing with a magnification that is dependent upon the determined distance of the object from the screen.

18. The interface of claim 16, wherein the object includes an extension of a finger used to determine to where on the screen a user is pointing.

19. The interface of claim 16, wherein the processor is configured to analyze the images to determine an extension of the object toward the screen to determine the position on the screen to which the object is pointing.

20. The interface of claim 16, wherein the processor is configured to analyze the images to track movement of the object to determine the position on the screen to which the object is pointing.