Gesture recognition interface system

• US 20080013826A1
• Filed: 07/13/2006
• Published: 01/17/2008
• Est. Priority Date: 07/13/2006
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Corporation
• Inventors: Hsu, Shi-Ping, Hillis, William Daniel, Nishihara, H Keith
• Granted Patent: US 8,589,824 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system and method is provided for a gesture recognition interface system. The interface system may comprise a first and second light source positioned to illuminate a background surface. The interface system may also comprise at least one camera operative to receive a first plurality of images based on a first reflected light contrast difference between the background surface and a sensorless input object caused by the first light source and a second plurality of images based on a second reflected light contrast difference between the background surface and the sensorless input object caused by the second light source. The interface system may further comprise a controller operative to determine a given input gesture based on changes in relative locations of the sensorless input object in the first plurality of images and the second plurality of images. The controller may further be operative to initiate a device input associated with the given input gesture.

Claims

1. A gesture recognition interface system comprising:

a first light source and a second light source positioned to illuminate a background surface;

at least one camera operative to receive a first plurality of images based on a first reflected light contrast difference between the background surface and a sensorless input object caused by the first light source and a second plurality of images based on a second reflected light contrast difference between the background surface and the sensorless input object caused by the second light source; and

a controller operative to determine a given input gesture based on changes in relative locations of the sensorless input object in the first plurality of images and the second plurality of images, and being further operative to initiate a device input associated with the given input gesture.

2. The interface system of claim 1, wherein the sensorless input object is a user'"'"'s hand.

3. The interface system of claim 2, wherein the controller is further operative to determine a three-dimensional physical location of each extended fingertip associated with the user'"'"'s hand, and wherein the one or more of the user'"'"'s hand gestures are associated with changes in the three-dimensional physical location of each extended fingertip.

4. The interface system of claim 1, wherein the given input gesture corresponds to one of a mouse command, a zoom command, a rotate command, an environment command, and a laser pointer gesture.

5. The interface system of claim 1, wherein the given input gesture is a six-degree of freedom input gesture based on changes in a three-dimensional location and an orientation of the sensorless input object.

6. The interface system of claim 1, wherein the controller is operative to determine a location of at least one end-point of the sensorless input object associated with each of the first plurality of images and the second plurality of images of the sensorless input object, and wherein the controller comprises a plurality of peak detectors that operate to determine a three-dimensional physical location of the at least one end-point of the sensorless input object.

7. The interface system of claim 6, wherein the controller is further operative to determine a length, a thickness, and an orientation of the sensorless input object at elongated regions that comprise the at least one end-point of the sensorless input object.

8. The interface system of claim 6, wherein the background surface is non-planar, and wherein the controller determines the three-dimensional physical location of the at least one end-point of the sensorless input object relative to a three-dimensional height function associated with the background surface.

9. The interface system of claim 6, wherein each of the plurality of peak detectors is associated with the first plurality of images and the second plurality of images of the sensorless input object, respectively, such that each of the plurality of peak detectors determines a two-dimensional physical location of the at least one end-point of the sensorless input object, and further comprising a location resolver operative to interpolate the three-dimensional physical location of the at least one end-point of the sensorless input object based on each two-dimensional physical location of the at least one end-point of the sensorless input object associated with the respective one of the plurality of peak detectors.

10. The interface system of claim 9, wherein the controller further comprises a two-dimensional Laplacian of Gaussian convolution filter associated with each of the plurality of peak detectors, the two-dimensional Laplacian of Gaussian convolution filter being operative to input either the first and second plurality of images or a contrast enhanced version of the first and second plurality of images, such that the plurality of peak detectors determine the two-dimensional physical location of the at least one end-point of the sensorless input object based on a convolution filtered output from the respective two-dimensional Laplacian of Gaussian convolution filter.

11. The interface system of claim 9, wherein the controller further comprises a spatial bandpass filter associated with each of the plurality of peak detectors, the spatial bandpass filter being operative to input either the first and second plurality of images or a contrast enhanced version of the first and second plurality of images, such that the plurality of peak detectors determine the two-dimensional physical location of the at least one end-point of the sensorless input object based on a filtered output from the respective spatial bandpass filter.

12. The interface system of claim 1, wherein the background surface is also operative as a display device, such that the background surface operates as a touchscreen input/output device.

13. The interface system of claim 1, wherein the first plurality of images and the second plurality of images comprise a plurality of matched pairs of images of the sensorless input object, and wherein the controller determines a three-dimensional physical location of the sensorless input object relative to the background surface based on an amount of parallax separation between the matched pairs of images of the sensorless input object.

14. The interface system of claim 13, wherein the first light source and the second light source are first and second infrared (IR) light sources, and wherein the at least one camera comprises a first IR filtered camera and a second IR filtered camera, each of the first IR filtered camera and the second IR filtered camera being closely coupled, respectively, to the first IR light source and the second IR light source, and wherein the background surface is retroreflective of IR light, such that the first IR filtered camera receives the first plurality of images and the second IR filtered camera receives the second plurality of images of the sensorless input object, the first plurality of images and the second plurality of images being silhouette images of the sensorless input object.

15. The interface system of claim 13, wherein the at least one camera is IR filtered, and wherein the first light source and the second light source are first and second IR light sources spaced apart from the at least one camera, the first and second IR light sources sequentially pulsing to generate the first plurality of images and the second plurality of images of the sensorless input object as shadows of the sensorless input object against the background surface.

16. The interface system of claim 13, wherein the controller comprises an automated calibration unit operative to calibrate the gesture recognition interface system based on an amount of parallax separation between matched pairs of images of a calibration pattern on the background surface.

17. The interface system of claim 1, wherein the controller comprises a memory operative to store a plurality of pre-defined gestures, and wherein the controller determines the given input gesture based on a comparison of the relative locations of the first plurality of images and the second plurality of images with the plurality of pre-defined gestures.

18. The interface system of claim 17, wherein the memory is further operative to be programmed with additional pre-defined gestures, the additional pre-defined gestures being input to the memory based on a user performing gestures to which the additional pre-defined gestures correspond.

19. A method of providing device inputs, the method comprising:

illuminating a background surface with a first light source and a second light source;

providing simulated inputs over the background surface via gestures associated with a sensorless input object;

generating a first plurality of images associated with the sensorless input object based on a reflected light contrast between the sensorless input object and the illuminated background surface caused by the first light source;

generating a second plurality of images associated with the sensorless input object based on a reflected light contrast between the sensorless input object and the illuminated background surface caused by the second light source;

determining a plurality of three-dimensional physical locations of the sensorless input object based on a relative separation of the sensorless input object in the first plurality of images relative to the second plurality of images;

determining if changes in the plurality of three-dimensional physical locations of the sensorless input object correspond to any of a plurality of pre-defined gestures; and

providing at least one device input based on a given one of the plurality of pre-defined gestures upon determining that the changes in the plurality of three-dimensional physical locations of the sensorless input object correspond to the given one of the plurality of pre-defined gestures.

20. The method of claim 19, wherein providing simulated inputs over the background surface comprises providing simulated inputs via gestures associated with a user'"'"'s hand over the background surface.

21. The method of claim 20, wherein determining the plurality of three-dimensional physical locations of the sensorless input object comprises determining a plurality of three-dimensional physical locations of each extended fingertip associated with the user'"'"'s hand.

22. The method of claim 19, wherein generating the first plurality of images associated with the sensorless input object and generating the second plurality of images associated with the sensorless input object comprises generating a matched pair of images associated with the sensorless input object, and wherein determining the plurality of three-dimensional physical locations of the sensorless input object comprises determining a plurality of three-dimensional physical locations of at least one end-point associated with-the sensorless input object.

23. The method of claim 22, wherein illuminating the background surface comprises illuminating a non-planar background surface, and wherein determining the plurality of three-dimensional physical locations of the at least one end-point associated with the sensorless input object comprises determining a plurality of three-dimensional physical locations of the at least one end-point associated with the sensorless input object relative to a three-dimensional height function associated with the non-planar background surface.

24. The method of claim 22, wherein determining the plurality of three-dimensional physical locations of the at least one end-point associated with the sensorless input object comprises either performing a two-dimensional Laplacian of Gaussian convolution operation on each of the first plurality of images associated with the sensorless input object and each of the second plurality of images associated with the sensorless input object, or performing a two-dimensional Laplacian of Gaussian convolution operation on a contrast enhanced version of each of the first plurality of images associated with the sensorless input object and a contrast enhanced version of each of the second plurality of images associated with the sensorless input object.

25. The method of claim 22, wherein determining the plurality of three-dimensional physical locations of the at least one end-point associated with the sensorless input object comprises determining a height of the at least one end-point associated with the sensorless input object relative to the illuminated background surface based on an amount of parallax separation between the first plurality of images and the second plurality of images of the sensorless input object.

26. The method of claim 25, further comprising calibrating the determined height of the at least one end-point based on an amount of parallax separation between a first plurality of images of a calibration pattern on the background surface and a second plurality of images of the calibration pattern on the background surface.

27. The method of claim 19, further comprising displaying output data on the illuminated background surface, such that the illuminated background surface operates as a touchscreen input/output device.

28. The method of claim 19, wherein illuminating the background surface with a first light source and a second light source comprises illuminating a retroreflective background surface with a first infrared (IR) light source and a second IR light source, and wherein generating the first plurality of images associated with the sensorless input object comprises generating a first plurality of silhouettes associated with the sensorless input object and generating the second plurality of images associated with the sensorless input object comprises generating a second plurality of silhouettes associated with the sensorless input object.

29. The method of claim 19, wherein illuminating the background surface comprises sequentially pulsing a first IR light source and a second IR light source, and wherein generating the first plurality of images associated with the sensorless input object and generating the second plurality of images associated with the sensorless input object comprises generating a first plurality of shadows of the sensorless input object against the illuminated background surface and generating a second plurality of shadows of the sensorless input object against the illuminated background surface.

30. A gesture recognition interface system comprising:

means for providing a first brightness contrast between a background surface and a sensorless input object;

means for providing a second brightness contrast between a background surface and a sensorless input object;

means for generating a first plurality of images of the sensorless input object based on the first brightness contrast;

means for generating a second plurality of images of the sensorless input object based on the second brightness contrast, the first plurality of images and the second plurality of images forming a plurality of matched pairs of images of the sensorless input object;

means for generating two-dimensional location information associated with at least one end-point of the sensorless input object for each of the first plurality of images and the second plurality of images;

means for interpolating three-dimensional location information associated with the sensorless input object based on the two-dimensional location information associated with the at least one end-point of the sensorless input object for each of the plurality of matched pairs of images of the sensorless input object;

means for translating changes in the three-dimensional location information associated with at least one end-point of the sensorless input object to a given input gesture; and

means for providing device inputs based on matching the given input gesture with one of a plurality of pre-defined gestures.

31. The interface system of claim 30, wherein the means for providing device inputs comprises means for providing device inputs based on matching the given input gesture with one of a plurality of pre-defined gestures associated with a user'"'"'s hand.

32. The interface system of claim 30, further comprising means for displaying output information on the background surface, such that the background surface operates as a touchscreen input/output device.

33. The interface system of claim 30, wherein the means for generating a first plurality of images comprises means for generating a first plurality of silhouette images of the sensorless input object in the foreground of the background surface and the means for generating the second plurality of images comprises means for generating a second plurality of silhouette images of the sensorless input object in the foreground of the background surface.

34. The interface system of claim 30, wherein the means for generating a first plurality of images comprises means for generating a first plurality of shadow images of the sensorless input object on the background surface and the means for generating the second plurality of images comprises means for generating a second plurality of shadow images of the sensorless input object on the background surface.

35. The interface system of claim 30, further comprising means for calibrating the means for interpolating three-dimensional location information associated with the sensorless input object based on an amount of parallax separation of a matched pair of images of a calibration pattern on the background surface.