Gesture recognition interface system

• US 8,589,824 B2
• Filed: 07/13/2006
• Issued: 11/19/2013
• Est. Priority Date: 07/13/2006
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Hillis, William Daniel, Nishihara, H Keith, Hsu, Shi-Ping
• Primary Examiner(s): TRAN, TUYETLIEN T
• Application Number:
  US11/485,788
• Publication Number:
  US 20080013826A1
• Time in Patent Office: 2,686 Days
• Field of Search: 715/856, 715/863, 382/103
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system and method is provided for a gesture recognition interface system. The interface system may comprise a first and second light source positioned to illuminate a background surface. The interface system may also comprise at least one camera operative to receive a first plurality of images based on a first reflected light contrast difference between the background surface and a sensorless input object caused by the first light source and a second plurality of images based on a second reflected light contrast difference between the background surface and the sensorless input object caused by the second light source. The interface system may further comprise a controller operative to determine a given input gesture based on changes in relative locations of the sensorless input object in the first plurality of images and the second plurality of images. The controller may further be operative to initiate a device input associated with the given input gesture.

Claims

1. A gesture recognition interface system comprising:

a first IR light source and a second IR light source positioned to illuminate a retroreflective background surface;

a first IR filtered camera operative to receive a first plurality of images based on a first reflected light contrast difference between the retroreflective background surface and a sensorless input object caused by the first IR light source and a second IR filtered camera operative to receive a second plurality of images based on a second reflected light contrast difference between the retroreflective background surface and the sensorless input object caused by the second IR light source, the first and second pluralities of images being respective silhouette images of the sensorless input object; and

a controller operative to determine a given input gesture based on changes in relative locations of the sensorless input object in the first plurality of images and the second plurality of images, and being further operative to initiate a device input associated with the given input gesture.

2. The interface system of claim 1, wherein the sensorless input object is a user'"'"'s hand.

3. The interface system of claim 2, wherein the controller is further operative to determine a three-dimensional physical location of each extended fingertip associated with the user'"'"'s hand, and wherein the one or more of the user'"'"'s hand gestures are associated with changes in the three-dimensional physical location of each extended fingertip.

4. The interface system of claim 1, wherein the given input gesture corresponds to one of a mouse command, a zoom command, a rotate command, an environment command, and a laser pointer gesture.

5. The interface system of claim 1, wherein the given input gesture is a six-degree of freedom input gesture based on changes in a three-dimensional location and an orientation of the sensorless input object.

6. The interface system of claim 1, wherein the controller comprises a plurality of peak detectors that are operative to determine a three-dimensional physical location of at least one end-point of the sensorless input object associated with each of the first plurality of images and the second plurality of images of the sensorless input object.

7. The interface system of claim 6, wherein the controller is further operative to determine a length, a thickness, and an orientation of the sensorless input object at elongated regions that comprise the at least one end-point of the sensorless input object.

8. The interface system of claim 6, wherein the retroreflective background surface is non-planar, and wherein the controller determines the three-dimensional physical location of the at least one end-point of the sensorless input object relative to a three-dimensional height function associated with the retroreflective background surface.

9. The interface system of claim 6, wherein each of the plurality of peak detectors is associated with the first plurality of images and the second plurality of images of the sensorless input object, respectively, such that each of the plurality of peak detectors determines a two-dimensional physical location of the at least one end-point of the sensorless input object, and further comprising a location resolver operative to interpolate the three-dimensional physical location of the at least one end-point of the sensorless input object based on each two-dimensional physical location of the at least one end-point of the sensorless input object associated with the respective one of the plurality of peak detectors.

10. The interface system of claim 9, wherein the controller further comprises a two-dimensional Laplacian of Gaussian convolution filter associated with each of the plurality of peak detectors, the two-dimensional Laplacian of Gaussian convolution filter being operative to input either the first and second plurality of images or a contrast enhanced version of the first and second plurality of images, such that the plurality of peak detectors determine the two-dimensional physical location of the at least one end-point of the sensorless input object based on a convolution filtered output from the respective two-dimensional Laplacian of Gaussian convolution filter.

11. The interface system of claim 9, wherein the controller further comprises a spatial bandpass filter associated with each of the plurality of peak detectors, the spatial bandpass filter being operative to input either the first and second plurality of images or a contrast enhanced version of the first and second plurality of images, such that the plurality of peak detectors determine the two-dimensional physical location of the at least one end-point of the sensorless input object based on a filtered output from the respective spatial bandpass filter.

12. The interface system of claim 1, wherein the retroreflective background surface is also operative as a display device, such that the retroreflective background surface operates as a touchscreen input/output device that allows a user to interact with each portion of displayed visual content on the retroreflective background surface via touch gestures.

13. The interface system of claim 1, wherein the first plurality of images and the second plurality of images comprise a plurality of matched pairs of images of the sensorless input object, and wherein the controller determines a three-dimensional physical location of the sensorless input object relative to the retroreflective background surface based on an amount of parallax separation between the matched pairs of images of the sensorless input object.

14. The interface system of claim 13, wherein the controller comprises an automated calibration unit operative to calibrate the gesture recognition interface system based on an amount of parallax separation between matched pairs of images of a calibration pattern on the retroreflective background surface.

15. The interface system of claim 1, wherein the controller comprises a memory operative to store a plurality of pre-defined gestures, and wherein the controller determines the given input gesture based on a comparison of the relative locations of the first plurality of images and the second plurality of images with the plurality of pre-defined gestures.

16. The interface system of claim 15, wherein the memory is further operative to be programmed with additional pre-defined gestures, the additional pre-defined gestures being input to the memory based on a user performing gestures to which the additional pre-defined gestures correspond.

17. A method of providing device inputs, the method comprising:

illuminating a retroreflective background surface with a first IR light source and a second IR light source;

providing simulated inputs over the retroreflective background surface via gestures associated with a sensorless input object;

generating a first plurality of silhouette images associated with the sensorless input object based on a reflected light contrast between the sensorless input object and the illuminated retroreflective background surface caused by the first IR light source;

generating a second plurality of silhouette images associated with the sensorless input object based on a reflected light contrast between the sensorless input object and the illuminated retroreflective background surface caused by the second IR light source;

determining a plurality of three-dimensional physical locations of the sensorless input object based on a relative separation of the sensorless input object in the first plurality of silhouette images relative to the second plurality of silhouette images;

determining if changes in the plurality of three-dimensional physical locations of the sensorless input object correspond to any of a plurality of pre-defined gestures; and

providing at least one device input based on a given one of the plurality of pre-defined gestures upon determining that the changes in the plurality of three-dimensional physical locations of the sensorless input object correspond to the given one of the plurality of pre-defined gestures.

18. The method of claim 17, wherein providing simulated inputs over the background surface comprises providing simulated inputs via gestures associated with a user'"'"'s hand over the background surface.

19. The method of claim 18, wherein determining the plurality of three-dimensional physical locations of the sensorless input object comprises determining a plurality of three-dimensional physical locations of each extended fingertip associated with the user'"'"'s hand.

20. The method of claim 17, wherein generating the first plurality of silhouette images associated with the sensorless input object and generating the second plurality of silhouette images associated with the sensorless input object comprises generating a matched pair of silhouette images associated with the sensorless input object, and wherein determining the plurality of three-dimensional physical locations of the sensorless input object comprises determining a plurality of three-dimensional physical locations of at least one end-point associated with the sensorless input object.

21. The method of claim 20, wherein illuminating the background surface comprises illuminating a non-planar background surface, and wherein determining the plurality of three-dimensional physical locations of the at least one end-point associated with the sensorless input object comprises determining a plurality of three-dimensional physical locations of the at least one end-point associated with the sensorless input object relative to a three-dimensional height function associated with the non-planar background surface.

22. The method of claim 20, wherein determining the plurality of three-dimensional physical locations of the at least one end-point associated with the sensorless input object comprises either performing a two-dimensional Laplacian of Gaussian convolution operation on each of the first plurality of silhouette images associated with the sensorless input object and each of the second plurality of silhouette images associated with the sensorless input object, or performing a two-dimensional Laplacian of Gaussian convolution operation on a contrast enhanced version of each of the first plurality of silhouette images associated with the sensorless input object and a contrast enhanced version of each of the second plurality of silhouette images associated with the sensorless input object.

23. The method of claim 20, wherein determining the plurality of three-dimensional physical locations of the at least one end-point associated with the sensorless input object comprises determining a height of the at least one end-point associated with the sensorless input object relative to the illuminated background surface based on an amount of parallax separation between the first plurality of silhouette images and the second plurality of silhouette images of the sensorless input object.

24. The method of claim 23, further comprising calibrating the determined height of the at least one end-point based on an amount of parallax separation between a first plurality of images of a calibration pattern on the background surface and a second plurality of images of the calibration pattern on the background surface.

25. The method of claim 17, further comprising displaying output data on the illuminated background surface, such that the illuminated background surface operates as a touchscreen input/output device, wherein providing at least one device input comprises providing the device input as a touch input corresponding to interaction with displayed visual content displayed at any point on the background surface.

26. A gesture recognition interface system comprising:

a first IR light source configured to provide a first brightness contrast between a retroreflective background surface and a sensorless input object;

a second IR light source configured to provide a second brightness contrast between the retroreflective background surface and a sensorless input object;

a first IR filtered camera configured to generate a first plurality of silhouette images of the sensorless input object based on the first brightness contrast;

a second IR filtered camera configured to generate a second plurality of silhouette images of the sensorless input object based on the second brightness contrast, the first plurality of images and the second plurality of images forming a plurality of matched pairs of silhouette images of the sensorless input object;

a peak detector configured to generate two-dimensional location information associated with at least one end-point of the sensorless input object for each of the first plurality of images and the second plurality of images;

a location resolver configured to interpolate three-dimensional location information associated with the sensorless input object based on the two-dimensional location information associated with the at least one end-point of the sensorless input object for each of the plurality of matched pairs of images of the sensorless input object;

a gesture recognition device configured to translate changes in the three-dimensional location information associated with at least one end-point of the sensorless input object to a given input gesture, and to provide device inputs based on matching the given input gesture with one of a plurality of pre-defined gestures.

27. The interface system of claim 26, wherein the gesture recognition device is configured to provide the device inputs based on matching the given input gesture with one of a plurality of pre-defined gestures associated with a user'"'"'s hand.

28. The interface system of claim 26, further comprising a projector configured to display output information on the background surface, such that the background surface operates as a touchscreen input/output device that allows a user to interact with each portion of displayed visual content on the background surface via touch gestures.

29. The interface system of claim 26, further comprising a calibration component configured to calibrate the location resolver based on an amount of parallax separation of a matched pair of silhouette images of a calibration pattern on the background surface.