User gesture recognition

• US 10,310,619 B2
• Filed: 03/05/2015
• Issued: 06/04/2019
• Est. Priority Date: 03/21/2014
• Status: Active Grant

• Current Assignee: iNUI Studio SA
• Original Assignee: Artnolens SA
• Inventors: Raulot, Olivier
• Primary Examiner(s): Patel, Sanjiv D.
• Application Number:
  US15/128,074
• Publication Number:
  US 20170090586A1
• Time in Patent Office: 1,552 Days
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

A method and device for gesture recognition, wherein the gesture is executed by a user in a gesture region which may be defined relative to a display surface. In an embodiment, the gesture comprises a select gesture and the device comprises at least three cameras operating in the visual range where a first camera is used to determine a horizontal location of the select gesture and the other cameras are used to determine a vertical location thereof. A device for providing input to a computing device comprises a rectangular display having a viewing surface and at least three cameras having respective fields of view. A first camera and a second camera are located at respective adjacent corners of the display and a third camera is located at an edge of the display between the first and second cameras.

Claims

1. A method of providing input to a computing device, the computing device comprising a processor, memory, a display having a viewing surface and at least three cameras having respective fields of view, the method comprising the steps of:

capturing corresponding images with the cameras;

identifying input gestures using the captured images, wherein the input gestures are spaced from the viewing surface of the display, and wherein said identifying input gestures includes processing each image by designating a region of the image as a gesture region and determining changes in the gesture region, wherein the gesture regions are spaced from the viewing surface of the display; and

interpreting the identified input gestures as input to the computing device;

wherein at least one of the cameras is orientated such that the respective field of view lies adjacent to the viewing surface of the display;

wherein said identifying input gestures in the captured images comprises first choosing between images captured by either a first camera or a second camera of the at least three cameras and then using the chosen images to identify the input gestures,

wherein said identifying input gestures in the captured images is further based on images captured by a third camera of the at least three cameras,

wherein said choosing between images captured by either the first camera or the second camera comprises comparing image information between images captured by the first camera and images captured by the second camera, and

wherein said comparing image information comprises comparing a standard deviation of information contained in an image captured by the first camera to a standard deviation of information contained in an image captured by the second camera,

wherein each of said at least three cameras operates to capture only two-dimensional image information.

2. The method according to claim 1 wherein a determination of a first dimension for use in identifying input gestures is based on images captured by the third camera.

3. The method according to claim 2 wherein an input gesture is recognized by determining a change in images captured by the first camera at different times and calculating a threshold based on the change.

4. The method according to claim 2 where a determination of a second dimension for use in identifying input gestures is based on images captured by the first camera or the second camera.

5. The method according to claim 4 wherein an input gesture is performed by a user'"'"'s appendage and wherein the step of determining the second dimension of the input gesture comprises recognizing the user'"'"'s appendage.

6. The method according to claim 5 wherein the user'"'"'s appendage is recognized according to a predetermined range of hue, saturation and value of pixels in images captured by the second camera.

7. The method according to claim 4, further comprising establishing a mapping between a co-ordinate system of the first camera and/or the second camera and a co-ordinate system of the viewing surface.

8. The method according to claim 7 wherein the mapping comprises a linear mapping.

9. The method according to claim 8 wherein the mapping comprises a first linear mapping applied to a first section of the co-ordinates of the viewing surface and a second linear mapping applied to a second section of the viewing surface.

10. The method according to claim 1, further comprising tracking an input gesture and determining a slide action in dependence on detected motion of the input gesture.

11. The method according to claim 1 wherein all of the cameras are situated near an edge of the display and orientated with intersecting fields of view.

12. The method according to claim 1 wherein the computing device further comprises a fourth camera which is positioned adjacent the third camera and wherein a determination of a first dimension comprises choosing between images captured by the third camera and images captured by the fourth camera.

13. A device for providing input to a computing device, the computing device comprising a processor and memory, the device comprising:

a rectangular display having a viewing surface; and

at least three cameras having respective fields of view, each of the cameras capturing information in two spatial dimensions only,

wherein a first camera and a second camera of the at least three cameras are located at respective adjacent corners of the display, and a third camera of the at least three cameras is located at an edge of the display between the first and second cameras,

wherein each camera is orientated to capture image information of a gesture region spaced from the viewing surface of the display to thereby recognize gestures in the gesture region from said image information,

wherein the processor is configured to recognized the gestures in the gesture region by first choosing between images captured by either the first camera or the second camera of the at least three cameras and then using the chosen images together with images captured by the third camera of the at least three cameras to recognize the gestures,

wherein said choosing between images captured by either the first camera or the second camera comprises comparing image information between images captured by the first camera and images captured by the second camera,

wherein said comparing image information comprises comparing a standard deviation of information contained in an image captured by the first camera to a standard deviation of information contained in an image captured by the second camera, and

wherein each of said first, second and third cameras operates to capture only two-dimensional image information.

14. The device according to claim 13, further comprising a fourth camera located at an edge of the display between the first and second cameras.

15. The device according to claim 14, further comprising a mounting frame located at a periphery of the display, wherein at least said first and second camera are mounted to the mounting frame.

16. The device according to claim 15 wherein each camera is mounted to the mounting frame.

17. The device according to claim 14 wherein at least one of the cameras is orientated such that the respective field of view lies adjacent to the viewing surface of the display.

18. The device according to claim 14 wherein the input comprises an input gesture and wherein the memory stores instructions which, when processed by the processor, identify input gestures using the captured images,

wherein identifying input gestures in the captured images comprises choosing between images captured by either the first camera or the second camera and further using the chosen image to identify input gestures.

19. The device according to claim 18 wherein identifying input gestures in the captured images is further based on images captured by the third camera.

20. The device according to claim 18 wherein choosing between images captured by either the first camera or the second camera comprises comparing information between images captured by the first camera and images captured by the second camera.

21. The device according to claim 20 wherein the input gesture is performed by a user'"'"'s appendage and wherein the instructions further comprise instructions for recognizing the user'"'"'s appendage.

22. The device according to claim 21 wherein the user'"'"'s appendage is recognized according to a predetermined range of hue, saturation and value of pixels in images captured by one or more of the cameras.

23. The device according to claim 18 wherein the instructions, when processed by the processor, determine a first dimension for use in identifying input gestures based on images captured by the third camera and determine a second dimension for use in identifying input gestures based on images captured by the first camera or the second camera, wherein the input gesture is performed by a user'"'"'s appendage, wherein the step of determining the second dimension of the input gesture comprises recognizing the user'"'"'s appendage, and wherein the user'"'"'s appendage is recognized according to a predetermined range of hue, saturation and value of pixels in images captured by the second camera.

24. The device according to claim 14 wherein at least one of the cameras is orientated so that the respective field of view lies adjacent to the viewing surface of the display.

25. The device according to claim 14 wherein all of the cameras are situated near an edge of the display and orientated with intersecting fields of view.

26. The device according to claim 18 wherein the instructions, when processed by the processor, process each image by designating a region of the image as a gesture region and determining changes in the gesture region.

27. The device according to claim 26 wherein the instructions, when processed by the processor, further define respective gesture regions adjacent to the viewing surface of the display.

28. The device according to claim 26 wherein the instructions, when processed by the processor, further define respective gesture regions spaced from the viewing surface of the display.