Creating a virtual environment for touchless interaction

• US 9,501,810 B2
• Filed: 09/12/2014
• Issued: 11/22/2016
• Est. Priority Date: 09/12/2014
• Status: Active Grant

• Current Assignee: GE Digital Holdings LLC (GE Aerospace)
• Original Assignee: General Electric Company
• Inventors: Maximo, Andre de Almeida
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US14/485,407
• Publication Number:
  US 20160078679A1
• Time in Patent Office: 802 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

This disclosure is directed to a touchless interactive environment. An input device may be configured to capture electronic images corresponding to physical objects detectable within a physical three-dimensional region. A computer system may establish a virtual three-dimensional region mapped to the physical three-dimensional region, with the virtual three-dimensional region defining a space where a plurality of virtual objects are instantiated based on the plurality of electronic images. The computer system may select a virtual object from the plurality of virtual objects as one or more commanding objects, with the one or more commanding objects indicating a command of a graphical user interface to be performed based on a position of the one or more commanding objects. The computer system may then perform the command of the graphical user interface based on the position of the one or more commanding objects.

Claims

1. A computer-implemented method comprising:

capturing, with a touchless input device, a first plurality of images corresponding to a plurality of physical objects detected within a physical three-dimensional region;

establishing, with one or more processors, a virtual three-dimensional region mapped to the physical three-dimensional region, the virtual three-dimensional region defining a space where a plurality of virtual objects are instantiated based on the first plurality of images;

displaying, on a display in communication with the one or more processors, a second plurality of images, each displayed image corresponding to a different virtual object selected from the plurality of virtual objects;

selecting, with the one or more processors, a virtual object from the plurality of virtual objects as one or more commanding objects, the one or more commanding objects indicating a command of a graphical user interface to be performed based on a position of the commanding object; and

performing the command of the graphical user interface based on the position of the one or more commanding objects.

2. The computer-implemented method of claim 1, further comprising:

mapping the virtual-three dimensional region to the physical three-dimensional region by applying a transformation matrix to convert a plurality of coordinates from the physical three-dimensional region to a corresponding plurality of coordinates from the virtual three-dimensional region.

3. The computer-implemented method of claim 1, wherein the plurality of objects captured by the input device comprise a human hand segmented into a plurality of segments.

4. The computer-implemented method of claim 3, wherein the instantiated virtual objects each represent a segment of the human hand or a physical tool being held by the human hand.

5. The computer-implemented method of claim 1, further comprising:

establishing a region of interaction displayed on a display, the region of interaction comprising a two-dimensional plane in which the plurality of virtual objects are displayed as the second plurality of images.

6. The computer-implemented method of claim 5, further comprising applying a transformation matrix to a plurality of coordinates of the virtual three-dimensional region to obtain a plurality of coordinates of the region of interaction.

7. The computer-implemented method of claim 1, further comprising:

displaying at least one image selected from the second plurality of images as having a size that is dependent on its corresponding virtual object, the size of the virtual object being dependent on a distance that its corresponding physical object is located from the graphical user interface.

8. A system comprising:

an input device configured to capture a first plurality of images corresponding to a plurality of physical objects detectable within a physical three-dimensional region;

a non-transitory;

computer-readable medium storing computer-executable instructions; and

one or more processors in communication with the input device and the non-transitory, computer-readable medium that, having executed the computer-executable instructions, are configured to;

establish a virtual three-dimensional region mapped to the physical three-dimensional region, the virtual three-dimensional region defining a space where a plurality of virtual objects are instantiated based on the first plurality of images;

display, on a display in communication with the one or more processors, a second plurality of images, each displayed image corresponding to a different virtual object selected from the plurality of virtual objects;

select a virtual object from the plurality of virtual objects as one or more commanding objects, the one or more commanding objects indicating a command of a graphical user interface to be performed based on a position of the one or more commanding objects; and

perform the command of the graphical user interface based on the position of the commanding object.

9. The system of claim 8, wherein the one or more processors are further configured to:

map the virtual-three dimensional region to the physical three-dimensional region by applying a transformation matrix to convert a plurality of coordinates from the physical three-dimensional region to a corresponding plurality of coordinates from the virtual three-dimensional region.

10. The system of claim 8, wherein the plurality of objects captured by the input device comprise a human hand segmented into a plurality of segments.

11. The system of claim 10, wherein the instantiated virtual objects each represent a segment of the human hand or a physical tool being held by the human hand.

12. The system of claim 8, wherein the one or more processors are further configured to:

establish a region of interaction displayed on a display, the region of interaction comprising a two-dimensional plane in which the plurality of virtual objects are displayed as the second plurality of images.

13. The system of claim 12, wherein the one or more processors are further configured to apply a transformation matrix to a plurality of coordinates of the virtual three-dimensional region to obtain a plurality of coordinates of the region of interaction.

14. The system of claim 8, wherein the one or more processors are further configured to:

display at least one image selected from the second plurality of images as having a size that is dependent on its corresponding virtual object, the size of the virtual object being dependent on a distance that its corresponding physical object is located from the graphical user interface.

15. A non-transitory, computer-readable medium having computer-executable instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

receiving a first plurality of images captured by an input device, the first plurality of images corresponding to a plurality of physical objects detected within a physical three-dimensional region;

establishing a virtual three-dimensional region mapped to the physical three-dimensional region, the virtual three-dimensional region defining a space where a plurality of virtual objects are instantiated based on the first plurality of images;

displaying, on a display in communication with the one or more processors, a second plurality of images, each displayed image corresponding to a different virtual object selected from the plurality of virtual objects;

selecting a virtual object from the plurality of virtual objects as one or more commanding objects, the one or more commanding objects indicating a command of a graphical user interface to be performed based on a position of the one or more commanding objects; and

performing the command of the graphical user interface based on the position of the one or more commanding objects.

16. The non-transitory, computer-readable medium of claim 15, wherein the method further comprises:

mapping the virtual-three dimensional region to the physical three-dimensional region by applying a first transformation matrix to convert a plurality of coordinates from the physical three-dimensional region to a corresponding plurality of coordinates from the virtual three-dimensional region;

establishing a region of interaction displayed on a display, the region of interaction comprising a two-dimensional plane in which the plurality of virtual objects are displayed; and

applying a second transformation matrix to the plurality of coordinates of the virtual three-dimensional region to obtain a plurality of coordinates of the region of interaction.

17. The non-transitory, computer-readable medium of claim 15, wherein the plurality of objects captured by the input device comprise a human hand segmented into a plurality of segments.

18. The non-transitory, computer-readable medium of claim 17, wherein the instantiated virtual objects each represent a segment of the human hand or a physical tool being held by the human hand.

19. The non-transitory, computer-readable medium of claim 15, wherein the method further comprises:

displaying at least one image selected from the second plurality of images as having a size that is dependent on its corresponding virtual object, the size of the virtual object being dependent on a distance that its corresponding physical object is located from the graphical user interface.