REMOTE MANIPULATION DEVICE AND METHOD USING A VIRTUAL TOUCH OF A THREE-DIMENSIONALLY MODELED ELECTRONIC DEVICE

• US 20140184499A1
• Filed: 07/06/2012
• Published: 07/03/2014
• Est. Priority Date: 07/11/2011
• Status: Active Grant

• Current Assignee: Vtouch Co Ltd
• Original Assignee: Vtouch Co Ltd
• Inventors: Kim, Seok-Joong
• Granted Patent: US 9,367,138 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 17/02 using a radio link
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/91 Remote control based on location and proximity
• CPC Class Codes: G08C 23/04 using light waves, e.g. infrared

Abstract

Disclosed is a remote control apparatus and method using a virtual touch of a three-dimensionally modeled electronic device. The remote control apparatus includes a 3D coordinate calculation unit, an electronic device detecting unit, a user detecting unit, an electronic device driving control unit, and an electronic device information database. The remote control apparatus enables the user to remotely control an operation of an electronic device using a virtual touch through a motion or gesture of the user'"'"'s finger.

Claims

1. A remote control apparatus using a virtual touch of a three-dimensionally modeled electronic device, comprising:

a 3D coordinate calculation unit extracting 3D coordinate data of a user'"'"'s body and the electronic device using an image received through image photographing;

an electronic device detecting unit recognizing spatial location information and product information of the electronic device using the 3D coordinate data of the electronic device extracted by the 3D coordinate calculation unit and generating a virtual contact point region for a virtual touch with a user and the electronic device based on direction information corresponding to a straight line between the user and the electronic device;

a user detecting unit calculating a virtual touch location and a change of the location for operating the electronic device using the 3D coordinate data of the user'"'"'s body calculated by the 3D coordinate calculation unit and generating a command code for operating a predetermined electronic device based on the change of the virtual touch location that is calculated;

an electronic device driving control unit converting the command code generated by the user detecting unit into a control command signal for operating a corresponding electronic device to transmit an operation command signal to the corresponding electronic device when the command code generated by the user detecting unit is inputted; and

an electronic device information database (DB) storing product information comprising a manufacturer and a model name of the electronic device, protocol information applied to the operation of each product, and operation method information of each electronic device.

2. The remote control apparatus of claim 1, wherein the 3D coordinate calculation unit comprises:

an image acquisition unit comprising two or more image sensors disposed at different locations to photograph the user'"'"'s body and the electronic device from different angles; and

a spatial coordinate calculation unit for calculating the 3D coordinate data of the user'"'"'s body and the electronic device using optical triangulation based on the images received from the image acquisition unit and photographed from the different angles.

3. The remote control apparatus of claim 1, wherein the 3D coordinate calculation unit comprises:

a lighting assembly comprising a light source and a light diffuser to project a speckle pattern on the user'"'"'s body and the electronic device;

an image acquisition unit comprising an image sensor and a lens to capture the speckle pattern on the user'"'"'s body and the electronic device projected by the lighting assembly; and

a spatial coordinate calculation unit calculating the 3D coordinate data of the user'"'"'s body and the electronic device using the speckle pattern captured by the image acquisition unit.

4. The remote control apparatus of claim 1, wherein the electronic detecting unit comprises:

a 3D modeling unit performing 3D modeling according to a type of the electronic device based on the 3D spatial coordinate of the electronic device calculated by the 3D coordinate calculation unit;

an electronic device setting unit, based on the type of the 3D-modeled electric device, comparing the 3D-modeled electronic device with the electronic device information pre-stored in the electronic device information DB to detect the corresponding electronic device, and setting a control signal according to the operation of the detected electronic device; and

a touch location matching unit for matching direction information corresponding to a straight line between the user and the electronic device to generate a virtual two-dimensional contact point region for the virtual touch between the user and the electronic device based on the direction information that is matched

5. The remote control apparatus of claim 1, wherein the user detecting unit comprises:

a touch location calculation unit for calculating a first spatial coordinate and a second spatial coordinate based on a 3D spatial coordinate of the user'"'"'s body calculated by the 3D coordinate calculation unit to calculate contact point coordinate data where a straight line connecting the first spatial coordinate and the second spatial coordinate meets the virtual contact point region; and

a virtual touch processing unit determining whether or not there is a change of the contact point coordinate data calculated by the touch location calculation unit and generating a command code for operating a predetermined electronic device when it is determined that there is a change of the contact point coordinate data.

6. The remote control apparatus of claim 5, wherein the first spatial coordinate is a three-dimensional coordinate of a tip of one of user'"'"'s fingers or a tip of a pointer gripped by the user'"'"'s fingers, and the second spatial coordinate is a three-dimensional coordinate of a central point of one of user'"'"'s eyes.

7. The remote control apparatus of claim 5, wherein the virtual touch processing unit determines whether or not there is a change of the contact point coordinate data for a predetermined time or more after the initial contact point coordinate data is calculated, and when there is a change of the contact point coordinate data for the predetermined time or more, it is determined that there is a change of the contact point coordinate data

8. The remote control apparatus of claim 7, wherein:

the virtual touch processing unit determines whether or not there is a change of the contact point coordinate data for a predetermined time or more after the initial contact point coordinate data is calculated, and when there is no change of the contact point coordinate data for the predetermined time or more, the virtual touch processing unit determines whether or not there is a change in distance equal to or larger than a predetermined distance between the first spatial coordinate and the second spatial coordinate; and

when there is a change in distance equal to or larger than a predetermined distance, the virtual touch processing unit generates a command code for operating a predetermined electronic device.

9. The remote control apparatus of claim 1, wherein the electronic device driving control unit transmits the operation command signal by a short range wireless communication method comprising Infrared Ray (IR) signal, WiFi, and Bluetooth.

10. A remote control method using a virtual touch of a three-dimensionally modeled electronic device, comprising:

(A) extracting 3D coordinate data of a user'"'"'s body and the electronic device using an image received through image photographing;

(B) scanning the electronic device using 3D coordinate data according to the electronic device among the calculated 3D coordinate data to perform 3D modeling according to a type of the electronic device;

(C) based on the type of the 3D-modeled electric device, comparing the 3D-modeled electronic device with the electronic device information pre-stored in an electronic device information database to detect the corresponding electronic device, and setting a control signal according to the operation of the detected electronic device;

(D) matching direction information corresponding to a straight line between the user and the electronic device to generate a virtual contact point region for the virtual touch between the user and the electronic device based on the direction information that is matched;

(E) calculating a first spatial coordinate having a 3D coordinate of a tip of a user'"'"'s finger and a second spatial coordinate having a 3D coordinate of a central point of a user'"'"'s eye using the 3D coordinate data of the user'"'"'s body among the calculated 3D coordinate data and calculating contact point coordinate data where a straight line connecting the first spatial coordinate and the second spatial coordinate meets the virtual contact point region that is generated;

(F) determining whether or not there is a change of the contact point coordinate data that meets the virtual 2D contact point region that is generated, and when there is a change of the contact point coordinate data, generating and outputting a command code of a predetermined electronic device so as to correspond to the change of the contact point coordinate data; and

(G) converting the generated command code into a control command signal for operating a corresponding electronic device to transmit an operation command signal to the corresponding electronic device.

11. The remote control method of claim 10, wherein (A) the extracting of the 3D coordinate data comprises:

photographing the user'"'"'s body and the electronic device from different angles through an image acquisition unit comprising two or more image sensors disposed at different locations; and

calculating the 3D coordinate data of the user'"'"'s body and the electronic device using optical triangulation based on images photographed from the different angles.

12. The remote control method of claim 10, wherein (A) the extracting of the 3D coordinate data comprises:

projecting a speckle pattern on the user'"'"'s body and the electronic device through a light assembly;

capturing the speckle pattern on the user'"'"'s body and the electronic device projected by the light assembly; and

calculating the 3D coordinate data of the user'"'"'s body and the electronic device using the speckle pattern that is captured.

13. The remote control method of claim 10, wherein (F) the determining of whether or not there is a change of the contact point coordinate data comprises determining whether or not there is a change of the contact point coordinate data for a predetermined time or more after the initial contact point coordinate data are calculated, and when there is a change of the contact point coordinate data for the predetermined time or more, determining that there is a change of the contact point coordinate data.

14. The remote control method of claim 13, wherein (F) the determining of whether or not there is a change of the contact point coordinate data comprises:

determining whether or not there is a change of the contact point coordinate data for a predetermined time or more after the initial contact point coordinate data is calculated, and when there is no change of the contact point coordinate data for the predetermined time or more, determining whether or not there is a change in distance equal to or larger than a predetermined distance between the first spatial coordinate and the second spatial coordinate; and

generating a command code for operating a predetermined electronic device when there is a change in distance equal to or larger than a predetermined distance.

15. The remote control method of claim 10, wherein (G) the converting of the generated command code into a control command signal comprises transmitting the operation command signal by a short range wireless communication method comprising Infrared Ray (IR) signal, WiFi, and Bluetooth.

16. The remote control method of claim 10, wherein (F) the determining of whether or not there is a change of the contact point coordinate data comprises:

setting a touch region for a virtual touch operation in the generated 2D contact point region according to the operation method information pre-stored in the electronic device information database;

calculating the contact point data where the straight line connecting the first spatial coordinate and the second spatial coordinate meets the virtual contact point region; and

performing, by a user, at least one virtual touch operation of upward, downward, left, right, forward, and backward directions in the preset touch region of the virtual contact point region to request an operation of the electric device, and when it is determined that there is a change of the calculated contact point coordinate data according to a change of the first spatial coordinate, generating a command code for operating a predetermined electric device so as to correspond to the change of the contact point coordinate data.

17. The remote control method of claim 10, wherein the contact point region is located between the electric device and the user regardless of the front, lateral, rear direction of the electric device to allow the user to operate the electric device through the virtual touch in any direction of the electronic device.