MOBILE TERMINAL AND METHOD FOR GENERATING CONTROL COMMAND USING MARKER ATTACHED TO FINGER

• US 20150078617A1
• Filed: 01/21/2014
• Published: 03/19/2015
• Est. Priority Date: 09/13/2013
• Status: Active Grant

• Current Assignee: SungKyunKwan University's Research & Business Foundation (SungKyunKwan University)
• Original Assignee: SungKyunKwan University's Research & Business Foundation (SungKyunKwan University)
• Inventors: HONG, Kwang-Seok, OH, Byung-Hun, CHANG, Kyung-Won
• Granted Patent: US 9,304,598 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/21375 involving differential geometry, e.g. embedding of pattern manifold
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 1/6005 with luminance or chrominance signals, e.g. LC1C2, HSL or YUV

Abstract

A mobile terminal 200 generating a control command using a marker attached to a finger includes a camera 210 configured to receive a target image including a finger region, a fingertip region detection module 220 configured to detect a finger region in the target image using an image processing technique, and a fingertip region in the finger region, a marker region detection module 230 configured to detect a color marker region attached to the finger in the target image using a color detection technique, a control module 240 configured to operate the fingertip region detection module or the marker region detection module, based on an input from a user, and a coordinate calculation module 250 configured to calculate three-dimensional coordinate values using the fingertip region or the color marker region.

Claims

1. A mobile terminal generating a control command using a marker attached to a finger, comprising:

a camera configured to receive a target image including a color marker region attached to a finger;

a marker region detection module configured to detect the color marker region in the target image using a color detection technique; and

a coordinate calculation module configured to calculate three-dimensional coordinate values using the color marker region.

2. The mobile terminal of claim 1, wherein the marker region detection module detects the color marker region in the target image using a CAMshift algorithm.

3. The mobile terminal of claim 2, wherein, when the color marker region is designated as a region of interest, the marker region detection module converts an RGB image to an HSV image, generates a base image using maximum frequency characteristics of a histogram for an H channel of the HSV image, generates a mask image by summing results in which threshold values for both of an S channel and a V channel of the HSV image are derived, and detects the color marker region by combining the base image and the mask image.

4. The mobile terminal of claim 1, wherein the coordinate calculation module calculates a Z coordinate value (FingerPoint.Z) of the three-dimensional coordinate values based on a width or diameter of the color marker region, using the following equation:

5. The mobile terminal of claim 4, wherein the coordinate calculation module calculates an X coordinate value (FingerPoint.X) of the three-dimensional coordinate values based on the width or diameter of the color marker region, using the following equation:

FingerPoint.X=ColorPoint.x (pixel)×

DistancePerPixel (cm)

(wherein ColorPoint.x is an x-pixel coordinate value for a center pixel of the color marker region in the target image,

6. The mobile terminal of claim 4, wherein the coordinate calculation module calculates a Y coordinate value (FingerPoint.Y) of the three-dimensional coordinate values based on the width or diameter of the color marker region, using the following equation:

FingerPoint.Y=ColorPoint.y (pixel)×

DistancePerPixel (cm)

(wherein ColorPoint.y is a y-pixel coordinate value for a center pixel of the color marker region in the target image,

7. A mobile terminal generating a control command using a marker attached to a finger, comprising:

a camera configured to receive a target image including a finger region;

a fingertip region detection module configured to detect a finger region in the target image using an image processing technique, and a fingertip region in the finger region;

a marker region detection module configured to detect a color marker region attached to a finger in the target image using a color detection technique;

a control module configured to operate the fingertip region detection module or the marker region detection module based on an input from a user; and

a coordinate calculation module configured to calculate three-dimensional coordinate values using the fingertip region or the color marker region.

8. The mobile terminal of claim 7, wherein the fingertip region detection module detects the fingertip region in the target image using a skin color detection technique or an edge detection technique, and the marker region detection module detects the color marker region in the target image using a CAMshift algorithm.

9. The mobile terminal of claim 7, wherein the coordinate calculation module calculates an X coordinate value and a Y coordinate value using a reference point of the fingertip region, and a Z coordinate value using at least one of a width, a height, and an area of the fingertip region.

10. The mobile terminal of claim 7, wherein the coordinate calculation module calculates a Z coordinate value using at least one of a width, a diameter, a height, and an area of the color marker region, and an X coordinate value and a Y coordinate value by calculating an actual distance which a center pixel of the color marker region travels using the Z coordinate value.

11. A method of generating a control command in a terminal using a marker attached to a finger, comprising:

receiving, by a terminal, a target image including a finger region to which a color marker is attached from a camera;

inputting, by the terminal, a command to use the finger region or a color marker region included in the target image;

detecting, by the terminal, a fingertip region in the target region using an image processing technique when the command to use the finger region is input, or detecting the color marker region when the command to use the color marker region is input; and

calculating, by the terminal, three-dimensional coordinate values using the fingertip region or the color marker region.

12. The method of claim 11, wherein the command to use the color marker region is a touch input to the color marker region output on a touch display device of the terminal.

13. The method of claim 11, wherein in the detecting of the color marker region, the color marker region is detected in the target image using a CAMshift algorithm, and the detecting of the color marker region comprises:

converting an RGB image including the color marker region to an HSV image;

generating a base image using maximum frequency characteristics of a histogram for an H channel of the HSV image;

generating a mask image by summing results in which threshold values for both of an S channel and a V channel of the HSV image are derived; and

generating a final image by combining the base image and the mask image.

14. The method of claim 11, wherein the calculating of the coordinate values includes, when using the finger region, calculating an X coordinate value and a Y coordinate value using a reference point of the fingertip region, and calculating a Z coordinate value using at least one of a width, a height, and an area of the fingertip region.

15. The method of claim 11, wherein the calculating of the coordinate values includes, when using the color marker region, calculating a Z coordinate value using at least one of a width, a diameter, a height, and an area of the color marker region, and calculating an X coordinate value and a Y coordinate value by calculating an actual distance which a center pixel of the color marker region travels using the Z coordinate value.

16. The method of claim 11, wherein, when calculating the coordinate values using the color marker region, the terminal calculates a Z coordinate value (FingerPoint.Z) of the three-dimensional coordinate values based on a width or diameter of the color marker region, using the following equation:

17. The method of claim 16, wherein, when calculating the coordinate values, the terminal calculates an X coordinate value (FingerPoint.X) of the three-dimensional coordinate values based on the width of the color marker region, using the following equation:

FingerPoint.X=ColorPoint.x (pixel)×

DistancePerPixel (cm)

(wherein ColorPoint.x is an x-pixel coordinate value for a center pixel of the color marker region in the target image,

18. The method of claim 16, wherein, when calculating the coordinate values, the terminal calculates a Y coordinate value (FingerPoint.Y) of the three-dimensional coordinate values based on the width of the color marker region, using the following equation:

FingerPoint.Y=ColorPoint.y (pixel)×

DistancePerPixel (cm)

(wherein ColorPoint.y is a y-pixel coordinate value for the center pixel of the color marker region in the target image,