Portable terminal and user interface control method thereof based on pattern recognition and analysis of image captured by camera

• US 7,693,333 B2
• Filed: 07/18/2007
• Issued: 04/06/2010
• Est. Priority Date: 08/07/2006
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Ryu, Dong Seok, Hwang, Jae Joon, Hong, Nho Kyung
• Primary Examiner(s): Mariam; Daniel G
• Application Number:
  US11/779,538
• Publication Number:
  US 20080034302A1
• Time in Patent Office: 993 Days
• Field of Search: 382/107, 382/181, 382/298, 382/299, 382/305, 382/313, 382/154, 382/218, 345/156, 715/763, 715863-864
• US Class Current: 382/181
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: H04N 1/00281 with a telecommunication apparatus, e.g. a switched network of teleprinters for the distribution of text-based information, a selective call terminal details of transmission H04N1/00095; establishing a communication with one of a facsimile machine or another apparatus sharing a single line H04N1/32704; interfacing cordless telephone terminals with an accessory to increase the functionality of user interface H04M1/72409
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/0035 User-machine interface; Control console input or output arrangements for computers G06F3/00
• CPC Class Codes: H04N 1/00352 Input means H04N1/00411 takes precedence; input arrangements for computers G06F3/00
• CPC Class Codes: H04N 2101/00 Still video cameras

Abstract

A portable terminal and a user interface control method thereof are provided. In the terminal and method, patterns of a subject image captured by a camera are recognized and analyzed. First and second images changed according to motion (movement, holding away from or drawing close to a user) of the portable terminal are captured by the camera module. Patterns of the first and second images are recognized. The motion of the portable terminal is analyzed based on the recognized patterns. Control commands corresponding to the analyzed motion are selected. A function corresponding to the selected control command is performed. The portable terminal can be intuitively and simply used as its motion is directly applied to user interface control. User interface can be controlled without an additional part, using a camera installed to the portable terminal. Also, the present invention can resolve the problems caused when the conventional navigation key is operated.

Claims

1. A user interface control method of a portable terminal including a camera module, the method comprising:

capturing first and second images through a camera module;

recognizing, by a pattern recognition unit, patterns of the first and second images by calculating positions of target points extracted from the first and second images respectively;

analyzing, by a pattern analysis unit, the recognized patterns by calculating a displacement angle θ

based on the positions of the target points;

selecting, by the pattern analysis unit, a control command based on the analysis result; and

performing by a controller, a function corresponding to the control command.

2. The method of claim 1, wherein the capturing of the first and second images comprises inputting a pattern recognition executing key.

3. The method of claim 1, wherein the calculating of the displacement angle θ

comprises determining θ

by

θ

=tan<sup>−

1</sup>(b/a),

wherein the position of the target point in the first image is (x, y) and the position of the compared target point in the second image is (x+a, y+b).

4. The method of claim 3, wherein the analyzing of the recognized patterns further comprises determining a type of an application currently executed.

5. The method of claim 4, wherein the application comprises at least one of a menu list, a file list, an image file, a message write/check window, a document file, a music file, and a game.

6. The method of claim 1, wherein the selecting of the control command comprises selecting at least one of a control command related to movement of four directions according to the displacement angle θ

and a control command related to movement of free direction.

7. The method of claim 1, wherein the control command comprises at least one of display previous list, move focus to down menu, display next list, and move focus to up menu, which respectively correspond to conditions of −

45°

<

<

45°

, 450<

θ

<

135°

, 135°

<

θ

<

225°

, and 225°

<

θ

<

315°

.

8. The method of claim 1, wherein the control command comprises at least one of move focus to previous file, move focus to down file, move focus to next file, and move focus to up file, which respectively correspond to conditions of −

45°

<

θ

<

45°

, 45°

<

θ

<

135°

, 135°

<

θ

<

225°

, and 225°

<

θ

<

315°

.

9. The method of claim 1, wherein the control command corresponds to the displacement angles to move a focus.

10. The method of claim 1, further comprising:

determining whether a currently executed application can perform pattern recognition; and

displaying a state of whether the currently executed application can perform pattern recognition.

11. The method of claim 10, wherein the displaying of the state comprises displaying a pattern recognition icon when the currently executed application can perform pattern recognition.

12. The method of claim 1, further comprising:

determining whether the pattern recognition has succeeded; and

displaying that the pattern is not recognizable when the pattern recognition has not succeeded.

13. The method of claim 12, wherein the displaying that the pattern is not recognizable comprises displaying at least one of an icon and a phrase.

14. A user interface control method of a portable terminal including a camera module, the method comprising:

capturing first and second images through a camera module;

recognizing, by a pattern recognition unit, patterns of the first and second images by calculating positions of target points extracted from the first and second images respectively;

analyzing, by a pattern analysis unit, the recognized patterns by calculating a distance increment/decrement Δ

d from the positions of the target points and determining whether the distance increment/decrement Δ

d comprises a positive or negative value;

selecting, by the pattern analysis unit, a control command based on the analysis result; and

performing, by a controller, a function corresponding to the control command.

15. The method of claim 14, wherein the calculating of the distance increment/decrement Δ

d comprises determining Δ

d by

d=√

{square root over ((x−

u)²+(y−

v)²)}{square root over ((x−

u)²+(y−

v)²)}, d′

=√

{square root over ((x′

−

u′

)²+(y′

−

v′

)²)}{square root over ((x′

−

u′

)²+(y′

−

v′

)²)}, and Δ

d=d−

d′

wherein the positions of the target points in the first image are (x, y) and (u, v), and the positions of the compared target points in the second image are (x′

, y′

) and (u′

, v′

) wherein (x′

, y′

) are (x+a, y+b) and (u′

, v′

) are (u+c, v+d).

16. The method of claim 14, wherein the selecting of the control command comprises selecting a screen magnification when the distance increment/decrement is positive and selecting a screen reduction when the distance increment/decrement is negative.

17. A user interface control method of a portable terminal including a camera module, the method comprising:

capturing first and second images through a camera module;

recognizing, by a pattern recognition unit, patterns of the first and second images by calculating positions of target points extracted from the first and second images respectively;

analyzing, by a pattern analysis unit, the recognized patterns by calculating an average displacement m and a distance change from the positions of the target points and comparing the average displacement m with the distance change;

selecting, by the pattern analysis unit, a control command based on the analysis result; and

performing, by a controller, a function corresponding to the control command.

18. The method of claim 17, wherein the calculating of the average displacement m comprises calculating m by

$p=\sqrt{{(x-x^{\prime }}^{}}$

) 2 + ( y - y ′

) 2 , ⁢

q = ( u - u ′

) 2 + ( v - v ′

) 2 , ⁢

and $m=\frac{p+q}{2}$

wherein the distance change is an absolute value of a distance increment/decrement Δ

d that is calculated by

d=√

{square root over ((x−

u)²+(y−

v)²)}{square root over ((x−

u)²+(y−

v)²)}, d′

=√

{square root over ((x′

−

u′

)²+(y′

−

v′

)²)}{square root over ((x′

−

u′

)²+(y′

−

v′

)²)}, and Δ

d=d−

d′

,

wherein the positions of the compared target points in the first image are (x, y) and (u, v), and the positions of the target points in the second image are (x+a, y+b) and (u+c, v+d), p is a distance between target points (x, y) in the first image and (x+a, y+b) in the second image wherein x′

=x+a, and y′

=y+b, and q is a distance between target points (u,v) in the first image and (u+c, v+d) in the second image wherein u′

=u+c and v′

=v+d.

19. The method of claim 17, wherein the selecting of the control command comprises selecting a direction movement when the average displacement is greater than the distance change and selecting a screen magnification/reduction when the distance change is greater than the average displacement.

20. A user interface control method of a portable terminal including a camera module, the method comprising:

capturing first and second images corresponding to motion of a portable terminal through a camera module;

recognizing, by a pattern recognition unit, patterns of the first and second images by calculating positions of target points extracted from the first and second images respectively;

analyzing, by a pattern analysis unit, the motion of the portable terminal based on the recognized patterns by calculating a displacement angle θ

based on the positions of the target points;

selecting, by the pattern analysis unit, a control command corresponding to the analyzed motion of the portable terminal; and

performing, by a controller, a function corresponding to the control command.

21. The method of claim 20, wherein the selecting of the control command comprises selecting a direction movement corresponding to the direction that the portable terminal is moved in, when the analysis determines that the motion of the portable terminal is movement.

22. The method of claim 20, wherein the selecting the control command comprises selecting a direction movement that is opposite to the movement direction of the portable terminal, when the analysis determines that the motion of the portable terminal is movement.

23. The method of claim 20, wherein the selecting of the control command comprises selecting a reduction/magnification of a display image, corresponding to pushing/pulling of the portable terminal, respectively, when the analysis determines that the motion of the portable terminal is moving away from/drawing closer to a user.

24. The method of claim 20, wherein the selecting of the control command comprises selecting a magnification/reduction of a display image, corresponding to pushing/pulling of the portable terminal, respectively, when the analysis determines that the motion of the portable terminal is moving away from/drawing closer to a user.

25. A portable terminal comprising:

a camera module for capturing first and second images;

a pattern recognition unit for recognizing patterns of the first and second images by calculating positions of target points extracted from the first and second images respectively;

a pattern analysis unit for analyzing the recognized patterns, based on the pattern recognition result of the pattern recognition unit by calculating a displacement angle θ

based on the positions of the target points, and for selecting a control command based on the analysis result; and

a controller for performing a function corresponding to the control command.