Method and apparatus for controlling user interface of electronic device using virtual plane

• US 20080120577A1
• Filed: 08/01/2007
• Published: 05/22/2008
• Est. Priority Date: 11/20/2006
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Ma, Gengyu, Moon, Young-su, Yoo, Yong-seok
• Granted Patent: US 9,052,744 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image

Abstract

Provided is a method and apparatus for controlling a user interface (UI) of an electronic device by using a specific region of a virtual plane. In the method, the virtual plane is initialized according to spatial coordinates of two or more images detected by using locations of the images displayed in an image display unit of a first image capture device and an image display unit of a second image capture device. Furthermore, the location of a marker is determined and is indicated by a specific image on the virtual plane. Furthermore, a screen pointer corresponding to the marker is displayed at a position where the marker is located in an image display unit corresponding to the virtual plane. Furthermore, a menu of the electronic device is selected which is pointed to by the screen pointer while moving the marker. Accordingly, a user can conveniently control the UI of the electronic device by performing a simple operation.

Claims

1. A method of controlling a UI (user interface) of an electronic device by using a virtual plane, the method comprising:

(a) initializing the virtual plane according to spatial coordinates of two or more images detected by using locations of the images displayed in an image display unit of a first image capture device and an image display unit of a second image capture device;

(b) determining a location of a marker indicated by a specific image on the virtual plane;

(c) displaying a screen pointer corresponding to the marker at a position where the marker is located in an image display unit corresponding to the virtual plane; and

(d) selecting a menu of the electronic device pointed to by the screen pointer while moving the marker.

2. The method of claim 1, further comprising determining a correlation between the locations of the images displayed in the image display units and the spatial coordinates of the images according to locations where images captured by the first and second image capture units are displayed in the image display units, wherein (a) further comprises:

(a1) detecting the spatial coordinates of two or more images displayed in the image display units of the image capture devices according to the determined correlation; and

(a2) initializing the virtual plane according to the detected spatial coordinates.

3. The method of claim 1, wherein, in (a), the virtual plane is initialized so that the virtual plane has coordinate axes in which, when four images are detected based on the their locations and are displayed in the first and second image capture devices, spatial coordinates of the four images are defined as four vertices of the virtual plane and one of the four vertices is defined as an origin.

4. The method of claim 3, wherein, in (a), the four images correspond to four vertices of a checkerboard in which check patterns are arranged having uniform intervals.

5. The method of claim 3, wherein, in (a), the four images correspond to tips of index fingers and thumbs when a user makes a gesture to form a specific figure by using the index fingers and thumbs.

6. The method of claim 5, wherein (a) further comprises:

(a1) determining a plane nearest to spatial coordinates of the four images corresponding to the tips of the index fingers and thumbs in a gesture in which the index fingers and the thumbs are perpendicular to each other;

(a2) determining axis coordinates of the virtual plane so that a direction perpendicular to the determined plane is defined as a z-axis direction; and

(a3) initializing the virtual plane so that vertices correspond to four spatial coordinates obtained based on the determined coordinate axes.

7. The method of claim 1, wherein the two images in (a) correspond to a user'"'"'s eyes.

8. The method of claim 7, wherein the virtual plane has a predetermined size and is spaced apart from the center between both eyes by a predetermined distance.

9. The method of claim 1, further comprising detecting a spatial coordinate of the marker according to where images of the marker captured by the first and second image capture units are displayed in the image display units of the respective image capture units, wherein, in (b), the detected spatial coordinate of the marker is mapped to the virtual plane so as to determine the location of the marker on the virtual plane.

10. The method of claim 9, wherein the marker image is an object having a distinctive color.

11. The method of claim 9, wherein the image of the marker corresponds to a fingertip.

12. The method of claim 11, wherein the image corresponding to the fingertip is specified by at least one of a skin color, a shape, and a depth.

13. The method of claim 3, wherein, in (b), the location of the marker is determined according to the coordinate axes of the initialized virtual plane.

14. The method of claim 1, wherein, in (c), the screen pointer is displayed at a position obtained by multiplying the location of the marker detected by using the virtual plane by a resolution of the electronic device.

15. The method of claim 1, wherein in (d), when the marker passes through the virtual plane as the marker moves, a menu of the electronic device pointed to by the screen pointer is selected.

16. The method of claim 1, wherein the screen pointer is displayed in the image display units of the electronic device in the form of an icon.

17. The method of claim 16, wherein the screen pointer increases in size as the marker approaches the virtual plane.

18. The method of claim 1, wherein the virtual plane includes a virtual image region corresponding to the image display units of the electronic device.

19. The method of claim 1, wherein the virtual plane includes the virtual image region, which corresponds to the image display units of the electronic device, and a virtual writing region.

20. The method of claim 1, wherein the menu of the electronic device includes a popup menu which appears according to the location of the screen pointer and a normal menu which appears in the image display units of the electronic device.

21. The method of claim 1, wherein the popup menu appears together with a sub-menu of the popup menu when the screen pointer is positioned over the popup menu.

22. The method of claim 1, wherein the menu of the electronic device appears in the form of a keyboard through which various characters can be input.

23. The method of claim 19, wherein, in (d), a command corresponding to the menu is directly written in the format of text in the virtual writing region while moving the marker so as to select the menu on the screen.

24. A computer-readable medium having embodied thereon a computer program for executing the method of any one of claims 1 to 23.

25. An apparatus for controlling a UI (user interface) of an electronic device by using a virtual plane, the apparatus comprising:

a first image capture unit and a second unit capture unit which capture images;

a virtual plane initializing unit which initializes the virtual plane according to spatial coordinates of two or more images detected by using locations of the images displayed in an image display unit of a first image capture device and an image display unit of a second image capture device when the first and second image capture units capture the images;

a marker positioning unit which determines a spatial location of a marker according to a specific image captured by the first and second image capture units and determines a location of the marker on the virtual plane by mapping the spatial location of the marker to a position on the virtual plane; and

an arithmetic unit which computes a location of a screen pointer, which appears on the image display units of the electronic device and corresponds to the location of the marker, and which selects a menu of the electronic device pointed to by the screen pointer that moves along with the movement of the marker.

26. The apparatus of claim 25, wherein the first and second image capture units determine a correlation between the locations of the images displayed in the image display units and the spatial coordinates of the images according to locations where images captured by the first and second image capture units are displayed in the image display units.

27. The apparatus of claim 25, wherein the virtual plane initializing unit initializes the virtual plane so that the virtual plane has coordinate axes in which, when four images are detected based on the their locations and are displayed in the first and second image capture devices, spatial coordinates of the four images are defined as four vertices of the virtual plane and one of the four vertices is defined as an origin.

28. The apparatus of claim 27,

wherein the first and second image capture units capture four images corresponding to four vertices of a checkerboard in which check patterns are arranged having uniform intervals, and

wherein the virtual plane initializing unit initializes the virtual plane so that the virtual plane has coordinate axes in which, when four images are detected based on the their locations and are displayed in the first and second image capture devices, spatial coordinates of the four images are defined as four vertices of the virtual plane and one of the four vertices is defined as an origin.

29. The apparatus of claim 27,

wherein the first and second image capture units capture four images corresponding to tips of index fingers and thumbs when a user makes a gesture to form a specific figure by using the index fingers and thumbs, and

wherein the virtual plane initializing unit further comprises:

a virtual plane positioning unit which determines a location of the virtual plane to be a plane nearest to spatial coordinates of the four images corresponding to the tips of the index fingers and thumbs in a gesture in which the index fingers and the thumbs are perpendicular to each other; and

a virtual plane coordinate axes determining unit which determines axis coordinates of the virtual plane so that a direction perpendicular to the determined plane is defined as a z-axis direction, wherein the virtual plane is initialized so that vertices correspond to four spatial coordinates determined based on the determined coordinate axes.

30. The apparatus of claim 25,

wherein the first and second image capture units capture two images corresponding to a user'"'"'s eyes, and

wherein the virtual plane initializing unit initializes the virtual plane so that the virtual plane is spaced apart by a distance from the center between the spatial coordinates of the images determined according to the locations of the two images displayed in the respective image display units.

31. The apparatus of claim 25, wherein the specific image is an object having a distinctive color.

32. The apparatus of claim 25, wherein the specific image corresponds to a fingertip.

33. The apparatus of claim 32, wherein the image corresponding to the fingertip is specified by at least one of a skin color, a shape, and a depth.

34. The apparatus of claim 25, wherein the arithmetic unit computes a location of the screen pointer to be a position obtained by multiplying the location of the marker detected according to the virtual plane by a resolution of the electronic device.

35. The apparatus of claim 25, wherein the arithmetic unit selects a menu of the electronic device pointed to by the screen pointer when the marker passes through the virtual plane as the marker moves.

36. The apparatus of claim 25, wherein the screen pointer is displayed in the image display units of the electronic device in the form of an icon.

37. The apparatus of claim 36, where in the screen pointer increases in size as the marker approaches the virtual plane.

38. The apparatus of claim 25, wherein the virtual plane includes a virtual image region corresponding to the image display units of the electronic device.

39. The apparatus of claim 25, wherein the virtual plane includes a virtual image region, which corresponds to the image display units of the electronic device, and a virtual writing region.

40. The apparatus of claim 25, wherein the menu of the electronic device includes a popup menu which appears according to the location of the screen pointer and a normal menu which appears in the image display units of the electronic device.

41. The apparatus of claim 40, wherein the popup menu appears together with a sub-menu of the popup menu when the screen pointer is positioned over the popup menu.

42. The apparatus of claim 25, wherein the menu of the electronic device appears in the form of a keyboard through which various characters can be input.

43. The apparatus of claim 39, wherein the arithmetic unit selects the menu on the screen by directly writing a command corresponding to the menu in the format of text in the virtual writing region while moving the marker.