Method and apparatus for controlling user interface of electronic device using virtual plane

• US 9,052,744 B2
• Filed: 08/01/2007
• Issued: 06/09/2015
• Est. Priority Date: 11/20/2006
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Ma, Gengyu, Moon, Young-su, Yoo, Yong-seok
• Primary Examiner(s): ALVESTEFFER, STEPHEN D
• Application Number:
  US11/882,443
• Publication Number:
  US 20080120577A1
• Time in Patent Office: 2,869 Days
• Field of Search: 715/863, 715/964
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image

Abstract

Provided is a method and apparatus for controlling a user interface (UI) of an electronic device by using a specific region of a virtual plane. In the method, the virtual plane is initialized according to spatial coordinates of two or more images detected by using locations of the images displayed in an image display unit of a first image capture device and an image display unit of a second image capture device. Furthermore, the location of a marker is determined and is indicated by a specific image on the virtual plane. Furthermore, a screen pointer corresponding to the marker is displayed at a position where the marker is located in an image display unit corresponding to the virtual plane. Furthermore, a menu of the electronic device is selected which is pointed to by the screen pointer while moving the marker. Accordingly, a user can conveniently control the UI of the electronic device by performing a simple operation.

Claims

1. A method of controlling a UI (user interface) of an electronic device by using a virtual plane, the method comprising:

detecting at least four points corresponding to at least four fingertips of a user from two or more images displayed in an image display unit of a first image capture device and an image display unit of a second image capture device;

calculating three-dimensional (3D) spatial coordinates of each of the detected at least four points;

initializing the virtual plane, used to control the UI of the electronic device;

determining a location of a marker indicated by a specific image on the virtual plane;

displaying a screen pointer corresponding to the marker at a position where the marker is located in an image display unit corresponding to the virtual plane; and

selecting a menu of the electronic device pointed to by the screen pointer while moving the marker,

wherein the virtual plane is initialized based on the three-dimensional spatial coordinates of the detected at least four points such that the virtual plane passes through a point corresponding to a center of the detected at least four points.

2. The method of claim 1, further comprising determining a correlation between the locations of the images displayed in the image display units and the spatial coordinates of the at least four points detected from the two or more images according to locations where images captured by the first and second image capture units are displayed in the image display units, wherein the initializing further comprises:

detecting the spatial coordinates of two or more images displayed in the image display units of the image capture devices according to the determined correlation; and

initializing the virtual plane according to the detected spatial coordinates.

3. The method of claim 1, wherein the detected at least four points comprise tips of an index finger and a thumb of a right hand of the user and tips of an index finger and a thumb of a left hand of the user when the user makes a gesture to form a specific figure by using the index fingers and thumbs.

4. The method of claim 3, wherein the initializing of the virtual plane ether comprises:

determining a plane nearest to spatial coordinates of the at least four points detected from the four tips of the index fingers and thumbs in a gesture in which the index fingers and the thumbs are perpendicular to each other;

determining axis coordinates of the virtual plane so that a direction perpendicular to the determined plane is defined as a z-axis direction; and

initializing the virtual plane so that vertices correspond to four spatial coordinates obtained based on the determined coordinate axes.

5. The method of claim 1, further comprising detecting a spatial coordinate of the marker according to where an image of the marker captured by the first and second image capture units are displayed in image display units of the respective image capture units, wherein, in the determining, the marker is mapped to the virtual plane to determine the location of the marker on the virtual plane.

6. The method of claim 5, wherein the marker image is an object having a distinctive color.

7. The method of claim 5, wherein the image of the marker corresponds to a fingertip.

8. The method of claim 7, wherein the image corresponding to the fingertip is specified by at least one of a skin color, a shape, and a depth.

9. The method of claim 1, wherein, in the determining, the location of the marker is determined according to coordinate axes of the initialized virtual plane.

10. The method of claim 1, wherein, in the displaying, the screen pointer is displayed at a position obtained by multiplying the location of the marker detected by using the virtual plane by a resolution of the electronic device.

11. The method of claim 1, wherein in the selecting, when the marker passes through the virtual plane as the marker moves, a menu of the electronic device pointed to by the screen pointer is selected.

12. The method of claim 1, wherein the screen pointer is displayed in the image display units of the electronic device as an icon.

13. The method of claim 12, wherein the screen pointer increases in size as the marker approaches the virtual plane.

14. The method of claim 1, wherein the virtual plane comprises a virtual image region corresponding to the image display units of the electronic device.

15. The method of claim 1, wherein the virtual plane comprises the virtual image region, which corresponds to the image display units of the electronic device, and a virtual writing region.

16. The method of claim 15, wherein, in the selecting, a command corresponding to the menu is directly written as text in the virtual writing region while moving the marker to select the menu on the screen.

17. The method of claim 1, wherein the menu of the electronic device comprises a popup menu which appears according to the location of the screen pointer and a normal menu which appears in the image display units of the electronic device.

18. The method of claim 17, wherein the popup menu appears together with a sub-menu of the popup menu when the screen pointer is positioned over the popup menu.

19. The method of claim 1, wherein the menu of the electronic device appears as a keyboard through which various characters can be input.

20. A non-transitory computer-readable medium having embodied thereon a computer program for executing the method of claim 1.

21. The method of claim 1 wherein the user sets the detected at least four points using hand gestures.

22. The method of claim 1, wherein the calculated 3D spatial coordinates are stored.

23. The method of claim 1, wherein the boundary points are set by a user, and the calculated 3D spatial coordinates match a physical location of the boundary points set by the user.

24. An apparatus for controlling a UI (user interface) of an electronic device by using a virtual plane, the apparatus comprising:

a first image capture unit and a second image capture unit which capture images and detect at least four points corresponding to at least four fingertips of a user from two or more images displayed in an image display unit of the first image capture device and an image display unit of the second image capture device when the first and second image capture units capture the images;

a virtual plane initializing unit which initializes the virtual plane used to control the UI of the electronic device;

a marker positioning unit which determines a spatial location of a marker according to a specific image captured by the first and second image capture units and determines a location of the marker on the virtual plane by mapping the spatial location of the marker to a position on the virtual plane; and

an arithmetic unit which computes a location of a screen pointer, which appears on the image display units of the electronic device and corresponds to the location of the marker, and which selects a menu of the electronic device pointed to by the screen pointer that moves along with the movement of the marker,

wherein the virtual plane is initialized based on three-dimensional spatial coordinates of the detected at least four points such that the virtual plane passes through a point corresponding to a center of the detected at least four points.

25. The apparatus of claim 24, wherein the first and second image capture units determine a correlation between the locations of the images displayed in the image display units and the spatial coordinates of the at least four points detected from the two or more images according to locations where images captured by the first and second image capture units are displayed in the image display units.

26. The apparatus of claim 24,

wherein the detected at least four points comprise to tips of an index fingers and a thumb of a right hand of the user and tips of an index finger and a thumb of a left hand of the user when the user makes a gesture to form a specific figure by using the index fingers and thumbs, and

wherein the virtual plane initializing unit comprises:

a virtual plane positioning unit which determines a location of the virtual plane to be a plane nearest to spatial coordinates of the at least four points detected from the four tips of the index fingers and thumbs in a gesture in which the index fingers and the thumbs are perpendicular to each other; and

a virtual plane coordinate axes determining unit which determines axis coordinates of the virtual plane so that a direction perpendicular to the determined plane is defined as a z-axis direction, wherein the virtual plane is initialized so that vertices correspond to four spatial coordinates determined based on the determined coordinate axes.

27. The apparatus of claim 24, wherein the specific image is an object having a distinctive color.

28. The apparatus of claim 24, wherein the specific image corresponds to a fingertip.

29. The apparatus of claim 28, wherein the image corresponding to the fingertip is specified by at least one of a skin color, a shape, and a depth.

30. The apparatus of claim 24, wherein the arithmetic unit computes a location of the screen pointer to be a position obtained by multiplying the location of the marker detected according to the virtual plane by a resolution of the electronic device.

31. The apparatus of claim 24, wherein the arithmetic unit selects a menu of the electronic device pointed to by the screen pointer when the marker passes through the virtual plane as the marker moves.

32. The apparatus of claim 24, wherein the screen pointer is displayed in the image display units of the electronic device as an icon.

33. The apparatus of claim 32, where in the screen pointer increases in size as the marker approaches the virtual plane.

34. The apparatus of claim 24, wherein the virtual plane comprises a virtual image region corresponding to the image display units of the electronic device.

35. The apparatus of claim 24, wherein the virtual plane comprises a virtual image region, which corresponds to the image display units of the electronic device, and a virtual writing region.

36. The apparatus of claim 35, wherein the arithmetic unit selects the menu on the screen by directly writing a command corresponding to the menu as text in the virtual writing region while moving the marker.

37. The apparatus of claim 24, wherein the menu of the electronic device comprises a popup menu which appears according to the location of the screen pointer and a normal menu which appears in the image display units of the electronic device.

38. The apparatus of claim 37, wherein the popup menu appears together with a sub-menu of the popup menu when the screen pointer is positioned over the popup menu.

39. The apparatus of claim 24, wherein the menu of the electronic device appears as a keyboard through which various characters can be input.