Apparatus and method for controlling interface

• US 9,141,189 B2
• Filed: 08/18/2011
• Issued: 09/22/2015
• Est. Priority Date: 08/26/2010
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Yoo, Byung In, Choi, Chang Kyu, Han, Jae Joon, Park, Du Sik, Ryu, Hee Seob
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Gray, Ryan M
• Application Number:
  US13/212,838
• Publication Number:
  US 20120050273A1
• Time in Patent Office: 1,496 Days
• Field of Search: 715/848, 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

In an apparatus and method for controlling an interface, a user interface (UI) may be controlled using information on a hand motion and a gaze of a user without separate tools such as a mouse and a keyboard. That is, the UI control method provides more intuitive, immersive, and united control of the UI. Since a region of interest (ROI) sensing the hand motion of the user is calculated using a UI object that is controlled based on the hand motion within the ROI, the user may control the UI object in the same method and feel regardless of a distance from the user to a sensor. In addition, since positions and directions of view points are adjusted based on a position and direction of the gaze, a binocular 2D/3D image based on motion parallax may be provided.

Claims

1. An apparatus for controlling an interface, comprising:

a receiver to receive image information comprising a depth image related to a user from a sensor;

a processor to determine a region of interest (ROI) based on depth information of the depth image, and generate, based on the image information, at least one of motion information regarding a hand motion within the ROI and gaze information regarding a gaze of the user; and

a controller to control a 2-dimensional or 3-dimensional graphical user interface(2D/3D GUI) based on at least one of the motion information and the gaze information,

wherein a resolution of the ROI is calculated based on a resolution of the sensor, a distance between the user and the sensor, and a predetermined size of ROI window in the air.

2. The apparatus of claim 1, wherein the processor calculates a size of a field of view of the sensor corresponding to the distance between the user and the sensor, and calculates the resolution of the ROI based on the resolution of the sensor, the size of the field of view of the sensor corresponding to the distance between the user and the sensor, and the predetermined size of the ROI window in the air.

3. The apparatus of claim 1, wherein the processor calculates a depth difference value between a current frame and at least one previous frame of the depth image, and generates the motion information based on the depth difference value.

4. The apparatus of claim 1, wherein the processor extracts, from the image information, head position information regarding a position of a head of the user and head rotation information regarding rotation of the head of the user, and generates the gaze information based on the head position information and the head rotation information.

5. The apparatus of claim 1, wherein

the controller starts controlling the 2D/3D GUI when the motion information corresponds to a control start interaction, and

the controller ends controlling of the 2D/3D GUI when the motion information corresponds to a control end interaction.

6. The apparatus of claim 5, wherein the control start interaction comprises at least one of a motion of raising a hand of the user and a motion of pushing the hand of the user.

7. The apparatus of claim 5, wherein the control finish interaction comprises a motion of lowering a hand of the user.

8. The apparatus of claim 1, wherein the controller displays an operation region corresponding to the motion information and controls the 2D/3D GUI within the operation region when the motion information corresponds to a high precision interaction.

9. The apparatus of claim 1, wherein

the controller controls the 2D/3D GUI when there is an interaction corresponding to the motion information among a plurality of interactions controlling the 2D/3D GUI, based on the interaction corresponding to the motion information, and

the controller controls movement of a control point of the 2D/3D GUI based on the motion information when there is no interaction corresponding to the motion information among the plurality of interactions.

10. The apparatus of claim 9, wherein the plurality of interactions comprises at least one of an execution interaction, a cancel interaction, a drag-and-drop interaction, a scrolling interaction, a rotation interaction, and a scaling interaction.

11. The apparatus of claim 1, wherein the image information further comprises a color image and IR image related to the user.

12. The apparatus of claim 1, further comprising:

a generator to generate the 2D/3D GUI based on the image information; and

an outputter to output the 2D/3D GUI to a display apparatus.

13. The apparatus of claim 12, wherein the generator comprises:

a view point adjustment unit to extract information regarding a left eye position of a left eye of the user and regarding a right eye position of a right eye of the user from the image information, to adjust a position of a left view point corresponding to the left eye position, and to adjust a position of a right view point corresponding to the right eye position;

a 2D/3D scene rendering unit to render a left 2D/3D scene based on the left view point position and to render a right 2D/3D scene based on the right view point position; and

a 2D/3D GUI generation unit to generate the 2D/3D GUI by combining the rendered left 2D/3D scene and the rendered right 2D/3D scene.

14. The apparatus of claim 13, wherein, when a plurality of users are within a sensing range of the sensor and, among the plurality of users, a main user exists that inputs a control start interaction, the view point adjustment unit extracts information on a left eye position regarding a left eye of the main user and on a right eye position regarding a right eye of the main user, adjusts the left view point position corresponding to the left eye position of main user, and adjusts the right view point position corresponding to the right eye position of main user.

15. The apparatus of claim 13, wherein, when a plurality of users are within a sensing range of the sensor, the view point adjustment unit extracts information on an average position of left eyes of the plurality of users and an average position of right eyes of the plurality of users, adjusts the left view point position corresponding to the average position of the left eyes, and adjusts the right view point position corresponding to the average position of the right eyes.

16. The apparatus of claim 1, wherein the 2D/3D GUI displays at least one UI object using at least one of a 2D/3D cube view that displays at least one UI object in the form of a cube, a 2D/3D thumbnail view that displays at least one UI object in the form of a thumbnail image, and a 2D/3D slide view that displays at least one UI object in the form of a slide.

17. An apparatus for controlling an interface, comprising:

a receiver to receive image information comprising a depth image related to a user from a sensor;

a processor to generate, based on the image information, at least one of motion information regarding a hand motion of the user and gaze information regarding a gaze of the user; and

a controller to control a 2-dimensional or 3-dimensional a graphical user interface(2D/3D GUI) based on at least one of the motion information and the gaze information,

wherein the processor determines a region of interest (ROI) and generates the motion information within the ROI, and

wherein the processor calculates a width of the ROI using Equations 1 and calculates a height of the ROI using Equations 2 as follows;

18. An apparatus for controlling an interface, comprising:

a receiver to receive image information related to a user from a sensor;

a generator to generate the 2D/3D GUI based on the image information; and

an outputter to output the 2D/3D GUI to a display apparatus,

wherein the generator comprises;

a view point adjustment unit to extract information regarding a left eye position of a left eye of the user and regarding a right eye position of a right eye of the user from the image information, to adjust a position of a left view point corresponding to the left eye position, and to adjust a position of a right view point corresponding to the right eye position;

a 2D/3D scene rendering unit to render a left 2D/3D scene based on the left view point position and to render a right 2D/3D scene based on the right view point position; and

a 2D/3D GUI generation unit to generate the 2D/3D GUI by combining the rendered left 2D/3D scene and the rendered right 2D/3D scene,

wherein, when a plurality of users are within a sensing range of the sensor and, among the plurality of users, the main user does not exist, the view point adjustment unit extracts information on an average position of left eyes of the plurality of users and an average position of right eyes of the plurality of users, adjusts the left view point position corresponding to the average position of the left eyes, and adjusts the right view point position corresponding to the average position of the right eyes.

19. The apparatus of claim 18, wherein, among the plurality of users, a main user exists that inputs a control start interaction, the view point adjustment unit extracts information on a left eye position regarding a left eye of the main user and on a right eye position regarding a right eye of the main user, adjusts the left view point position corresponding to the left eye position of main user, and adjusts the right view point position corresponding to the right eye position of main user.

20. A method of controlling an interface, comprising:

receiving, by a processor, image information comprising a depth image related to a user;

determining, by the processor, a region of interest (ROI) based on depth information of the depth image;

generating, by the processor, based on the image information, at least one of motion information regarding a hand motion within the ROI and gaze information regarding a gaze of the user; and

controlling, by the processor, a 2D/3D GUI based on at least one of the motion information and the gaze information,

wherein a resolution of the ROI is calculated based on a resolution of the sensor, a distance between the user and the sensor, and a predetermined size of ROI window in the air.

21. The method of claim 20, wherein determining of the ROI comprises:

calculating, by the processor, a size of a field of view of the sensor corresponding to the distance between the user and the sensor; and

calculating, by the processor, the resolution of the ROI based on the resolution of the sensor, the size of the field of view of the sensor corres ndin o he distance between the user and the sensor, and the predetermined size of the ROI window in the air.

22. A non-transitory computer readable recording medium storing a program to cause a computer to implement the method of claim 20.