Apparatus and method for controlling interface

• US 9,710,068 B2
• Filed: 09/01/2015
• Issued: 07/18/2017
• Est. Priority Date: 08/26/2010
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Han, Jae Joon, Ryu, Hee Seob, Yoo, Byung In, Choi, Chang Kyu, Park, Du Sik
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US14/842,448
• Publication Number:
  US 20150370337A1
• Time in Patent Office: 686 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

In an apparatus and method for controlling an interface, a user interface (UI) may be controlled using information on a hand motion and a gaze of a user without separate tools such as a mouse and a keyboard. That is, the UI control method provides more intuitive, immersive, and united control of the UI. Since a region of interest (ROI) sensing the hand motion of the user is calculated using a UI object that is controlled based on the hand motion within the ROI, the user may control the UI object in the same method and feel regardless of a distance from the user to a sensor. In addition, since positions and directions of view points are adjusted based on a position and direction of the gaze, a binocular 2D/3D image based on motion parallax may be provided.

Claims

1. An apparatus for controlling an interface, comprising:

a sensor configured to obtain a image related to a user; and

a processor configured to

determine a resolution of a region of interest (ROI) based on a distance between the user and the sensor, a resolution of the sensor, and a size of an ROI window in the air,

generate information regarding a user input based on the resolution of the ROI, and

control the interface based on the information.

2. The apparatus of claim 1, wherein the resolution of the ROI is further calculated a size of a field of view of the sensor corresponding to the distance between the user and the sensor.

3. The apparatus of claim 1, wherein a width of the resolution of the ROI is calculated based on Equation 1 and a height of the resolution of the ROI is calculated based on Equation 2 as follows:

4. The apparatus of claim 1, wherein the information comprises either one or both of motion information regarding a hand motion within the ROI and gaze information regarding a gaze of the user.

5. The apparatus of claim 1, wherein the processor is further configured to calculate a depth difference value between a current frame and at least one previous frame of the depth image, and generate motion information based on the depth difference value.

6. The apparatus of claim 1, wherein the processor is further configured to extract, from the image, head position information regarding a position of a head of the user and head rotation information regarding rotation of the head of the user, and generate gaze information based on the head position information and the head rotation information.

7. The apparatus of claim 1, wherein the processor is further configured to start controlling the interface in response to the information corresponding to a control start interaction, and end controlling of the interface in response to the information corresponding to a control end interaction.

8. The apparatus of claim 7, wherein the control start interaction comprises either one or both of a motion of raising a hand of the user and a motion of pushing the hand of the user.

9. The apparatus of claim 7, wherein the control end interaction comprises a motion of lowering a hand of the user.

10. The apparatus of claim 1, wherein the processor is further configured to display an operation region corresponding to the information and controls the interface within the operation region in response to the information corresponding to a high precision interaction.

11. The apparatus of claim 1, wherein the processor is further configured to

control the interface in response to there being an interaction corresponding to any one or any combination of any two or more of an execution interaction, a cancel interaction, a drag-and-drop interaction, a scrolling interaction, a rotation interaction, and a scaling interaction, and

control movement of a control point of the interface in response to there being no interaction corresponding to the any one or any combination of any two or more of an execution interaction, a cancel interaction, a drag-and-drop interaction, a scrolling interaction, a rotation interaction, and a scaling interaction.

12. The apparatus of claim 1, wherein the image comprises any one or any combination of any two or more of a depth image related to the user, a color image related to the user, and an IR image related to the user.

13. The apparatus of claim 1, wherein the processor is further configured to:

adjust a position of a left view point corresponding to a position of a left eye of the user in the image;

adjust a position of a right view point corresponding to a position of a right eye of the user in the image;

render a left scene based on the position of the left view point;

render a right scene based on the position of the right view point; and

combine the left scene and the right scene to generate the interface.

14. The apparatus of claim 13, wherein, in response to users being within a sensing range of the sensor and, among the users, a main user existing that inputs a control start interaction, the processor extracts information on a left eye position regarding a left eye of the main user and on a right eye position regarding a right eye of the main user, adjusts the left view point position corresponding to the left eye position of the main user, and adjusts the right view point position corresponding to the right eye position of the main user.

15. The apparatus of claim 13, wherein, in response to users being within a sensing range of the sensor and, among the users, a main user not existing, the processor extracts information on an average position of left eyes of the users and an average position of right eyes of the users, adjusts the left view point position corresponding to the average position of the left eyes, and adjusts the right view point position corresponding to the average position of the right eyes.

16. The apparatus of claim 13, wherein, in response to users being within a sensing range of the sensor, the processor extracts information on an average position of left eyes of the users and an average position of right eyes of the users, adjusts the left view point position corresponding to the average position of the left eyes, and adjusts the right view point position corresponding to the average position of the right eyes.

17. The apparatus of claim 1, wherein the processor is further configured to display at least one UI object based on any one or any combination of any two or more of a cube view, a thumbnail view, and a slide view.

18. A method of controlling an interface, comprising:

obtaining a image related to a user from a sensor;

determining a resolution of a region of interest (ROI) based on a distance between the user and the sensor, a resolution of the sensor, and a size of an ROI window in the air;

generating information regarding a user input based on the resolution of the ROI; and

controlling the interface based the information.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 18.