APPARATUS AND METHOD FOR CONTROLLING USER INTERFACE USING SOUND RECOGNITION

• US 20120304067A1
• Filed: 05/23/2012
• Published: 11/29/2012
• Est. Priority Date: 05/25/2011
• Status: Abandoned Application

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: HAN, Jae Joon, CHOI, Chang Kyu, YOO, Byung In
• Application Number:
  US13/478,635
• Publication Number:
  US 20120304067A1
• Time in Patent Office: Days
• US Class Current: 715/728
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

An apparatus and method for controlling a user interface using sound recognition are provided. The apparatus and method may detect a position of a hand of a user from an image of the user, and may determine a point in time for starting and terminating the sound recognition, thereby precisely classifying the point in time for starting the sound recognition and the point in time for terminating the sound recognition without a separate device. Also, the user may control the user interface intuitively and conveniently.

Claims

1. An apparatus for controlling a user interface, the apparatus comprising:

a reception unit to receive an image of a user from a sensor;

a detection unit to detect a position of a face of the user, and a position of a hand of the user, from the received image;

a processing unit to calculate a difference between the position of the face and the position of the hand; and

a control unit to start sound recognition corresponding to the user when the calculated difference is less than a threshold value, and to control a user interface based on the sound recognition.

2. The apparatus of claim 1, wherein

the detection unit detects a posture of the hand from the received image, and

the control unit starts the sound recognition when the calculated difference is less than the threshold value, and the posture of the hand corresponds to a posture for starting the sound recognition.

3. The apparatus of claim 2, wherein the control unit terminates the sound recognition when the posture of the hand corresponds to a posture for terminating the sound recognition.

4. The apparatus of claim 1, wherein the control unit outputs a visual indicator corresponding to the sound recognition, to a display apparatus associated with the user interface, and starts the sound recognition when the visual indicator is output.

5. The apparatus of claim 1, wherein

the detection detects a gesture of the user from the received image, and

the control unit starts the sound recognition when the calculated difference is less than the threshold value, and the gesture of the user corresponds to a gesture for starting the sound recognition.

6. The apparatus of claim 5, wherein the gesture for starting the sound recognition is predetermined by the user.

7. The apparatus of claim 1, wherein

the control unit outputs one of a posture for starting the sound recognition and a gesture for starting the sound recognition to a display apparatus associated with the user interface,

the sensor senses an image of the user,

the detection unit detects the posture of the hand and the gesture of the user from the received image, and

the control unit starts the sound recognition when the calculated difference is less than the threshold value, and when the detected gesture of the user corresponds to the gesture for starting the sound recognition or the detected posture of the hand corresponds to the posture for starting the sound recognition.

8. The apparatus of claim 1, wherein the control unit terminates the sound recognition corresponding to the user when a sound signal fails to be input within a predetermined time period.

9. The apparatus of claim 1, wherein

the reception unit receives an image of the user from the sensor continuously after the sound recognition is started,

the detection unit detects the posture of the hand and the gesture of the user from the received image, and

the control unit terminates the sound recognition when the detected gesture of the user corresponds to a gesture for terminating the sound recognition or the detected posture of the hand corresponds to a posture for terminating the sound recognition.

10. The apparatus of claim 1, wherein

to the control unit outputs one of a posture for terminating the sound recognition and a gesture for terminating the sound recognition to a display apparatus associated with the user interface after the sound recognition is started,

the sensor senses an image of the user,

the detection unit detects the posture of the hand and the gesture of the user from the received image, and

the control unit terminates the sound recognition when the detected gesture of the user corresponds to the gesture for terminating the sound recognition or the detected posture of the hand corresponds to the posture for terminating the sound recognition.

11. An apparatus for controlling a user interface, the apparatus comprising:

a reception unit to receive images of a plurality of users from a sensor;

a detection unit to detect positions of faces of each of the plurality of users, and positions of hands of each of the plurality of users, from the received images;

a processing unit to calculate differences between the positions of the faces and the positions of the hands, respectively associated with each of the plurality of users; and

a control unit to start sound recognition corresponding to a user matched to a difference that is less than a threshold value when there is a user matched to the difference that is less than the threshold value, among the plurality of users, and to control a user interface based on the sound recognition.

12. The apparatus of claim 11, wherein

the reception unit receives sounds of the plurality of users from the sensor, and

the control unit segments, from the received sounds, a sound of the user having a difference that is less than the threshold value, based on at least one of the positions of the faces, and the positions of the hands, and controls the user interface based on the segmented sound.

13. The apparatus of claim 11, further comprising:

a database to store a sound signature of a main user who controls the user interface,

wherein

the reception unit receives sounds of the plurality of users from the sensor, and

the control unit segments, from the received sounds, a sound corresponding to the sound signature, and controls the user interface based on the segmented sound.

14. An apparatus for controlling a user interface, the apparatus comprising:

a reception unit to receive an image of a user from a sensor;

a detection unit to detect a position of a face of the user from the received image, and to detect a lip motion of the user based on the detected position of the face; and

a control unit to start sound recognition when the detected lip motion corresponds to a lip motion for starting the sound recognition corresponding to the user, and to control a user interface based on the sound recognition.

15. An apparatus for controlling a user interface, the apparatus comprising:

a reception unit to receive images of a plurality of users from a sensor;

a detection unit to detect positions of faces of each of the plurality of users from the received images, and to detect lip motions of each of the plurality of users based on the detected positions of the faces; and

a control unit to start sound recognition when there is a user having a lip motion corresponding to a lip motion for starting the sound recognition, among the plurality of users, and to control a user interface based on the sound recognition.

16. A method of controlling a user interface, the method comprising:

receiving an image of a user from a sensor;

detecting a position of a face of the user, and a position of a hand of the user, from the received image;

calculating a difference between the position of the face and the position of the hand;

starting sound recognition corresponding to the user when the calculated difference is less than a threshold value; and

controlling a user interface based on the sound recognition.

20. A non-transitory computer-readable medium comprising a program for instructing a computer to perform the method of claim 16.

17. A method of controlling a user interface, the method comprising:

receiving images of a plurality of users from a sensor;

detecting positions of faces of each of the plurality of users, and positions of hands of each of the plurality of users, from the received images;

calculating differences between the positions of the faces and the positions of the hands, respectively associated with each of the plurality of users;

starting sound recognition corresponding to a user matched to a difference that is less than a threshold value when there is a user matched to the difference that is less than the threshold value, among the plurality of users; and

controlling a user interface based on the sound recognition.

18. A method of controlling a user interface, the method comprising:

receiving an image of a user from a sensor;

detecting a position of a face of the user from the received image;

detecting a lip motion of the user based on the detected position of the face;

starting sound recognition when the detected lip motion corresponds to a lip motion for starting the sound recognition corresponding to the user; and

controlling a user interface based on the sound recognition.

19. A method of controlling a user interface, the method comprising:

receiving images of a plurality of users from a sensor;

detecting positions of faces of each of the plurality of users from the received images;

detecting lip motions of each of the plurality of users based on the detected positions of the faces;

starting sound recognition when there is a user having a lip motion corresponding to a lip motion for starting the sound recognition, among the plurality of users; and

controlling a user interface based on the sound recognition.