DISPLAY APPARATUS, REMOTE CONTROL APPARATUS, SYSTEM AND CONTROLLING METHOD THEREOF

• US 20150339098A1
• Filed: 05/21/2015
• Published: 11/26/2015
• Est. Priority Date: 05/21/2014
• Status: Abandoned Application

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: KA, Jee-hoon, KIM, Kwang-soo, RYU, Hee-seob, LEE, Sang-jun
• Application Number:
  US14/718,995
• Publication Number:
  US 20150339098A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G08C 17/02 using a radio link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/71 Directional beams
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection

Abstract

A display apparatus includes a display which displays a plurality of items, a communicator which receives a pointing signal from a remote control apparatus, a recognizer which recognizes at least one of a voice command and a gesture, and a processor which selects one item among the plurality of items based on at least one of the pointing signal and the gesture, and in response to receiving the voice command regarding the selected one item, performs a control operation based on a keyword extracted to execute the voice command.

Claims

1. A display apparatus, comprising:

a display configured to display a plurality of items;

a communicator configured to receive a pointing signal from a remote control apparatus;

an input device configured to receive at least one of a voice command and a gesture; and

a processor configured to select one item among the plurality of items based on at least one of the pointing signal and the gesture, and in response to receiving the voice command regarding the selected one item, perform an operation based on a keyword extracted to execute the received voice command.

2. The display apparatus of claim 1, wherein the processor is further configured to extract the keyword by analyzing the selected one item.

3. The display apparatus of claim 1, wherein the processor is further configured to control the communicator to transmit information regarding the selected one item to an external server, and receive the keyword from the external server.

4. The display apparatus of claim 1, wherein the processor is further configured to control the input device to stop the selecting of the one item and control the input device to perform voice recognition in response to the selection of the one item being determined.

5. The display apparatus of claim 4, wherein the processor is further configured to control the input device to resume the determining of the one item in response to the voice command not being received for a preset time.

6. The display apparatus of claim 1, wherein the processor is further configured to control the input device to stop performing gesture recognition and to perform voice recognition in response to the selection of the one item being determined based on the gesture.

7. The display apparatus of claim 6, wherein the processor is further configured to control the input device to stop performing voice recognition and to perform gesture recognition in response to not receiving the voice command for a preset time while the voice recognition is performed.

11. The display apparatus of claim 1, wherein the processor is further configured to control the input device to simultaneously perform voice recognition and gesture recognition.

12. The display apparatus of claim 1, further comprising:

an object register configured to store a shape of an object,

wherein the processor is further configured to control the input device to receive an object gesture performed by the object as the gesture and perform a control operation corresponding to the object gesture.

13. The display apparatus of claim 1, wherein the input device comprises an apparatus configured to receive voice and motion, which is attachable to and detachable from the display apparatus.

8. A remote control apparatus, comprising:

a communicator configured to communicate with a display apparatus displaying a plurality of items;

a voice input device configured to receive a voice command; and

a controller configured to control the communicator to transmit a pointing signal indicating at least one item among the plurality of the items, to control the communicator to stop transmitting the pointing signal in response to a preset event, to control the voice input device to receive a voice command, and to control the communicator to transmit the voice command to the display apparatus.

9. The remote control apparatus of claim 8, wherein the controller is further configured to stop the voice input device and resume transmitting the pointing signal, in response to the voice command not being received for a preset time.

10. The remote control apparatus of claim 8, wherein the preset event comprises at least one of an event in which a user selection command is input while the pointing signal is transmitted and an event in which the pointing signal is transmitted for more than a preset time.

14. A system comprising a display apparatus and a remote control apparatus,

wherein the display apparatus is configured to select one item among a plurality of displayed items based on at least one of a pointing signal received from the remote control apparatus and a recognized gesture, and perform a control operation based on a keyword extracted to execute a voice command, and

the remote control apparatus is configured to stop transmitting the pointing signal in response to a preset event, perform voice recognition, receive the voice command, and transmit the voice command to the display apparatus.

15. A method of controlling a display apparatus including a display configured to display a plurality of items and an input device configured to receive at least one of a voice command and a gesture, the method comprising:

communicating with a remote control apparatus;

selecting one item among the plurality of items based on at least one of a pointing signal received from the remote control apparatus and the received gesture;

receiving a voice command corresponding to the selected one item;

extracting a keyword; and

performing a control operation based on the extracted keyword.

16. The method of claim 15, wherein the extracting comprises analyzing the selected one item and extracting the keyword based on the analyzing.

17. The method of claim 15, wherein the extracting comprises transmitting information corresponding to the selected item to an external server, and receiving the extracted keyword from the external server.

18. The method of claim 15, further comprising selecting the one item and stopping the determining in response to selecting of the one item.

19. A method of controlling a remote control apparatus, comprising:

communicating with a display apparatus displaying a plurality of items; and

transmitting a pointing signal to indicate one item among the plurality of the items;

stopping the transmitting in response to a preset event;

receiving a voice command corresponding to the indicated one item; and

transmitting the voice command to the display apparatus.

20. A non-transitory computer readable medium containing program instructions for causing at least one processor to perform a method in combination with a display apparatus configured to display a plurality of items, wherein the method comprises:

communicating with a remote control apparatus;

selecting one item among the plurality of displayed items based on at least one of a pointing signal received from the remote control apparatus and a received user motion;

receiving a voice command corresponding to the selected one item;

extracting a keyword; and

performing a control operation based on the extracted keyword.

21. A non-transitory computer readable medium containing program instructions for causing at least one processor to perform a method in combination with a remote control apparatus, wherein the method comprises:

communicating with a display apparatus displaying a plurality of items; and

transmitting a pointing signal to indicate one item among the plurality of the items;

stopping the transmitting in response to a preset event;

receiving a voice command corresponding to the indicated one item; and

transmitting the voice command to the display apparatus.

22. A display device comprising:

a display configured to display an image;

an input device configured to receive a gesture performed by a user;

an audio capturer configured to capture a voice command performed by the user;

a communicator; and

a processor configured to determine a selected object of the image corresponding to the gesture, to control the communicator to transmit a search request based on the selected object and the voice command, to receive a search result and to control the image displayed on the display to correspond to the search result.

23. The display device of claim 22, wherein the input device comprises a gaze sensor and the processor is further configured to recognize the gesture based on a gaze of the user.

24. The display device of claim 22, wherein the processor is further configured to control the audio capturer to convert the voice command to a text command, and

the search request comprises the text command.

25. A remote control device comprising:

an imaging device configured to analyze a received image;

an input device configured to receive a user input;

a display configured to display search information;

a communicator; and

a processor configured to determine a selected object of the received image corresponding to the user input, to control the communicator to transmit a search request based on the selected object, to receive a search result and to control the display to display the search result as the search information.