ELECTRONIC DEVICE AND METHOD FOR CONTROLLING THE ELECTRONIC DEVICE THEREOF

• US 20190325224A1
• Filed: 03/29/2019
• Published: 10/24/2019
• Est. Priority Date: 04/20/2018
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: LEE, Jungmin
• Granted Patent: US 11,954,150 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

An artificial intelligence (AI) system utilizing a machine learning algorithm such as deep learning for controlling an electronic device when a video is reproduced and a user'"'"'s voice instruction is received, to acquire a frame corresponding to the time point when the input of the user'"'"'s voice instruction is received, and obtain a search result for information on objects in the frame using an AI model trained according to at least one of machine learning, a neural network or a deep learning algorithm.

Claims

1. A method for controlling an electronic device comprising:

reproducing a video;

storing a plurality of frames of the reproduced video for a specific time period while reproducing the video;

receiving input of a user'"'"'s voice instruction while reproducing the video;

acquiring a frame from among the stored plurality of frames corresponding to a time point when the input of the user'"'"'s voice instruction is received; and

providing a search result for information about a plurality of objects included in the frame based on information on the acquired frame and information on the input of the user'"'"'s voice instruction.

2. The method for controlling an electronic device of claim 1, further comprising:

inputting the frame corresponding to the time point when the input of the user'"'"'s voice instruction is received into an artificial intelligence model trained through an artificial intelligence algorithm; and

acquiring information on the plurality of objects included in the frame.

3. The method for controlling an electronic device of claim 2, wherein the user'"'"'s voice instruction comprises a trigger voice for initiating an inquiry on information on the plurality of objects included in the frame and an inquiry voice inquiring information on the plurality of objects, and

wherein acquiring information on the plurality of objects comprises:

inputting the frame based on the time point when input of the trigger voice is received into the artificial intelligence model; and

acquiring information on the plurality of objects included in the frame.

4. The method for controlling an electronic device of claim 2, wherein the frame includes image frames and audio frames,

wherein the storing comprises:

matching the image frames and the audio frames and storing the frames, and

wherein the acquiring comprises;

inputting at least one of the image frames or the audio frames into the artificial intelligence model, and acquiring information on the plurality of objects included in the frame.

5. The method for controlling an electronic device of claim 2, further comprising:

matching information on the plurality of objects with the image frame in which the plurality of objects appeared, and storing the information and the frame.

6. The method for controlling an electronic device of claim 2, further comprising:

determining information on an object corresponding to the user'"'"'s voice instruction among the information on the plurality of objects, and

the providing comprises;

transmitting the determined information on the object to an external search server and providing the acquired search result.

7. The method for controlling an electronic device of claim 6, wherein the determining further comprises:

displaying a UI identifying whether the information on an object is information on an object corresponding to the user'"'"'s voice instruction among the information on the plurality of objects, or identifying whether there is an additional inquiry for inquiring additional information.

8. The method for controlling an electronic device of claim 1, wherein the providing comprises:

providing the search result and a frame corresponding to the search result in an area of the video while the video is being reproduced.

9. The method for controlling an electronic device of claim 1, comprising:

transmitting the acquired frame to an external server for acquiring information on frames; and

acquiring information on the acquired frame from the external server.

10. The method for controlling an electronic device of claim 9, wherein the external server recognizes a fingerprint included in the frame.

11. An electronic device comprising:

a display;

a communicator;

a microphone;

memory storing at least one instruction; and

a processor coupled to the display, the communicator, the microphone and the memory, and controlling the electronic device,

wherein the processor controls the electronic device to store in the memory a plurality of frames of reproduced video for a specific time period while reproducing the video by executing the at least one instruction,

acquires, based on receiving input of a user'"'"'s voice instruction while the video is being reproduced, a frame from among the stored plurality of frames corresponding to a time point when the input of the user'"'"'s voice instruction is received, and

provides a search result for information about a plurality of objects included in the frame based on information on the acquired frame and information on the input of the user'"'"'s voice instruction.

12. The electronic device of claim 11, wherein the processor inputs a frame corresponding to the time point when the input of the user'"'"'s voice instruction is received into an artificial intelligence model trained through an artificial intelligence algorithm, and acquires information on the plurality of objects included in the frame.

13. The electronic device of claim 12, wherein the user'"'"'s voice instruction comprises a trigger voice for initiating an inquiry on information on the plurality of objects included in the frame and an inquiry voice inquiring information on the plurality of objects, and

the processor inputs the frame based on the time point when input of the trigger voice is received into the artificial intelligence model, and acquires information on the plurality of objects included in the frame.

14. The electronic device of claim 12, wherein the frame includes image frames and audio frames,

the memory matches the image frames and the audio frames and stores the frames, and

the processor inputs at least one of the image frames or the audio frames into the artificial intelligence model, and acquires information on the plurality of objects included in the frame.

15. The electronic device of claim 12, wherein the processor matches information on the plurality of objects with the image frame in which the plurality of objects appeared, and stores the information and the frame in the memory.

16. The electronic device of claim 12, wherein the processor determines information on an object corresponding to the user'"'"'s voice instruction among the information on the plurality of objects, and

transmits the determined information on the object to an external search server through the communicator and provides the acquired search result.

17. The electronic device of claim 16, wherein the processor controls the display to display a UI identifying whether the information on an object is information on an object corresponding to the user'"'"'s voice instruction among the information on the plurality of objects, or identifying whether there is an additional inquiry for inquiring additional information.

18. The electronic device of claim 11, wherein the processor provides the search result and a frame corresponding to the search result in an area of the video while the video is being reproduced.

19. The electronic device of claim 11, wherein the processor controls the communicator to transmit the acquired frame to an external server for acquiring information on frames, and

acquires information on the acquired frame from the external server.

20. The electronic device of claim 19, wherein the external server recognizes a fingerprint included in the frame.