Object search method and apparatus

• US 10,311,115 B2
• Filed: 05/15/2014
• Issued: 06/04/2019
• Est. Priority Date: 05/15/2014
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Li, Yan, Li, Xiaojuan, Gao, Wenmei
• Primary Examiner(s): Spieler, William
• Application Number:
  US14/902,227
• Publication Number:
  US 20160147882A1
• Time in Patent Office: 1,846 Days
• CPC Class Codes: G06F 16/2428 Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

An object search method and apparatus, where the method includes receiving voice input and gesture input that are of a user; determining, according to the voice input, a name of a target object for which the user expects to search and a characteristic category of the target object; extracting characteristic information of the characteristic category from an image area selected by the user by means of the gesture input; and searching for the target object according to the extracted characteristic information and the name of the target object. The solutions provided in the embodiments of the present disclosure can provide a user with a more flexible search manner, and reduce a restriction on an application scenario during a search.

Claims

1. An object search method implemented by a terminal communicatively coupled to a server, the method comprising:

receiving, by the terminal, voice input and gesture input from a user;

determining, by the terminal, a name of a target object for which the user expects to search and a characteristic category of the target object according to the voice input;

determining, by the terminal, whether the name of the target object and the characteristic category of the target object correspond to a preset category;

extracting, by the terminal, extracted characteristic information locally on the terminal according to the name of the target object, the characteristic category of the target object, and an image corresponding to the gesture input and sending the extracted characteristic information and the name of the target object to the server when the name of the target object and the characteristic category of the target object correspond to the preset category;

sending, by the terminal, the name of the target object, the characteristic category of the target object, and the image corresponding to the gesture input to the server to enable the server to extract the extracted characteristic information when the name of the target object and the characteristic category of the target object do not correspond to the preset category;

receiving, by the terminal, a search result from the server that corresponds to the extracted characteristic information and the image corresponding to the gesture input; and

displaying, by the terminal, the search result.

2. The method of claim 1, wherein receiving the voice input and the gesture input from the user comprises receiving the voice input and the gesture input that are received simultaneously from the user.

3. The method of claim 1, further comprising acquiring an image area from the gesture input of the user by:

acquiring the image area that is selected from a designated image corresponding to the gesture input; and

using the image area as the image from the user.

4. The method of claim 1, wherein receiving the voice input and the gesture input from the user comprises at least one of:

receiving the voice input and the gesture input that are received simultaneously from the user;

receiving the voice input from the user, and when no gesture input is performed by the user and no image area is selected, instructing the user to perform an operation of selecting an image area, and receiving the gesture input from the user;

or

receiving the gesture input from the user, and when no voice input is performed by the user, instructing the user to perform a voice input operation, and receiving the voice input from the user.

5. The method of claim 1, wherein receiving the voice input and the gesture input from the user comprises receiving the voice input of the user, and when no gesture input is performed by the user and no image area is selected, instructing the user to perform an operation of selecting an image area, and receiving the gesture input from the user.

6. The method of claim 1, wherein receiving the voice input and the gesture input from the user comprises receiving the gesture input of the user, and when no voice input is performed by the user, instructing the user to perform a voice input operation, and receiving the voice input from the user.

7. The method of claim 1, further comprising acquiring an image area selected by the user as the gesture input, and acquiring the image area comprises at least one of:

acquiring an image area that is selected from a designated image by the user as the gesture input, and using the image area as the image area selected by the user;

or

acquiring an image that is obtained by photographing from the user as the gesture input, and using the obtained image as the image area selected by the user.

8. The method of claim 1, further comprising acquiring an image area selected by the user as the gesture input, and acquiring the image area comprises:

acquiring an image that is obtained by photographing from the user as the gesture input; and

using the obtained image as the image area selected by the user.

9. The method of claim 1, wherein the gesture input from the user comprises a graph of a profile of an irregular shape input by the user.

10. The method of claim 1, wherein the gesture input from the user comprises a brand logo in a photograph taken by the user.

11. The method of claim 1, wherein the gesture input from the user comprises a pixel value selected by the user from a color histogram displayed on the terminal.

12. A terminal communicatively coupled to a server, comprising:

a receiver configured to receive voice input and gesture input from a user;

a processor coupled to the receiver and configured to;

determine, according to the voice input, a name of a target object for which the user expects to search and a characteristic category of the target object;

determine whether the name of the target object and the characteristic category of the target object correspond to a preset category;

extract extracted characteristic information locally on the terminal according to the name of the target object, the characteristic category of the target object, and an image corresponding to the gesture input and send the extracted characteristic information and the name of the target object to the server when the name of the target object and the characteristic category of the target object correspond to the preset category;

send the name of the target object, the characteristic category of the target object, and the image corresponding to the gesture input to the server to enable the server to extract the extracted characteristic information when the name of the target object and the characteristic category of the target object do not correspond to the preset category;

receive a search result from the server that corresponds to the extracted characteristic information and the image corresponding to the gesture input; and

display the search result.

13. The terminal of claim 12, wherein the receiver is further configured to perform at least one of:

receive the voice input and the gesture input that are received simultaneously from the user;

receive the voice input from the user, and when no gesture input is received from the user and no image area is selected, instruct the user to perform an operation of selecting an image area, and receive the gesture input from the user;

or

receive the gesture input from the user, and when no voice input is received from the user, instruct the user to perform a voice input operation, and receive the voice input from the user.

14. The terminal of claim 12, wherein the processor is further configured to perform at least one of:

acquire an image area that is selected from a designated image corresponding to the gesture input, and use the image area as the image from the user;

or

acquire an image that is obtained by photographing from the user as the gesture input, and use the obtained image as the image from the user.

15. An object search method implemented by a terminal communicatively coupled to a server, the method comprising:

receiving, by the terminal, voice input and gesture input from a user;

determining, by the terminal, a name of a target object for which the user expects to search and a characteristic category of the target object according to the voice input;

determining, by the terminal, whether the name of the target object and the characteristic category of the target object correspond to a preset category;

extracting, by the terminal, extracted characteristic information locally on the terminal according to the name of the target object, the characteristic category of the target object, and an image corresponding to the gesture input and sending the extracted characteristic information and the name of the target object to the server when the name of the target object and the characteristic category of the target object correspond to the preset category;

sending, by the terminal, the name of the target object, the characteristic category of the target object, and an image area that corresponds to the gesture input to the server to enable the server to extract the extracted characteristic information when the name of the target object and the characteristic category of the target object do not correspond to the preset category; and

receiving, by the terminal, a search result from the server, the search result being obtained by the server by searching for the target object represented by the name of the target object, and a characteristic of the characteristic category that is of the image area and represented by the category information is used as a search criterion.

16. The method of claim 15, wherein receiving the voice input and the gesture input from the user comprises at least one of:

receiving the voice input and the gesture input that are received-simultaneously from the user;

receiving the voice input of the user, and when no gesture input is received from the user and no image area is selected, instructing the user to perform an operation of selecting an image area, and receiving the gesture input from the user;

or

receiving the gesture input from the user, and when no voice input is received from the user, instructing the user to perform a voice input operation, and receiving the voice input from the user.

17. The method of claim 15, further comprising acquiring an image area selected by the user corresponding to the gesture input by:

acquiring the image area that is selected from a designated image by the user corresponding to the gesture input, and using the image area as the image area selected by the user;

or

acquiring an image that is obtained by photographing from the user as the gesture input, and using the obtained image as the image area selected by the user.

18. A terminal communicatively coupled to a server, comprising:

a receiver configured to receive voice input and gesture input from a user;

a processor coupled to the receiver and configured to;

determine, according to the voice input, a name of a target object for which the user expects to search and a characteristic category of the target object;

determine whether the name of the target object and the characteristic category of the target object correspond to a preset category;

extract extracted characteristic information locally on the terminal according to the name of the target object, the characteristic category of the target object, and an image corresponding to the gesture input and send the extracted characteristic information and the name of the target object to the server when the name of the target object and the characteristic category of the target object correspond to the preset category;

send, to the server, the name of the target object, the characteristic category of the target object, and an image area that corresponds to the gesture input to enable the server to extract the extracted characteristic information when the name of the target object and the characteristic category of the target object do not correspond to the preset category; and

receive a search result from the server, the search result being obtained by the server by searching for the target object represented by the name of the target object, and a characteristic of the characteristic category that is of the image area and represented by the category information is used as a search criterion.

19. The terminal of claim 18, wherein the receiver is further configured to perform at least one of:

receive the voice input and the gesture input that are received simultaneously from the user;

receive the voice input from the user, and when no gesture input is performed by the user and no image area is selected, instruct the user to perform an operation of selecting an image area, and receive the gesture input from the user;

or

receive the gesture input from the user, and when no voice input is performed by the user, instruct the user to perform a voice input operation, and receive the voice input from the user.

20. The terminal of claim 18, wherein the processor is further configured to:

acquire an image area that is selected from a designated image by the user corresponding to the gesture input, and use the image area as the image area selected by the user;

or

acquire an image that is obtained from photographing by the user as the gesture input, and use the obtained image as the image area selected by the user.