Voice-assisted scanning

• US 9,767,501 B1
• Filed: 11/07/2013
• Issued: 09/19/2017
• Est. Priority Date: 11/07/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Schaaf, Thomas, Salvador, Stan Weidner
• Primary Examiner(s): Iwarere, Seye
• Application Number:
  US14/074,346
• Time in Patent Office: 1,412 Days
• Field of Search: 705 28, 23546245
• CPC Class Codes: G06F 40/00 Handling natural language data speech analysis or synthesis, speech recognition G10L
• CPC Class Codes: G06Q 30/0623 Item investigation

Abstract

In some cases, a handheld device that includes a microphone and a scanner may be used for voice-assisted scanning. For example, a user may provide a voice input via the microphone and may activate the scanner to scan an item identifier (e.g., a barcode). The handheld device may communicate voice data and item identifier information to a remote system for voice-assisted scanning. The remote system may perform automatic speech recognition (ASR) operations on the voice data and may perform item identification operations based on the scanned identifier. Natural language understanding (NLU) processing may be improved by combining ASR information with item information obtained based on the scanned identifier. An action may be executed based on the likely user intent.

Claims

1. A system comprising:

one or more processors;

memory; and

one or more computer-executable instructions stored in the memory and executable by the one or more processors to;

receive, from a handheld electronic device, voice data and item identifier information, wherein the handheld electronic device includes at least a microphone to receive a voice input from a user and a scanner to scan an identifier of an item;

determine, based at least in part on the item identifier information, information about the item;

generate one or more transcriptions of the voice data using a speech recognition model;

generate a semantic representation of the one or more transcriptions using a natural language understanding model;

identify a reference to the item in the semantic representation;

identify a user intent in the semantic representation;

determine an action based at least in part on the information about the item, the reference to the item in the semantic representation, and the user intent in the semantic representation; and

perform the action.

2. The system as recited in claim 1, wherein the action includes at least one of:

adding a particular quantity of the item to a virtual cart of the user;

removing the item from the virtual cart of the user;

communicating, to the handheld electronic device, product information about the item;

or

communicating, to the handheld electronic device, product information associated with another item that is identified as being related to the item based on one or more properties of the item.

3. The system as recited in claim 1, further comprising:

an item database communicatively coupled to at least one of the memory or the one or more processors, the item database including information associated with a plurality of items, each item of the plurality of items being associated with an individual barcode,

wherein the item identifier information comprises a barcode and the one or more computer-executable instructions are further executable by the one or more processors to query the item database based on the barcode to identify the information about the item.

4. The system as recited in claim 1, wherein the reference to the item comprises an anaphora.

5. The system as recited in claim 1, wherein at least one of the speech recognition model or the natural language understanding model was created using at least a portion of the information about the item.

6. A computer-implemented method comprising:

receiving, via a network, voice data and an identifier of an item, wherein the identifier of the item was obtained by scanning the item;

determining, based at least in part on the identifier of the item, information about the item;

generating a semantic representation of the voice data using at least one of a speech recognition model or a natural language understanding model;

identifying a reference to the item in the semantic representation;

identifying a user intent in the semantic representation; and

performing an action that is determined based at least in part on the information about the item, the reference to the item in the semantic representation, and the user intent in the semantic representation.

7. The computer-implemented method as recited in claim 6, wherein the reference to the item comprises an anaphora.

8. The computer-implemented method as recited in claim 6, wherein the speech recognition model is obtained using the information about the item.

9. The computer-implemented method as recited in claim 6, wherein the natural language understanding model is obtained using the information about the item.

10. The computer-implemented method as recited in claim 6, further comprising determining the identifier of the item based at least partly on a barcode, a quick response (QR) code, a radio-frequency identification (RFID), a near-field communication (NFC) identifier, a product logo, or an image on a product package of the item.

11. The computer-implemented method as recited in claim 6, wherein performing the action comprises:

querying an item database to determine product information associated with the item; and

initiating communication of the product information associated with the item via the network.

12. The computer-implemented method as recited in claim 6, wherein performing the action comprises:

identifying second item that is related to the item based on one or more properties of the item;

determining product information associated with the second item; and

initiating communication of the product information associated with the second item via the network.

13. The computer-implemented method as recited in claim 6, wherein the user intent comprises an indication of a quantity of the item, a modification of a characteristic of the item, a request for the information about the item, or an indication to add the item to a virtual shopping cart.

14. The computer-implemented method as recited in claim 6, wherein the semantic representation comprises one or more named entities.

21. The computer-implemented method as recited in claim 6, wherein scanning of the item is performed by a scanner that scans at least one of a barcode, a QR code, an RFID, or an NFC identifier associated with the item.

15. A method comprising:

receiving, from a handheld electronic device, voice data associated with a voice input from a user and an identifier of an item, wherein the identifier of the item was obtained by scanning the item;

determining, based at least in part on the identifier of the item, information about the item;

generating a semantic representation of the voice data using at least one of a speech recognition model or a natural language understanding model;

identifying a reference to the item in the semantic representation;

identifying a user intent in the semantic representation; and

performing an action that is determined based at least in part on the information about the item, the reference to the item in the semantic representation, and the user intent in the semantic representation.

16. The method as recited in claim 15, wherein the reference to the item comprises an anaphora.

17. The method as recited in claim 15, wherein the identifier of the item is scanned prior to receiving the voice input from the user, after receiving the voice input from the user, or while at least a portion of the voice input is being received from the user.

18. The method as recited in claim 15, wherein the voice data includes a recording of the voice input from the user that is stored as an audio file in a memory of the handheld electronic device.

19. The method as recited in claim 15, wherein performing the action comprises initiating communication of information to be presented in a visual format via a user interface of the handheld electronic device.

20. The method as recited in claim 15, wherein performing the action comprises initiating communication of information to be presented in an audible format via a speaker of the handheld electronic device.