Voice-assisted restocking method and system

• US 10,614,805 B2
• Filed: 03/09/2018
• Issued: 04/07/2020
• Est. Priority Date: 03/09/2018
• Status: Active Grant

• Current Assignee: Midea Group Co., Ltd
• Original Assignee: Midea Group Co., Ltd
• Inventors: Huang, Yingbiao, Tang, Tian
• Primary Examiner(s): Riley, Marcus T
• Application Number:
  US15/917,304
• Publication Number:
  US 20190279625A1
• Time in Patent Office: 760 Days
• Field of Search: 704 1-504
• CPC Class Codes: F25D 2400/361 Interactive visual displays
• CPC Class Codes: F25D 2500/06 Stock management
• CPC Class Codes: F25D 2700/04 Sensors detecting the presence of a person
• CPC Class Codes: F25D 2700/06 Sensors detecting the presence of a product
• CPC Class Codes: F25D 2700/08 Sensors using Radio Frequency Identification [RFID]
• CPC Class Codes: F25D 29/00 Arrangement or mounting of control or safety devices
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0202 Constructional details or processes of manufacture of the input device
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 20/40145 Biometric identity checks
• CPC Class Codes: G06Q 30/0255 based on user history
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06V 40/1365 Matching; Classification
• CPC Class Codes: G07F 17/0092 for assembling and dispensing of pharmaceutical articles medical dispensers for private use, A61J7/00
• CPC Class Codes: G07F 9/009 User recognition or proximity detection
• CPC Class Codes: G07F 9/026 : for alarm, monitoring and auditing in vending machines or means for indication, e.g. when empty
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 : Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: H01H 2219/026 : with programming switches
• CPC Class Codes: H03K 17/18 : Modifications for indicating state of switch
• CPC Class Codes: H03K 17/965 : Switches controlled by moving an element forming part of the switch
• CPC Class Codes: H03K 2217/96031 : Combination of touch switch and LC display
• CPC Class Codes: H03K 2217/96042 : with illumination
• CPC Class Codes: H03K 2217/9653 : with illumination

Abstract

A voice-assisted restocking system includes: a first compartment for storing a plurality of consumable items; a restocking suggestion unit that is configured to receive voice-based restocking suggestions from a remotely located digital assistant server over a network communication interface; a voice output interface configured to output the voice-based restocking suggestions to a user; and a first hardware button that is located in the interior of the first compartment, wherein the first hardware button is configured to: after a first voice-based restocking suggestion from the digital assistant is output by the voice output interface, prompt the user to provide a confirmation input regarding the first voice-based restocking suggestion; and send a respective confirmation signal to the remotely located digital assistant when the first hardware button is activated by the user. A corresponding voice-assisted restocking method is also disclosed.

Claims

1. A voice-assisted restocking system, comprising:

a first compartment for storing a plurality of consumable items;

a restocking suggestion unit that is configured to receive voice-based restocking suggestions from a remotely located digital assistant server over a network communication interface;

a voice output interface configured to output the voice-based restocking suggestions to a user; and

a first hardware button that is located in an interior of the first compartment, wherein the first hardware button is configured to;

after a first voice-based restocking suggestion from the remotely located digital assistant server is output by the voice output interface, prompt the user to provide a confirmation input regarding the first voice-based restocking suggestion;

send a respective confirmation signal to the remotely located digital assistant server when the first hardware button is activated by the user within a threshold amount of time after the first voice-based restocking suggestion is output by the voice output interface; and

send a respective rejection signal to the remotely located digital assistant server when the first hardware button is not activated within the threshold amount of time after the first voice-based restocking suggestion is output by the voice output interface.

2. The voice-assisted restocking system of claim 1, including:

a voice input interface configured to capture voice inputs uttered by the user before the voice inputs are sent by a network communication unit to the remotely located digital assistant server.

3. The voice-assisted restocking system of claim 1, wherein:

the first compartment is an enclosed compartment of a refrigerator;

the enclosed compartment of the refrigerator has a door with an interior side facing an interior of the enclosed compartment when the door is in a closed state, and with an exterior side facing an exterior of the enclosed compartment when the door is in an open state; and

the first hardware button is located on the interior side of the door within the interior of the enclosed compartment when the door is in the closed state.

4. The voice-assisted restocking system of claim 1, wherein the first hardware button includes a visual feedback component that is configured to generate a respective light pattern corresponding to a selected one of a plurality of confirmation request types in accordance with a suggestion type of the first voice-based restocking suggestion.

5. The voice-assisted restocking system of claim 1, including:

a second hardware button that is located adjacent to the first hardware button in the interior of the first compartment, wherein the second hardware button is configured to;

after a second voice-based restocking suggestion from the remotely located digital assistant server is output by the voice output interface, prompt the user to provide a rejection input regarding the second voice-based restocking suggestion; and

send a rejection signal to the remotely located digital assistant server when the second hardware button is activated by the user.

6. The voice-assisted restocking system of claim 1, including:

a touch-screen display that is configured to display the first voice-based restocking suggestion and a confirmation of the first voice-based restocking suggestion in accordance with user activation of the first hardware button.

7. The voice-assisted restocking system of claim 6, wherein the restocking suggestion unit is configured to:

cause display of the first voice-based restocking suggestion on the touch-screen display in accordance with a determination that the first compartment is in a closed state; and

cause the first hardware button to generate a light pattern to prompt for the confirmation input regarding the first voice-based restocking suggestion in accordance with a determination that the first compartment has switched from the closed state to an open state.

8. The voice-assisted restocking system of claim 6, including:

a third hardware button that is located on an exterior of the first compartment, wherein the third hardware button is configured to;

after a third voice-based restocking suggestion from the remotely located digital assistant server is output by the voice output interface, prompt the user to provide a confirmation input regarding the third voice-based restocking suggestion; and

send a respective confirmation signal to the remotely located digital assistant server when the third hardware button is activated by the user.

9. The voice-assisted restocking system of claim 6, including:

a second compartment that is distinct from the first compartment;

a fourth hardware button that is located in the interior of the second compartment, wherein the fourth hardware button is configured to;

after a fourth voice-based restocking suggestion from the digital assistant is output by the voice output interface, prompt the user to provide a confirmation input regarding the fourth voice-based restocking suggestion; and

send a respective confirmation signal to the remotely located digital assistant server when the fourth hardware button is activated by the user, wherein;

the restocking suggestion unit is configured to cause the voice output interface to output a respective restocking suggestion with regard to an item stored in the first compartment in accordance with a determination that the first compartment is opened; and

cause the voice output interface to output a respective restocking suggestion with regard to an item stored in the second compartment in accordance with a determination that the second compartment is opened.

10. The voice-assisted restocking system of claim 6, including:

a user presence detection unit, wherein the user presence detection unit is configured to detect a user'"'"'s presence in a vicinity of the first compartment, wherein the user presence detection unit is configured to trigger the voice output interface to output the first voice-based restocking suggestion that has been received from the remotely located digital assistant sever.

11. The voice-assisted restocking system of claim 10, wherein:

the user presence detection unit is configured to detect that a door of the first compartment is opened, and

the user presence detection unit is configured to, in accordance with the detection that the door of the first compartment is opened and that the first voice-based restocking suggestion has been output by the voice output interface, cause the first hardware button to prompt the user to provide the confirmation input regarding the first voice-based restocking suggestion and to wait for the user'"'"'s confirmation input.

12. The voice-assisted restocking system of claim 6, wherein the first hardware button has a built-in fingerprint sensor, and wherein the voice-assisted restocking system further includes an authentication unit that is configured to:

in accordance with a determination that the first voice-based restocking suggestion is of a first suggestion type that requires user authentication, transmit a fingerprint captured by the built-in fingerprint sensor to the remotely located digital assistant server for authentication; and

in accordance with a determination that the first voice-based restocking suggestion is of a second suggestion type that does not require user authentication, forgo transmitting the fingerprint captured by the built-in fingerprint sensor to the remotely located digital assistant server.

13. A method of providing restocking suggestions, comprising:

at an appliance including one or more processors, memory, and a first compartment for storing a plurality of consumable items;

detecting a user presence in front of the first compartment;

in response to detecting the user presence and in accordance with a determination that the user presence meets first criteria, outputting a first voice-based restocking suggestion through a voice output interface;

after the first voice-based restocking suggestion is output by the voice output interface, providing a first prompt, via a first hardware button that is located in an interior of the first compartment, to request a confirmation input from a user regarding the first voice-based restocking suggestion;

while providing the first prompt, detecting activation of the first hardware button by the user; and

in response to detecting the activation of the first hardware button while the first prompt is provided, sending a respective confirmation signal regarding the first voice-based restocking suggestion to a remotely located digital assistant.

14. The method of claim 13, including:

while providing the first prompt via the first hardware button, receiving a voice input rejecting the first voice-based restocking suggestion through a voice input interface; and

in response to detecting a user'"'"'s voice input rejecting the first voice-based restocking suggestion, ceasing to providing the first prompt via the first hardware button.

15. The method of claim 13, wherein:

the first compartment is an enclosed compartment of a refrigerator;

the enclosed compartment of the refrigerator has a door with an interior side facing an interior of the enclosed compartment when the door is in a closed state, and with an exterior side facing an exterior of the enclosed compartment when the door is in an open state; and

the first hardware button is located on an interior side of the door within the interior of the enclosed compartment when the door is in the closed state.

16. The method of claim 13, including:

generating a respective light pattern on the first hardware button that corresponds to a selected one of a plurality of confirmation request types in accordance with a suggestion type of the first voice-based restocking suggestion.

17. The method of claim 13, wherein:

the first prompt is provided for no more than a threshold amount of time, and the respective confirmation signal is sent to the remotely located digital assistant server when the first hardware button is activated by the user within the threshold amount of time after the first voice-based restocking suggestion is output by the voice output interface.

18. The method of claim 13, including:

displaying the first voice-based restocking suggestion and a confirmation of the first voice-based restocking suggestion in accordance with user activation of the first hardware button.

19. The method of claim 18, including:

displaying the first voice-based restocking suggestion on a touch-screen display in accordance with a determination that the first compartment is in a closed state; and

generating a light pattern to prompt for the confirmation input regarding the first voice-based restocking suggestion in accordance with a determination that the first compartment has switched from the closed state to an open state.