Multi-state model for robot and user interaction

• US 8,818,556 B2
• Filed: 01/13/2011
• Issued: 08/26/2014
• Est. Priority Date: 01/13/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sanchez, Russell Irvin, Chalabi, Malek M., Krauskopf, Kordell B.
• Primary Examiner(s): Tran, Dalena
• Application Number:
  US13/005,839
• Publication Number:
  US 20120185090A1
• Time in Patent Office: 1,321 Days
• Field of Search: 700/253, 700/257, 700/245, 700/247, 706/14, 706/45, 382/155, 318/567, 318/568.12, 318/568.15, 318/568.16
• US Class Current: 700/253
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 9/161 Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• CPC Class Codes: G05B 2219/40202 Human robot coexistence
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour

Abstract

The subject disclosure is directed towards a robot device including a model that controls a robot'"'"'s task-related operations to perform tasks and user-engagement operations to interact with the robot. The model controls the operating states, including transitioning from an autonomous task state to an engaged state based on current context determined from various stimuli, e.g., information received via sensors of the robot and/or learned data. The robot may seek to engage the user, the user may seek to engage the robot, or the user and robot may meet by chance in which either may attempt to initiate the engagement.

Claims

1. In a computing environment, a robotic device comprising:

a sensor set that inputs information into the robotic device;

an output mechanism comprising one or more output modalities for communicating with a user; and

a model configured to control the operation of the robotic device with respect to operating states, the states including an autonomous task state in which the robot performs a task independent of direct user input, and an engaged state in which the robot interacts with the user via the sensor set and the output mechanism, the model configured to determine when to transition from the autonomous task state to the engagement state, and from the engagement state to the autonomous task state, based on a priority level ordering and a current context including information received via the sensor set.

2. The robotic device of claim 1 wherein the autonomous task state includes a user-directed mode in which the robot performs a task set comprising one or more user tasks based upon user input, and a self-directed mode in which the robot performs one or more tasks that are not based upon user input.

3. The robotic device of claim 2 further comprising a rest state, and wherein the model controls the operation of the robotic device according to a priority level ordering in which the engaged state has a higher priority level than the user-directed mode in the autonomous task state, the user-directed mode has a higher priority level than the self-directed mode in the autonomous task state, and the self-directed mode in the autonomous task state has a higher priority level than the rest state.

4. The robotic device of claim 1 further comprising a rest state, wherein the model is further configured to exit the rest state and enter the engagement state based upon detecting an explicit engagement request, and to exit the rest state and enter the autonomous task state upon detecting a triggering event.

5. The robotic device of claim 1 wherein the model transitions the robot from the autonomous task state to the engagement state when the robot initiates an engagement attempt with the user and the attempt succeeds.

6. The robotic device of claim 1 wherein the model transitions the robot from the autonomous task state to the engagement state when the robot detects the user by a chance encounter, and the robot further detects that the user is possibly interested in interaction with the robot.

7. The robotic device of claim 1 wherein the model transitions the robot from the autonomous task state to the engagement state when the user initiates an engagement attempt with the robot while the robot is performing a task, and while in the engagement state the robot communicates with the user to determine whether to resume the task or remain in the engagement state.

8. In a computing environment, a method performed at least in part on at least one processor, comprising, running one or more programs in a robot that cause the robot to operate autonomously, including performing a task set while in an autonomous task state, detecting a user, attempting to engage with the user, and if the attempt to engage is successful, entering into an engagement state to interact with the user.

9. The method of claim 8 further comprising, entering into the engagement state, interacting with the user, and disengaging from the engagement state based upon a change in current state data.

10. The method of claim 9 wherein disengaging from the engagement state based upon a change in current state data comprises detecting that the user has moved away or otherwise directed attention away from the robot for a sustained period, or receiving communication data corresponding to an disengage command from a user.

11. The method of claim 8 wherein attempting to engage with that user includes pausing a task of the task set and initiating an offer of engagement.

12. The method of claim 8 wherein detecting the user comprises detecting one of a plurality of stimuli, including detecting a remote proxy communication, detecting audio data, or detecting motion, and wherein attempting to engage with the user comprises changing at least one behavior corresponding to the attempt to engage based upon which of the stimuli was detected.

13. The method of claim 8 wherein detecting the user comprises sensing nearby presence of the user by chance, or by sensing nearby presence of the user by actively seeking the user.

14. The method of claim 8 wherein attempting to engage with the user comprises initiating the engagement attempt by moving towards the user, or outputting audible or visible communication data, or both, directed towards the user, or both moving towards the user and outputting audible or visible communication data, or both, directed towards the user.

15. The method of claim 8 wherein attempting to engage with the user comprises adapting the engagement attempt based on previous engagement history with the user.

16. The method of claim 8 wherein adapting the engagement attempt comprises waiting until a frequency-determined time for the engagement attempt is reached.

17. The method of claim 8 wherein the attempt to engage is successful, and further comprising, remaining engaged with the user based upon detecting that the user is looking at the robot, speaking to the robot, or gesturing to the robot, or any combination of looking at the robot, speaking to the robot, or gesturing to the robot.

18. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:

running one or more programs in a robot that cause the robot to operate autonomously, including operating in a self-directed mode of an autonomous task state in which the robot independently performs a task;

operating in a user-directed mode of the autonomous task state in which the robot performs a user task based on previous input from a user and detection of a triggering event;

completing the user task; and

operating in another instance of the user-directed mode in which the robot performs another user task, including transitioning to an engagement state from the autonomous task state, receiving information corresponding to the other user task while in the engagement state, and transitioning to the autonomous task state to perform the other user task.

19. The one or more computer-readable media of claim 18 having further computer-executable instructions comprising, transitioning from a rest state to the engagement state, or transitioning from a rest state to the autonomous task state.

20. The one or more computer-readable media of claim 18 having further computer-executable instructions comprising, sensing input data to determine a current context, and wherein the triggering event is based on information corresponding to the current context.