Method and system for providing remote robotic control

• US 10,678,264 B2
• Filed: 10/10/2018
• Issued: 06/09/2020
• Est. Priority Date: 10/10/2018
• Status: Active Grant

• Current Assignee: Midea Group Co., Ltd
• Original Assignee: Midea Group Co., Ltd
• Inventors: Tian, Yuan, Mani, Suresh
• Primary Examiner(s): Shafi, Muhammad
• Application Number:
  US16/156,969
• Publication Number:
  US 20200117212A1
• Time in Patent Office: 608 Days
• Field of Search: 701 26
• CPC Class Codes: B25J 13/025 comprising haptic means
• CPC Class Codes: B25J 13/085 Force or torque sensors B25J13/082, B25J13/084 take precedence
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/005 by providing the operator with signals other than visual, e.g. acoustic, haptic
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

A virtualized environment corresponding to a physical environment currently surrounding a robot is displayed. The virtualized environment is updated in accordance with streaming environment data received from sensors collocated with the robot. A first user input inserting a first virtual object at a first location in the virtualized environment is detected. The virtualized environment is modified in accordance with the insertion of the first virtual object at the first location. The first virtual object at the first location causes the robot to execute a first navigation path in the physical environment. A second user input is detected that moves the first virtual object along a movement path to a second location in the virtualized environment. The movement path is constrained by simulated surfaces in the virtualized environment, and the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment.

Claims

1. A method of providing mixed-initiative robotic control, including:

at a computing device having one or more processors and memory, wherein the computing device is communicably coupled to a robot and is configured to generate a planned path for the robot in accordance with a first set of preprogrammed path-planning instructions, and the robot is configured to navigate within a physical environment in accordance with the planned path received from the computing device and locally-stored path-execution instructions;

displaying a control user interface via a display generation component coupled to the computing device, including displaying a virtualized environment corresponding to a first physical environment currently surrounding the robot, wherein the virtualized environment is generated and updated in accordance with streaming environment data received from a first set of sensors collocated with the robot;

while displaying the virtualized environment, detecting a first user input inserting a first virtual object at a first location in the virtualized environment;

in response to detecting the first user input, modifying the first virtualized environment in accordance with the insertion of the first virtual object at the first location, wherein the first virtual object at the first location causes the robot to execute a first navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions;

while displaying the first virtual object at the first location in the virtualized environment and while the robot is executing the first navigation path in the physical environment, detecting a second user input, including detecting a first movement input directed to the first virtual object via a haptic-enabled input device; and

in response to detecting the second user input;

moving the first virtual object along a first movement path to a second location in the virtualized environment in accordance with the first movement input, wherein the first movement path is constrained by one or more simulated surfaces in the virtualized environment, wherein the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions.

2. The method of claim 1, including:

during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via the haptic-enabled input device in accordance with simulated friction forces between the first virtual object and a simulated surface at a current location of the first virtual object.

3. The method of claim 1, wherein the first virtual object has a predefined size and shape, and wherein the method includes:

during movement of the first virtual object along the first movement path to the second location in the virtualized environment, detecting that the first virtual object has come in contact with a first simulated surface in a current direction of movement of the first virtual object; and

in response to detecting that the first virtual object has come in contact with the first simulated surface in the current direction of movement of the first virtual object, providing haptic feedback via the haptic-enabled input device in accordance with simulated reactive forces from the first simulated surface to the first virtual object.

4. The method of claim 1, wherein the first virtual object has a predefined size and shape, and wherein the method includes:

during movement of the first virtual object along the first movement path to the second location in the virtualized environment, detecting that the first virtual object has come in contact with a second virtual object that was previously placed in the simulated environment in the current direction of movement of the first virtual object; and

in response to detecting that the first virtual object has come in contact with the second virtual object in the current direction of movement of the first virtual object, providing haptic feedback via the haptic-enabled input device in accordance with simulated reactive forces from the second virtual object to the first virtual object.

5. The method of claim 1, wherein the first virtual object is marked as an obstacle and the robot avoids the first virtual object during navigation on a path generated in accordance with the first set of pre-programmed instructions.

6. The method of claim 1, wherein the first virtual object is marked as a target and the robot approaches the first labeled object during navigation on a path generated in accordance with the first set of pre-programmed instructions.

7. The method of claim 1, wherein the first virtual object is marked as a passage, the presence of the first virtual object overrides presence of a physical obstacle during the robot'"'"'s navigation on a path generated in accordance with the first set of preprogrammed path-planning instructions.

8. A computing device for providing mixed-initiative robotic control, including:

one or more processors; and

memory storing instructions, wherein;

the computing device is communicably coupled to a robot and is configured to generate a planned path for the robot in accordance with a first set of preprogrammed path-planning instructions,

the robot is configured to navigate within a physical environment in accordance with the planned path received from the computing device and locally-stored path-execution instructions, and

the instructions, when executed by the one or more processors, cause the processors, to perform operations comprising;

displaying a control user interface via a display generation component coupled to the computing device, including displaying a virtualized environment corresponding to a first physical environment currently surrounding the robot, wherein the virtualized environment is generated and updated in accordance with streaming environment data received from a first set of sensors collocated with the robot;

while displaying the virtualized environment, detecting a first user input inserting a first virtual object at a first location in the virtualized environment;

in response to detecting the first user input, modifying the first virtualized environment in accordance with the insertion of the first virtual object at the first location, wherein the first virtual object at the first location causes the robot to execute a first navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions;

while displaying the first virtual object at the first location in the virtualized environment and while the robot is executing the first navigation path in the physical environment, detecting a second user input, including detecting a first movement input directed to the first virtual object via a haptic-enabled input device; and

in response to detecting the second user input;

moving the first virtual object along a first movement path to a second location in the virtualized environment in accordance with the first movement input, wherein the first movement path is constrained by one or more simulated surfaces in the virtualized environment, wherein the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions.

9. The computing device of claim 8, wherein the operations include:

during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via the haptic-enabled input device in accordance with simulated friction forces between the first virtual object and a simulated surface at a current location of the first virtual object.

10. The computing device of claim 8, wherein the first virtual object has a predefined size and shape, and wherein the method includes:

during movement of the first virtual object along the first movement path to the second location in the virtualized environment, detecting that the first virtual object has come in contact with a first simulated surface in a current direction of movement of the first virtual object; and

in response to detecting that the first virtual object has come in contact with the first simulated surface in the current direction of movement of the first virtual object, providing haptic feedback via the haptic-enabled input device in accordance with simulated reactive forces from the first simulated surface to the first virtual object.

11. The computing device of claim 8, wherein the first virtual object has a predefined size and shape, and wherein the operations include:

during movement of the first virtual object along the first movement path to the second location in the virtualized environment, detecting that the first virtual object has come in contact with a second virtual object that was previously placed in the simulated environment in the current direction of movement of the first virtual object; and

in response to detecting that the first virtual object has come in contact with the second virtual object in the current direction of movement of the first virtual object, providing haptic feedback via the haptic-enabled input device in accordance with simulated reactive forces from the second virtual object to the first virtual object.

12. The computing device of claim 8, wherein the first virtual object is marked as an obstacle and the robot avoids the first virtual object during navigation on a path generated in accordance with the first set of pre-programmed instructions.

13. The computing device of claim 8, wherein the first virtual object is marked as a target and the robot approaches the first labeled object during navigation on a path generated in accordance with the first set of pre-programmed instructions.

14. The computing device of claim 8, wherein the first virtual object is marked as a passage, the presence of the first virtual object overrides presence of a physical obstacle during the robot'"'"'s navigation on a path generated in accordance with the first set of preprogrammed path-planning instructions.

15. A computer-readable storage medium storing instructions, the instructions, when executed by one or more processors of a computing device, cause the computing device to perform operations, wherein:

the computing device is communicably coupled to a robot and is configured to generate a planned path for the robot in accordance with a first set of preprogrammed path-planning instructions,

the robot is configured to navigate within a physical environment in accordance with the planned path received from the computing device and locally-stored path-execution instructions, and

the operations include;

displaying a control user interface via a display generation component coupled to the computing device, including displaying a virtualized environment corresponding to a first physical environment currently surrounding the robot, wherein the virtualized environment is generated and updated in accordance with streaming environment data received from a first set of sensors collocated with the robot;

while displaying the virtualized environment, detecting a first user input inserting a first virtual object at a first location in the virtualized environment;

in response to detecting the first user input, modifying the first virtualized environment in accordance with the insertion of the first virtual object at the first location, wherein the first virtual object at the first location causes the robot to execute a first navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions;

while displaying the first virtual object at the first location in the virtualized environment and while the robot is executing the first navigation path in the physical environment, detecting a second user input, including detecting a first movement input directed to the first virtual object via a haptic-enabled input device; and

in response to detecting the second user input;

moving the first virtual object along a first movement path to a second location in the virtualized environment in accordance with the first movement input, wherein the first movement path is constrained by one or more simulated surfaces in the virtualized environment, wherein the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions.

16. The computer-readable storage medium of claim 15, wherein the operations include:

during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via the haptic-enabled input device in accordance with simulated friction forces between the first virtual object and a simulated surface at a current location of the first virtual object.

17. The computer-readable storage medium of claim 15, wherein the first virtual object has a predefined size and shape, and wherein the operations include:

during movement of the first virtual object along the first movement path to the second location in the virtualized environment, detecting that the first virtual object has come in contact with a first simulated surface in a current direction of movement of the first virtual object; and

in response to detecting that the first virtual object has come in contact with the first simulated surface in the current direction of movement of the first virtual object, providing haptic feedback via the haptic-enabled input device in accordance with simulated reactive forces from the first simulated surface to the first virtual object.

18. The computer-readable storage medium of claim 15, wherein the first virtual object has a predefined size and shape, and wherein the method includes:

during movement of the first virtual object along the first movement path to the second location in the virtualized environment, detecting that the first virtual object has come in contact with a second virtual object that was previously placed in the simulated environment in the current direction of movement of the first virtual object; and

in response to detecting that the first virtual object has come in contact with the second virtual object in the current direction of movement of the first virtual object, providing haptic feedback via the haptic-enabled input device in accordance with simulated reactive forces from the second virtual object to the first virtual object.

19. The computer-readable storage medium of claim 15, wherein the first virtual object is marked as an obstacle and the robot avoids the first virtual object during navigation on a path generated in accordance with the first set of pre-programmed instructions.

20. The computer-readable storage medium of claim 15, wherein the first virtual object is marked as a target and the robot approaches the first labeled object during navigation on a path generated in accordance with the first set of pre-programmed instructions.