INTERFACING WITH A MOBILE TELEPRESENCE ROBOT

• US 20120197439A1
• Filed: 01/27/2012
• Published: 08/02/2012
• Est. Priority Date: 01/28/2011
• Status: Active Grant

• Current Assignee: iRobot Corporation, Jonata Sub Two, Inc. (Teladoc Health, Inc.), Teladoc Health, Inc.
• Original Assignee: iRobot Corporation, Touch Technologies Incorporated
• Inventors: Wang, Yulun, Jordan, Charles S., Pinter, Marco, Hanrahan, Kevin, Whitney, Blair, Lai, Fuji, Kearns, Justin H., Wright, Tim, Chan, Michael, Sanchez, Daniel, Ballantyne, James, Herzog, Cody, Temby, Kelton, Rauhut, Eben Christopher, Wong, Cheuk Wah, Farlow, Timothy Sturtevant
• Granted Patent: US 8,965,579 B2
• Time in Patent Office: Days
• US Class Current: 700/259
• CPC Class Codes: B25J 11/009 Nursing, e.g. carrying sick persons, pushing wheelchairs, distributing drugs
• CPC Class Codes: B25J 11/0095 Manipulators transporting wafers
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A telepresence robot may include a drive system, a control system, an imaging system, and a mapping module. The mapping module may access a plan view map of an area and tags associated with the area. In various embodiments, each tag may include tag coordinates and tag information, which may include a tag annotation. A tag identification system may identify tags within a predetermined range of the current position and the control system may execute an action based on an identified tag whose tag information comprises a telepresence robot action modifier. The telepresence robot may rotate an upper portion independent from a lower portion. A remote terminal may allow an operator to control the telepresence robot using any combination of control methods, including by selecting a destination in a live video feed, by selecting a destination on a plan view map, or by using a joystick or other peripheral device.

Claims

1. A telepresence robot, comprising:

an upper portion;

a lower portion rotatably connected to the upper portion;

a drive system configured to move the telepresence robot according to drive instructions;

a control system in communication with the drive system, the control system configured to generate drive instructions to cause the drive system to move the telepresence robot; and

a rotation system configured to rotate the robot from a first heading to a second heading by rotating the upper portion and the lower portion independently.

2. The telepresence robot of claim 1, wherein the rotation system is configured to rotate the robot toward a second heading by:

rotating the upper portion of the robot toward the second heading;

detecting that the upper portion of the robot has reached a panning limit of the upper portion of the robot relative to the lower portion of the robot;

beginning to rotate the lower portion of the robot toward the second heading at the panning limit of the upper portion of the robot;

detecting that the upper portion of the robot has reached the second heading; and

continuing to rotate the lower portion of the robot toward the second heading while simultaneously counter-rotating the upper portion of the robot, such that the upper portion of the robot maintains the second heading.

3. The telepresence robot of claim 2, wherein the panning limit is reached when the upper portion cannot physically rotate anymore with respect to the lower portion of the robot.

4. The telepresence robot of claim 2, wherein the panning limit is reached when the upper portion is misaligned with respect to the lower portion a predefined number of rotational degrees.

5. The telepresence robot of claim 2, wherein the panning limit is a function of the number of degrees the upper portion is misaligned with respect to the lower portion and the length of time the upper portion has been misaligned with respect to the lower portion.

6. The telepresence robot of claim 1, wherein the rotation system is configured to rotate the robot toward a second heading by:

rotating the upper portion of the robot toward the second heading at a first rotational velocity;

rotating the lower portion of the robot toward the second heading at a second rotation velocity;

detecting that the upper portion of the robot has reached the second heading; and

continuing to rotate the lower portion of the robot toward the second heading while simultaneously counter-rotating the upper portion of the robot, such that the upper portion of the robot maintains the second heading.

7. The telepresence robot of claim 1, further comprising:

an imaging system in communication with the control system; and

a positioning system in communication with the control system configured to provide a current position of the robot relative to a plan view map and a current alignment of the upper portion with respect to the plan view map, and

wherein the control system is configured to transmit a video feed from the imaging system, the current position of the robot, and the current alignment of the upper portion to a remote terminal, such that the remote terminal can;

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot;

apply the distortion to a tag having coordinates associated with the plan view map in order to determine corresponding video coordinates and perspective data describing the location and perspective of the tag relative to the video feed; and

display a three-dimensional rendition of the tag overlaid on the video feed using the video coordinates.

8. The telepresence robot of claim 1, further comprising:

an imaging system in communication with the control system; and

a positioning system in communication with the control system configured to provide a current position of the robot relative to a plan view map and a current alignment of the upper portion with respect to the plan view map, and

wherein the control system is configured to;

transmit a video feed from the imaging system, the current position of the robot, and the current alignment of the upper portion to a remote terminal;

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot; and

apply the distortion to a tag having coordinates associated with the plan view map in order to determine corresponding video coordinates and perspective data describing the location and perspective of the tag relative to the video feed, such that the remote terminal can display a three-dimensional rendition of the tag overlaid on the video feed using the video coordinates.

9. A telepresence robot system local terminal comprising:

an electronic display;

a processor in communication with the electronic display; and

a memory in communication with the processor, the memory comprising instructions executable by the processor configured to cause the processor to;

retrieve at least a portion of a plan view map representative of robot-navigable areas of a robot operating surface;

retrieve at least one of a plurality of tags, each of the plurality of tags comprising tag coordinates describing the relative location of the tag and tag information;

receive a video feed from an imaging system of a remote telepresence robot;

receive positioning information associated with a current position of the remote telepresence robot;

display the video feed from the imaging system of the remote telepresence robot;

display a rendition of the tag information of the at least one tag on the video feed using the tag coordinates; and

transmit a command to the remote telepresence robot.

10. The telepresence robot system of claim 9, wherein the instructions executable by the processor are further configured to cause the processor to:

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot;

apply the distortion to the tag coordinates of the at least one tag to determine corresponding video coordinates and perspective data describing a location and perspective of the at least one tag relative to the video feed; and

display a three-dimensional rendition of the tag information of the at least one tag overlaid on the video feed using the tag video coordinates.

11. The telepresence robot system of claim 10, wherein the three-dimensional rendition of the tag information is dynamically re-rendered based on the current position of the remote telepresence robot and the perspective of the at least one tag relative to the video feed.

12. The telepresence robot system of claim 10, wherein the three-dimensional rendition of the tag information is overlaid on the video feed with respect to an object detected in the video feed.

13. The telepresence robot system of claim 12, wherein the three-dimensional rendition of the tag information is overlaid along a wall detected in the video feed.

14. The telepresence robot system of claim 9, wherein the tag information of the at least one tag comprises a robot action modifier, and

wherein the robot action modifier is configured to provide execution instructions to a control system of the telepresence robot to execute a first action in response to the telepresence robot being within a predetermined range of the tag coordinates of the at least one tag.

15. The telepresence robot system of claim 14, wherein the instructions executable by the processor are further configured to cause the processor to transmit the execution instruction to the control system of the telepresence robot when the telepresence robot is within a predetermined range of the tag coordinates of the at least one tag.

16. The telepresence robot system of claim 14, wherein the robot action modifier further comprises instructions regarding one of a time and a location on the plan view map that the control system of the telepresence robot should execute the first action.

17. The telepresence robot system of claim 9, wherein the instructions executable by the processor are further configured to cause the processor to:

receive a sequence of coordinates relative to the plan view map forming a path along which the remote telepresence robot has traveled;

store the sequence of coordinates forming the path as a path tag comprising tag coordinates and tag information;

retrieve the path tag when the remote telepresence robot arrives within a predetermined distance of the tag coordinates; and

display a rendition of the tag information of the path tag on the video feed using the tag coordinates.

18. The telepresence robot system of claim 17, wherein the telepresence robot system local terminal further comprises at least one user input device, and

wherein the sequence of coordinates forming the path is provided by the user input device.

19. The telepresence robot system of claim 17, wherein the sequence of coordinates forming the path is provided by the remote telepresence robot.

20. The telepresence robot system of claim 9, further comprising a communication system configured to facilitate communication between the telepresence robot system local terminal and the remote telepresence robot.

21. The telepresence robot system of claim 9, wherein the local terminal further comprises at least one user input device, and

wherein the user input device is configured to allow a user to provide an indication of a desired destination of the remote telepresence robot on at least one of the plan view map and the video feed from the imaging system of the remote telepresence robot, and

wherein the command transmitted to the remote telepresence robot comprises the desired destination.

22. The telepresence robot system of claim 21, wherein the sequence of coordinates forming the robot path is based at least in part on tagging information associated with the at least one tag.

23. The telepresence robot system of claim 21, wherein the instructions executable by the processor are further configured to cause the processor to:

determine a sequence of coordinates relative to the plan view map to create a robot path between the current position of the remote telepresence robot and the desired destination of the remote telepresence robot, and

wherein the command transmitted to the remote telepresence robot comprises the sequence of coordinates forming the robot path.

24. The telepresence robot system of claim 23, wherein the instructions executable by the processor are further configured to cause the processor to display the sequence of coordinates forming the robot path overlaid on the plan view map.

25. The telepresence robot system of claim 23, wherein the instructions executable by the processor are further configured to cause the processor to:

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot;

apply the distortion to the sequence of coordinates forming the robot path to determine corresponding video coordinates and perspective data describing a location and perspective of the sequence of coordinates relative to the video feed; and

display a three-dimensional rendition of the sequence of coordinates forming the robot path overlaid on the video feed.

26. The telepresence robot system of claim 25, wherein the three-dimensional rendition of the sequence of coordinates forming the robot path is overlaid on the video feed with respect to a floor detected in the video feed.

27. The telepresence robot system of claim 9, wherein the instructions executable by the processor are further configured to cause the processor to:

receive a sequence of coordinates relative to the plan view map from a navigation system of the remote telepresence robot, the sequence of coordinates forming a robot path between the current position of the remote telepresence robot and a desired destination of the remote telepresence robot; and

display the sequence of coordinates forming the robot path overlaid on the plan view map.

28. The telepresence robot system of claim 27, wherein the instructions executable by the processor are further configured to cause the processor to:

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot;

apply the distortion to the sequence of coordinates forming the robot path to determine corresponding video coordinates and perspective data describing the location and perspective of the sequence of coordinates relative to the video feed; and

display a three-dimensional rendition of the sequence of coordinates forming the robot path overlaid on the video feed.

29. The telepresence robot system of claim 28, wherein the three-dimensional rendition of the sequence of coordinates forming the robot path is overlaid on the video feed with respect to a floor detected in the video feed.

30. The telepresence robot system of claim 9, wherein the tag information relates to at least one of a position, a path, and a volume, and

wherein the control system is configured to execute an action relative to the at least one of the position, the path, and the volume.

31. The telepresence robot system of claim 9, wherein the instructions executable by the processor are further configured to cause the processor to receive coordinates on the plan view map of an obstacle detected by a sensor system of the remote telepresence robot.

32. The telepresence robot system of claim 9, wherein the plan view map and the plurality of tags are stored remotely.

33. The telepresence robot system of claim 31, wherein the plan view map and the plurality of tags are stored within the remote telepresence robot.

34. The telepresence robot system of claim 9, wherein the instructions executable by the processor are further configured to cause the processor to:

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot; and

generate a hybrid map view comprising a blended view of the plan view map and the video feed from the imaging system of the remote telepresence robot.

35. The telepresence robot system of claim 34, wherein the hybrid map view comprises a three-dimensional representation of the plan view map overlaid on the video feed.

36. The telepresence robot system of claim 9, wherein the telepresence robot system local terminal further comprises at least one user input device, and

wherein the instructions executable by the processor are further configured to cause the processor to:

receive a request via the at least one input device for a rendered look ahead for a virtual location of the remote telepresence robot on the plan view map;

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot;

generate a virtual three-dimensional video feed based on a virtual location of the remote telepresence robot; and

display the virtual three-dimensional video feed based on the virtual location of the remote telepresence robot.

37. The telepresence robot system of claim 9, wherein the tag information of the at least one tag comprises a set of coordinates with respect to the plan view map defining a protected region, and

wherein the tag information of the at least one tag is configured to indicate the presence of the protected region.

38. The telepresence robot system of claim 9, wherein the instructions executable by the processor are further configured to cause the processor to:

receive a request to create a new tag;

associate tag coordinates describing a relative location of the new tag and tag information with the new tag; and

display a rendition of the tag information of the new tag on the video feed using the tag coordinates.

39. The telepresence robot system of claim 38, wherein the request to create the new tag is generated by the remote telepresence robot.

40. The telepresence robot system of claim 38, wherein the request to create the new tag is automatically generated based on a detected object in the video feed.

41. The telepresence robot system of claim 40, wherein the new tag is a temporary tag configured to expire once the detected object is no longer present in the video feed.

42. The telepresence robot system of claim 40, wherein the object is a person and the tag information of the new tag comprises identification information associated with the person.

43. The telepresence robot system of claim 40, wherein the object is a person and the tag information of the new tag comprises potential actions the remote telepresence robot can execute with respect to the person.

44. The telepresence robot system of claim 38, wherein the request to create the new tag is generated by a user input device in communication with the telepresence robot system local terminal.

45. The telepresence robot system of claim 40, wherein the request to create the new tag is made with respect to the video feed.

46. The telepresence robot system of claim 40, wherein the request to create the new tag is made with respect to the plan view map.

47. The telepresence robot system of claim 40, wherein the request to create a new tag is made with respect to the current position of the remote telepresence robot.

48. The telepresence robot system of claim 38, wherein the tag information relates to at least one of a position, a path, and a volume, and

wherein the control system is configured to execute an action relative to the at least one of the position, the path, and the volume.

49. The telepresence robot system of claim 9, wherein the instructions executable by the processor are further configured to cause the processor to display the plan view map with an indication of the current position of the telepresence robot on the plan view map.

50. The telepresence robot system of claim 49, wherein the instructions executable by the processor are further configured to cause the processor to display a rendition of the tag information on the plan view map using the tag coordinates.

51. The telepresence robot system of claim 9, wherein the tag information comprises a tag annotation, and wherein displaying a rendition of the tag information comprises displaying a rendition of the tag annotation.

52. A method for changing a heading of a robot comprising:

transmitting a heading to a control system of a robot, the control system of the robot in communication with a drive system configured to move the robot according to drive instructions; and

rotating, via the drive system, an upper portion of the robot toward the heading independently from the lower portion of the robot.

53. A method for controlling a remote telepresence robot comprising:

retrieving at least a portion of a plan view map representative of robot-navigable areas of a robot operating surface;

retrieving at least one of a plurality of tags, each of the plurality of tags comprising tag coordinates describing the relative location of the tag and tag information;

receiving a video feed from an imaging system of a remote telepresence robot;

receiving positioning information associated with a current position of the remote telepresence robot;

displaying, via an electronic display, the video feed from the imaging system of the remote telepresence robot;

displaying, via the electronic display, a rendition of the tag information of the at least one tag on the video feed using the tag coordinates; and

transmitting a command to the remote telepresence robot.