Interfacing with a mobile telepresence robot

• US 9,469,030 B2
• Filed: 10/27/2015
• Issued: 10/18/2016
• Est. Priority Date: 01/28/2011
• Status: Active Grant

• Current Assignee: iRobot Corporation, Jonata Sub Two, Inc. (Teladoc Health, Inc.), Teladoc Health, Inc.
• Original Assignee: iRobot Corporation, Touch Technologies Incorporated
• Inventors: Wang, Yulun, Jordan, Charles S., Wright, Tim, Chan, Michael, Pinter, Marco, Hanrahan, Kevin, Sanchez, Daniel, Ballantyne, James, Herzog, Cody, Whitney, Blair, Lai, Fuji, Temby, Kelton, Rauhut, Eben Christopher, Kearns, Justin H., Wong, Cheuk Wah, Farlow, Timothy Sturtevant
• Primary Examiner(s): WONG, YUEN H
• Application Number:
  US14/924,471
• Publication Number:
  US 20160046021A1
• Time in Patent Office: 357 Days
• US Class Current: 1/1
• CPC Class Codes: B25J 11/009 Nursing, e.g. carrying sick persons, pushing wheelchairs, distributing drugs
• CPC Class Codes: B25J 11/0095 Manipulators transporting wafers
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A telepresence robot may include a drive system, a control system, an imaging system, and a mapping module. The mapping module may access a plan view map of an area and tags associated with the area. In various embodiments, each tag may include tag coordinates and tag information, which may include a tag annotation. A tag identification system may identify tags within a predetermined range of the current position and the control system may execute an action based on an identified tag whose tag information comprises a telepresence robot action modifier. The telepresence robot may rotate an upper portion independent from a lower portion. A remote terminal may allow an operator to control the telepresence robot using any combination of control methods, including by selecting a destination in a live video feed, by selecting a destination on a plan view map, or by using a joystick or other peripheral device.

Claims

1. A telepresence robot system comprising:

an electronic display;

a processor in communication with the electronic display;

a memory device in communication with the processor, the memory device containing instructions, that when executed by the processor, cause the processor to;

retrieve a destination list including destination locations within a facility;

receive a video feed from an imaging system of a remote telepresence robot at a first perspective;

display the video feed from the imaging system of the remote telepresence robot; and

control a movement of the remote telepresence robot by transmitting a command to the remote telepresence robot specifying the movement for the remote telepresence robot from a current location to a destination location via a drive motor; and

a user input device in communication with the processor, the user input device configured to enable a user to select the movement for the remote telepresence robot from the current location to the destination location via one of a plurality of options on a single user interface, the plurality of options comprising;

selecting a velocity-based control specifying a direction of the movement of the remote telepresence robot;

selecting a destination location of the remote telepresence robot with respect to the destination list; and

selecting a destination location of the remote telepresence robot with respect to the video feed.

2. The telepresence robot system of claim 1, wherein the selection of the movement for the remote telepresence robot from the current location to the destination location via the one of the plurality of options on the single user interface results in selection of an alternative perspective of the video feed.

3. The telepresence robot system of claim 1, wherein the selection of the movement further comprises selecting an alternative perspective of the video feed by selecting a point within the video feed.

4. The telepresence robot system of claim 1, wherein the selection of the movement further comprises selecting an alternative perspective of the video feed by incrementally panning or tilting the imaging system while the remote telepresence robot remains stationary.

5. The telepresence robot system of claim 1, wherein the selection of the movement relates to rotating one of a lower portion of the remote telepresence robot and an upper portion of the remote telepresence robot.

6. The telepresence robot system of claim 1, wherein the processor is further configured to:

receive the selection of the destination location of the remote robot from the user input device;

determine a sequence of coordinates relative to a plan view map to create a navigation path between the current location of the remote telepresence robot and the selected destination location of the remote telepresence robot; and

transmit a command to the remote telepresence robot, the command comprising the sequence of coordinates forming the navigation path.

7. The telepresence robot system of claim 6, wherein the processor is further configured to display the sequence of coordinates forming the navigation path overlaid on the plan view map.

8. The telepresence robot system of claim 6, wherein the processor is further configured to:

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot;

apply the distortion to the sequence of coordinates forming the navigation path to determine corresponding video coordinates and perspective data describing a location and perspective of the sequence of coordinates relative to the video feed; and

display a three-dimensional rendition of the sequence of coordinates forming the navigation path overlaid on the video feed.

9. The telepresence robot system of claim 8, wherein the three-dimensional rendition of the sequence of coordinates forming the navigation path is overlaid on the video feed with respect to a floor detected in the video feed.

10. The telepresence robot system of claim 1, wherein the processor is further configured to:

receive the selection of the destination location of the remote robot from the user input device;

transmit destination coordinates relative to a plan view map to the remote telepresence robot, the destination coordinates corresponding to the selected destination location;

receive a sequence of coordinates relative to the plan view map from a navigation system of the remote telepresence robot, the sequence of coordinates forming a navigation path between the current location of the remote telepresence robot and the selected destination location of the remote telepresence robot; and

display the sequence of coordinates forming the navigation path overlaid on the plan view map.

11. The telepresence robot system of claim 10, wherein the processor is further configured to:

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot;

apply the distortion to the sequence of coordinates forming the navigation path to determine corresponding video coordinates and perspective data describing a location and perspective of the sequence of coordinates relative to the video feed; and

display a three-dimensional rendition of the sequence of coordinates forming the navigation path overlaid on the video feed.

12. The telepresence robot system of claim 11, wherein the three-dimensional rendition of the sequence of coordinates forming the navigation path is overlaid on the video feed with respect to a floor detected in the video feed.

13. The telepresence robot system of claim 1, wherein the processor is further configured to receive coordinates on the plan view map of an obstacle detected by a sensor system of the remote telepresence robot.

14. The telepresence robot system of claim 1, wherein the destination list is stored remotely.

15. The telepresence robot system of claim 14, wherein the destination list is stored within the remote telepresence robot.

16. The telepresence robot system of claim 1, wherein the processor is further configured to:

determine a distortion between a plan view map and the video feed received from the imaging system of the remote telepresence robot; and

generate a hybrid map view comprising a blended view of the plan view map and the video feed from the imaging system of the remote telepresence robot.

17. The telepresence robot system of claim 16, wherein the hybrid map view comprises a three-dimensional representation of the plan view map overlaid on the video feed.

18. The telepresence robot system of claim 1, wherein the processor is further configured to:

receive a request via the user input device for a rendered look ahead for a virtual location of the remote telepresence robot on a plan view map;

determine a distortion between the plan view map and the video feed received from the imaging system of the remote telepresence robot;

generate a virtual three-dimensional video feed based on a virtual location of the remote telepresence robot; and

display the virtual three-dimensional video feed based on the virtual location of the remote telepresence robot.

19. The telepresence robot system of claim 1, wherein the processor is further configured to display a plan view map with an indication of the current location of the remote telepresence robot on a plan view map.

20. A method for controlling a remote telepresence robot, comprising:

receiving a video feed from an imaging system of the remote telepresence robot at a first perspective;

retrieving a destination list from a memory device including destination locations within a facility;

displaying, on an electronic display, the video feed from the imaging system of the remote telepresence robot;

controlling, by a processor, movement of the remote telepresence robot by transmitting a command to the remote telepresence robot to move from a current location to a destination location via a drive motor; and

receiving, by the processor, a selection of the movement for the remote telepresence robot from the current location to the destination location via one of a plurality of options on a single user interface, the plurality of options comprising;

selecting a velocity-based control specifying a direction of the movement of the remote telepresence robot;

selecting a destination location of the remote telepresence robot with respect to the destination list; and

selecting a destination location of the remote telepresence robot with respect to the video feed.