Mobile robots moving on a visual display

• US 9,372,095 B1
• Filed: 05/08/2014
• Issued: 06/21/2016
• Est. Priority Date: 05/08/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Chen, Charles L., Raman, Tiruvilwamalai Venkatram
• Primary Examiner(s): Frejd, Russell
• Application Number:
  US14/273,107
• Time in Patent Office: 775 Days
• Field of Search: 701/431, 701/523, 701/516, 701/519, 701/538, 345/584, 345/173, 345/175, 345/157, 340/407.2, 700/245, 700/248, 700/259, 700/264, 715/702
• US Class Current: 1/1
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: G01C 21/3652 Guidance using non-audiovisual output, e.g. tactile, haptic or electric stimuli
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/03548 Sliders, in which the moving part moves in a plane
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G09B 21/007 using both tactile and audible presentation of the information
• CPC Class Codes: G09B 29/106 using electronic means

Abstract

A system for using mobile robots that track visual features of a drawing to create a tangible tactile interface for a user. The user places a hand or finger on a mobile robot to trace and consequently cognize features of interest. The mobile robot tracks visual features and reports a location of the mobile robot to a guidance system. The guidance system presents location information to the user.

Claims

1. A system comprising:

a display surface displaying a plurality of visual features;

a guidance system; and

a mobile robot comprising;

a drive system configured to drive the mobile robot over the display surface, such that the mobile robot physically traverses across the display surface;

a communication system configured to communicate with the guidance system;

a sensor system configured to detect the visual features on the display surface; and

a control system configured to cause the mobile robot to track, using the drive system and the sensor system, a first visual feature on the display surface, and to report, using the communication system, a location of the mobile robot to the guidance system while tracking the first visual feature;

wherein the guidance system is configured to receive the location of the mobile robot and, in response, determine location information associated with the location of the mobile robot on the display surface and present the location information to a user in physical contact with a surface of the mobile robot.

2. The system of claim 1, wherein the display surface is an electronic display device, and wherein the guidance system is coupled to the electronic display device and configured to cause the electronic display device to display the visual features.

3. The system of claim 2, wherein the guidance system is a tablet computer and the electronic display device is a touchscreen.

4. The system of claim 1, wherein the first visual feature is a line, and wherein causing the mobile robot to track the first visual feature comprises executing a line following routine.

5. The system of claim 1, wherein the display surface displays a map, and wherein determining the location information associated with the location of the mobile robot on the display surface comprises accessing map data associating location information for geographic locations with corresponding locations on the map.

6. The system of claim 1, wherein presenting the location information to the user comprises announcing the location information using an audio speaker.

7. The system of claim 6, wherein the display surface displays a map, and wherein determining the location information associated with the location of the mobile robot on the display surface comprises determining a street name marked on the map near the location of the mobile robot, and wherein announcing the location information comprises announcing the street name.

8. The system of claim 1, wherein the mobile robot comprises a haptic feedback system, and wherein presenting the location information to the user comprises sending an instruction to the mobile robot to actuate the haptic feedback system.

9. A system comprising:

a display surface displaying a plurality of visual features;

a guidance system; and

a mobile robot comprising;

a drive system configured to drive the mobile robot over the display surface;

a communication system configured to communicate with the guidance system;

a sensor system configured to detect the visual features on the display surface;

a control system configured to cause the mobile robot to track, using the drive system and the sensor system, a first visual feature on the display surface, and to report, using the communication system, a location of the mobile robot to the guidance system while tracking the first visual feature;

wherein the guidance system is configured to receive the location of the mobile robot and, in response, determine location information associated with the location of the mobile robot on the display surface and present the location information to a user in physical contact with a surface of the mobile robot, and

a second mobile robot comprising a second drive system, a second communication system configured to communicate with the guidance system, a second sensor system, and a second control system;

wherein the mobile robot comprises a first tactile feature and the second mobile robot comprises a second tactile feature different from the first tactile feature so that the user can distinguish the mobile robot and the second mobile robot by sense of touch.

10. The system of claim 9, wherein the display surface displays a map comprising first and second layers of information, and wherein the first visual feature represents a first element in the first layer of information, and wherein the second mobile robot is configured to track a second visual feature that represents a second element in the second layer of information.

11. A method performed by a guidance system comprising one or more computers configured to perform operations comprising:

receiving a location from a mobile robot tracking a visual feature displayed on a display surface, the tracking including the mobile robot physically traversing along the visual feature and across the display surface of an electronic device;

determining location information associated with the location of the mobile robot on the display surface; and

presenting the location information to a user in physical contact with a surface of the mobile robot.

12. The method of claim 11, comprising:

instructing the user to place the robot in a known location on the display surface; and

sending an initial location corresponding to the known location to the mobile robot.

13. The method of claim 11, wherein the display surface displays a map, and wherein determining the location information associated with the location of the mobile robot on the display surface comprises accessing map data associating location information for geographic locations with corresponding locations on the map.

14. The method of claim 11, wherein presenting the location information to the user comprises announcing the location information using an audio speaker.

15. The method of claim 14, wherein the display surface displays a map, and wherein determining the location information associated with the location of the mobile robot on the display surface comprises determining a street name marked on the map near the location of the mobile robot, and wherein announcing the location information comprises announcing the street name.

16. The method of claim 11, wherein presenting the location information to the user comprises sending an instruction to the mobile robot to actuate a haptic feedback system.

17. The method of claim 11, wherein the display surface is an electronic display device, and wherein the guidance system is coupled to the electronic display device and configured to cause the electronic display device to display the visual features.

18. The method of claim 17, wherein the guidance system is a tablet computer and the electronic display device is a touchscreen.

19. A method performed by a guidance system comprising one or more computers configured to perform operations comprising:

receiving a location from a mobile robot tracking a visual feature on a display surface;

determining location information associated with the location of the mobile robot on the display surface;

presenting the location information to a user in physical contact with a surface of the mobile robot; and

storing a list of robot identifiers for a plurality of robots on the display surface.

20. The method of claim 19, comprising storing a robot-layer mapping list that specifies which robot corresponds to which layer of a plurality of layers of information displayed on the display surface.