Visual communication using a robotic device

• US 9,578,076 B2
• Filed: 05/02/2011
• Issued: 02/21/2017
• Est. Priority Date: 05/02/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Olivier, Charles, Fouillade, Jean Sebastien, Crow, William M., Burianek, Francois, Sanchez, Russ
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): MESSMORE, JONATHAN R
• Application Number:
  US13/098,492
• Publication Number:
  US 20120281092A1
• Time in Patent Office: 2,122 Days
• Field of Search: 348/51, 348/61, 348/148, 348/207.99, 348/211.99, 348/211.11, 348/211.12, 348/211.13
• US Class Current: 1/1
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

Technology is described for visually communicating using a robotic device. An example of a method can include a video feed sent from the video camera of the robotic device to the remote user. A projection surface identified in the video feed can then be sent to the remote user using an application. Another operation can be obtaining an image from the remote user using the application. The image created by the remote user can then be projected on the projection surface.

Claims

1. A method implemented by one or more computer processing devices, the method comprising:

receiving, at a first location, a video feed from a video camera of a robotic device that is at a second location;

automatically selecting a drawing projection surface at the second location from multiple possible projection areas that are visible in the video feed, the automatically selecting comprising using edge detection to detect a contiguous bordered area that is suitable as the drawing projection surface;

obtaining a drawing by a first user that is at the first location;

instructing the robotic device to project the drawing at the second location;

receiving a user instruction from the first user, the user instruction being relative to the contiguous bordered area;

applying a drawing function to the contiguous bordered area based at least on the user instruction, the drawing function being applied by instructing the robotic device to project the drawing function into the contiguous bordered area;

receiving an updated video feed from the video camera of the robotic device, wherein the updated video feed shows the drawing function applied to the contiguous bordered area and the projected drawing by the first user; and

displaying the updated video feed at the first location.

2. The method as in claim 1, further comprising:

receiving adjustments to the automatically selected drawing projection surface from the first user; and

sending alignment instructions for aligning the video camera to the robotic device, wherein the alignment instructions are based at least on the adjustments.

3. The method as in claim 1, further comprising:

receiving input from a second user at the second location, the input representing interaction of the second user with the projected drawing on the drawing projection surface; and

causing the robotic device to modify the projected drawing based at least on the input received from the second user.

4. The method of claim 1, further comprising:

identifying a shape of the drawing projection surface using data obtained from a depth camera of the robotic device; and

modifying the projected drawing to accommodate the shape of the drawing projection surface.

5. The method as in claim 1, wherein the drawing function comprises auto-filling of the contiguous bordered area with a particular color.

7. The mobile robotic device as in claim 2, wherein the computer readable instructions further cause the processor to:

communicate an updated video feed showing the drawing function performed in the contiguous bordered area back to the remote device.

8. The mobile robotic device as in claim 2, wherein the drawing function comprises auto-filling the contiguous bordered area with a particular color.

9. The mobile robotic device as in claim 2, wherein the computer readable instructions further cause the processor to:

receive a freehand drawing from the remote device;

apply a graphical transformation to the freehand drawing to obtain a transformed freehand drawing; and

cause the transformed freehand drawing to be projected as light ink onto a selected projection surface, wherein the graphical transformation conforms the light ink to a shape of the selected projection surface.

10. The mobile robotic device as in claim 9, wherein the computer readable instructions further cause the processor to:

use edge detection on aspects of the selected projection surface to assist the remote user in creating the freehand drawing.

6. A mobile robotic device, comprising:

a video camera;

a depth camera;

a projector;

a processor; and

memory storing computer readable instructions which, when executed by the processor, cause the processor to;

send a video feed from the video camera to a remote device that displays the video feed to a remote user;

cause the mobile robotic device to navigate to a location based at least on coordinates received from the remote device;

cause the mobile robotic device to set a point of view of the video camera;

use the depth camera to identify a contiguous bordered area at the location;

receive, from the remote device, an instruction to perform a drawing function within the contiguous bordered area; and

responsive to the instruction, use the projector to perform the drawing function by projecting light into the contiguous bordered area.

11. A system comprising:

a processing device; and

a hardware memory device storing computer readable instructions which, when executed by the processing device, cause the processing device to;

obtain a video feed from a video camera of a robotic device;

use edge detection to select a contiguous bordered area shown in the video feed as a projection surface from multiple possible projection areas at a location of the robotic device;

obtain a drawing instruction from a remote user at another location;

send the drawing instruction received from the remote user to the robotic device, wherein the drawing instruction is performed by projecting light onto the selected projection surface by a projector of the robotic device;

obtain a continued video feed from the video camera of the robotic device, wherein the continued video feed shows the projected light in the contiguous bordered area; and

display the continued video feed.

12. The system as in claim 11, wherein the computer readable instructions further cause the processing device to:

instruct the robotic device to set a field of view of the video camera, wherein the contiguous bordered area is within the field of view.

13. The system as in claim 11, embodied as a mobile computing device.

14. The system as in claim 11, embodied as a gaming console.

15. The system as in claim 11, wherein the drawing instruction instructs the robotic device to auto-fill the contiguous bordered area with a particular color.

16. The system as in claim 11, wherein the computer readable instructions further cause the processing device to:

use input received from a depth camera of the robotic device to identify a shape of the selected projection surface; and

cause the robotic device to conform the projected light to the shape of the selected projection surface.

17. The system as in claim 11, wherein the computer readable instructions further cause the processing device to:

present a plurality of drawing tools to the remote user;

receive an input from the remote user identifying a selected drawing tool;

receive a freehand drawing created by the remote user with the selected drawing tool; and

instruct the robotic device to project the freehand drawing.

18. The system as in claim 17, wherein the selected drawing tool is a paint brush.

19. The system as in claim 18, wherein the computer readable instructions further cause the processing device to:

receive another input from the remote user identifying a selected color for the freehand drawing painted with the paint brush; and

instruct the robotic device to use the selected color for the freehand drawing.

20. The system as in claim 17, wherein the computer readable instructions further cause the processing device to:

instruct the robotic device to conform the freehand drawing to the contiguous bordered area.