System and method for 3D projection mapping with robotically controlled objects

• US 10,447,899 B2
• Filed: 10/18/2017
• Issued: 10/15/2019
• Est. Priority Date: 11/16/2011
• Status: Active Grant

• Current Assignee: X Development LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Linnell, Jeff
• Primary Examiner(s): Abaza, Ayman A
• Application Number:
  US15/787,656
• Publication Number:
  US 20180041667A1
• Time in Patent Office: 727 Days
• Field of Search: 348 51, 348 61
• CPC Class Codes: B25J 11/00 Manipulators not otherwise provided for
• CPC Class Codes: B25J 9/1633 compliant, force, torque control, e.g. combined with position control
• CPC Class Codes: G05B 2219/36429 Power assisted positioning
• CPC Class Codes: G05B 2219/39342 Adaptive impedance control
• CPC Class Codes: G05B 2219/39346 Workspace impedance control
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

A system for motion control is presented. In one embodiment, a motion control 3D projection system includes a projector; and a projection surface coupled to a robotic arm, where the robotic arm moves the projection surface through a set of spatial coordinates, and a 3D projection from the projector is projected onto a set of coordinates of the projection surface and matches the 3D projection to the set of coordinates of the projection surface as the projection surface moves through the set of spatial coordinates. In additional embodiments, a master control system may integrate additional robotic arms and other devices to create a motion control scene with a master timeline.

Claims

1. A computer-implemented method comprising:

determining, by a computing device, a first pose of a camera that is located at a first location in a set of spatial coordinates, wherein the camera is moveable through at least a portion of the set of spatial coordinates independently from a projector;

when the camera has the first pose at a first location in the set of special coordinates, causing the projector to render graphic content such that a three-dimensional effect of a projection of the graphic content on a projection surface corresponds to a first perspective of the projection surface from the first pose of the camera;

determining, by the computing device, a movement of the camera to a second pose at a second location in the set of spatial coordinates, wherein the movement to the second pose changes at least one of orientation or position of the camera with respect to the projector;

determining, by the computing device, a second perspective of the projection surface from the camera, wherein the second perspective corresponds to the second pose of the camera at the second location; and

updating, by the computing device, the graphic content so as to maintain a three-dimensional effect of the projection of the graphic content on the projection surface as perspective of the projection surface from the camera changes from the first perspective to the second perspective.

2. The method of claim 1, wherein the camera is coupled to a robotic arm, and wherein causing the movement of the camera to the second pose at the second location in the set of spatial coordinates comprises operating the robotic arm to move the camera to the second location.

3. The method of claim 1, further comprising:

in order to maintain the three-dimensional effect of the projection of the graphic content on the projection surface, causing a movement of the projector in conjunction with the updating of the graphic content.

4. The method of claim 3, wherein the projector is coupled to a robotic arm, and wherein causing the movement of the projector comprises operating the robotic arm to move the projector within the set of spatial coordinates.

5. The method of claim 1, wherein the projection surface is coupled to a robotic arm, the method further comprising:

causing a movement of the projection surface comprises operating the robotic arm to move the projector within the set of spatial coordinates, such that perspective of the projection surface from the camera changes; and

updating the graphic content so as to maintain a three-dimensional effect of the projection of the graphic content on the projection surface as perspective of the projection surface from the camera changes due to movement of the projection surface.

6. A system comprising:

at least one projector located in an environment;

a projection surface located in the environment and coupled to a first device actor, wherein the first device actor is configured to move the projection surface through a set of spatial coordinates in the environment;

a camera located in the environment and coupled to a second device actor, wherein the camera is configured to capture image data of the environment or portions thereof as the projection surface moves through the set of spatial coordinates in the environment, and wherein the second device actor is configured to move the camera independently from the projector, through at least a portion of the set of spatial coordinates; and

a control system configured to;

(i) when the camera has a first pose at a first location in the set of spatial coordinates, cause the projector to render graphic content such that a three-dimensional effect of the projection of the graphic content on the projection surface corresponds to a first perspective of the projection surface from the first pose of the camera;

(ii) cause a movement of the camera to a second location in the set of spatial coordinates, which changes the position of the camera with respect to the projector;

(iii) determine a second perspective of the projection surface from the camera, wherein the second perspective corresponds to a second pose of the camera at the second location; and

(iv) coordinate an update of the graphic content so as to maintain the three-dimensional effect of the projection of the graphic content on the projection surface as perspective of the projection surface from the camera changes from the first perspective to the second perspective of the projection surface from the camera.

7. A computer-implemented method comprising:

determining, by a computing device, a first pose of a camera with respect to a projection surface that is located at a first location in a set of spatial coordinates, wherein the camera is moveable through at least a portion of the set of spatial coordinates independently from a projector;

when the camera has the first pose with respect to the projection surface, causing a projector to render graphic content such that a three-dimensional effect of a projection of the graphic content on the projection surface corresponds to a first perspective of the projection surface from the first pose of the camera;

causing a movement of the projection surface a second location in the set of spatial coordinates, which changes the position of the camera with respect to the projector;

determining, by the computing device, a second perspective of the projection surface from the camera, when the projection surface is at the second location; and

updating, by the computing device, the graphic content so as to maintain a three-dimensional effect of the projection of the graphic content on the projection surface as perspective of the projection surface from the camera changes due to the movement of the projection surface.

8. The method of claim 7, wherein the projection surface is coupled to a robotic arm, and wherein causing the movement of the projection surface comprises operating the robotic arm to move the camera to the second location.

9. A method comprising:

determining, by a computing system, a first pose of a device with respect to a projection surface, wherein the projection surface is coupled to a first device actor such that the projection surface is movable through a set of spatial coordinates, and wherein the device is movable through at least a portion of the set of spatial coordinates independently from thea projector;

when the device has a first pose at a first location in the set of special coordinates, causing the projector to render graphic content such that a three-dimensional effect of the projection of the graphic content on the projection surface corresponds to a first perspective of the projection surface from the first pose of the device;

determining, by the computing system, a movement of the device to a second pose, which changes at least one of orientation or position of the device with respect to the projector;

determining, by the computing system, a second perspective of the projection surface from the device, wherein the second perspective corresponds to the second pose of the device; and

updating, by the computing device, the graphic content so as to maintain a three-dimensional effect of the projection of the graphic content on the projection surface as perspective of the projection surface from the device changes from the first perspective to the second perspective.

10. The method of claim 9, further comprising, the control system computing system periodically:

re-determining pose of the device;

re-determining, based at least in part on the re-determined pose, the perspective of the projection surface that is associated with the device; and

initiating a process to render the graphic content in accordance with the re-determined perspective of the projection surface.

11. The method of claim 9, wherein the projection surface is a first projection surface;

the method further comprising the computing system simultaneously performing the method of claim 9 to determine a respective perspective of each of two or more projection surfaces from the device, and to render respective graphic content for projection onto each of the two or more projection surfaces in accordance with the respectively associated perspective, wherein the two or more projection surfaces comprise the first projection surface and at least one other projection surface.

12. A system comprising:

a projection surface located in an environment and coupled to a first device actor, wherein the first device actor is configured to move the projection surface through a set of spatial coordinates in the environment;

a projector located in the environment;

a device located in the environment and moveable independently from the projector through at least a portion of the set of spatial coordinates, wherein a pose of the device defines a perspective of the projection surface from the device; and

a control system configured to;

when the device has a first pose at a first location in the set of special coordinates, cause the projector to render graphic content such that a three-dimensional effect of the projection of the graphic content on the projection surface corresponds to a first perspective of the projection surface from the first pose of the device;

determine a movement of the device to a second pose, which changes at least one of orientation or position of the device with respect to the projector;

determine a second perspective of the projection surface from the device, wherein the second perspective corresponds to the second pose of the device; and

coordinate an update of the graphic content so as to maintain the three-dimensional effect of the projection of the graphic content on the projection surface as perspective of the projection surface from the device changes from the first perspective to the second perspective.

13. The system of claim 12, wherein the device comprises a camera configured to capture image data of the at least one projection surface as the projection surface moves through the set of spatial coordinates, wherein the camera is coupled to a second device actor that is configured to move the camera through the set of spatial coordinates.

14. The system of claim 13, wherein a perspective of the projection surface from the camera changes according to movement of at least one of the camera and the projection surface, and wherein the control system graphic content is rendered such that the projection of the graphic content changes over time such that the graphic content is directed to the perspective from the camera as the camera moves and/or the projection surface moves.

15. The system of claim 13, wherein the pose of the camera at a given time comprises coordinates of the camera, within the set of spatial coordinates, at the given time.

16. The system of claim 13, wherein the pose of the camera at a given time comprises an orientation of the camera at the given time.

17. The system of claim 12, wherein the control system is further configured to periodically:

determine a current perspective of the projection surface, which is associated with the device; and

cause the projector to update the rendered graphic content in accordance with the current perspective of the projection surface.

18. The system of claim 12, wherein the device comprises a wearable computing device, and wherein the perspective of the projection surface that is associated with the device comprises a perspective of a wearer of the wearable computing device.

19. The system of claim 18, wherein the wearable computing device is a head-mountable device (HMD).

20. The system of claim 18, wherein the wearable computing device is a mobile phone.