Kinetic input/output

• US 8,751,049 B2
• Filed: 05/24/2011
• Issued: 06/10/2014
• Est. Priority Date: 05/24/2010
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Linder, Natan, Maes, Patricia
• Primary Examiner(s): Tran, Khoi
• Assistant Examiner(s): PATTON, SPENCER D
• Application Number:
  US13/114,587
• Publication Number:
  US 20110288964A1
• Time in Patent Office: 1,113 Days
• Field of Search: 700/245, 700/246, 700/250, 700/253, 700/257, 700/258, 700/259, 700/264, 345/156, 345/168, 345/170, 345/175, 345/173, 361/679.21, 361/679.22, 361/679.23
• US Class Current: 700/264
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/36169 Remote, host controlled, operated manual data input, keyboard
• CPC Class Codes: G05B 2219/40411 Robot assists human in non-industrial environment like home or office
• CPC Class Codes: G06Q 30/0641 Shopping interfaces

Abstract

In exemplary implementations of this invention, an input/output device (“Bulb”) is attached to on an articulated, actuated robotic arm. The robotic arm can move the Bulb by translating it along three axes and by rotating it about the arm'"'"'s base. In addition, the I/O can rotate about its own vertical axis. The Bulb comprises at least a pico-projector, two cameras, a depth sensor and an onboard computer. The onboard computer controls actuators in the robotic arm and Bulb that move the Bulb. It also processes visual data captured by the cameras in order to recognize objects or events, and to respond to them. This response may include changing the position of the Bulb or changing the parameters of an image projected by the pico-projector.

Claims

1. An apparatus comprising:

(a) an articulated arm with actuators for moving an I/O device that is attached to the articulated arm;

(b) the I/O device, which I/O device comprises (1) a plurality of cameras for capturing visual data, and (2) a projector for projecting a projected multi-pixel image; and

(c) at least one processor for

(i) analyzing the visual data to recognize an object or event, which object or event is neither the projected multi-pixel image nor distortion of the projected multi-pixel image,

(ii) selecting a movement of the articulated arm in response to the object or event that is recognized,

(iii) outputting movement control signals to cause the actuators to effect the movement, and

(iv) also analyzing the visual data to detect distortion of the projected multi-pixel image and outputting-image control signals to correct the distortion;

wherein the plurality of cameras comprises a first camera and a second camera, the first camera is aligned with the projector, and the second camera is configured to swivel to image objects that are not in a line of sight of the first camera.

2. The apparatus of claim 1, wherein the at least one processor is adapted:

to analyze the visual data to recognize a planar uncluttered surface,

to output movement control signals to cause the actuators to move the I/O device nearer to the uncluttered surface, and

to output image control signals to cause the multi-pixel image to be projected on an area that is entirely within the uncluttered surface.

3. The apparatus of claim 1, wherein the robotic arm can move with at least four degrees of freedom.

4. The apparatus of claim 1, wherein the I/O device further comprises a depth sensor for detecting distance from the I/O device to an item in the vicinity of the I/O device, which distance changes as a result of the movement of the articulated arm.

5. The apparatus of claim 1, wherein the image control signals cause the projector to correct for a keystone effect.

6. The apparatus of claim 1, wherein the image control signals dynamically control projection parameters for homographic calibration of a projection surface with a camera view-port.

7. The apparatus of claim 1, wherein the at least one processor is adapted:

to output movement control signals to cause the actuators to alter the distance between the I/O device and a surface on which the multi-pixel image is being or will be projected, and

to output image control signals to digitally zoom the multi-pixel image.

8. The apparatus of claim 1, wherein the at least one processor is further adapted to output the motion control signals or image control signals in response, at least in part, to direct physical manipulation of the apparatus.

9. The apparatus of claim 1, wherein the at least one processor is further adapted:

to output control signals to retrieve information from a network, which information is associated with the object or event that is recognized,

to accept data indicative of the information, and

to output signal control data to cause the multi-pixel image to display content that is based at least in part on the information.

10. The apparatus of claim 1, wherein

the apparatus is adapted to receive and transmit data to and from a network,

the multi-pixel image comprises a projected graphical user interface, and

the at least one processor is adapted to process the visual data to detect when and where a touch interaction occurs, the touch interaction comprising a human user touching a surface on which the projected graphical user interface is projected.

11. The apparatus of claim 10, wherein the at least one processor is also adapted to process data from a microphone or an accelerometer to detect the touch interaction.

12. The apparatus of claim 10, wherein the projected graphical user interface is for displaying an email or other electronic message and for receiving input from a human user to send an email or other electronic message.

13. The apparatus of claim 10, wherein the projected graphical user interface is for displaying information about an item for sale and for receiving input from a human user to purchase that item.

14. The apparatus of claim 1, wherein the image control signals control projection mapping.

15. The apparatus of claim 1, wherein the apparatus is further adapted to transmit at least part of the visual data to at least one separate device for display or storage by that at least one separate device.

16. The apparatus of claim 15, wherein the at least one separate device is at least one other apparatus that meets the description of claim 1.

17. An apparatus comprising:

(a) an articulated arm with actuators for moving an I/O device that is attached to the articulated arm;

(b) the I/O device, which I/0 device comprises at least one sensor for capturing visual data and a projector for projecting a multi-pixel image; and

(c) at least one processor for

(i) analyzing the visual data to recognize a gesture,

(ii) selecting a movement of the articulated arm in response to the gesture,

(iii) outputting movement control signals to cause the actuators to effect the movement for a distance, in a direction or with a stopping point indicated by the gesture, the movement being to a new position, and

(iv) outputting image control signals to cause the projector to project, from the new position, the multi-pixel image.

18. A process comprising:

(a) using actuators to move an articulated arm and thus an I/O device that is attached to the articulated arm, which I/O device comprises a plurality of cameras and a projector;

(b) using the plurality of cameras to capture visual data

(c) using the projector to project a projected multi-pixel image; and

(c) using at least one processor

(i) to analyze the visual data to recognize an object or event, which object or event is neither the projected multi-pixel image nor distortion of the projected multi-pixel image,

(ii) to select a movement of the articulated arm in response to the object or event that is recognized,

(iii) to output movement control signals to cause the actuators to effect the movement,

(iv) to also analyze the visual data to detect distortion of the projected multi-pixel image, and

(v) to output image control signals to correct the distortion;

wherein

(A) the plurality of cameras comprises a first camera and a second camera,

(B) the first camera is aligned with the projector, and

(C) the process further comprises using servo motors to swivel the second camera to image objects that are not in a line of sight of the first camera.

19. The process of claim 18, wherein the object that is recognized is an uncluttered surface, the movement brings the I/O device closer to the uncluttered surface, and the multi-pixel image is projected on an area that falls entirely within the uncluttered surface.

20. The process of claim 18, wherein the process further comprises:

(a) using the processor to analyze the visual data to recognize an object with a first level of confidence;

(b) translating the I/O device to different spatial positions;

(c) using a camera, out of the plurality of cameras, to capture images of the object from different perspectives when the I/O device is in the different spatial positions; and

(d) using the processor to analyze the images to recognize the object with a second level of confidence, the second level of confidence being higher than the first level of confidence.