Combined Surface User Interface

• US 20110191690A1
• Filed: 02/03/2010
• Published: 08/04/2011
• Est. Priority Date: 02/03/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhao, Chunshui, Cai, Xiong-Fei, Zhao, Ji, Zhang, Chunhui, Wang, Min, Gao, Rui
• Granted Patent: US 9,110,495 B2
• Time in Patent Office: Days
• US Class Current: 715/746
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 9/31 Projection devices for colour picture display , e.g. using electronic spatial light modulators [ESLM]
• CPC Class Codes: H04N 9/3147 Multi-projection systems displays in general H04N9/12; video walls G06F3/1446, G09G2300/026

Abstract

Techniques for utilizing two or more mobile devices equipped with projectors to generate a combined seamless user interfaces by stitching projection areas generated by the projectors.

Claims

1. A mobile computing device comprising:

a projector that displays a user interface on a surface external to the mobile computing device, the user interface occupying a projection area on the surface;

an illuminator that illuminates the projection area with non-visible light;

a camera that monitors the non-visible illumination of the projection area to detect touch-type interaction with the displayed user interface and to detect portions of the projection area that overlap with projection areas of one or more neighboring computing devices;

a communication system allowing the mobile computing device to communicate with the one or more neighboring computing devices;

collaboration logic that interacts with the one or more neighboring computing devices through the communication system to (a) graphically stitch the projection area of the mobile computing device with projection areas of the one or more neighboring computing devices based on the detection of overlapping portions of the projection area and (b) create a combined seamless user interface utilizing the projection area of the mobile computing device and the projection areas of the one or more neighboring computing devices; and

wherein the combined seamless user interface allows graphical representations to span the projection areas of the mobile computing device and the one or more neighboring computing devices in response to the touch-type interaction with the combined seamless user interface.

2. A mobile computing device as recited in claim 1, wherein the combined seamless user interface allows the graphical representations to move seamlessly between the projection areas of the mobile computing device and the one or more neighboring computing devices in response to the touch-type interaction with the combined seamless user interface.

3. A mobile computing device as recited in claim 1, wherein the collaboration logic is configured to graphically stitch the projection areas in response to physical proximity of the one or more neighboring computing devices with the mobile computing device.

4. A mobile computing device as recited in claim 1, wherein:

the combined seamless user interface has a home area corresponding to the mobile computing device and one or more neighboring areas corresponding to the neighboring computing devices, respectively;

the collaboration logic visually represents computing device resources of the respective computing devices on their corresponding user interface areas; and

in response to a user moving a visual representation of a particular device resource from the user interface area corresponding to a first of the computing devices to the user interface area corresponding to a second of the computing devices, the collaboration logic copies the particular device resource from the first of the computing devices to the second of the computing devices.

5. A mobile computing device as recited in claim 1, wherein:

the camera is configured to capture identifying information projected by the one or more neighboring computing devices on their projection areas; and

the collaboration logic is configured to recognize the captured identifying information and in response to communicate with the one or more neighboring computing devices and to include the projection areas of the one or more neighboring computing devices in the combined user interface.

6. A mobile computing device as recited in claim 1, wherein the projection areas of the neighboring computing devices expand the user interface of the mobile computing device.

7. A method of collaborating between first and second mobile computing devices having user interfaces that are projected in respective projection areas on a common projection surface, comprising:

optically detecting an overlapping portion of the projection area of the first mobile computing device that overlaps with the projection area of the second mobile computing device;

clipping at least part of the detected overlapping portion from the projection area of the first mobile computing device to leave a remaining projection area of the first mobile computing device; and

graphically stitching the remaining projection area of the first mobile computing device with the projection area of the second mobile computing device to create a seamless user interface that includes at least portions of the projection areas of the first and second mobile computing devices.

8. A method as recited in claim 7, further comprising:

capturing identifying information projected the second mobile computing device;

communicating with the second mobile computing device based on the captured identifying information.

9. A method as recited in claim 7, further comprising:

communicating with the second mobile computing device to coordinate clipping and stitching of the projection areas of the first and second mobile computing devices.

10. A method as recited in claim 7, wherein the first and second mobile computing devices illuminate their projection areas with infrared light to detect user interaction, and wherein optically detecting the overlapping portion comprises detecting overlapping areas of infrared light illumination.

11. A method as recited in claim 7, wherein optically detecting the overlapping portion comprises sensing brightness the projection area of the first mobile computing device.

12. A method as recited in claim 7, further comprising:

illuminating the projection area of the first mobile computing device with infrared light; and

monitoring an infrared camera to detect user interaction with the projected user interface of the first computing device.

13. A method as recited in claim 7, further comprising:

optically monitoring the projected user interface of the first computing device to detect user interaction and to detect the overlapping portion.

14. A mobile computing device comprising:

a projector that displays a user interface on a surface external to the mobile computing device, the user interface occupying a projection area on the surface;

an input sensor that senses user interaction with the displayed user interface;

a communication system allowing the mobile computing device to communicate with one or more neighboring computing devices; and

collaboration logic that stitches the projection area of the mobile computing device with projection areas of the one or more neighboring computing devices to create a combined user interface utilizing the projection area of the mobile computing device and the projection areas of the one or more neighboring computing devices.

15. A mobile computing device as recited in claim 14, wherein the collaboration logic is configured to stitch the projection areas in response to physical proximity of the one or more neighboring computing devices with the mobile computing device.

16. A mobile computing device as recited in claim 14, wherein:

the combined user interface has a home area corresponding to the mobile computing device and one or more neighboring areas corresponding to the neighboring computing devices, respectively;

the collaboration logic visually represents computing device resources of the respective computing devices on their corresponding user interface areas; and

in response to a user moving a visual representation of a particular device resource from the user interface area corresponding to a first of the computing devices to the user interface area corresponding to a second of the computing devices, the collaboration logic copies the particular device resource from the first of the computing devices to the second of the computing devices.

17. A mobile computing device as recited in claim 14, wherein:

the input sensor comprises a camera configured to capture identifying information projected by the one or more neighboring computing devices on their projection areas; and

the collaboration logic is configured to recognize the captured identifying information and in response to communicate with the one or more neighboring computing devices and to include the projection areas of the one or more neighboring computing devices in the combined user interface.

18. A mobile computing device as recited in claim 14, wherein the projection areas of the neighboring computing devices expand the user interface of the mobile computing device.

19. A mobile computing device as recited in claim 14, wherein:

the input sensor comprises a camera configured to detect physical interaction by a user; and

the collaboration logic is responsive to the camera to detect overlapping projection areas and to stitch the projection area of the mobile computing device with the projection areas of the one or more neighboring computing devices based on the detection of the overlapping projection areas.

20. A mobile computing device as recited in claim 14, further comprising:

an illuminator that illuminates the projection area of the mobile computing device with infrared light; and

the input sensor comprising a camera that monitors the infrared illumination of the projection area of the mobile computing device to detect physical interaction by a user.