Using physical objects in conjunction with an interactive surface

• US 9,372,552 B2
• Filed: 03/27/2013
• Issued: 06/21/2016
• Est. Priority Date: 09/30/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hartmann, Bjrn U., Wilson, Andrew D., Benko, Hrvoje, Morris, Meredith J.
• Primary Examiner(s): Siddiqui, Md Saiful A
• Application Number:
  US13/851,797
• Publication Number:
  US 20130229353A1
• Time in Patent Office: 1,182 Days
• Field of Search: 345/163, 345/168, 345173-179, 345/650, 345/676, 345156-158, 715/700, 715/750, 715/757, 715/764, 715/807, 463 37- 38, 178 1801- 1809
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/171 by use of digital ink

Abstract

An interaction management module (IMM) is described for allowing users to engage an interactive surface in a collaborative environment using various input devices, such as keyboard-type devices and mouse-type devices. The IMM displays digital objects on the interactive surface that are associated with the devices in various ways. The digital objects can include input display interfaces, cursors, soft-key input mechanisms, and so on. Further, the IMM provides a mechanism for establishing a frame of reference for governing the placement of each cursor on the interactive surface. Further, the IMM provides a mechanism for allowing users to make a digital copy of a physical article placed on the interactive surface. The IMM also provides a mechanism which duplicates actions taken on the digital copy with respect to the physical article, and vice versa.

Claims

1. A system comprising:

one or more processing devices; and

one or more computer-readable memory devices or storage devices storing computer-readable instructions which, when executed by the one or more processing devices, cause the one or more processing devices to;

detect placement of a physical input device on an interactive surface;

determine an absolute position of the physical input device and an orientation of the physical input device;

define a frame of reference based on the absolute position and the orientation of the physical input device;

obtain a relative offset measurement from the physical input device; and

display a cursor on the interactive surface relative to the frame of reference that has been defined, wherein the cursor is placed based on both the absolute position and the relative offset measurement obtained from the physical input device.

2. The system of claim 1, wherein the computer-readable instructions cause the one or more processing devices to:

identify a reference object on the interactive surface and define the frame of reference based on that reference object.

3. The system of claim 1, wherein the computer-readable instructions cause the one or more processing devices to:

define the frame of reference relative to the orientation of the physical input device.

4. The system of claim 1, wherein the computer-readable instructions cause the one or more processing devices to:

display a visual attribute which associates the physical input device and the cursor.

5. The system of claim 1, wherein the computer-readable instructions cause the one or more processing devices to:

detect that a control provided by the physical input device has been activated; and

operate the physical input device in a touch mode, wherein, in the touch mode, the cursor serves as a simulated contact point that engages the interactive surface.

6. The system of claim 1, wherein the computer-readable instructions cause the one or more processing devices to:

identify a stationary feature of the interactive surface and define the frame of reference based on the stationary feature of the interactive surface.

7. The system of claim 1, further comprising:

a global position sensing mechanism configured to determine the absolute position of the physical input device.

8. A system comprising:

one or more processing devices; and

one or more computer-readable memory devices or storage devices storing computer-readable instructions which, when executed by the one or more processing devices, cause the one or more processing devices to;

determine whether a touch mode of a mouse-type device has been activated, the mouse-type device having;

the touch mode where the mouse-type device is used as a simulated finger contact, and

another mode where the mouse-type device provides mouse functionality including moving a cursor; and

when the touch mode is activated, manipulate an image displayed on an interactive surface that is separate from the mouse-type device in accordance with movement of the mouse-type device.

9. The system of claim 8, wherein the mouse-type device comprises a mouse button to activate the touch mode.

10. The system of claim 9, wherein the computer-readable instructions cause the one or more processing devices to:

process the movement of the mouse-type device in the touch mode when the mouse button is depressed, and

process the movement of the mouse-type device in the another mode when the mouse button is not depressed.

11. The system of claim 9, wherein the computer-readable instructions cause the one or more processing devices to:

when the touch mode has been activated by depressing the mouse button, process the movement of the mouse-type device as a finger drag on the interactive surface that is separate from the mouse-type device.

12. The system of claim 8, wherein the image is a digital object that is manipulated by the movement of the mouse-type device by moving the digital object, changing a size of the digital object, or rotating the digital object on the interactive surface that is separate from the mouse-type device.

13. The system of claim 8, wherein the computer-readable instructions cause the one or more processing devices to:

when the touch mode is activated, process input received from the mouse-type device as a contact point on the interactive surface that is separate from the mouse-type device.

14. The system of claim 13, wherein the computer-readable instructions cause the one or more processing devices to:

when the touch mode is activated, process other input received from a finger of a user as another contact point on the interactive surface that is separate from the mouse-type device.

15. A system comprising:

one or more processing devices; and

one or more computer-readable memory devices or storage devices storing computer-readable instructions which, when executed by the one or more processing devices, cause the one or more processing devices to;

detect that multiple contact points are concurrently engaging a digital object, the multiple contact points concurrently engaging the digital object including a cursor contact point and a finger contact point, wherein the cursor contact point corresponds to input received from an input device that controls a cursor and the finger contact point corresponds to a finger; and

manipulate the digital object in accordance with the multiple contact points.

16. The system of claim 15, wherein the input device that controls the cursor is a mouse-type device.

17. The system of claim 16, wherein the digital object is manipulated by rotating the digital object in accordance with movement of the cursor contact point and movement of the finger contact point.

18. The system of claim 16, wherein the digital object is manipulated by changing a size of the digital object in accordance with movement of the cursor contact point and movement of the finger contact point.

19. The system of claim 18, further comprising the input device.

20. The system of claim 19, further comprising an interactive surface configured to display the digital object.