Using physical objects in conjunction with an interactive surface

• US 10,346,529 B2
• Filed: 05/18/2016
• Issued: 07/09/2019
• Est. Priority Date: 09/30/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hartmann, Bjorn U., Wilson, Andrew D., Benko, Hrvoje, Morris, Meredith J.
• Primary Examiner(s): Siddiqui, Md Saiful A
• Application Number:
  US15/157,992
• Publication Number:
  US 20170024370A1
• Time in Patent Office: 1,147 Days
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/171 by use of digital ink

Abstract

An interaction management module (IMM) is described for allowing users to engage an interactive surface in a collaborative environment using various input devices, such as keyboard-type devices and mouse-type devices. The IMM displays digital objects on the interactive surface that are associated with the devices in various ways. The digital objects can include input display interfaces, cursors, soft-key input mechanisms, and so on. Further, the IMM provides a mechanism for establishing a frame of reference for governing the placement of each cursor on the interactive surface. Further, the IMM provides a mechanism for allowing users to make a digital copy of a physical article placed on the interactive surface. The IMM also provides a mechanism which duplicates actions taken on the digital copy with respect to the physical article, and vice versa.

Claims

1. A system comprising:

one or more processing devices; and

one or more computer-readable memory devices or storage devices storing computer-readable instructions which, when executed by the one or more processing devices, cause the one or more processing devices to;

detect a physical object placed on an interactive surface;

display a digital object on the interactive surface, the digital object being a digital copy of the physical object;

detect multiple contact points on the interactive surface, the multiple contact points engaging the digital object displayed on the interactive surface, at least one of the multiple contact points being a cursor contact point simulating a finger contact point, the cursor contact point being controlled by an input device that is operating in a touch mode and is operable in at least one other mode; and

manipulate an orientation, size, or location of the digital object on the interactive surface responsive to the multiple contact points.

2. The system of claim 1, wherein:

the orientation of the digital object is manipulated; and

the computer-readable instructions, when executed by the one or more processing devices, cause the one or more processing devices to rotate the digital object around a fixed pivot defined by one of the multiple contact points.

3. The system of claim 1, further comprising:

an infrared emitter-detector configured to detect the multiple contact points.

4. The system of claim 1, further comprising:

a projector disposed above the interactive surface and configured to project the digital object onto the interactive surface.

5. The system of claim 1, further comprising:

a projector integrated into the interactive surface and configured to project the digital object on the interactive surface.

6. A system comprising:

one or more processing devices; and

one or more computer-readable storage media storing computer-readable instructions which, when executed by the one or more processing devices, cause the one or more processing devices to;

display a digital copy of a physical article placed on an interactive surface;

receive a finger touch contact at a first part of the digital copy displayed on the interactive surface; and

responsive to the finger touch contact, display a digital mark at a second part of the physical article,

wherein a position of the second part on the physical article maps to a corresponding position of the first part on the digital copy.

7. The system of claim 6, wherein the digital mark is a digital highlight.

8. The system of claim 6, wherein the computer-readable instructions further cause the one or more processing devices to:

detect that the finger touch contact has been removed from the digital copy; and

responsive to removal of the finger touch contact from the digital copy, remove the digital mark from the physical article.

9. The system of claim 6, wherein the computer-readable instructions further cause the one or more processing devices to:

detect that the finger touch contact has been removed from the digital copy; and

preserve the digital mark on the physical article after removal of the finger touch contact.

10. The system of claim 6, wherein the computer-readable instructions further cause the one or more processing devices to:

responsive to the finger touch contact, display a matching digital mark at the first part of the digital copy.

11. The system of claim 6, further comprising:

an infrared emitter-detector configured to detect the finger touch contact.

12. The system of claim 6, further comprising:

a projector disposed above the interactive surface and configured to project the digital copy of the physical article onto the interactive surface.

13. The system of claim 6, wherein the computer-readable instructions further cause the one or more processing devices to:

display another digital copy of the physical article that includes the digital mark.

14. The system of claim 6, wherein the interactive surface includes an editing region for creating the digital mark.

15. A method, comprising:

detecting a physical object placed on an interactive surface;

displaying a digital object on the interactive surface, the digital object being a digital copy of the physical object;

detecting multiple contact points on the interactive surface, the multiple contact points engaging the digital object displayed on the interactive surface, at least one of the multiple contact points being a cursor contact point simulating a finger contact point, the cursor contact point being controlled by an input device that is operating in a touch mode and is operable in at least one other mode; and

manipulating an orientation, size, or location of the digital object on the interactive surface responsive to the multiple contact points.

16. The method of claim 15, wherein manipulating the orientation of the digital object comprises:

rotating the digital object around a fixed pivot defined by one of the multiple contact points.

17. The method of claim 15, further comprising:

detecting a movement of the input device when the input device is operating in the at least one other mode; and

in response to detecting the movement, moving a cursor on the interactive surface in correspondence with the movement of the input device.

18. A method, comprising:

displaying a digital copy of a physical article placed on an interactive surface;

receiving a finger touch contact at a first part of the digital copy displayed on the interactive surface; and

responsive to the finger touch contact, displaying a digital mark at a second part of the physical article,

wherein a position of the second part on the physical article maps to a corresponding position of the first part on the digital copy.

19. The method of claim 18, further comprising:

responsive to the finger touch contact, displaying a matching digital mark at the first part of the digital copy.

20. The method of claim 18, further comprising:

displaying another digital copy of the physical article that includes the digital mark.