Using physical objects in conjunction with an interactive surface

• US 8,427,424 B2
• Filed: 09/30/2008
• Issued: 04/23/2013
• Est. Priority Date: 09/30/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D., Benko, Hrvoje, Morris, Meredith J., Hartmann, Bjrn U.
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): SIDDIQUI, MD SAIFUL A
• Application Number:
  US12/241,281
• Publication Number:
  US 20100079369A1
• Time in Patent Office: 1,666 Days
• Field of Search: 345173-179, 345/156, 345/168, 345/650, 345/676, 345/163, 345/172, 178 1801- 1809, 463 37- 38, 715/700, 715/750, 715/757, 715/764
• US Class Current: 345/156
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/171 by use of digital ink

Abstract

An interaction management module (IMM) is described for allowing users to engage an interactive surface in a collaborative environment using various input devices, such as keyboard-type devices and mouse-type devices. The IMM displays digital objects on the interactive surface that are associated with the devices in various ways. The digital objects can include input display interfaces, cursors, soft-key input mechanisms, and so on. Further, the IMM provides a mechanism for establishing a frame of reference for governing the placement of each cursor on the interactive surface. Further, the IMM provides a mechanism for allowing users to make a digital copy of a physical article placed on the interactive surface. The IMM also provides a mechanism which duplicates actions taken on the digital copy with respect to the physical article, and vice versa.

Claims

1. A method for engaging an interactive surface, the method comprising:

detecting placement of an input device at a selected location and orientation on the interactive surface, the input device configured to provide alphanumeric text entry via keystrokes;

displaying a digital object on the interactive surface in positional association with the input device such that the digital object and the input device are at different locations on the interactive surface; and

displaying a first visual attribute with the digital object and a second visual attribute with the input device, wherein the first visual attribute and the second visual attribute convey an affiliation between the digital object and the input device,

wherein the digital object provides a display interface that presents a visual representation of information entered when a user enters the keystrokes using the input device, and

wherein at least the detecting is performed by one or more processing devices.

2. The method of claim 1, wherein:

the first visual attribute comprises a first border around the digital object and the second visual attribute comprises a second border around the input device, and

the first border around the digital object and the second border around the input device share a common color to convey the affiliation between the digital object and the input device.

3. The method of claim 1, wherein the input device comprises a first input device, the method further comprising:

detecting placement of a second input device on the interactive surface in proximity to the first input device; and

in response to the detecting, automatically associating the second input device with the digital object, such that both the first input device and the second input device are associated with the digital object.

4. The method of claim 1, wherein the input device comprises a first input device and the digital object comprises a first digital object, the method further comprising:

detecting placement of a second input device on the interactive surface, the interactive surface displaying a second digital object in positional association with the second input device;

detecting movement of the second input device in proximity to the first input device or of the first input device in proximity to the second input device; and

in response to detecting the movement, merging content provided by the first digital object with content provided by the second digital object.

5. The method of claim 1, wherein the displaying of the digital object comprises:

placing the digital object on the interactive surface at a location which;

a) satisfies at least one placement constraint to an extent deemed appropriate; and

b) reduces interference between the digital object and other existing objects on the interactive surface.

6. The method of claim 5, wherein the location at which the digital object is placed is determined using an integral image technique.

7. A system comprising:

an interactive management module configured to manage an interactive surface, the interactive management module comprising;

logic configured to detect placement of a physical article at a first location on the interactive surface;

logic configured to display an interface object on the interactive surface for use in association with the physical article, the interface object surrounding the physical article;

logic configured to detect when a user identifies the physical article for copying by touching the interface object surrounding the physical article with a finger and dragging the finger away from the physical article; and

logic configured to, responsive to the user dragging the finger away from the physical article, generate a digital image of the physical article and display the digital image of the physical article on the interactive surface at a second location identified by the user, wherein the second location of the digital image of the physical article on the interactive surface is different than the first location of the physical article on the interactive surface; and

one or more processing devices configured to execute the interactive management module,

wherein the logic configured to display the interface object is configured to display the interface object responsive to the placement of the physical article on the interactive surface and before the user identifies the physical article for copying.

8. The system of claim 7, wherein the second location where the digital image is displayed is where the user lifts the finger from the interactive surface.

9. The system of claim 7, the interactive management module further comprising:

logic configured to detect a selection by the user of a particular digital part of the digital image of the physical article; and

logic configured to highlight or mark a physical part of the physical article that is a counterpart to the particular digital part of the digital image of the physical article.

10. The system of claim 7, the interactive management module further comprising:

logic configured to detect that the user has moved the digital image of the physical article to an editing region of the interactive surface;

logic configured to detect a physical mark applied by the user to the digital image of the physical article within the editing region; and

logic configured to generate another digital image of the physical article, the another digital image including a digital representation of the physical mark.

11. The system of claim 7, wherein the logic configured to generate the digital image is configured to crop a larger image of the interactive surface to obtain the digital image of the physical article.

12. The system of claim 11, wherein the larger image is an image of the entire interactive surface.

13. The system of claim 7, wherein the interface object is displayed as a halo or border around the physical article.

14. The system according to claim 7, wherein the physical article comprises a device.

15. A method for managing an interactive surface, the method comprising:

detecting placement of a physical article at a first location on the interactive surface;

displaying an interface object on the interactive surface for use in association with the physical article, the interface object being displayed proximate to the physical article;

detecting when a user identifies the physical article for copying by contacting the interface object that is proximate to the physical article with a contact point and moving the contact point away from the physical article;

obtaining, a digital image of the physical article, the digital image of the physical article being obtained responsive to detecting the moving of the contact point away from the physical article; and

displaying the digital image of the physical article on the interactive surface at a second location, wherein the second location of the digital image of the physical article on the interactive surface is different than the first location of the physical article on the interactive surface,

wherein at least the obtaining is performed by one or more processing devices, and

wherein the interface object is displayed responsive to the placement of the article on the interactive surface and is displayed before the user identifies the physical article for copying.

16. The method according to claim 15, wherein the contact point is a finger contact point, the method further comprising:

distinguishing the physical article from the finger contact point using contour analysis.

17. The method according to claim 15, wherein the physical article is not an input device.

18. The method according to claim 15, further comprising:

determining that the physical article is not an input device by determining that the physical article lacks a marking that identifies input devices.

19. The method according to claim 15, wherein the interface object comprises a halo or border formed around the physical article.

20. The method according to claim 15, wherein the contact point is a cursor contact point.

21. The method according to claim 15, wherein the contact point is a finger contact point by a user'"'"'s finger on the interactive surface, the method further comprising:

capturing the digital image of the physical article responsive to the user'"'"'s finger being removed from the interactive surface.