Interactive input system having a 3D input space

• US 9,489,040 B2
• Filed: 07/19/2011
• Issued: 11/08/2016
• Est. Priority Date: 07/19/2010
• Status: Active Grant

• Current Assignee: Smart Technologies ULC (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Smart Technologies ULC (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Antonyuk, Viktor, Xin, Min, Lai, Henry, Tse, Edward, Hudson, Carl, Wu, Hui Xiu, Chen, Jingwen
• Primary Examiner(s): MA, CALVIN
• Application Number:
  US13/810,850
• Publication Number:
  US 20130215148A1
• Time in Patent Office: 1,939 Days
• Field of Search: 345/156, 345/158, 345/165, 345/204, 348/333.01, 343/395
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

An interactive input system comprises computing structure; and an input device detecting at least one physical object carrying a recognizable pattern within a three-dimensional (3D) input space and providing output to the computing structure, wherein the computing structure processes the output of the input device to: recognize the pattern carried by the at least one physical object in the 3D input space; and modify an image presented on a display surface by applying a transition to digital content associated with the at least one physical object based on a detected state of the at least one physical object.

Claims

1. An interactive input system, comprising:

a computing structure; and

an input device configured to detect at least one physical marker having a recognizable pattern thereon within a three-dimensional (3D) input space and to provide output to said computing structure, wherein said computing structure is configured to process the output of the input device to;

visually recognize the pattern on the at least one physical marker in the 3D input space partially based on reference images in a pattern library;

assign a tracking state to the at least one physical marker indicating a level of detection by the input device of the at least one physical marker in the 3D input space;

determine digital content associated with the recognized pattern; and

modify an image presented on a display surface by applying a transition to the determined digital content based on the assigned tracking state in response to movement of the at least one physical marker within the 3D input space.

2. The interactive input system of claim 1, wherein the computing structure is configured to process the output of the input device to detect at least one of orientation, position and movement of the at least one physical marker in the 3D input space.

3. The interactive input system of claim 1, wherein when the detected state of the at least one physical marker signifies that a new recognizable physical marker is positioned in the 3D input space, the image is modified to add associated digital content to the image, the added associated digital content appearing in the image using a visual effect.

4. The interactive input system of claim 3, wherein the visual effect is a fade-in visual effect or a scale-up visual effect.

5. The interactive input system of claim 3, wherein when the detected state of the at least one physical marker signifies a lost physical marker in the 3D input space whose position has been relocated, the image is modified to add associated digital content to the image using the visual effect.

6. The interactive input system of claim 1, wherein when the detected state of the at least one physical marker signifies removal of a previously recognized physical marker from the 3D input space, the image is modified to remove associated digital content from the image, the associated digital content disappearing from the image using a different visual effect.

7. The interactive input system of claim 6, wherein the different visual effect is a fade-out visual effect or a scale-down visual effect.

8. The interactive input system of claim 6, wherein when the detected state of the at least one physical marker signifies a physical marker in the 3D input space whose tracked position has been lost, the image is modified to remove the associated digital content from the image using the different visual effect.

9. The interactive input system of claim 1, wherein the computing structure further process the output of the input device to detect physical marker gestures.

10. The interactive input system of claim 9, wherein said computing structure is configured to manipulate digital content associated with the physical marker in response to detected gestures.

11. The interactive input system of claim 10, wherein said computing structure, in response to a detected gesture, is configured to one of (i) changes associated digital content, (ii) merges associated digital content, (iii)clone associated digital content, (iv) disassociate digital content, (v) associate digital content, and (vi) rotates digital content.

12. The interactive input system of claim 1, wherein the at least one physical marker is a card, cube or other physical object having a least one surface carrying said pattern thereon.

13. The interactive input system of claim 12, wherein the at least one physical marker comprises more than one surface, at least some of said surfaces carrying different patterns thereon.

14. The interactive input system of claim 12, wherein said at least one physical marker comprises at least one display surface on which a pattern is presented.

15. The interactive input system of claim 1, wherein said input device is a camera device capturing images of said 3D input space.

16. The interactive input system of claim 1, further comprising an interactive surface on which said image is displayed.

17. An apparatus, comprising:

a processing device receiving data representing a physical marker having a recognizable pattern thereon positioned within a three-dimensional (3D) input space; and

one or more memory devices storing computer program code executable by the processing device, the processing device communicating with the one or more memory devices and executing the program code causing the apparatus at least to;

visually recognize the pattern on the physical marker in the 3D input space partially based on reference images in a pattern library;

assign a tracking state to the physical marker indicating a level of detection of the physical marker within the 3D input space;

determine digital content associated with the recognized pattern;

modify image data by applying a transition to the determined digital content based on the assigned tracking state of the physical marker and in response to movement of the physical marker within the 3D space; and

output the modified image data for presentation on a display surface.

18. A method comprising:

detecting a physical marker having a unique pattern thereon within a three-dimensional (3D) input space;

visually recognizing the physical marker partially based on a pattern library that includes reference images associated with the unique pattern;

assigning a tracking state of the physical marker indicating a level of detection of the physical marker in the 3D input space;

determining digital content associated with the recognized unique pattern; and

modifying image data used to present an image on a display surface based on the assigned tracking state of the physical marker and in response to movement of the physical marker within the 3D input space, wherein said modifying comprises applying a transition effect to the determined digital content.

19. An interactive input system, comprising:

a computing structure; and

an input device configured to detect at least one physical marker having a recognizable pattern thereon within a three-dimensional (3D) input space and to provide output to said computing structure, wherein said computing structure is configured to process the output of the input device to;

visually recognize the pattern on the at least one physical marker in the 3D input space partially based on reference images in a pattern library;

determine when the location of the pattern within the 3D input space and the location of digital content in an image presented on a display surface satisfy defined criteria; and

associate the pattern to the digital content.

20. The interactive input system of claim 19, wherein a cursor is assigned to said pattern and is presented in said image at a location corresponding to the position of said pattern within said 3D input space, said computing device configured to determine when the pattern is moved within the 3D input space to bring said cursor into a threshold distance from said digital content.

21. The interactive input system of claim 20, wherein the computing structure is further configured to determine if the cursor is positioned from said digital content by said threshold distance for a threshold period of time before associating the pattern with the digital content.

22. The interactive input system of claim 19 wherein said input device is a camera device configured to capture images of said 3D input space.

23. The interactive input system of claim 19, further comprising an interactive surface on which said image is presented.

24. The interactive input system of claim 19, wherein the computing structure is further configured to process the output of the input device to detect physical marker gestures.

25. The interactive input system of claim 24, wherein said computing structure is configured to manipulate digital content associated with the physical marker in response to detected gestures.

26. The interactive input system of claim 25, wherein said computing structure, in response to a detected gesture, is configured to one of (i) change associated digital content, (ii) merge associated digital content, (iii) clone associated digital content, (iv) disassociate digital content, (v) associate digital content, and (vi) rotate digital content.

27. An apparatus comprising:

a processing device receiving data representing a physical marker having a recognizable pattern thereon positioned within a three-dimensional (3D) input space; and

one or more memory devices storing computer program code executable by the processing device, the processing device communicating with the one or more memory devices and executing the program code causing the apparatus at least to;

visually recognize the pattern on the at least one physical marker in the 3D input space partially based on reference images in a pattern library;

determine when the location of the pattern within the 3D input space and the location of digital content in an image presented on a display surface satisfy defined criteria; and

associate the pattern to the digital content.

28. The apparatus according to claim 27, wherein said processing device generates an indication signifying said association.

29. The apparatus according to claim 28, wherein said indication is a visual indicator.

30. A method, comprising:

detecting a physical marker having a unique pattern thereon within a three-dimensional input space;

recognizing the physical marker partially based on a pattern library that includes reference images associated with the unique pattern;

determining when the location of the pattern within the input space and the location of digital content within a presented image satisfy defined criteria; and

associating the pattern to the digital content so that manipulation of the physical marker within the input space manipulates the digital content.

31. An interactive input system comprising:

a computing structure; and

an input device configured to detect at least two physical markers, each having a recognizable pattern thereon, within a three-dimensional (3D) input space and to provide output to said computing structure, wherein said computing structure is configured to process the output of the input device to;

visually recognize the patterns on the at least two physical markers in the 3D input space partially based on reference images in a pattern library; and

associate the patterns with one digital content.

32. The interactive input system of claim 31, wherein the computing structure further process the output of the input device to detect physical marker gestures.

33. The interactive input system of claim 32, wherein said computing structure is configured to manipulate digital content associated with the physical marker in response to detected gestures.

34. An apparatus, comprising:

a processing device receiving data representing at least two physical markers, each having a recognizable pattern thereon, positioned within a three-dimensional (3D) input space; and

one or more memory devices storing computer program code executable by the processing device, the processing device communicating with the one or more memory devices and executing the program code causing the apparatus at least to;

visually recognize the patterns on the at least two physical markers in the 3D input space partially based on reference images in a pattern library; and

associate the patterns with one digital content.

35. An interactive input system, comprising:

a computing structure; and

an input device configured to detect a physical marker having a recognizable pattern thereon within a three-dimensional (3D) input space and to provide output to said computing structure, wherein said computing structure is configured to process said output of said input device to;

visually recognize said pattern on said marker in said 3D input space partially based on reference images in a pattern library;

modify an image presented on a display surface associated with said recognized pattern;

determine which digital content object presented on the display surface is closest to the marker on the display surface; and

select the digital content object closest to said marker to associate the digital content object with said marker.

36. The interactive input system of claim 35, wherein said computing device is configured to determine a distance between each digital content object and said marker.

37. The interactive input system of claim 36, wherein said computing device is configured to determine a first geometric center of said marker, and to determine a second geometric center of each of said at least one of digital content objects, and to measure the distance between said first geometric center and each second geometric center.

38. The interactive input system of claim 37, wherein said computing structure is further configured to determine if said marker is positioned from said digital content within a threshold distance.

39. A method, comprising:

detecting a marker having a unique pattern thereon within a three-dimensional (3D) input space;

visually recognizing said unique pattern on said marker partially based on reference images in a pattern library;

modifying an image presented on a display surface to display a marker associated with said marker when said unique pattern is visually recognized;

determining when said physical object is positioned within said 3D input space at a location to position the marker within a threshold distance of at least one digital content object presented on said display surface.

40. The method of claim 39, further comprising:

further modifying said image based on the velocity of said marker, wherein the determined velocity of said marker is based on projecting said marker on to a two-dimensional plane.

41. The method of claim 39, wherein the determining the velocity of said marker further comprises:

calculating a geometric center of said marker; and

calculating a geometric center of said at least one of digital content object.

42. The method of claim 41, wherein the determining the velocity of the said marker further comprises:

maintaining a timer; and

setting a time threshold for detecting the velocity of said marker.

43. An apparatus, comprising:

a processing device receiving data representing a marker having a recognizable pattern thereon and positioned within a three-dimensional (3D) input space; and

one or more memory devices storing computer program code executable by said processing device, said processing device communicating with said one or more memory devices and executing said program code causing said apparatus at least to;

present an image including at least one digital content object associated with said recognizable pattern on a display surface, presenting the image partially based on reference images in a pattern library, the reference images associated with said recognizable pattern.

44. The apparatus according to claim 43, wherein said processing device generates an indication signifying said marker is moving.

45. The apparatus according to claim 44, wherein said indication is a visual indicator.

46. An interactive input system, comprising:

a computing structure;

a display coupled to said computing structure, said display configured to present a three-dimensional (3D) image including at least one 3D digital content object; and

an input device configured to detect a physical marker having a recognizable pattern thereon within a three-dimensional (3D) input space and to provide output to said computing structure, wherein said recognizable pattern is associated with reference images in a pattern library, wherein said computing structure is configured to process said output of said input device to;

visually recognize said pattern on said marker partially based on said recognizable pattern associated with said reference images in said pattern library; and

select at least one digital content object closest to said marker to associate with said marker.

47. An interactive input system, comprising:

a computing structure; and

an input device configured to detect at least one marker having a recognizable pattern thereon within a three-dimensional (3D) input space and to provide output to said computing structure, wherein said computing structure is configured to process the output of the input device to;

visually recognize the pattern on the at least one marker in the 3D input space partially based on reference images in a pattern library;

determine digital content associated with the recognized pattern;

detect manipulation of the at least one marker in the 3D input space; and

manipulate the digital content associated with the at least one physical object in response to the detected manipulation of the at least one marker.

48. The interactive input system of claim 47, wherein said computing structure in response to a detected gesture is configured to one of (i) change associated digital content, (ii) merge associated digital content, (iii) clone associated digital content, (iv) disassociate digital content, (v) associates digital content, and (vi) rotate digital content.