Virtualization of tangible interface objects

• US 9,158,389 B1
• Filed: 06/26/2013
• Issued: 10/13/2015
• Est. Priority Date: 10/15/2012
• Status: Active Grant

• Current Assignee: Tangible Play, Inc. (Think & Learn Pvt Ltd.)
• Original Assignee: Tangible Play, Inc. (Think & Learn Pvt Ltd.)
• Inventors: Scholler, Jerome, Sharma, Pramod Kumar
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US13/928,111
• Time in Patent Office: 839 Days
• Field of Search: 345156-160, 345173-178, 345204-215
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

An example system includes a computing device located proximate to a physical activity surface, a video capture device, and a detector. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically intractable with by a user. The detector processes the video stream to detect the one or more interface objects included in the activity scene, to identify the one or more interface objects that are detectable, to generate one or more events describing the one or more interface objects, and to provide the one or more events to an activity application configured to render virtual information on the one or more computing devices based on the one or more events.

Claims

1. A physical activity surface visualization system comprising:

a computing device located proximate to a physical activity surface, the computing device having one or more processors;

a video capture device coupled for communication with the computing device, the video capture device being adapted to capture a video stream that includes a physical activity scene of the physical activity surface and one or more physical interface objects placed in the physical activity scene and physically interactable with by a user; and

a detector executable by the one or more processors to process the video stream to detect the one or more physical interface objects included in the physical activity scene, to identify the one or more physical interface objects that are detectable, to generate one or more events describing the one or more physical interface objects, and to provide the one or more events to an activity application configured to render virtual information on the computing device based on the one or more events, the detector being coupled to the video capture device to receive the video stream.

2. The physical activity surface visualization system of claim 1, further comprising:

an adapter covering at least a portion of a field of view of the video capture device, the adapter configured to redirect at least a portion of the field of view of the video capture device to the physical activity surface.

3. The physical activity surface visualization system of claim 2, wherein the video capture device is integrated with the computing device, the computing device includes a housing, and the adapter is coupled to the housing of the computing device to cover at least the portion of the field of view of the video capture device.

4. The physical activity surface visualization system of claim 3, wherein the adapter includes a housing having a slot that is configured to receive and removeably retain an edge of the housing of the computing device.

5. The physical activity surface visualization system of claim 1, further comprising:

a stand placeable on the physical activity surface and configured to position a display of the computing device for viewing by the user, the display including the video capture device and the video capture device facing the user.

6. The physical activity surface visualization system of claim 1, further comprising:

a calibrator executable by the one or more processors to calibrate the video capture device for one or more of a geometry of the physical activity surface visualization system and image processing performed by the detector.

7. The physical activity surface visualization system of claim 1, further comprising:

a stand placeable on the physical activity surface and configured to position a display of the computing device for viewing by the user; and

an adapter covering at least a portion of a field of view of the video capture device, the video capture device facing the user and the adapter configured to redirect at least a portion of the field of view of the video capture device toward the physical activity surface.

8. A method comprising:

capturing, using a video capture device, a video stream that includes a physical activity scene of a physical activity surface and one or more physical interface objects placed in the physical activity scene and physically interactable with by a user;

processing, using one or more computing devices, the video stream to detect the one or more physical interface objects included in the physical activity scene;

identifying, using the one or more computing devices, the one or more physical interface objects that are detected; and

presenting virtual information on the one or more computing devices based on the one or more physical interface objects that are identified.

9. The method of claim 8, wherein processing the video stream further includes receiving an image of the physical activity scene, processing the image for one or more contours of each of the one or more physical interface objects included in the physical activity scene, constructing an object description for each of the one or more physical interface objects based on the one or more contours, and inferring an object shape for each of the one or more physical interface objects based on the object description associated with each of the one or more physical interface objects.

10. The method of claim 9, wherein constructing the object description for each of the one or more physical interface objects includes mapping the one or more contours associated with each of the one or more physical interface objects using a coordinate system, and inferring the object shape for each of the one or more physical interface objects includes inferring the object shape for each of the one or more physical interface objects based on coordinates of the one or more contours associated with each of the one or more physical interface objects.

11. The method of claim 9, further comprising:

determining one or more of an orientation and a position within the physical activity scene for each of the one or more physical interface objects based on the object description of each of the one or more physical interface objects, wherein the virtual information is generated based on the one or more of the orientation and the position of each of the one or more physical interface objects.

12. The method of claim 9, wherein identifying the one or more physical interface objects includes identifying each of the one or more physical interface objects based on the object shape of each of the one or more physical interface objects, respectively.

13. The method of claim 9, wherein at least one physical interface object of the one or more physical interface objects includes an image and processing the video stream further includes processing the image of the physical activity scene for one or more contours of the image of the at least one physical interface object and constructing a description of the image of the at least one physical interface object based on the one or more contours of the image of the at least one physical interface object.

14. The method of claim 13, wherein identifying the at least one physical interface object includes identifying the image of the at least one physical interface object based on the description associated with the image of the at least one physical interface object.

15. The method of claim 13, wherein constructing the description of the image of the at least one physical interface object includes mapping the one or more contours associated with the image of the at least one physical interface object using a coordinate system.

16. The method of claim 9, wherein at least one physical interface object of the one or more physical interface objects includes an image and processing the video stream further includes calculating one or more of a moment for the image of the at least one physical interface object and a color histogram for the image of the at least one physical interface object, and identifying the at least one physical interface object includes identifying the image of the at least one physical interface object based on one or more of the moment and the color histogram.

17. The method of claim 8, wherein capturing, using the video capture device, the video stream includes receiving an image frame portraying the physical activity scene, comparing the image frame portraying the physical activity scene to a previously received image frame portraying the physical activity scene, determining whether a substantial difference exists between the image frame and the previously received image frame, proceeding to process the video stream to detect the one or more physical interface objects included in the physical activity scene if the substantial difference is determined to exist, and waiting for a subsequent image frame portraying the physical activity scene if the substantial difference is determined not to exist.

18. The method of claim 8, further comprising:

receiving one or more events for the one or more physical interface objects present in the physical activity scene;

determining the one or more physical interface objects based on the one or more events; and

executing one or more routines based on the one or more physical interface objects to generate object-related information visualizing the one or more physical interface objects, wherein presenting the virtual information on the one or more computing devices includes presenting the object-related information to visualize the one or more physical interface objects to the user.

19. The method of claim 8, further comprising:

generating one or more events for the one or more physical interface objects, respectively, based on an identity of each of the one or more physical interface objects; and

executing one or more routines based on the one or more events to generate object-related information related to the one or more physical interface objects for presentation, wherein presenting the virtual information on the one or more computing devices includes presenting the object-related information.

20. The method of claim 8, wherein the video stream further includes a split view having the physical activity scene including the physical activity surface and a user scene including one or more of a face and one or more extremities of the user.

21. The method of claim 20, further comprising:

processing the user scene for a user input; and

generating the virtual information for presentation based on the one or more physical interface objects that are identified and the user input processed from the user scene.

22. An adapter for adapting a camera integrated with a computing device, the adapter comprising:

a housing including a slot on a first side, the slot configured to receive and retain an edge of a body of the computing device, the housing configured to cover at least a portion of a field of view of the camera of the computing device; and

a reflective element recessed at an angle into the first side of the housing to redirect the field of view of the camera toward an activity surface located proximate the computing device.