Virtualization of tangible interface objects

• US 9,552,081 B1
• Filed: 05/10/2016
• Issued: 01/24/2017
• Est. Priority Date: 10/15/2012
• Status: Active Grant

• Current Assignee: Tangible Play, Inc. (Think & Learn Pvt Ltd.)
• Original Assignee: Tangible Play, Inc. (Think & Learn Pvt Ltd.)
• Inventors: Sharma, Pramod Kumar, Scholler, Jerome
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US15/151,213
• Time in Patent Office: 259 Days
• Field of Search: 345156-160, 345173-178, 345204-215
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

An example system includes a computing device located proximate to a physical activity surface, a video capture device, and a detector. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically intractable with by a user. The detector processes the video stream to detect the one or more interface objects included in the activity scene, to identify the one or more interface objects that are detectable, to generate one or more events describing the one or more interface objects, and to provide the one or more events to an activity application configured to render virtual information on the one or more computing devices based on the one or more events.

Claims

1. A computer implemented method for virtualizing a physical activity scene, the method comprising:

receiving, using a video capture device, a video stream depicting the physical activity scene;

processing, using one or more processors, a first tangible interface object from the video stream;

identifying, using the one or more processors, a first attribute of the first tangible interface object;

generating, using the one or more processors, a digital representation of the first tangible interface object based on the first attribute;

presenting, via a display, the digital representation of the first tangible interface object in a virtual scene;

processing, using the one or more processors, a second tangible interface object from the video stream;

identifying, using the one or more processors, a second attribute of the second tangible interface object;

enhancing, using the one or more processors, the digital representation of the first tangible interface object based on the second attribute of the second tangible interface object; and

updating, via the display, the virtual scene with the enhanced digital representation of the first tangible interface object.

2. The method of claim 1, wherein each of the first attribute and the second attribute includes one of a color, a position, an alignment, a pattern, a shape, a size, a sound, and a location.

3. The method of claim 1, further comprising:

determining, using the one or more processors, that the first tangible interface object and the second tangible interface object form a cluster; and

determining, using the one or more processors, an event based on the cluster, wherein updating, via the display, the virtual scene with the enhanced digital representation of the first tangible interface object includes rendering a unique visual representation of the enhanced digital representation of the first tangible interface object based on the determined event.

4. The method of claim 3, wherein determining that the first tangible interface object and the second tangible interface object form the cluster includes determining that the first tangible interface object is situated adjacent to the second tangible interface object in the physical activity scene.

5. The method of claim 3, further comprising:

processing, using the one or more processors, a third tangible interface object from the video stream;

determining, using the one or more processors, that the third tangible interface object is separate from the cluster;

generating, using the one or more processors, a digital representation of the third tangible interface object; and

incorporating, using the one or more processors, the digital representation of the third tangible interface object into the virtual scene.

6. The method of claim 1, further comprising:

identifying, by processing the video stream using the one or more processors, a physical manipulation of the second tangible interface object by a user in the physical activity scene; and

determining, using the one or more processors, an event associated with the physical manipulation, wherein the digital representation of the first tangible interface object is enhanced based on the event.

7. The method of claim 1, wherein enhancing the digital representation of the first tangible interface object based on the second attribute of the second tangible interface object further comprises:

identifying, using the one or more processors, a portion of the digital representation of the first tangible interface object enhanced based on the first attribute; and

enhancing, using the one or more processors, the portion of the digital representation of the first tangible object based on the second attribute.

8. A method for virtualizing a physical activity scene, the method comprising:

displaying, on a display of a computing device, a graphical user interface embodying a virtual scene;

capturing, using a video capture device coupled to the computing device, a video stream of the physical activity scene proximate to the computing device;

processing, using one or more processors, a tangible interface object from the video stream, the tangible interface object being physically present in the physical activity scene and depicted in the video stream;

identifying, using the one or more processors, an attribute of the tangible interface object;

generating, using the one or more processors, a digital representation of the tangible interface object in the virtual scene based on the attribute; and

presenting, on the display of the computing device, the virtual scene including the digital representation of the tangible interface object.

9. The method of claim 8, wherein the attribute is one of a color, a position, an alignment, a pattern, a shape, a size, a sound, and a location.

10. The method of claim 9, wherein generating the digital representation of the attribute of the tangible interface object in the virtual scene based on the attribute further includes:

enhancing, using the one or more processors, the digital representation of the tangible interface object based on the attribute.

11. The method of claim 8, wherein the tangible interface object is a first tangible interface object, the attribute is a first attribute, and the method further includes:

processing, using the one or more processors, a second tangible interface object from the video stream, the second tangible interface object being physically present in the physical activity scene and depicted in the video stream;

identifying, using the one or more processors, a second attribute of the second tangible interface object;

enhancing, using the one or more processors, the digital representation of the first tangible interface object based on the second attribute; and

updating, using the one or more processors, the virtual scene to include the enhanced digital representation of the first tangible interface object.

12. The method of claim 11, wherein the first tangible interface object and the second tangible interface object form a cluster that comprises a logical construct for enhancing the digital representation of the first tangible interface object.

13. The method of claim 12, wherein the second attribute is an instruction for enhancing the digital representation of the first tangible interface object.

14. A computer implemented method for virtualizing a physical activity scene, the method comprising:

receiving, using a video capture device, a video stream depicting the physical activity scene;

processing, using one or more processors, a tangible interface object from the video stream;

identifying, using the one or more processors, an attribute of the tangible interface object processed from the video stream;

generating, using the one or more processors, a virtual scene including a virtualization of the tangible interface object based on the attribute;

identifying, using the one or more processors, a manipulation of the attribute of the tangible interface object from the video stream; and

enhancing, using the one or more processors, the virtualization of the tangible interface object based on the manipulation of the attribute.

15. The method of claim 14, wherein the attribute is one of a color, a position, an alignment, a pattern, a shape, a size, a sound, and a location.

16. The method of claim 14, wherein the manipulation of the attribute is a gesture.

17. The method of claim 16, further comprising:

generating, using the one or more processors, an event associated with the tangible interface object based on the gesture; and

updating, using the one or more processors, the virtual scene in association with the enhanced virtualization of the tangible interface object based on the event.

18. The method of claim 17, wherein updating the virtual scene includes displaying complementary images related to the enhanced virtualization of the tangible interface object.

19. The method of claim 17, wherein updating the virtual scene includes rendering complementary sounds related to the enhanced virtualization of the tangible interface object.

20. The method of claim 14, wherein processing the tangible interface object from the video stream occurs at a first instance and identifying the manipulation of the attribute of the tangible interface object from the video stream occurs at a second instance after the first instance.