Activity surface detection, display and enhancement

• US 10,033,943 B1
• Filed: 12/10/2015
• Issued: 07/24/2018
• Est. Priority Date: 10/15/2012
• Status: Active Grant

• Current Assignee: Tangible Play, Inc. (Think & Learn Pvt Ltd.)
• Original Assignee: Tangible Play, Inc. (Think & Learn Pvt Ltd.)
• Inventors: Sharma, Pramod Kumar, Scholler, Jerome, Verkler, Conrad
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US14/965,776
• Time in Patent Office: 957 Days
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G09B 11/00 Teaching hand-writing, shorthand, drawing, or painting
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Activity surface detection, display, and enhancement implementations are described. In an example implementation, a method determines, using a processor of a computing device, a traceable image and presenting the traceable image in an interface on a display of the computing device; captures, using a video capture device coupled to the computing device, a video stream of a physical activity surface proximate to the computing device; and

displays, on the display of the computing device, the captured video stream overlaid with the traceable image in the interface.

Claims

1. A computer-implemented method for monitoring user activity in a physical activity scene, comprising:

determining, using a processor of a computing device, a traceable image and presenting the traceable image in an interface on a display of the computing device;

capturing, using a video capture device coupled to the computing device, a video stream of a physical activity surface proximate to the computing device;

displaying, on the display of the computing device, the captured video stream overlaid with the traceable image in the interface;

processing, using the processor of the computing device, the video stream captured by the video capture device to detect a contour of a work being created by a user on the physical activity surface;

comparing, using the processor of the computing device, the contour of the work with a corresponding portion of the traceable image;

determining, using the processor of the computing device, an accuracy of the contour relative to the corresponding portion of the traceable image based on the comparison;

determining, using the processor of the computing device, a progress of the user in creating a work on the physical activity surface based on the accuracy; and

updating, using the processor of the computing device, the interface to include the progress of the user.

2. The method of claim 1, further comprising:

determining, using the processor of the computing device, directive feedback based on the comparison between the contour of the work and the corresponding portion of the traceable image; and

updating, using the processor of the computing device, the interface to include the directive feedback.

3. The method of claim 2, wherein the comparison indicates a difference between the contour of the work and the corresponding portion of the traceable image, and the directive feedback highlights the difference.

4. The method of claim 1 further comprising:

detecting, using the processor of the computing device, a new contour on the physical activity surface;

generating, using the processor of the computing device, a visualization of the new contour; and

updating the interface with the visualization of the new contour overlaid on a corresponding portion of the traceable image.

5. The method of claim 1, wherein displaying the captured video stream overlaid with the traceable image includes displaying an image frame from the captured video stream overlaid with the traceable image.

6. The method of claim 1, wherein the traceable image is semi-transparent.

7. The method of claim 1 wherein generating a traceable image further comprises:

receiving, using the processor of the computing device, a captured image;

detecting, using the processor of the computing device, contours in the captured image; and

generating, using the processor of the computing device, the traceable image based on the contours in the captured image.

8. The method of claim 1, further comprising:

detecting, using the processor of the computing device, one or more of a resize instruction and a reposition instruction;

modifying, using the processor of the computing device, the traceable image based on the one or more of the resize instruction and the reposition instruction; and

updating the interface to display the modified traceable image.

9. A computer-implemented method comprising:

determining, using a processor of a computing device, a traceable image and presenting the traceable image in an interface on a display of the computing device;

capturing, using a video capture device coupled to the computing device, a video stream of a physical activity surface proximate to the computing device;

displaying, on the display of the computing device, the captured video stream overlaid with the traceable image in the interface, the video stream depicting a work being created by a user on the physical activity surface;

determining, using the processor of the computing device, an accuracy of the work relative to the traceable image;

determining, using the processor of the computing device, a progress of the user in creating the work on the physical activity surface based on the accuracy; and

updating, using the processor of the computing device, the interface to include the progress of the user.

10. A computer-implemented method of claim 9, further comprising:

processing, using a processor of a computing device, the video stream to detect a change in the work being created by the user in the physical activity surface;

tracking, using the computing device, the progress of the work in comparison to the traceable image depicted in the interface displayed on the computing device, by evaluating which portion of the traceable image the change in the work relates to; and

updating, using a processor of a computing device, the interface displayed on the computing device with a visual indication of the tracked progress.

11. The method of claim 9, further comprising:

generating, using a processor of a computing device, the traceable image by detecting contour lines of an image and generating the traceable image based on the contour lines.

12. The method of claim 10, wherein detecting the change in the work includes detecting, using the processor of the computing device, a change in a contour.

13. The method of claim 9, wherein the computing device is a tablet computer, a smartphone, a laptop computer, or a desktop computer.

14. The method of claim 10, wherein the visual indication of the tracked progress includes generating corresponding virtual contours based on the change in the contour and rendering the corresponding virtual contours overlaid on the traceable image displayed in the interface.

15. The method of claim 9, further comprising:

processing, using the processor of the computing device, the video stream captured by the video capture device to detect a contour of a work being created by a user on the physical activity surface;

comparing, using the processor of the computing device, the contour of the work with a corresponding portion of the traceable image;

determining, using the processor of the computing device, directive feedback based on the comparison between the contour of the work and the corresponding portion of the traceable image; and

updating, using the processor of the computing device, the interface to include the directive feedback.

16. The method of claim 15, wherein the comparison indicates a difference between the contour of the work and the corresponding portion of the traceable image, and the directive feedback highlights the difference.

17. The method of claim 9, further comprising:

processing, using the processor of the computing device, the video stream to detect an object in the physical activity surface;

determining, using the processor of the computing device, a relative position of the object in relation to the traceable image displayed on the interface; and

displaying on the interface, a visualization of the object in relation to the traceable image.

18. The method of claim 9, wherein the physical activity surface is located in front of the display of the computing device or behind the display of the computing device.

19. A physical activity surface visualization system, comprising:

a detection engine coupled via a communications bus to a video capture device configured to capture a video stream of a physical activity surface proximate to a computing device, the video stream depicting a work being created by a user on the physical activity surface, the detection engine further configured to process the video stream to detect a change in the work being created by a user in the physical activity surface; and

an activity application executable by a processor of the computing device and configured to display an instructional image in an interface on a display of the computing device, the activity application further configured to determine an accuracy of the work relative to the instructional image, determine a progress of the work based on the accuracy, the interface displaying a visual indicator of the tracked progress and the instructional image.

20. The physical activity surface visualization system of claim 19, wherein the computing device is a tablet computer, a smartphone, a laptop computer, or a desktop computer.

21. The physical activity surface visualization system of claim 19, further comprising:

a detachable camera adapter coupled to adapt the video capture device, the detachable camera adapter configured to alter a field of view of the video capture device to include the physical activity surface.

22. The physical activity surface visualization system of claim 21, wherein the detachable camera adapter includes one or more mirrors for redirecting the field of view of the video capture device to the physical activity surface.