Vision-based interactive projection system

• US 9,030,445 B2
• Filed: 10/04/2012
• Issued: 05/12/2015
• Est. Priority Date: 10/07/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Shamaie, Atid
• Primary Examiner(s): CERULLO, LILIANA P
• Application Number:
  US13/645,390
• Publication Number:
  US 20130088461A1
• Time in Patent Office: 950 Days
• Field of Search: 345156-178, 382181-231, 382312-324, 348135-141
• US Class Current: 345/175
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G03B 21/14 Details
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Techniques are provided to improve interaction between a user and a projection system. In some embodiments, an image of a user in front of a display screen is captured. An inference can then be made as to whether a user is touching a display screen based on an analysis of shadows and/or variation of brightness (i.e., intensities) across pixels in the image. For example, it may be inferred that the object is: (1) approaching the screen when a region surrounding a top of the object is characterized by a relatively small brightness variation; (2) hovering near the screen when the brightness variation is large and the region includes a dark extremum (caused by a shadow); and (3) touching the screen when the brightness variation is large and the region includes a light extremum.

Claims

1. A method comprising:

projecting a display onto a surface;

illuminating a region near the surface during a plurality of different time periods;

capturing a plurality of images of the surface during the plurality of different time periods;

analyzing a variation of intensities of pixels in a same area across the plurality of images;

determining a change in polarity between the plurality of images the polarity indicating whether the area is associated with a predetermined intensity value; and

determining whether an object is touching the surface based at least partly on the analysis of the variation and on the determined change in polarity.

2. The method of claim 1, wherein the area comprises a region surrounding an estimated location of at least part of the object.

3. The method of claim 1, wherein the analyzing the variation of intensities comprises calculating one or more standard deviations of the intensities.

4. The method of claim 1, wherein the analyzing the variation of intensities comprises calculating a plurality of standard deviations associated with non-overlapping regions in the area.

5. The method of claim 4, wherein the analyzing the variation of intensities further comprises averaging the plurality of standard deviations.

6. The method of claim 1, wherein determining the change in polarity comprises identifying a sign of an extremum in the area of the image.

7. The method of claim 1, wherein determining that the object is touching the surface requires that the polarity is positive after the determined change.

8. The method of claim 1, further comprising:

illuminating a front region near the surface;

capturing a reduced-shadow image of the surface during the illumination of the front region; and

identifying a location associated with the object based on the reduced-shadow image,

wherein the front-region illumination is configured to minimize shadows associated with an object hovering near the surface in the reduced-shadow image.

9. The method of claim 8, further comprising ceasing illuminating the region prior to illuminating the front region.

10. The method of claim 1, wherein a substantially enclosed projecting device projects the display, illuminates the region, and captures the image.

11. A projector system comprising:

projection optics configured to project a display on a surface;

a light source configured to emit light in a direction towards the surface during a plurality of different time periods;

a camera configured to capture a plurality of images of the surface during the plurality of different time periods;

an image analyzer configured to detect a variation of intensities across pixels in at least a portion of the plurality of images and to determine a change in polarity of pixels in a same area across between the plurality of images the polarity indicating whether the area is associated with a predetermined intensity value; and

a state detector configured to determine whether an object is touching the surface based at least partly on the variation detection and on the determined change in polarity.

12. The projector system of claim 11, wherein the image analyzer is further configured to estimate a location associated with the object.

13. The projector system of claim 12, wherein the area of the image comprises a region surrounding the estimated location.

14. The projector system of claim 11, wherein the image analyzer is further configured to identify a plurality of standard deviations of intensities associated with the area of the image.

15. The projector system of claim 11, further comprising:

a second light source configured to emit light in a second direction towards the surface, the second direction being different from the direction; and

a position detector configured to detect a position of the object based on an image captured by the camera during the second light'"'"'s emission of light.

16. The projector system of claim 11, wherein the determination of the change in polarity comprises identifying a sign of an extremum in the area of the image.

17. The projector system of claim 11, further comprising a program interactor configured to allow the object to interact with a program being displayed on the surface upon a determination that the object is touching the surface.

18. The projector system of claim 11, further comprising an image pre-processor configured to filter the image based at least partly on one or more control images.

19. The projector system of claim 11, further comprising a timing circuit coupled to the camera and the light source, the timing circuit configured to coordinate times that the light source emits light with times that the camera captures images.

20. A projection system comprising:

means for projecting a display onto a surface;

means for illuminating a region near the surface during a plurality of different time periods;

means for capturing a plurality of images of the surface during the plurality of different time periods;

means for analyzing a variation of intensities of pixels in a same area across the plurality of images;

means for determining a change in polarity between the plurality of images the polarity indicating whether the area is associated with a predetermined intensity value; and

means for determining whether an object is touching the surface based at least partly on the analysis and on the determined change in polarity.

21. The projection system of claim 20, wherein the area comprises a region surrounding an estimated location of at least part of the object.

22. The projection system of claim 20, wherein the means for analyzing the variation of intensities comprises means for calculating one or more standard deviations of the intensities.

23. The projection system of claim 20, wherein the means for analyzing the variation of intensities comprises means for calculating a plurality of standard deviations associated with non-overlapping regions in the area.

24. The projection system of claim 23, wherein the means for analyzing the variation of intensities further comprises means for averaging the plurality of standard deviations.

25. The projection system of claim 20, wherein the means for determining the change in polarity comprises means for identifying a sign of an extremum in the area of the image.

26. The projection system of claim 20, wherein the means for determining that the object is touching the surface comprises means for enforcing a requirement that the determined polarity be above a threshold.

27. The projection system of claim 20, further comprising:

means for illuminating a front region near the surface;

means for capturing a reduced-shadow image of the surface during the illumination of the front region; and

means for identifying a location associated with the object based on the reduced-shadow image,

wherein the front-region illumination is configured to minimize shadows associated with an object hovering near the surface in the reduced-shadow image.

28. The projection system of claim 27, further comprising means for ceasing illuminating the region prior to illuminating the front region.

29. The projection system of claim 20, wherein a substantially enclosed projecting device includes the means for projecting the display, means for illuminating the region, and means for capturing the image.

30. A non-transitory computer-readable medium containing instructions which, when executed by a processor, cause the processor to perform steps of:

accessing, during a plurality of different time periods, a plurality of images of a surface on which a display is projected;

analyzing a variation of intensities of pixels in a same area across the plurality of images;

determining a change in polarity between the plurality of-images the polarity indicating whether the area is associated with a predetermined intensity; and

determining whether an object is touching the surface based at least partly on the analysis of the variation and on the determined change in polarity.

31. The computer-readable medium of claim 30, wherein determining the change in polarity comprises identifying a sign of an extremum in the area of the image.

32. The computer-readable medium of claim 30, wherein determining that the object is touching the surface requires that the polarity is positive after the determined change.

33. The computer-readable medium of claim 30, wherein the analyzing the variation of intensities comprises calculating one or more standard deviations of the intensities.