Methods, systems, and computer program products for imperceptibly embedding structured light patterns in projected color images for display on planar and non-planar surfaces

• US 20050254726A1
• Filed: 02/25/2005
• Published: 11/17/2005
• Est. Priority Date: 02/25/2004
• Status: Active Grant

• Current Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Original Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Inventors: Fuchs, Henry, Gross, Markus, Cotting, Daniel, Naef, Martin
• Granted Patent: US 7,182,465 B2
• Time in Patent Office: Days
• US Class Current: 382/285
• CPC Class Codes: H04N 5/74 Projection arrangements for image reproduction, e.g. using eidophor
• CPC Class Codes: H04N 9/3182 Colour adjustment, e.g. white balance, shading or gamut white balance per se H04N9/73; control of amplitude of colour signals H04N9/68; colour control circuits for displays in general G09G5/02, G09G3/2003, G09G3/3607
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

Methods, systems, and computer program products for imperceptibly embedding structured light patterns in projected color images for display on planar and non-planar surfaces are disclosed. According to one method, an image exposure period for detecting an embedded structured light patterns in a projected image is selected based on analysis of pixel polarities for different pixel intensities of a pixel color. Pixel intensities for the color are varied in the user image so that pixel polarities encode the structured light patterns during image exposure period. The user image is projected with the structured light patterns onto the surface. Depth information is continuously acquired and used to adjust display of the user image.

Claims

1. A method for embedding a structured light pattern in a color user image and for projecting the user image, the method comprising:

(a) varying pixel intensity values for a first visible light color in a color user image so that pixel polarities encode a first structured light pattern during a selected image exposure period and so that varied pixel intensity values have a predetermined relationship with respect to original pixel intensity values in the user image;

(b) projecting the user image with the embedded structured light pattern onto a surface; and

(c) detecting the first structured light pattern during the image exposure period.

2. The method of claim 1 wherein varying the pixel intensity values includes, for each pixel in the first structured light pattern:

(a) determining whether a polarity of a pixel in the user image encodes a desired value corresponding to a pixel in the first structured light pattern during the selected image exposure period, the pixel in the user image having a first intensity value; and

(b) in response to determining that the polarity of the pixel in the user image does not encode the desired value, changing the first intensity value to a second intensity value, the second intensity value being the closest intensity value to the first intensity value that encodes the desired value during the image exposure period.

3. The method of claim 2 comprising calculating a difference between the second pixel intensity value and the first pixel intensity value and propagating the difference to pixels adjacent to the pixel in the user image.

4. The method of claim 1 wherein varying pixel intensity values comprises:

(a) compressing a range of intensity values for the first visible light color in the user image to exclude intensities at a beginning and an end of the range for which pixel polarity has a constant value; and

(b) selecting, from the compressed range, the pixel intensity values whose polarities encode the first structured light pattern.

5. The method of claim 4 comprising compressing a range of pixel intensity values for at least one second visible light color in the user image proportionally to the compressing of the range of intensity values for the first visible light color.

6. The method of claim 1 wherein varying pixel intensity values includes varying the pixel intensity values without modifying a projector used to project the user image.

7. The method of claim 1 comprising:

(a) calculating depth information based on the detected structured light pattern; and

(b) adjusting projection of the user image based on the depth information.

8. The method of claim 7 wherein projecting the user image onto a surface includes projecting the user image onto a non-planar surface and wherein adjusting display of the user image includes pre-distorting the user image such that a projection of the user image on the non-planar surface appears undistorted.

9. The method of claim 1 wherein projecting the user image includes projecting an interface generated by a computer software application.

10. The method of claim 1 wherein projecting the user image comprises projecting the user image using a single projector and wherein detecting the first structured light pattern includes detecting the first structured light pattern using a single camera.

11. The method of claim 1 wherein projecting the user image includes projecting portions of the user image using a plurality of projectors and wherein detecting the first structured light pattern includes detecting the first structured light pattern using a plurality of cameras.

12. The method of claim 11wherein projecting portions of the user image using a plurality of projectors includes projecting a first portion of the user image including the first structured light pattern and projecting a second portion of the user image that overlaps with the first portion of the user image in a manner that does not interfere with detection of the first structured light pattern during the selected image exposure period.

13. The method of claim 12 wherein projecting a first portion of the user image including a first structured light pattern includes encoding the first structured light pattern using different intensities of the first visible light color and wherein projecting the second portion of the user image in a manner that does not interfere with detection of the first structured light pattern includes varying intensities of the first visible light color in the second portion to encode a common value during the selected image exposure period.

14. The method of claim 1 comprising selecting the image exposure period based on analysis of pixel polarities.

15. The method of claim 14 wherein selecting the image exposure period includes analyzing mirror flip sequences of a projector used to project the user image for different intensities of the first visible light color.

16. The method of claim 14 wherein selecting the image exposure period includes selecting a time during which pixel polarities vary between on and off states for a range of intensities of the first visible light color.

17. The method of claim 1 comprising imperceptibly projecting a second structured light pattern and tracking motion of a user based on the second structured light pattern.

18. The method of claim 17 comprising controlling projection of the user image based on the motion of the user.

19. A system for embedding a structured light pattern in a color user image and for projecting the user image, the method comprising:

(a) an embedding module for varying pixel intensity values for a first visible light color in a color user image so that pixel polarities encode a first structured light pattern during a selected image exposure period and so that varied pixel intensity values have a predetermined relationship with respect to original pixel intensity values in the user image;

(b) at least one projector for projecting the user image with the embedded structured light pattern onto a surface; and

(c) at least one detector for detecting the first structured light pattern during the image exposure period.

20. The system of claim 19 wherein, in varying the pixel intensity values, the embedding module is adapted to, for each pixel in the first structured light pattern:

(a) determine whether a polarity of a pixel in the user image encodes a desired value corresponding to a pixel in the first structured light pattern during the selected image exposure period, the pixel in the user image having a first intensity value; and

(b) in response to determining that the polarity of the pixel in the user image does not encode the desired value, change the first intensity value to a second intensity value, the second intensity value being the closest intensity value to the first intensity value that encodes the desired value during the image exposure period.

21. The system of claim 20 wherein the embedding module is adapted to calculate a difference between the second pixel intensity value and the first pixel intensity value and propagate the difference to pixels adjacent to the pixel in the user image.

22. The system of claim 19 wherein, in varying pixel intensity values, the embedding module is adapted to:

(a) compress a range of intensity values for the first visible light color in the user image to exclude intensities at a beginning and an end of the range for which pixel polarity has a constant value; and

(b) select, from the compressed range, the pixel intensity values whose polarities encode the first structured light pattern.

23. The system of claim 22 wherein the embedding module is adapted to compress a range of pixel intensity values for at least one second visible light color in the user image proportionally to the compressing of the range of intensity values for the first visible light color.

24. The system of claim 19 comprising:

(a) a depth extractor for calculating depth information based on the detected structured light pattern; and

(b) a projection undistortion module adjusting projection of the user image based on the depth information.

25. The system of claim 24 wherein the at least one projector is adapted to project the user image onto a non-planar surface and wherein, in adjusting projection of the user image, the projection undistortion module is adapted to pre-distort the user image such that a projection of the user image on the non-planar surface appears undistorted.

26. The system of claim 19 wherein the at least one projector is adapted to project an interface generated by a computer software application.

27. The system of claim 19 the at least one projector comprises a single projector and wherein the at least one detector comprises a single detector.

28. The system of claim 19 wherein the at least one projector comprises a plurality of projectors for projecting portions of the user image and wherein the at least one detector comprises a plurality of detectors for detecting portions of the first structured light pattern

29. The system of claim 28 wherein the plurality of projectors includes a first projector for projecting a first portion of the user image including the first structured light pattern and a second projector for projecting a second portion of the user image that overlaps with the first portion of the user image in a manner that does not interfere with detection of the first structured light pattern during the selected image exposure period.

30. The system of claim 29 wherein the first projector is adapted to project the first portion of the user image encoding the first structured light pattern using different intensities of the first visible light color and wherein the second projector is adapted to project the second portion of the user image with intensities of the first visible light color in the second portion encoding a common value during the selected image exposure period.

31. The system of claim 19 wherein the image exposure period is selected based on analysis of mirror flip sequences of the projector corresponding to different intensities of the first visible light color.

32. The system of claim 31 wherein the image exposure period comprises a time period during which pixel polarities vary between on and off states for a range of intensities of the first visible light color.

33. The system of claim 19 wherein the at least one projector comprises a first projector for projecting the user image and wherein the system further comprises a second projector for imperceptibly projecting a second structured light pattern onto a user and tracking motion of the user based on the second structured light pattern.

34. The system of claim 33 comprising a control module for controlling projection of the user image based on the motion of the user.

35. A computer program product comprising computer-executable instructions embodied in a computer-readable medium for performing steps comprising:

(a) varying pixel intensity values for a first visible light color in a color user image so that pixel polarities encode a first structured light pattern during a selected image exposure period and so that varied pixel intensity values have a predetermined relationship with respect to original pixel intensity values in the user image;

(b) projecting the user image with the embedded structured light pattern onto a surface; and

(c) detecting the first structured light pattern during the image exposure period.

36. The computer program product of claim 35 wherein varying the pixel intensity values includes, for each pixel in the first structured light pattern:

(a) determining whether a polarity of a pixel in the user image encodes a desired value corresponding to a pixel in the first structured light pattern during the selected image exposure period, the pixel in the user image having a first intensity value; and

(b) in response to determining that the polarity of the pixel in the user image does not encode the desired value, changing the first intensity value to a second intensity value, the second intensity value being the closest intensity value to the first intensity value that encodes the desired value during the image exposure period.

37. The computer program product of claim 36 comprising calculating a difference between the second pixel intensity value and the first pixel intensity value and propagating the difference to pixels adjacent to the pixel in the user image.

38. The computer program product of claim 35 wherein varying pixel intensity values comprises:

(a) compressing a range of intensity values for the first visible light color in the user image to exclude intensities at a beginning and an end of the range for which pixel polarity has a constant value; and

(b) selecting, from the compressed range, the pixel intensity values whose polarities encode the first structured light pattern.

39. The computer program product of claim 38 comprising compressing a range of pixel intensity values for at least one second visible light color in the user image proportionally to the compressing of the range of intensity values for the first visible light color.

40. The computer program product of claim 35 wherein varying pixel intensity values include varying pixel intensity values without modifying a projector used to project the user image.

41. The computer program product of claim 35 comprising:

(a) calculating depth information based on the detected structured light pattern; and

(b) adjusting projection of the user image based on the depth information.

42. The computer program product of claim 41 wherein projecting the user image onto a surface includes projecting the user image onto a non-planar surface and wherein adjusting display of the user image includes pre-distorting the user image such that a projection of the user image on the non-planar surface appears undistorted.

43. The computer program product of claim 35 wherein projecting the user image includes projecting an interface generated by a computer software application.

44. The computer program product of claim 35 wherein projecting the user image comprises projecting the user image using a single projector and wherein detecting the first structured light pattern includes detecting the first structured light pattern using a single camera.

45. The computer program product of claim 35 wherein projecting the user image includes projecting portions of the user image using a plurality of projectors and wherein detecting the first structured light pattern includes detecting the first structured light pattern using a plurality of cameras.

46. The computer program product of claim 45 wherein projecting portions of the user image using a plurality of projectors includes projecting a first portion of the user image including the first structured light pattern and projecting a second portion of the user image that overlaps with the first portion of the user image in a manner that does not interfere with detection of the first structured light pattern during the selected image exposure period.

47. The computer program product of claim 46 wherein projecting a first portion of the user image including a first structured light pattern includes encoding the first structured light pattern using different intensities of the first visible light color and wherein projecting the second portion of the user image in a manner that does not interfere with detection of the first structured light pattern includes varying intensities of the first visible light color in the second portion to encode a common value during the selected image exposure period.

48. The computer program product of claim 35 comprising selecting the image exposure period based on analysis of pixel polarities.

49. The computer program product of claim 48 wherein selecting the image exposure period includes analyzing mirror flip sequences of a projector used to project the user image for different intensities of the first visible light color.

50. The computer program product of claim 48 wherein selecting the image. exposure period includes selecting a time during which pixel polarities vary between on and off states for a range of intensities of the first visible light color.

51. The computer program product of claim 35 comprising imperceptibly projecting a second structured light pattern and tracking motion of a user based on the second structured light pattern.

52. The computer program product of claim 51 comprising controlling projection of the user image based on the motion of the user.