Interactive projection system

• US 8,842,096 B2
• Filed: 01/07/2011
• Issued: 09/23/2014
• Est. Priority Date: 01/08/2010
• Status: Active Grant

• Current Assignee: Crayola LLC (Hallmark Cards Incorporated)
• Original Assignee: Crayola LLC (Hallmark Cards Incorporated)
• Inventors: Nungester, Gregory R., Weiss, Stephen
• Primary Examiner(s): Zhou, Hong
• Application Number:
  US12/986,298
• Publication Number:
  US 20110169778A1
• Time in Patent Office: 1,355 Days
• Field of Search: 345/173, 345/175, 345/179, 345/158, 345/169, 345/207, 353/42, 353 46- 51, 353/122, 178/18.09, 178/19.05
• US Class Current: 345/175
• CPC Class Codes: G06F 3/03542 Light pens for emitting or receiving light
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

An interactive projection system, apparatus, and methods for their use are provided. The system includes a projection unit that projects an image of a user interface onto a projection surface. At least one vision sensor is provided that is configured to sense a location of an infrared light source. A computing device that is configured to execute applications to provide the user interface and to determine corresponding outputs based on inputs received by the vision sensor is also provided. An infrared input device, or stylus, is employed by a user to provide infrared light inputs at desired locations with respect to the projected user interface. Additionally, a plurality of styli, each emitting a different respective wavelength of infrared light is also useable to provide multiple simultaneous inputs to the system. An equal plurality of vision sensors may also be employed to separately sense inputs from a respective stylus.

Claims

1. An interactive projection system comprising:

a projection unit that projects at least one image onto a projection surface, wherein the projection surface is separate from the interactive projection system;

a first vision sensor configured to sense a location of a first infrared light source;

a first filter associated with the first vision sensor that allows only infrared light of a first predetermined range of wavelengths to contact the first vision sensor;

a second vision sensor configured to sense a location of a second infrared light source;

a second filter associated with the second vision sensor that allows only infrared light of a second predetermined range of wavelengths to contact the second vision sensor, wherein the first and second predetermined ranges of wavelengths are mutually exclusive,

a computing device having one or more processors and a memory that executes a computer application that provides a user interface image that is projected by the projection unit onto the projection surface; and

a first infrared input device that is configured to emit an infrared light upon activation by a user, the infrared light providing input to the computing device via the first vision sensor, the first infrared input device configured to emit infrared light having a wavelength within the first predetermined range, wherein the input provided to the computing device via the first vision sensor is associated with a first type of infrared input signal,

a second infrared input device that is configured to emit an infrared light upon activation by a user, the infrared light providing input to the computing device via the second vision sensor, the second infrared input device configured to emit infrared light having a wavelength within the second predetermined range, wherein the input provided to the computing device via the second vision sensor is associated with a second type of infrared input signal;

wherein the computing device identifies a location of the infrared light with respect to the user interface image projected onto the projection surface and provides a corresponding output;

wherein the corresponding output provided by the computing device in response to infrared light received from the first infrared input device is based at least in part on

     1) the identified location of the infrared light input from the first infrared input device with respect to the projected user interface image and

     2) the first type of infrared input signal;

wherein the corresponding output provided by the computing device in response to infrared light received from the second infrared input device is based at least in part on

     1) the identified location of the infrared light input from the second infrared input device with respect to the projected user interface image and

     2) the second type of infrared input signal.

2. The projection system of claim 1, wherein the vision sensor is calibrated to an area inclusive of the user interface image projected onto the projection surface.

3. The projection system of claim 2, wherein the corresponding output includes one or more of updating the user interface image, providing a audible output, and performing an executable task.

4. The projection system of claim 1, further comprising a removable computer memory, the removable computer memory comprising one or more of a universal serial bus (USB) flash drive, a Secure Digital (SD) memory card, a CompactFlash memory card, a Memory Stick, and an xD memory card.

5. The projection system of claim 1, wherein the vision sensor is a complementary metal-oxide-semiconductor (CMOS) sensor with a visible light filter.

6. The projection system of claim 1, wherein an input is received by the projection system from both the first input device and the second input device simultaneously.

7. The projection system of claim 1, wherein the computing device includes at least two processors, wherein a first processor determines the location of the input from the first infrared input device and a second processor determines a location of the input from the second infrared input device.

8. The projection system of claim 1, wherein at least one of the one or more computer applications is a drawing program useable by children to create an artwork.

9. A method for using a projected user interface comprising:

providing a projection system that includes a housing having disposed therein;

(1) a projection unit that projects at least one image onto a projection surface, wherein the projection surface is separate from the projection system;

(2) a first vision sensor capable of sensing a location of a first infrared light source;

(3) a first filter associated with the first vision sensor that allows only infrared light of a first predetermined range of wavelengths to contact the first vision sensor;

(4) a second vision sensor capable of sensing a location of a second infrared light source; and

(5) a computing device having a processor and a memory that executes a software application that provides a user interface image for projection onto the projection surface;

projecting the user interface image onto the projection surface via the projection unit;

providing a first infrared input device that is configured to provide a first infrared input signal upon activation by a user, the first infrared input signal providing input to the computing device via the first vision sensor, wherein the first infrared input device is configured to emit infrared light having a wavelength within the first predetermined range, wherein the input provided to the computing device via the first vision sensor is associated with a first type of infrared input signal;

providing a second infrared input device that is configured to provide a second infrared input signal upon activation by a user, the second infrared input signal providing input to the computing device via the second vision sensor, wherein the second infrared input device is configured to emit infrared light having a wavelength within the second predetermined range, wherein the input provided to the computing device via the second vision sensor is associated with a second type of infrared input signal;

receiving the first infrared input signal from the first infrared input device via the first vision sensor;

determining a location of the received first infrared input signal with respect to the projected user interface image;

determining a type of the received first infrared input signal, wherein the type comprises one or more of a click input, a drag input, a short stationary signal, and a prolonged non-stationary signal;

determining a first input command to the software application based at least in part on the determined location of the received first infrared input signal and the determined type of the received first infrared input signal;

receiving the second infrared input signal from the second infrared input device via the second vision sensor;

determining a location of the received second infrared input signal with respect to the projected user interface image;

determining a type of the received second infrared input signal, wherein the type comprises one or more of a click input, a drag input, a short stationary signal, and a prolonged non-stationary signal;

determining a second input command to the software application based at least in part on the determined location of the received second infrared input signal and the determined type of the received second infrared input signal;

updating the user interface image based on one or more of the first input command and the second input command; and

projecting an updated user interface image.

10. The method of claim 9, wherein at least one of the first infrared input device and the second infrared input device includes one or more of a switch or a button that, when actuated by a user, causes at least one of the first infrared input device and the second infrared input device to emit a corresponding infrared input signal.

11. The method of claim 9, wherein the first infrared input signal from the first infrared input device is reflected off of the projection surface toward the first vision sensor, and further wherein the second infrared input signal from the second infrared input device is reflected off of the projection surface toward the second vision sensor.

12. The method of claim 9, wherein determining at least one of a location of the first infrared input signal with respect to the projected user interface image and a location of the second infrared input signal with respect to the projected user interface image further comprises:

analyzing pixel data collected by at least one of the first vision sensor and the second vision sensor to identify pixels that indicate received infrared light;

filtering one or more of

     1) the pixel data received by the first vision sensor to remove data not associated with infrared light emitted by the first infrared input device, and

     2) the pixel data received by the second vision sensor to remove data not associated with the infrared light emitted by the second infrared input device;

identifying one or more of

     1) a largest grouping of pixels indicating received infrared light from the first infrared input device, and

     2) a largest grouping of pixels indicating received infrared light from the second infrared input device;

determining one or more of

     1) a central pixel in the largest grouping of pixels indicating received infrared light from the first infrared input device, and

     2) a central pixel in the largest grouping of pixels indicating received infrared light from the second infrared input device; and

identifying coordinates of at least one of the central pixel corresponding to the first infrared input device and the central pixel corresponding to the second infrared input device,

wherein the central pixel corresponding to the first infrared input device is associated with the first input command,

and further wherein the central pixel corresponding to the second infrared input device is associated with the second input command.

13. The method of claim 9, further comprising;

projecting a calibration image onto the projection surface, the calibration image including one or more calibration points;

receiving an input from the first infrared input device or the second infrared input device at each of the one or more calibration points;

determining a coordinate location of the received input at each of the one or more calibration points;

mapping the coordinate locations to the calibration image, thereby aligning an output of the projection unit with inputs received by the first vision sensor or the second vision sensor.

14. The method of claim 9, wherein more than one infrared input is received at one time, and wherein each infrared input device emits infrared light of a different wavelength.

15. The method of claim 14, wherein the first vision sensor and the second vision sensor each sense infrared light within a respective, mutually exclusive range of wavelengths of infrared light.

16. An interactive projection system comprising:

a projection unit that projects at least one image onto a projection surface, wherein the projection surface is separate from the interactive projection system;

a first vision sensor configured to sense the location of a first infrared light source, the first vision sensor including a first complementary metal-oxide-semiconductor (CMOS) sensor and a light filter that allows only a first predetermined range of wavelengths of infrared light to reach the CMOS sensor;

a second vision sensor configured to sense the location of a second infrared light source, the second vision sensor including a second CMOS sensor and a second light filter that allows only a second predetermined range of wavelengths of infrared light to reach the second CMOS sensor;

a computing device having a processor and a memory configured to execute one or more applications that provide a user interface image that is projected by the projection unit onto the projection surface;

a first infrared input device that is configured to provide a first input to the computing device by emitting infrared light of a first wavelength within the first predetermined range that is received by the first vision sensor, wherein the input provided to the computing device via the first vision sensor is associated with a location of the first input with respect to the user interface image and a first type of infrared input signal; and

a second infrared input device that is configured to provide a second input to the computing device by emitting infrared light of a second wavelength within the second predetermined range that is received by the second vision sensor, wherein the input provided to the computing device via the second vision sensor is associated with a location of the second input with respect to the user interface image and a second type of infrared input signal,

wherein the computing device is further configured to execute the one or more applications based on at least one of

     1) the location of the first input with respect to the user interface image and the first type of infrared input signal and

     2) the location of the second input with respect to the user interface image and the second type of infrared input signal.

17. The interactive projection unit of claim 16, wherein the first infrared input device emits infrared light of a wavelength of approximately 850 nanometers and the second infrared input device emits infrared light of a wavelength of approximately 940 nanometers.