Video actuated interactive environment

• US 7,775,883 B2
• Filed: 11/05/2003
• Issued: 08/17/2010
• Est. Priority Date: 11/05/2002
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: McArdle, William Joseph, Smoot, Lanny Starkes
• Primary Examiner(s): Hotaling; John M
• Assistant Examiner(s): Kim; Kevin
• Application Number:
  US10/703,052
• Publication Number:
  US 20040102247A1
• Time in Patent Office: 2,477 Days
• Field of Search: 463 1- 5, 463 30- 34, 463/36, 463/39, 463 49- 51, 702/150
• US Class Current: 463/36
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1062 being specially adapted to a type of game, e.g. steering wheel
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/8011 Ball
• CPC Class Codes: A63F 2300/8023 the game being played by multiple players at a common site, e.g. in an arena, theatre, shopping mall using a large public display
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: G02B 26/0816 by means of one or more reflecting elements
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G07F 17/32 for games, toys, sports, or amusements

Abstract

A surface-based interactive environment that utilizes low cost equipment to create a highly interactive environment for participants. The game can include electronic versions of traditional games, such as air hockey, football, soccer, and the like. The game can also include one or more electronically generated characters with whom the participants can interact. The system utilizes a high-mounted projector to create the game environment and/or generate the characters. A remotely mounted infrared light source and infrared camera are used to detect the participants as they move around within the interaction area of the game.

Claims

1. A video-actuated system for creating an interactive environment on an interaction surface for a participant, comprising:

at least one energy source capable of generating electromagnetic energy in a relatively limited wavelength range, wherein the at least one energy source is positioned such that the energy from the at least one energy source strikes at least a portion of the interaction surface;

at least one electromagnetic imaging device, wherein the at least one electromagnetic imaging device is capable of detecting electromagnetic energy in at least the relatively limited wavelength range generated by the at least one energy source and converting the detected energy into an output video signal including a plurality of frames, and wherein the at least one electromagnetic imaging device is positioned such that the at least one electromagnetic imaging device can detect the electromagnetic energy from at least a portion of the interaction surface;

at least one visible light projector having at least one input, wherein the at least one projector is capable of projecting visible light onto the interaction surface;

at least one computer; and

at least one computer program process operating on the at least one computer, capable of receiving the output video signal from the electromagnetic imaging device, detecting blobs of infrared energy within the output signal by scanning each pixel in each frame of the output signal and setting pixel intensity in each frame to zero for each pixel having an intensity less that a predetermined threshold, reporting information about the blobs, processing the blob information, and generating an input to the at least one visible light projector based on the processed blob information, wherein the blob information includes coordinates of a bounding box for each blob, determined by;

setting each pixel at or above the threshold as a blob pixel and each pixel below the threshold as a zeroed pixel,

determining a top and bottom edge of each blob by selecting a first blob pixel, moving up and down a frame one row at a time in a same column as the selected first blob pixel until a zeroed pixel is encountered, and saving coordinates of each last encountered blob pixel in the same column;

calculating and saving coordinates of a vertical mid-point pixel between the coordinates of the last encountered blob pixels in the same column;

setting the vertical mid-point pixel as a next blob pixel;

determining a right and left edge of each blob by moving left and right in the frame one column at a time in the same row as the next blob pixel until a zeroed pixel is encountered, and saving coordinates of the last encountered blob pixels in the same row;

calculating and saving coordinates of a horizontal mid-point pixel between the coordinates of the last encountered blob pixels in the same row;

determining a top and bottom edge of the blob by moving up and down the frame one row at a time in the same column as the horizontal midpoint pixel until a zeroed pixel is encountered, and saving the coordinates of the last encountered blob pixels in that column; and

calculating and saving the bounding box coordinates of each blob from an intersection of a vertical line through the horizontal midpoints and a horizontal line through the vertical midpoints as above determined.

2. The system of claim 1, further comprising at least one relatively narrow wavelength pass filter, wherein the at least one pass filter allows electromagnetic energy substantially equivalent to the relatively narrow wavelength range generated by the energy source to pass through effectively unattenuated while attenuating electromagnetic energy at other wavelengths, and wherein the at least one pass filter is positioned substantially between the at least one imaging device and the interaction surface, such that substantially all electromagnetic energy entering the at least one imaging device passes through at least one pass filter.

3. The system of claim 1, further comprising at least one relatively narrow wavelength stop filter, wherein the at least one stop filter attenuates electromagnetic energy substantially equivalent to the relatively narrow wavelength range generated by the electromagnetic energy source while allowing other electromagnetic wavelengths to pass through effectively unattenuated, and wherein the at least one stop filters are positioned substantially between the at least one visible light projector and the surface, such that substantially all light from the at least one visible light projector passes through the at least one stop filter.

4. The system of claim 1, further comprising:

at least one relatively narrow wavelength pass filter, wherein the at least one pass filter allows electromagnetic energy substantially equivalent to the relatively narrow wavelength range generated by the energy source to pass through effectively unattenuated while attenuating electromagnetic energy at other wavelengths, and wherein the at least one pass filter is positioned substantially between the at least one imaging device and the interaction surface, such that substantially all electromagnetic energy entering the at least one imaging device passes through at least one pass filter; and

at least one relatively narrow wavelength stop filter, wherein the at least one stop filter attenuates electromagnetic energy substantially equivalent to the relatively narrow wavelength range generated by the electromagnetic energy source while allowing other electromagnetic wavelengths to pass through effectively unattenuated, and wherein the at least one stop filters are positioned substantially between the at least one visible light projector and the interaction surface, such that substantially all light from the at least one visible light projector passes through the at least one stop filter.

5. The system of claim 1, wherein the energy source generates electromagnetic energy with wavelengths in the infrared range.

6. The system of claim 1, wherein the interaction surface is comprised of a floor, and the energy source is positioned substantially above the interaction surface.

7. The system of claim 1, wherein the electromagnetic imaging device is a grayscale CCD camera.

8. The system of claim 1, wherein the interaction surface is comprised of a floor, and the imaging device is positioned substantially above the interaction surface.

9. The system of claim 1, wherein blob detection occurs on a frame-by-frame basis.

10. The system of claim 1, wherein blob detection is based on at least one customizable threshold brightness level.

11. The system of claim 1, wherein blob detection is based on at least one customizable blob size threshold.

12. The system of claim 11, wherein blob detection is based on at least one customizable threshold brightness level.

13. The system of claim 1, wherein blob detection is controllable based on a customizable granularity setting.

14. The system of claim 1, wherein a target is associated with the at least one participant.

15. The system of claim 14, wherein the target is a retro-reflective target.

16. The system of claim 1, wherein a plurality of targets are associated with each of the at least one participants.

17. The system of claim 16, wherein three targets of a known orientation are associated to each of the at least one participants.

18. The system of claim 17, wherein the blob information includes the size and location of each blob.

19. The system of claim 18, wherein the blob information is used to determine the orientation of each of the at least one the participants.

20. The system of claim 19, wherein the orientation of at least one of the at least one participants is used by the at least one computer program process to generate the input to the visual light projector.

21. The system of claim 17, wherein the blob information is used to determine the height of a prop above the interaction surface.

22. The system of claim 17, wherein the blob information is used to determine the scale of at least a portion of the image projected by the visual light projector.

23. The system of claim 1, wherein an electromagnetic energy source is assigned to each of the at least one participants.

24. The system of claim 1, wherein a pan and tilt mirror is placed in the optical path of the visible light projector.

25. The system of claim 1, wherein the visible light projector projects light encoded using a three dimensional display technique.

26. The system of claim 1, wherein the electromagnetic energy source projects electromagnetic energy onto at least a portion of the interaction surface from a position different from that from which the at least one electromagnetic imaging device detects the at least a portion of the interaction surface such that at least one of the at least one participants appears to cast a shadow on the at least a portion of the interaction surface when the at least a portion of the interaction surface is detected by the at least one electromagnetic imaging device.

27. The system of claim 26, wherein the blob detection algorithm determines the position of at least one participant shadow within the at least a portion of the interaction surface detected by the at least one electromagnetic imaging device.

28. The system of claim 27, wherein the at least one computer program process causes the at least one visible light projector to project an animated character in the region of the at least one participant shadow.

29. The system according to claim 1 wherein:

the interaction surface is comprised of a floor;

at least one retro-reflective target is associated with the at least one participant;

the at least one energy source is an infrared energy source, wherein the infrared energy source is positioned substantially above the interaction surface and capable of projecting infrared energy onto the interaction surface;

the at least one electromagnetic imaging device is at least one grayscale CCD camera, wherein the camera is capable of detecting the infrared energy generated by the infrared energy source and converting the detected energy into an output signal, and wherein the at least one camera is positioned substantially above the interaction surface;

the system further comprising at least one relatively narrow wavelength pass filter, wherein the at least one pass filter allows infrared energy from the energy source to pass through effectively unattenuated while attenuating electromagnetic energy at other wavelengths, and wherein the at least one pass filter is positioned substantially between the at least one camera and the interaction surface, such that substantially all electromagnetic energy entering the at least one camera passes through at least one pass filter;

at least one relatively narrow wavelength stop filter, wherein the at least one stop filter attenuates infrared energy of a wavelength substantially equivalent to that generated by the electromagnetic energy source while allowing other electromagnetic wavelengths to pass through effectively unattenuated, and wherein the at least one stop filters are positioned substantially between the at least one visible light projector and the interaction surface, such that substantially all light from the at least one visible light projector passes through the at least one stop filter.

30. The system of claim 29, wherein blob detection occurs on a frame-by-frame basis.

31. The system of claim 29, wherein blob detection is based on at least one customizable threshold brightness level.

32. The system of claim 29, wherein blob detection is based on at least one customizable blob size threshold.

33. The system of claim 32, wherein blob detection is based on at least one customizable threshold brightness level.

34. The system of claim 29, wherein blob detection is controllable based on a customizable granularity setting.

35. The system of claim 29, wherein a target is associated with the at least one participant.

36. The system of claim 35, wherein the target is a retro-reflective target.

37. The system of claim 29, wherein a plurality of targets are associated with each of the at least one participants.

38. The system of claim 37, wherein three targets of a known orientation are associated to each of the at least one participants.

39. The system of claim 38, wherein the blob information includes the size and location of each blob.

40. The system of claim 39, wherein the blob information is used to determine the orientation of each of the at least one the participants.

41. The system of claim 40, wherein the orientation of at least one of the at least one participants is used by the at least one computer program process to generate the input to the visual light projector.

42. The system of claim 29, wherein the visible light projector includes a pan and tilt mirror.

43. The system of claim 29, wherein the visible light projector projects light encoded using a three dimensional display technique.

44. A video-actuated system for creating an interactive environment for at least one participant with respect to an interaction surface, comprising:

at least one imaging device, for generating an output video signal representative of at least a portion of the interaction surface;

a visible light projector having at least one input, wherein the projector is capable of projecting visible light onto only a subset of the interaction surface, wherein the interaction surface and the visible light projector are positioned such that the participant is located substantially between the interaction surface and the visible light projector;

a pan and tilt mirror system placed in the optical path of the projector, wherein the pan and tilt mirror system is capable of directing the output from the visible light projector to a desired position;

at least one computer;

at least one computer program process operating on the at least one computer, capable of receiving the output video signal from the imaging device, detecting the position of the participant within an area on the interaction surface by scanning each pixel in each frame of the video output signal and setting pixel intensity in each frame to zero for each pixel having an intensity less that a predetermined threshold and detecting the at least one participant position by determining a blob position, generating a signal for projecting a character in the proximity of the at least one participant, providing the signal to the input of the visible light projector, and providing approximately the detected position of the participant via the blob position to the pan and tilt mirror system, wherein the blob position includes coordinates of a bounding box for each blob, determined by;

setting each pixel at or above the threshold as a blob pixel and each pixel below the threshold as a zeroed pixel,

determining a top and bottom edge of each blob by selecting a first blob pixel, moving up and down a frame one row at a time in a same column as the selected first blob pixel until a zeroed pixel is encountered, and saving coordinates of each last encountered blob pixel in the same column;

calculating and saving coordinates of a vertical mid-point pixel between the coordinates of the last encountered blob pixels in the same column;

setting the vertical mid-point pixel as a next blob pixel;

determining a right and left edge of each blob by moving left and right in the frame one column at a time in the same row as the next blob pixel until a zeroed pixel is encountered, and saving coordinates of the last encountered blob pixels in the same row;

calculating and saving coordinates of a horizontal mid-point pixel between the coordinates of the last encountered blob pixels in the same row;

determining a top and bottom edge of the blob by moving up and down the frame one row at a time in the same column as the horizontal midpoint pixel until a zeroed pixel is encountered, and saving the coordinates of the last encountered blob pixels in that column; and

calculating and saving the bounding box coordinates of each blob from an intersection of a vertical line through the horizontal midpoints and a horizontal line through the vertical midpoints as above determined.

45. The system of claim 44, wherein the pan and tilt mirror system allows the generated character to follow the participant across the interaction surface.

46. A video-actuated system for creating an interactive environment on an interaction surface for a participant, comprising:

at least one energy source capable of generating electromagnetic energy in a relatively limited wavelength range, wherein the at least one energy source is positioned such that the energy from the at least one energy source strikes at least a portion of the interaction surface;

at least one electromagnetic imaging device, wherein the at least one electromagnetic imaging device is capable of detecting electromagnetic energy in at least the relatively limited wavelength range generated by the at least one energy source and converting the detected energy into an output video signal including a plurality of frames, and wherein the at least one electromagnetic imaging device is positioned such that the at least one electromagnetic imaging device can detect the electromagnetic energy from at least a portion of the interaction surface;

at least one visible light projector having at least one input, wherein the at least one projector is capable of projecting visible light onto the interaction surface;

at least one computer; and

at least one computer program process operating on the at least one computer, capable of receiving the output video signal from the electromagnetic imaging device, detecting blobs of infrared energy within the output signal by scanning each pixel in each frame of the output signal and setting pixel intensity in each frame to zero for each pixel having an intensity less that a predetermined threshold, reporting information about the blobs, processing the blob information, and generating an input to the at least one visible light projector based on the processed blob information, wherein the blob information includes coordinates of a midpoint for each blob, wherein the midpoint is determined by;

scanning each pixel to determine a top and bottom edge of each blob;

saving the coordinates along the top and bottom edge of each blob;

calculating the vertical midpoint of each blob from the coordinates of the top and bottom edge of each blob;

scanning each pixel to determine a right and left edge of each blob;

saving the coordinates along the right and left edge of each blob;

calculating the horizontal midpoint of each blob from the coordinates of the right and left edge of each blob; and

calculating the midpoint of each blob from an intersection of a vertical line through the horizontal midpoint and a horizontal line through the vertical midpoint.

47. A video-actuated system for creating an interactive environment for at least one participant with respect to an interaction surface, comprising:

at least one imaging device, for generating an output video signal representative of at least a portion of the interaction surface;

a visible light projector having at least one input, wherein the projector is capable of projecting visible light onto only a subset of the interaction surface, wherein the interaction surface and the visible light projector are positioned such that the participant is located substantially between the interaction surface and the visible light projector;

a pan and tilt mirror system placed in the optical path of the projector, wherein the pan and tilt mirror system is capable of directing the output from the visible light projector to a desired position;

at least one computer;

at least one computer program process operating on the at least one computer, capable of receiving the output video signal from the imaging device, detecting the position of the participant within an area on the interaction surface by scanning each pixel in each frame of the video output signal and setting pixel intensity in each frame to zero for each pixel having an intensity less that a predetermined threshold and detecting the at least one participant position by determining a blob position, generating a signal for projecting a character in the proximity of the at least one participant, providing the signal to the input of the visible light projector, and providing approximately the detected position of the participant via the blob position to the pan and tilt mirror system, wherein the blob information includes coordinates of a midpoint for each blob, wherein the midpoint is determined by;

scanning each pixel to determine a top and a bottom edge of each blob;

saving the coordinates along the top and the bottom edge of each blob;

calculating the vertical midpoint of each blob from the coordinates of the top and the bottom edge of each blob;

scanning each pixel to determine a right and a left edge of each blob;

saving the coordinates along the right and the left edge of each blob;

calculating the horizontal midpoint of each blob from the coordinates of the right and the left edge of each blob; and

calculating the midpoint of each blob from an intersection of a vertical line through the horizontal midpoint and a horizontal line through the vertical midpoint.