Method and apparatus for optimizing capture device settings through depth information

• US 7,663,689 B2
• Filed: 01/16/2004
• Issued: 02/16/2010
• Est. Priority Date: 01/16/2004
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard L.
• Primary Examiner(s): Ho; Tuan
• Assistant Examiner(s): WANG, KENT F
• Application Number:
  US10/758,817
• Publication Number:
  US 20050157204A1
• Time in Patent Office: 2,223 Days
• Field of Search: 348/42, 348/222.1, 348/223.1, 348/224.1, 348/335, 348/362, 348/363, 348/370, 348/371, 345/462, 345/611, 345/839, 375/240.08, 382/107, 382/115, 382/154, 382/173, 382/203, 382/283
• US Class Current: 348/370
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method for adjusting image capture settings for an image capture device is provided. The method initiates with identifying a scene. Then, an image of the scene is captured. The method includes generating a depth mask of the scene from data defining the image of the scene. Then, pixel values corresponding to objects within any one or both of a foreground region and a background region of the captured image are adjusted based upon bit values of the depth mask. An image capture device and a system are provided.

Claims

1. A method for differentiating between foreground objects and background objects within a scene being captured through an image capture device, comprising:

emitting a ray of light from a light source toward an object of the scene;

opening an aperture cover allowing access to a sensor of the image capture device for reflected light from the light source;

closing the aperture cover after a set time, the predefined amount of time corresponding to a maximum distance traveled by the light;

generating a depth mask identifying objects within a foreground region and a background region of the scene based upon the light captured during the set time, the depth mask identifying objects within the foreground region with a first bit value and identifying objects within the background region with a second bit value;

adjusting an image capture device parameter according to bit values of the depth mask for one or more of a sequence of captured image frames of the scene, wherein the image capture device parameter is selected from one of brightness, exposure or gain;

applying the adjusted image capture device parameter so that the brightness, exposure, or gain of the captured image frames is adjustable independently for both of the objects within the foreground region and the objects within the background region; and

tracking an object within the foreground region, the tracking being enhanced by the applied adjusted image capture device parameter.

2. The method of claim 1, further comprising:

storing the depth mask in memory of the image capture device.

3. The method of claim 1, wherein the light source is configured to emit infrared light.

4. The method of claim 1, wherein the method operation of opening an aperture cover allowing access to a sensor of the image capture device includes,

receiving reflected light from the objects within the foreground region.

5. The method of claim 1, wherein the method operation of adjusting image capture device parameters according to bit values of the depth mask includes,

determining an optimal amount of light based upon the depth mask;

and

adjusting the aperture cover to allow the optimal amount of light into the image capture device.

6. The method of claim 1, wherein the image capture device parameters are adjusted through mechanical adjustments.

7. The method operation of claim 1, wherein the method operation of emitting a ray of light from a light source toward an object of the scene includes,

pulsing infrared light from the light source.

8. The method of claim 1, further comprising, adjusting a focus parameter of the image capture device according to the bit values of the depth mask, so that the objects within the foreground region are in focus.

9. The method of claim 1, wherein the adjusted captured image frames facilitates tracking of the objects within the foreground region by reducing brightness of the background region.

10. A method for adjusting image capture settings for an image capture device, comprising:

identifying a scene;

capturing an image of the scene through the image capture device;

generating a depth mask of the scene from data defining the image of the scene;

adjusting pixel values of the data defining the image corresponding to objects within any one or both of a foreground region and a background region of the captured image, wherein the adjusting of pixel values is according to bit values of the depth mask, wherein the adjusting pixel values causes a change in brightness such that both of the pixel values associated with the foreground region and pixel values associated with the background region are adjustable independently for brightness; and

tracking an object within the foreground region, the tracking being enhanced by the adjusting of pixel values.

11. The method of claim 10, wherein the method operation of generating a depth mask of the scene from data defining the image of the scene includes,

segmenting the foreground and background regions of the scene.

12. The method of claim 10, wherein the data defining the image of the scene includes pixel data where each pixel is tagged with distance information.

13. The method of claim 10, wherein the image capture device is selected from the group consisting of a digital camera, a web cam, and a camcorder.

14. The method of claim 10, further comprising:

displaying a portion of the image of the scene having adjusted pixel values.

15. The method of claim 14, wherein the portion of the image of the scene is an image of a participant for use in an interactive gaming application.

16. The method of claim 10, wherein the adjusting of pixel values facilitates tracking of the objects within the foreground region by reducing brightness of the background region.

17. The method of claim 10, wherein the foreground and background regions define two layers of the scene;

wherein the scene includes additional layers; and

wherein the method operation of adjusting pixel values is performed for data defining the image corresponding to objects within any one of the layers of the scene.

18. An image capture device configured to provide an image of a scene, comprising:

depth logic configured to provide a depth mask associated with the scene, the depth mask configured to distinguish between foreground objects and background objects within the scene;

image capture logic configured to adjust an image capture device setting for a characteristic associated with the image based upon a corresponding bit value of the depth mask, wherein the bit value determines whether the respective pixel is associated with one of the foreground objects and the background objects, and the image capture logic is configured to apply the adjusted image capture device setting so that the pixels of the image associated with both of the foreground objects and the background objects are adjustable independently for the characteristic,

wherein the characteristic is selected from a group consisting of exposure, gain, and brightness; and

tracking logic configured to track an object within the foreground region, the tracking being enhanced by the applied adjusted image capture device setting.

19. The image capture device of claim 18, wherein the depth mask is a bit mask having a first logical value assigned to represent the foreground objects and a second logical value assigned to represent the background objects.

20. The image capture device of claim 18, further comprising:

a sensor in communication with the depth logic, the sensor configured to receive a light signal reflected from one of the foreground objects, the receipt of the light signal indicating a location corresponding to one of the foreground objects.

21. The image capture device of claim 18, wherein each logic element is one or a combination of hardware and software.

22. The image capture device of claim 18, wherein the image capture device is a video capture device.

23. The image capture device of claim 22, wherein the depth logic is further configured to periodically provide a depth mask for a sequence of video frames captured by the video capture device.

24. The image capture device of claim 18, wherein the image capture device setting is adjusted through one of a mechanical or electrical adjustment.

25. The image capture device of claim 18, wherein the image capture logic is further configured to adjust each pixel of image data of the scene.

26. The method of claim 18, further comprising,

image capture logic configured to adjust an image capture device setting for focus based upon a corresponding bit value of the depth mask, so that the foreground objects are in focus.

27. The method of claim 18, wherein the adjusted pixels of the image facilitates tracking of the foreground objects by reducing brightness of the background objects.

28. A system, comprising:

a computing device;

a display screen in communication with the computing device, the display screen configured to display an image of a scene;

a video capture device in communication with the computing device, the video capture device providing scene image data to the computing device for presentation on the display screen, the video capture device including,

depth logic configured to provide a depth mask associated with the scene, the depth mask configured to distinguish between foreground objects and background objects within the scene;

image capture logic configured to adjust an image capture device setting for a characteristic associated with each pixel of the image data based upon depth information;

image capture logic configured to apply the adjusted image capture device setting so that each pixel of the image is adjustable independently of another pixel for the characteristic, whether associated with the foreground objects or the background objects, and wherein the characteristic is selected from the group consisting of exposure, gain, and brightness; and

tracking logic configured to track an object within the foreground region, the tracking being enhanced by the applied adjusted image capture device setting.

29. The system of claim 28, wherein the computing device is a game console.

30. The system of claim 29, wherein the scene image data includes an image of a person, the image of the person being incorporated into a video game for interaction therein.

31. The system of claim 28, wherein the depth logic is further configured to periodically provide a single depth mask for a sequence of video frames captured by the video capture device.

32. The system of claim 28, wherein the image capture device setting is adjusted through one of a mechanical or electrical adjustment.

33. The system of claim 28, wherein the video capture device is a webcam.

34. The system of claim 28, wherein the image data defines data for each pixel, the data for each pixel including distance information.

35. The system of claim 28, wherein the depth information is obtained from a depth mask, the depth mask defining a relative distance between an object associated with the corresponding pixel and the video capture device.

36. The system of claim 28, further comprising,

image capture logic configured to adjust an image capture device setting for focus based upon depth information, so that the foreground objects are in focus.

37. The system of claim 28, wherein adjusted pixels of the image facilitates tracking of the foreground objects by reducing brightness of the background objects.