IMAGE PROCESSING DEVICE, STEREOSCOPIC DEVICE, INTEGRATED CIRCUIT, AND PROGRAM FOR DETERMINING DEPTH OF OBJECT IN REAL SPACE USING IMAGE PROCESSING

• US 20140071251A1
• Filed: 03/15/2013
• Published: 03/13/2014
• Est. Priority Date: 03/23/2012
• Status: Active Grant

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Nakamura, Tsuyoshi, Tezuka, Tadanori, Tanaka, Yoshiteru, Ohashi, Masahiro
• Granted Patent: US 9,754,357 B2
• Time in Patent Office: Days
• US Class Current: 348/51
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 5/002 Denoising; Smoothing noise processing or correction adapted to be used in an image pickup device containing and electronic image sensor H04N23/81, H04N25/60 - H04N25/67
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

A histogram generator generates a histogram that indicates a count of pixels of a designated colour in association with coordinates along a basic axis of a screen, for frame image data obtained by filming a real space. A histogram smoother performs smoothing of the generated histogram. A three-dimensional coordinates generator selects a value associated with particular coordinates from among the counts indicated in the smoothed histogram, and performs depth value determination for a target object using the selected value.

Claims

1. An image processing device determining a depth of a target object in real space by performing image processing, the image processing device comprising:

a generator generating a histogram for a designated colour in frame image data obtained by filming the real space, the histogram indicating each of a plurality of designated colour pixel counts at each of a plurality of coordinates in a reference axis of a screen;

a smoother performing smoothing of a generated histogram; and

a depth determiner selecting one among the designated colour pixel counts indicated in a smoothed histogram, a selected count being associated with a specific set of coordinates, and determining the depth of the target object using the selected count.

2. The image processing device of claim 1, wherein

the specific set of coordinates include a coordinate in the reference axis associated with a maximum count in the smoothed histogram, and a coordinate in the reference axis associated with a second-highest or lower count in the smoothed histogram.

3. The image processing device of claim 1, further comprising:

a registration table in which two or more designated colours serving as a basis for histogram generation are registered in advance; and

a histogram compositor that, when histograms have been generated for each of the two or more designated colours, sums the designated colour pixel counts for each of the designated colours at each of the coordinates and obtains a composite histogram indicating summed counts at each of the coordinates, wherein

the specific set of coordinates used by the depth determiner are coordinates associated with a count in the composite histogram that exceeds a predetermined threshold.

4. The image processing device of claim 3, wherein

one of the two or more designated colours is a specified colour having a weighting factor allocated thereto, and

the histogram compositor obtains the composite histogram by multiplying the designated colour pixel counts in the generated histogram of the specified colour by the weighting factor, and summing the result with the designated colour pixel counts at each of the coordinates.

5. The image processing device of claim 1, wherein

the target object in real space is an operation member having two or more designated colours,

the image processing device further comprises a rotation angle determiner determining a rotation angle of the operation member,

each histogram generated by the generator for the two or more designated colours is located at different coordinates in the reference axis, and

the rotation angle determiner determines the rotation angle of the operation member according to a difference value representing a separation, with respect to the reference axis, between maximum coordinates and minimum coordinates in the histogram for each of the designated colours.

6. The image processing device of claim 1, wherein

a designated colour is designated by a combination of luminance component and colour-difference component in a pixel,

the image processing device further comprises a pixel group extractor extracting a pixel group associated with a set of coordinates in the reference axis among pixels in the frame image data, the pixel group being a group of pixels matching the combination of luminance component and colour-difference component, and

the histogram generator generates the histogram by associating a count of extracted pixels with each of the coordinates in the reference axis.

7. The image processing device of claim 1, wherein

a designated colour is designated by a combination of a plurality of primary colour luminance components in a pixel,

the image processing device further comprises a pixel group extractor extracting a pixel group associated with a set of coordinates in the reference axis among pixels in the frame image data, the pixel group being a group of pixels matching the combination of primary colour luminance components, and

the histogram generator generates the histogram by associating a count of extracted pixels with each of the coordinates in the reference axis.

8. The image processing device of claim 1, wherein

the target object in real space is an operation member for operating a stereoscopic object projecting from the screen through a stereoscopic effect of a stereoscopic viewing device,

the reference axis includes an X axis and a Y axis of the frame image data,

the depth determined by the depth determiner is a Z coordinate of the operation member in terms of three-dimensional coordinates, and

the three-dimensional coordinates are used by the stereoscopic viewing device for generating an event in which behaviour of the stereoscopic object is changed.

9. The image processing device of claim 1, wherein

the smoothing is performed by applying an impulse response convolution to each of the coordinates appearing in the histogram.

10. An image processing device, comprising

a pixel group extractor extracting a pixel group among pixels of a specific designated colour in frame image data;

a smoother performing smoothing of pixel values for each of the pixels in the pixel group;

a generator generating a histogram indicating each of a plurality of designated colour pixel counts in a smoothed pixel group at each of a plurality of coordinates in a reference axis of a screen; and

a depth determiner selecting one among the designated colour pixel counts indicated in the histogram, a selected count being associated with a specific set of coordinates, and determining a depth of the target object using the selected count.

11. A stereoscopic viewing device used with an image processing device, comprising:

an execution unit executing an application;

a playback unit performing playback of stereoscopic video in accordance with an instruction from the application; and

an event manager generating an event in response to a user operation, the event indicating a depth produced by the image processing device, wherein

the application causes the playback unit to change playback content of the stereoscopic viewing device according to the event.

12. The stereoscopic viewing device of claim 11, wherein

the stereoscopic video is one of a multi-view video stream and graphics rendered by the application, and

the change of the playback content is switching the multi-view video streams or graphics subject to playback.

13. An integrated circuit determining a depth of a target object in real space by performing image processing, the integrated circuit comprising:

a generator generating a histogram for a designated colour in frame image data obtained by filming the real space, the histogram indicating each of a plurality of designated colour pixel counts at each of a plurality of coordinates in a reference axis of a screen;

a smoother performing smoothing of a generated histogram; and

a depth determiner selecting one among the designated colour pixel counts indicated in a smoothed histogram, a selected count being associated with a specific set of coordinates, and determining the depth of the target object using the selected count.

14. An image processing program causing a computer to execute processing for determining a depth of a target object in real space by performing image processing, the image processing program comprising:

generating a histogram for a designated colour in frame image data obtained by filming the real space, the histogram indicating each of a plurality of designated colour pixel counts at each of a plurality of coordinates in a reference axis of a screen;

smoothing a generated histogram; and

selecting one among the designated colour pixel counts indicated in a smoothed histogram, a selected count being associated with a specific set of coordinates, and determining the depth of the target object using the selected count.