Image processing device, stereoscoopic device, integrated circuit, and program for determining depth of object in real space generating histogram from image obtained by filming real space and performing smoothing of histogram

• US 9,754,357 B2
• Filed: 03/05/2013
• Issued: 09/05/2017
• Est. Priority Date: 03/23/2012
• Status: Expired due to Fees

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: Nakamura, Tsuyoshi, Tezuka, Tadanori, Tanaka, Yoshiteru, Ohashi, Masahiro
• Primary Examiner(s): Rahaman, Mohammed
• Assistant Examiner(s): Boylan, James
• Application Number:
  US14/003,084
• Publication Number:
  US 20140071251A1
• Time in Patent Office: 1,645 Days
• Field of Search: 348 51, 382154
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

A histogram generator generates a histogram that indicates a count of pixels of a designated color in association with coordinates along a basic axis of a screen, for frame image data obtained by filming a real space. A histogram smoother performs smoothing of the generated histogram. A three-dimensional coordinates generator selects a value associated with particular coordinates from among the counts indicated in the smoothed histogram, and performs depth value determination for a target object using the selected value.

Claims

1. An image processing device determining a depth of a target object in real space by performing image processing, the image processing device comprising:

a generator generating an X-axis histogram for a designated color and a Y-axis histogram for the designated color, the X-axis histogram indicating a pixel count at each of a plurality of coordinates along an X-axis, the Y-axis histogram indicating a pixel count at each of a plurality of coordinates along a Y-axis, the X-axis and the Y-axis being axes of a screen, a pixel count at a given coordinate indicating the number of pixels having the designated color at the given coordinate in frame image data obtained by filming the real space;

a smoother performing smoothing of the X-axis histogram and the Y-axis histogram to acquire a smoothed X-axis histogram and a smoothed Y-axis histogram, respectively; and

a depth determiner determining three-dimensional coordinates of the target object using the smoothed X-axis histogram and the smoothed Y-axis histogram, wherein

the three-dimensional coordinates are expressed as (Xu, Yu, f(h1(Xu), h2(Yu))) when Xu is an X coordinate in the smoothed X-axis histogram, h1(Xu) is a pixel count at the coordinate Xu, Yu is a Y coordinate in the smoothed Y-axis histogram, h2(Yu) is a pixel count at the coordinate Yu, and f(h1(Xu), h2(Yu)) is a function for determining a Z coordinate from the pixel count h1(Xu) and the pixel count h2(Yu),

the pixel count h1(Xu) and the pixel count h2(Yu) determine a centroid intersecting at a center of the target object,

the function f(h1(Xu), h2(Yu)) determines the Z coordinate of the target object based on a length of the centroid determined by the X-axis histogram and the Y-axis histogram,

the smoothing for the smoothed X-axis histogram and the smoothed Y-axis histogram involves applying an impulse response convolution to pixel counts at the coordinates in the X-axis histogram and pixel counts at the coordinates in the Y-axis histogram,

the smoothed X-axis histogram includes a peak pixel count at the center along the X-axis and pixel counts approaching zero at edges along the X-axis, and the smoothed Y-axis histogram includes a peak pixel count at the center along the Y-axis and pixel counts approaching zero at edges along the Y-axis,

an impulse response I[n] used in the impulse response convolution is obtained by converting a frequency response H(w),

the impulse response I[n] is obtained by;

(Math

     1)

I[n]=DTFT<sup>−

1</sup>[H(w)]

where DTFT denotes a discrete-time Fourier transform, and

the impulse response continues from 0 to M in terms of the X coordinates and an output I[n]*x[n] of the impulse response convolution applied to an nth X coordinate x[n] among coordinates 0 through M is calculated as;

(Math

     2)

2. The image processing device of claim 1, wherein

the pair of the coordinate Xu and the coordinate Yu is either a pair composed of a coordinate associated with a maximum pixel count in the smoothed X-axis histogram and a coordinate associated with a maximum pixel count in the smoothed Y-axis histogram or a pair composed of a coordinate associated with a second-highest or lower pixel count in the smoothed X-axis histogram and a coordinate associated with a second-highest or lower pixel count in the smoothed Y-axis histogram.

3. The image processing device of claim 1, wherein

the designated color is designated by a combination of pixel components including a luminance component and a color-difference component,

the image processing device further comprises a pixel group extractor extracting, among pixels in the frame image data, a group of pixels that match the combination of the luminance component and the color-difference component, the extracted pixels associated with the plurality of coordinates along the X-axis and the plurality of coordinates along the Y-axis, and

the generator generates the X-axis histogram by associating, with each of the plurality of coordinates along the X-axis, a number of pixels in the pixel group that are associated with the coordinates along the X-axis and generates the Y-axis histogram by associating, with each of the plurality of coordinates along the Y-axis, a number of pixels in the pixel group that are associated with the coordinates along the Y-axis.

4. The image processing device of claim 1, wherein

the designated color is designated by a combination of pixel components including a plurality of primary color luminance components,

the image processing device further comprises a pixel group extractor extracting, among pixels in the frame image data, a group of pixels that match the combination of the primary color luminance components, the extracted pixels associated with the plurality of coordinates along the X-axis and the plurality of coordinates along the Y-axis, and

the generator generates the X-axis histogram by associating, with each of the plurality of coordinates along the X-axis, a number of pixels in the pixel group that are associated with the coordinates along the X-axis and generates the Y-axis histogram by associating, with each of the plurality of coordinates along the Y-axis, a number of pixels in the pixel group that are associated with the coordinates along the Y-axis.

5. The image processing device of claim 1, wherein

the target object in real space is an operation member for operating a stereoscopic object projecting from the screen through a stereoscopic effect of a stereoscopic viewing device, and

the three-dimensional coordinates (Xu, Yu, f(h1(Xu), h2(Yu))) are used by the stereoscopic viewing device for generating an event which changes behavior of the stereoscopic object.

6. The image processing device of claim 1, wherein

the designated color is a color of a colored substance that protrudes from a tip of an operation body when the operation body is manipulated by a user.

7. The image processing device of claim 1, wherein

processing including determining a size of a window by using the function f(h1(Xu), h2(Yu) by taking a width and the length of the centroid, mapping of the window at a location in the frame image data, and a process related to pixels surrounded by the window is repeated.

9. A stereoscopic viewing device used with the image processing device of claim 1, comprising:

an execution unit executing an application;

a playback unit performing playback of stereoscopic video in accordance with an instruction from the application; and

an event manager generating an event in response to a user operation, the event indicating the three-dimensional coordinates determined by the image processing device, wherein

the application causes the playback unit to change playback content of the stereoscopic viewing device according to the event.

10. The stereoscopic viewing device of claim 9, wherein

the stereoscopic video is one of a multi-view video stream and graphics rendered by the application, and

the change of the playback content is one of a change in the multi-view video stream and a change in the graphics that are played back.

8. An image processing device, comprising

a pixel group extractor extracting, among pixels in the frame image data, a group of pixels of a designated color from each of an X-axis and a Y-axis, the X-axis and the Y-axis being axes of a screen;

a smoother performing smoothing of pixel values of the pixels in the pixel group from each of the X-axis and the Y-axis to acquire a smoothed pixel group;

a generator generating an X-axis histogram indicating a pixel count at each of a plurality of coordinates along the X-axis and a Y-axis histogram indicating a pixel count at each of a plurality of coordinates along the Y-axis, a pixel count at a given coordinate indicating the number of pixels at the given coordinate in the smoothed pixel group; and

a depth determiner determining three-dimensional coordinates of the target object using the X-axis histogram and the Y-axis histogram, wherein

the three-dimensional coordinates are expressed as (Xu, Yu, f(h1(Xu), h2(Yu))) when Xu is an X coordinate in the X-axis histogram, h1(Xu) is a pixel count at the coordinate Xu, Yu is a Y coordinate in the Y-axis histogram, h2(Yu) is a pixel count at the coordinate Yu, and f(h1(Xu), h2(Yu)) is a function for determining a Z coordinate from the pixel count h1(Xu) and the pixel count h2(Yu),

the pixel count h1(Xu) and the pixel count h2(Yu) determine a centroid intersecting at a center of the target object,

the function f(h1(Xu), h2(Yu)) determines the Z coordinate of the target object based on a length of the centroid determined by the X-axis histogram and the Y-axis histogram,

the smoothing for the smoothed pixel group involves applying an impulse response convolution to the pixel values of the pixels in the pixel group from each of the X-axis and the Y-axis,

an impulse response I[n] used in the impulse response convolution is obtained by converting a frequency response H(w),

the impulse response I[n] is obtained by;

(Math

     1)

I[n]=DTFT<sup>−

1</sup>[H(w)]

where DTFT denotes a discrete-time Fourier transform, and

the impulse response continues from 0 to M in terms of the X coordinates and an output I[n]*x[n] of the impulse response convolution applied to an nth X coordinate x[n] among coordinates 0 through M is calculated as;

(Math

     2)

11. An integrated circuit determining a depth of a target object in real space by performing image processing, the integrated circuit comprising:

a generator generating an X-axis histogram for a designated color and a Y-axis histogram for the designated color, the X-axis histogram indicating a pixel count at each of a plurality of coordinates along an X-axis, the Y-axis histogram indicating a pixel count at each of a plurality of coordinates along a Y-axis, the X-axis and the Y-axis being axes of a screen, a pixel count at a given coordinate indicating the number of pixels having the designated color at the given coordinate in frame image data obtained by filming the real space;

a smoother performing smoothing of the X-axis histogram and the Y-axis histogram to acquire a smoothed X-axis histogram and a smoothed Y-axis histogram, respectively; and

a depth determiner determining three-dimensional coordinates of the target object using the smoothed X-axis histogram and the smoothed Y-axis histogram, wherein

the three-dimensional coordinates are expressed as (Xu, Yu, f(h1(Xu), h2(Yu))) when Xu is an X coordinate in the smoothed X-axis histogram, h1(Xu) is a pixel count at the coordinate Xu, Yu is a Y coordinate in the smoothed Y-axis histogram, h2(Yu) is a pixel count at the coordinate Yu, and f(h1(Xu), h2(Yu)) is a function for determining a Z coordinate from the pixel count h1(Xu) and the pixel count h2(Yu),

the pixel count h1(Xu) and the pixel count h2(Yu) determine a centroid intersecting at a center of the target object,

the function f(h1(Xu), h2(Yu)) determines the Z coordinate of the target object based on a length of the centroid determined by the X-axis histogram and the Y-axis histogram,

the smoothing for the smoothed X-axis histogram and the smoothed Y-axis histogram involves applying an impulse response convolution to pixel counts at the coordinates in the X-axis histogram and pixel counts at the coordinates in the Y-axis histogram,

the smoothed X-axis histogram includes a peak pixel count at the center along the X-axis and pixel counts approaching zero at edges along the X-axis, and the smoothed Y-axis histogram includes a peak pixel count at the center along the Y-axis and pixel counts approaching zero at edges along the Y-axis,

an impulse response I[n] used in the impulse response convolution is obtained by converting a frequency response H(w),

the impulse response I[n] is obtained by;

(Math

     1)

I[n]=DTFT<sup>−

1</sup>[H(w)]

where DTFT denotes a discrete-time Fourier transform, and

the impulse response continues from 0 to M in terms of the X coordinates and an output I[n]*x[n] of the impulse response convolution applied to an nth X coordinate x[n] among coordinates 0 through M is calculated as;

(Math

     2)

12. A non-transitory recording medium recording thereon an image processing program causing a computer to execute processing for determining a depth of a target object in real space by performing image processing, the image processing program causing the computer to execute:

generating an X-axis histogram for a designated color and a Y-axis histogram for the designated color, the X-axis histogram indicating a pixel count at each of a plurality of coordinates along an X-axis, the Y-axis histogram indicating a pixel count at each of a plurality of coordinates along a Y-axis, the X-axis and the Y-axis being axes of a screen, a pixel count at a given coordinate indicating the number of pixels having the designated color at the given coordinate in frame image data obtained by filming the real space;

smoothing the X-axis histogram and the Y-axis histogram to acquire a smoothed X-axis histogram and a smoothed Y-axis histogram, respectively; and

determining three-dimensional coordinates of the target object using the smoothed X-axis histogram and the smoothed Y-axis histogram, wherein

the three-dimensional coordinates are expressed as (Xu, Yu, f(h1(Xu), h2(Yu))) when Xu is an X coordinate in the smoothed X-axis histogram, h1(Xu) is a pixel count at the coordinate Xu, Yu is a Y coordinate in the smoothed Y-axis histogram, h2(Yu) is a pixel count at the coordinate Yu, and f(h1(Xu), h2(Yu)) is a function for determining a Z coordinate from the pixel count h1(Xu) and the pixel count h2(Yu),

the pixel count h1(Xu) and the pixel count h2(Yu) determine a centroid intersecting at a center of the target object,

the function f(h1(Xu), h2(Yu)) determines the Z coordinate of the target object based on a length of the centroid determined by the X-axis histogram and the Y-axis histogram,

the smoothing for the smoothed X-axis histogram and the smoothed Y-axis histogram involves applying an impulse response convolution to pixel counts at the coordinates in the X-axis histogram and pixel counts at the coordinates in the Y-axis histogram,

the smoothed X-axis histogram includes a peak pixel count at the center along the X-axis and pixel counts approaching zero at edges along the X-axis, and the smoothed Y-axis histogram includes a peak pixel count at the center along the Y-axis and pixel counts approaching zero at edges along the Y-axis,

an impulse response I[n] used in the impulse response convolution is obtained by converting a frequency response H(w),

the impulse response I[n] is obtained by;

(Math

     1)

I[n]=DTFT<sup>−

1</sup>[H(w)]

where DTFT denotes a discrete-time Fourier transform, and

the impulse response continues from 0 to M in terms of the X coordinates and an output I[n]*x[n] of the impulse response convolution applied to an nth X coordinate x[n] among coordinates 0 through M is calculated as;

(Math

     2)