Image pickup apparatus, information processing system, mat, and image generation method

• US 10,477,175 B2
• Filed: 01/29/2016
• Issued: 11/12/2019
• Est. Priority Date: 02/06/2015
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Ogasawara, Hidehiko, Nagatsuka, Norio, Numaguchi, Naoki, Segawa, Hiroyuki, Ohba, Akio
• Primary Examiner(s): Nguyen, Luong T
• Application Number:
  US15/547,707
• Publication Number:
  US 20180027222A1
• Time in Patent Office: 1,383 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/32 using local area network [LAN] connections
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63H 2200/00 Computerized interactive toys, e.g. dolls
• CPC Class Codes: A63H 33/04 Building blocks, strips, or similar building parts
• CPC Class Codes: A63H 33/22 Optical, colour, or shadow toys puzzles with optical elements A63F9/0613; kaleidoscopes G02B27/08
• CPC Class Codes: G03B 15/00 Special procedures for taking photographs; Apparatus therefor
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 1/4076 dependent on references outside the picture
• CPC Class Codes: H04N 1/4078 using gradational references, e.g. grey-scale test pattern analysis
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 17/02 for colour television signals
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 : Combination of two or more compensation controls
• CPC Class Codes: H04N 23/88 : for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 9/69 : for modifying the colour signals by gamma correction controlling camera response for colour signals H04N23/83
• CPC Class Codes: H04N 9/77 : Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase circuits for matrixing H04N9/67

Abstract

An image of a block or the like entering a space above a play mat is picked up by an image pickup apparatus, and an information processing apparatus detects and recognizes the block on the basis of a picture of the block in a picked up image and performs information processing. The play mat includes a play field that defines a detection space for the block, and a calibration chart including at least a plurality of regions of colors of different luminances. The image pickup apparatus adjusts an exposure time period, a gain value for each color component and a correction rule upon gamma correction on the basis of the picture of the calibration chart in the picked up image.

Claims

1. An image pickup apparatus that outputs data of a picked up image to be used for recognition of a real object, comprising:

a computer having a microprocessor coupled to a storage medium containing a computer program, which when executed by the microprocessor, causes the computer to implement the following functional units;

an output image production unit configured to produce data of a picked up image to be outputted from data of a non-processed image picked up using an image pickup element;

an adjustment processing unit configured to evaluate the picked up image and adjust at least one of an image pickup condition in the image pickup element and a processing condition in the output image production unit; and

a data sending out unit configured to output the data of the picked up image to an apparatus that recognizes the real object, wherein

the adjustment processing unit adjusts the image pickup condition and the processing condition based on color information of a picture of a calibration chart that is provided in a space of a recognition target and is configured from a plurality of color regions having luminances different from each other.

2. The image pickup apparatus according to claim 1, wherein the adjustment processing unit evaluates the luminance of a picture of the color regions of the calibration chart and adjusts the processing condition so as to expand the difference between a plurality of luminances corresponding to the luminance of the color of the real object.

3. The image pickup apparatus according to claim 2, wherein the adjustment processing unit determines a luminance range within which the difference is to be expanded based on a luminance distribution of the picture of the real object in the picked up image.

4. The image pickup apparatus according to claim 1, wherein the adjustment processing unit evaluates a color of a picture of a color region having a color same as a color of the real object in the calibration chart and adjusts the processing condition so as to expand a distance between a plurality of colors of different pictures in a color space.

5. The image pickup apparatus according to claim 2, wherein the adjustment processing unit adjusts, as the adjustment of the processing condition, a correction rule upon gamma correction.

6. The image pickup apparatus according to claim 1, wherein the adjustment processing unit adjusts a gain value of each color component such that a picture of a color region in the calibration chart, which has a color same as one of chromatic colors of the real object, has an achromatic color.

7. The image pickup apparatus according to claim 1, wherein the adjustment processing unit evaluates a luminance of a picture of a color region in the calibration chart, which has a given luminance, and adjusts the image pickup condition and the processing condition in a direction in which the luminance is adjusted to a target luminance determined in advance.

8. The image pickup apparatus according to claim 1, wherein the adjustment processing unit adjusts a white balance to a picture of a color region of an achromatic color in the calibration chart.

9. The image pickup apparatus according to claim 1, wherein the data sending out unit parallelly outputs data of a picked up image produced in accordance with a processing condition adjusted by the adjustment processing unit and data of a picked up image produced in accordance with a processing condition that omits at least part of the adjustment by the adjustment processing unit.

10. The image pickup apparatus according to claim 1, wherein the data sending out unit switches and outputs data of a picked up image produced in accordance with a processing condition adjusted by the adjustment processing unit and data of a picked up image produced in accordance with a processing condition that omits at least part of the adjustment by the adjustment processing unit in accordance with a request from an apparatus that recognizes the real object.

12. The image pickup apparatus according to claim 1, further comprising a mat that is placed in an image pickup space, wherein:

the mat includes a play field that defines a bottom region of the space in which, when the real object is placed into the space, the real object is determined as the recognition target; and

the mat includes the calibration chart.

13. The image pickup apparatus according to claim 12, wherein the calibration chart includes a color region having a luminance corresponding to a supposed color of the real object of the recognition target.

14. The image pickup apparatus according to claim 12, wherein

the mat includes a first mat that includes the play field and a second mat that includes the calibration chart, and

the second mat is placed and removably mounted on the first mat.

11. An information processing system including an image pickup apparatus that outputs data of a picked up image obtained by image pickup of a real object and an information processing apparatus that recognizes the real object using the picked up image, wherein

the image pickup apparatus includes a computer having a microprocessor coupled to a storage medium containing a computer program, which when executed by the microprocessor, causes the computer to implement the following functional units:

an output image production unit configured to produce data of a picked up image to be outputted from data of a non-processed image picked up using an image pickup element,

an adjustment processing unit configured to evaluate the picked up image and adjust at least one of an image pickup condition in the image pickup element and a processing condition in the output image production unit, and

a data sending out unit configured to output the data of the picked up image to the information processing apparatus, and

the adjustment processing unit adjusts the image pickup condition and the processing condition based on color information of a picture of a calibration chart that is provided in a space of a recognition target and is configured from a plurality of color regions having luminances different from each other, and

the information processing apparatus

recognizes a block by comparison between color information of the picture of the real object in the picked up image and color information of real objects registered already with each other and performs a process set for the block.

15. An image production method, by an image pickup apparatus, of producing data of a picked up image to be used for recognition of a real object, comprising:

producing data of a picked up image to be outputted from data of a non-processed image picked up using an image pickup element;

evaluating the picked up image and adjusting at least one of an image pickup condition in the image pickup element and a processing condition in the producing; and

outputting the data of the picked up image to an apparatus that recognizes the real object, wherein

the adjusting adjusts the image pickup condition and the processing condition based on color information of a picture of a calibration chart that is provided in a space of a recognition target and is configured from a plurality of color regions having luminances different from each other.

16. A non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to carry out actions to implement a function for producing data of a picked up image to be used for recognition of a real object, the computer program causing the computer to carry out the actions of:

producing data of a picked up image to be outputted from data of a non-processed image picked up using an image pickup element;

evaluating the picked up image and adjusting at least one of an image pickup condition in the image pickup element and a processing condition in the production function; and

outputting the data of the picked up image to an apparatus that recognizes the real object, wherein

the adjusting adjusts the image pickup condition and the processing condition based on color information of a picture of a calibration chart that is provided in a space of a recognition target and is configured from a plurality of color regions having luminances different from each other.