Object-based color detection and correction

• US 9,304,582 B1
• Filed: 12/19/2013
• Issued: 04/05/2016
• Est. Priority Date: 12/19/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kamarshi, Vijay, Liu, Yue, Gopalan, Sowmya, Ramalho, Helder Jesus, Agrawal, Rahul, Flenniken, Robert Ramsey, Yao, Ning, Bombach, Paul Matthew
• Primary Examiner(s): Bost, Dwayne
• Assistant Examiner(s): Kohlman, Christopher
• Application Number:
  US14/134,721
• Time in Patent Office: 838 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics

Abstract

A system may utilize a projector, a camera, and a depth sensor to produce images within the environment of a user and to detect and respond to user actions. Depth data from the depth sensor may be analyzed to detect and identify items within the environment. A coordinate transformation may then be used to identify corresponding color values from camera data, which can in turn be analyzed to determine the colors of detected items. A similar coordinate transformation may be used to identify color values of a projected image that correspond to the detected items. In some cases, camera color values corresponding to an item may be corrected based on the corresponding color values of a projected image. In other cases, projected color values corresponding to an item may be corrected based on the corresponding camera color values.

Claims

1. A system, comprising:

a depth sensor configured to produce a depth image of a scene, the scene containing one or more surfaces, wherein the depth image indicates three-dimensional coordinates for the one or more surfaces;

a color camera configured to produce a camera image of the scene, the camera image comprising a camera color value for an individual one of two-dimensional camera coordinates;

a projector configured to project a display image onto the scene, the display image comprising a projector color value for an individual one of two-dimensional projector coordinates; and

operating logic configured to perform acts comprising;

obtaining coordinate transformations between the three-dimensional coordinates, the two-dimensional camera coordinates, and the two-dimensional projector coordinates;

analyzing the depth image to detect an item within the scene and to identify a set of the surface points that are formed by the detected item;

identifying a set of camera color values that correspond to the item based at least in part on the three-dimensional coordinates of surface points of the set of surface points and one or more of the coordinate transformations;

analyzing the set of camera color values to determine a color of the item; and

identifying a set of the projector color values that correspond to the item based at least in part on the three-dimensional coordinates of the surface points of the set of surface points and one or more of the coordinate transformations.

2. The system of claim 1, the acts further comprising correcting the set of projector color values based at least in part on the set of camera color values.

3. The system of claim 1, the acts further comprising correcting the set of camera color values based at least in part on the set of projector color values.

4. The system of claim 1, wherein the item comprises a display surface and the display image implement a user interface on the display surface.

5. A system comprising:

a depth sensor configured to produce a depth image of a scene, the depth image comprising three-dimensional coordinates for one or more surfaces of the scene;

a color camera configured to produce a camera image of the scene, the camera image comprising a camera color value for an individual one of two-dimensional camera coordinates;

a projector configured to project an image onto the scene, the image comprising a projector color value for an individual one of two-dimensional projector coordinates;

operating logic configured to perform acts comprising;

obtaining coordinate transformations between the three-dimensional coordinates, the two-dimensional camera coordinates, and the two-dimensional projector coordinates;

analyzing the depth image to identify a set of the surface points of an item within the scene;

applying one or more of the coordinate transformations to the three-dimensional coordinates of surface points of the set of surface points to identify a set of the camera color values that correspond to the item; and

analyzing the set of camera color values to determine a color of the item.

6. The system of claim 5,

wherein the acts further comprise:

applying one or more of the coordinate transformations to the three-dimensional coordinates of the surface points of the set of surface points to identify a set of the projector color values that correspond to the item; and

correcting the set of projector color values based at least in part on the set of camera color values.

7. The system of claim 5,

wherein the acts further comprise:

applying one or more of the coordinate transformations to the three-dimensional coordinates of the surface points of the set of surface points to identify a set of the projector color values that correspond to the item; and

correcting the set of camera color values based at least in part on the set of projector color values.

8. The system of claim 5, wherein at least one of the coordinate transformations is a function of intrinsic parameters of the depth sensor and color camera and extrinsic calibration parameters between the depth sensor and the color camera.

9. The system of claim 5, wherein the intrinsic parameters comprise one or more of:

focal lengths,

image formats,

principal points,

scaling factors, and

skew coefficients.

10. The system of claim 5, wherein the extrinsic parameters define relative positions and orientations of the depth sensor and the color camera.

11. The system of claim 5, wherein the item comprises a display surface, the acts further comprising:

projecting a user interface onto the display surface; and

color correcting the projected user interface based at least in part on the determined color of the display surface.

12. The system of claim 5, wherein the item comprises a tool for interacting with a user interface displayed on a display surface.

13. A method, comprising:

receiving a depth image of a scene, the depth image a comprising three-dimensional coordinates for one or more surfaces of the scene;

receiving a color image of the scene, the color image comprising a camera color value for an individual one of two-dimensional camera coordinates;

receiving two-dimensional coordinates associated with a display image projected onto the scene, the display image comprising a projector color value for an individual one of two-dimensional projector coordinates;

analyzing the depth image to identify a set of surface points of an item within the scene;

obtaining coordinate transformations between the three-dimensional coordinates, the two-dimensional camera coordinates, and the two-dimensional projector coordinates;

identifying a set of the camera color values that correspond to the item based at least in part on the identified set of surface points and one or more of the coordinate transformations; and

determining a color of the item based at least in part on the identified set of camera color values.

14. The method of claim 13, further comprising:

projecting the display image onto the scene; and

identifying a set of the projector color values that correspond to the item based at least in part on the identified set of surface points and one or more of the coordinate transformations.

15. The method of claim 13, wherein the item comprises a display surface, the method further comprising:

projecting a user interface onto the display surface, wherein the user interface comprises projector color values; and

color correcting the projector color values of the user interface based at least in part on the determined color of the display surface.

16. The method of claim 13, wherein the item comprises a display surface, the method further comprising:

projecting a display image onto the display surface, wherein the display image comprises projector color values; and

color correcting the identified set of the camera color values based at least in part on the projector color values.

17. The method of claim 13, wherein the item comprises a drawing tool for interacting with a user interface displayed on a display surface.

18. The method of claim 13, wherein one of the coordinate transformations is a function of intrinsic parameters of a depth sensor and a color camera and extrinsic calibration parameters between the depth sensor and the color camera.

19. The method of claim 13, wherein the intrinsic parameters comprise one or more of:

focal lengths,

image formats,

principal points,

scaling factors, and

skew coefficients.

20. The method of claim 13, wherein the extrinsic parameters define relative positions and orientations of the depth sensor and the color camera.