Integrated near field sensor for display devices

• US 9,652,083 B2
• Filed: 03/04/2016
• Issued: 05/16/2017
• Est. Priority Date: 03/28/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Baldwin, Leo B.
• Primary Examiner(s): Yodichkas, Aneeta
• Application Number:
  US15/061,868
• Publication Number:
  US 20160259483A1
• Time in Patent Office: 438 Days
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A transmissive element including a diffractive and/or refractive pattern can be used with a display screen of a computing device to allow the display screen to function as a near field optical sensor (NFOS). The NFOS can be used to detect shadows cast on the display screen, and/or bright objects detected near the display screen, which can be used to determine the relative position of one or more features with respect to the device. These features can be, for example, the fingers or thumbs of a user attempting to provide input to the computing device. The device can support motion or gesture detection using cameras of the device, and touch input using touch screen functionality, but the NFOS can enable the device to also track motion in the dead zone between the field of view of the cameras and the touch screen.

Claims

1. A computing device, comprising:

a processor;

a display screen that includes a transmissive layer having a diffractive pattern configured to direct at least a portion of light received over a plurality of angles incident on a plurality of regions of the display screen to a plurality of light sensors near at least one edge of the transmissive layer; and

memory including instructions that, when executed by the processor, cause the computing device to;

detect intensity values of the at least the portion of light incident on the plurality of regions of the display screen using the plurality of light sensors;

analyze variations in the intensity values caused by a shadow on the display screen to determine a difference between a first intensity value corresponding to a first region of the display screen and a second intensity value corresponding to at least a second region of the display screen; and

determine a position with respect to the computing device for one or more objects that cast the shadow on the display screen based on one or more characteristics of the first intensity value.

2. The computing device of claim 1, further comprising:

at least one camera operable to capture image information within a field of view, and

wherein the instructions, when executed, further cause the computing device to track movement of the one or more objects based on the variations in the intensity values when the one or more objects is outside the field of view of the camera.

3. The computing device of claim 1, further comprising:

at least one camera operable to capture image information within a field of view, and

wherein the instructions, when executed, further cause the computing device to determine the position with respect to the computing device for the one or more objects based on the image information.

4. The computing device of claim 1, wherein the instructions, when executed, further cause the computing device to:

determine at least one of a size, or a shape of the one or more objects based on the variations in the intensity values caused by the shadow on the display screen.

5. The computing device of claim 1, wherein the instructions, when executed, further cause the computing device to:

track the position for the one or more objects over time; and

determine at least one of a motion input, a gesture input, or a position input from a user based on the position of the one or more objects over time.

6. The computing device of claim 1, further comprising at least one camera, wherein the instructions, when executed, further cause the computing device to:

determine at least one of a size, or a shape of the one or more objects based on image information captured by the at least one camera; and

determine the position for the one or more objects based at least in part on the at least one of the size or the shape of the one or more objects.

7. The computing device of claim 1, wherein the display screen further includes an active display layer and at least one light source associated with the active display layer.

8. The computing device of claim 1, wherein the transmissive layer is formed from a plastic sheet, and wherein the diffractive pattern is embossed in the plastic sheet.

9. The computing device of claim 1, wherein the transmissive layer is further configured to function as a brightness enhancing film.

10. A non-transitory computer-readable storage medium for storing instructions, the instructions, when executed by a processor of a computing device, cause the computing device to:

detect, using one or more light sensors, intensity values for light received over a plurality of angles on a display screen of the computing device, the light directed to the one or more light sensors by a diffractive pattern of the display screen;

analyze variations in the intensity values caused by a shadow on the display screen to determine a difference between a first intensity value corresponding to a first region of the display screen and a second intensity value corresponding to at least one other region of the display screen; and

determine a position with respect to the computing device for one or more objects that cast the shadow on the display screen based at least in part on one or more characteristics of the first intensity value.

11. The non-transitory computer-readable storage medium of claim 10, wherein the instructions, when executed, further cause the computing device to:

capture image information by at least one camera of the computing device, the image information represents the one or more objects within a field of view of the at least one camera located at a distance from the display screen; and

determine the position of the one or more objects based on the variations in the intensity values when the one or more objects are outside the field of view of the camera.

12. The non-transitory computer-readable storage medium of claim 10, wherein the instructions, when executed, further cause the computing device to:

determine at least one of a size, or a shape of the one or more objects based on the variations in the intensity values caused by the shadow on the display screen.

13. The non-transitory computer-readable storage medium of claim 10, wherein the instructions, when executed, further cause the computing device to:

track the position for the one or more objects over time; and

determine at least one of a motion input, a gesture input, or a position input from a user based on the position of the one or more objects over time.

14. The non-transitory computer-readable storage medium of claim 10, wherein the instructions, when executed, further cause the computing device to:

capture image information using at least one camera of the computing device;

determine at least one of a size, a shape, or a location of the one or more objects based on the image information; and

determine the position with respect to the computing device for the one or more objects based on the at least one of the size, the shape, or the location of the one or more objects.

15. The non-transitory computer-readable storage medium of claim 10, wherein at least one of the one or more objects includes one of a finger, a thumb, a hand, an arm, a pen, or a stylus.

16. A method, comprising:

detecting, by one or more light sensors of a computing device, intensity values from light received over a plurality of angles on a display screen of the computing device, the light directed to the one or more light sensors by a diffractive pattern of the display screen;

analyzing, by a processor of the computing device, variations in the intensity values caused by a shadow on the display screen to determine a difference between a first intensity value corresponding to a first region of the display screen and a second intensity value corresponding to at least one other region of the display screen; and

determining, by the processor, a position with respect to the computing device for one or more objects that cast the shadow on the display screen based at least in part on one or more characteristics of the first intensity value.

17. The method of claim 16, further comprising:

tracking movement of the one or more objects based on the variations in the intensity values when the one or more objects is outside a field of view of a camera of the computing device.

18. The method of claim 16, further comprising:

determining at least one of a size, or a shape of the one or more objects based on the variations in the intensity values caused by the shadow on the display screen.

19. The method of claim 16, further comprising:

tracking, by the processor, the position for the one or more objects over time; and

determining, by the processor, at least one of a motion input, a gesture input, or a position input from a user based on the position of the one or more objects over time.

20. The method of claim 16, further comprising:

capturing image information within a field of view of at least one camera of the computing device, the image information represents the one or more objects located at a distance from the display screen; and

determining the position for the one or more objects based on the image information.