Mobile device with display overlaid with at least a light sensor

• US 10,432,872 B2
• Filed: 02/24/2017
• Issued: 10/01/2019
• Est. Priority Date: 10/30/2015
• Status: Active Grant

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Evans, V, David John, Jiang, Xinrui, Rubin, Andrew E., Hershenson, Matthew, Miao, Xiaoyu, Tate, Joseph Anthony, Gagne-Keats, Jason Sean, Zavin, Rebecca Schultz
• Primary Examiner(s): Mazumder, Tapas
• Application Number:
  US15/442,576
• Publication Number:
  US 20170171448A1
• Time in Patent Office: 949 Days
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1637 Details related to the display arrangement, including those related to the mounting of the display in the housing
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G09G 2320/0686 with two or more screen areas displaying information with different brightness or colours
• CPC Class Codes: G09G 2330/027 Arrangements or methods related to powering off a display
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 2370/025 LAN communication management
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 3/2003 Display of colours specific for liquid crystal displays G09G3/3607
• CPC Class Codes: G09G 3/3208 : organic, e.g. using organic light-emitting diodes [OLED]
• CPC Class Codes: G09G 3/3225 : using an active matrix
• CPC Class Codes: G09G 3/36 : using liquid crystals
• CPC Class Codes: G09G 5/377 : for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 23/63 : by using electronic viewfinders
• CPC Class Codes: H04N 23/72 : Combination of two or more compensation controls
• CPC Class Codes: H04N 7/144 : camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
• CPC Class Codes: H04N 7/15 : Conference systems

Abstract

Some embodiments include a mobile device with a light sensor overlaid under a display (e.g., a touchscreen). The mobile device can identify a command to capture an image. The mobile device can adjust at least a target portion of an opaqueness adjustable region of the display directly over the light sensor. The opaqueness adjustable region is capable of transforming from substantially opaque to substantially transparent. The mobile device can capture the image utilizing at least the light sensor while the target portion of the opaqueness adjustable region is substantially transparent.

Claims

1. A computer-implemented method of operating a mobile device having a light sensor underneath a display, comprising:

displaying a user interface on at least a portion of the display over the light sensor, the display comprising a plurality of pixels, wherein opaqueness of each pixel in the plurality of pixels is individually adjustable;

identifying, by a processor, a command to capture an image with the light sensor;

responsive to the command, adjusting, by the processor, opaqueness of at least a target portion of an opaqueness adjustable region of the display directly over the light sensor, wherein the opaqueness of each pixel in the opaqueness adjustable region is individually adjustable creating a visual effect comprising a grating or a lens flare;

based on the visual effect, determining, by the processor, a gesture from the image; and

further responsive to the command, the processor capturing the image with the light sensor while the target portion of the opaqueness adjustable region is at least partially transparent.

2. The computer-implemented method of claim 1, wherein adjusting the opaqueness includes turning at least the target portion substantially transparent.

3. The computer-implemented method of claim 1, wherein opaqueness and transparency of the opaqueness adjustable region correspond to detectability of external light outside of the mobile device by the light sensor.

4. The computer-implemented method of claim 1, wherein the display is an organic light emitting diode (OLED) display that is substantially transparent when light emitting pixels of OLED are turned off, and wherein said adjusting the opaqueness includes turning off light emitting pixels of the OLED at the target portion to make the target portion substantially transparent.

5. The computer-implemented method of claim 4, wherein the target portion of the OLED display is turned off whenever the light sensor is active.

6. The computer-implemented method of claim 1, wherein the display is a liquid crystal display (LCD), and wherein said adjusting the opaqueness includes causing one or more liquid crystals of the LCD at the target portion to optically rotate light passing through the LCD toward the light sensor.

7. The computer-implemented method of claim 1, further comprising detecting a touch event utilizing a touch sensor integrated with the display of the mobile device, and wherein said adjusting the opaqueness is responsive to detecting the touch event.

8. The computer-implemented method of claim 1, wherein the light sensor is part of a video camera, a still image camera, a proximity sensor system, an optical biometric sensor system, an optical gesture detection system, or any combination thereof.

9. The computer-implemented method of claim 1, wherein said capturing the image is performed while the mobile device is displaying a video on the display.

10. The computer-implemented method of claim 9, further comprising:

analyzing the image to identify the gesture; and

providing the gesture to an operating system of the mobile device to trigger a command associated with the gesture.

11. The computer-implemented method of claim 1, wherein the target portion of the opaqueness adjustable region is kept substantially transparent for a duration sufficiently short to be substantially undetectable by human perception.

12. The computer-implemented method of claim 1, wherein said adjusting the opaqueness includes keeping the target portion substantially transparent for substantially a duration needed to capture the image according to the command.

13. The computer-implemented method of claim 1, further comprising displaying an opaque portion within the opaqueness adjustable region of the display while the target portion of the opaqueness adjustable region is rendered substantially transparent.

14. The computer-implemented method of claim 1, further comprising turning at least the target portion substantially translucent.

15. The computer-implemented method of claim 14, wherein the captured image is a first image;

and further comprising displaying a second image over the target portion while capturing the first image.

16. The computer-implemented method of claim 1, further comprising:

utilizing the captured image to recognize a biometric pattern; and

providing access to at least a subset of the user interface based on the recognized biometric pattern.

17. The computer-implemented method of claim 1, further comprising:

utilizing the captured image to videoconference with an external device; and

wherein the light sensor is positioned in the middle of the display to enable a user of the mobile device to maintain eye contact with an external user of the external device while videoconferencing.

18. The computer-implemented method of claim 1, further comprising:

identifying a real world object from the captured image; and

rendering, in substantially real-time relative to said capturing the image, a video representation of the real world object on the display of the mobile device.

19. The computer-implemented method of claim 18, wherein said capturing occurred between two consecutive frames of the video representation when the target portion of the opaqueness adjustable region of the display is substantially transparent.

20. The computer-implemented method of claim 1, further comprising flashing a light source from underneath the display immediately before or during said capturing of the image.

21. The computer-implemented method of claim 1, further comprising flashing the portion of the display immediately before or during said capturing of the image.

22. A computer readable data storage memory storing computer-executable instructions that, when executed by a computer system, cause the computer system to perform a computer-implemented method, the computer-executable instructions comprising:

instructions for identifying a command to capture an image with a light sensor of a mobile device;

instructions for, responsive to the command, adjusting opaqueness of at least a target portion of an opaqueness adjustable region of a display directly over the light sensor from substantially opaque to substantially transparent, the display comprising a plurality of pixels, wherein opaqueness of each pixel in the plurality of pixels is individually adjustable, wherein the opaqueness of each pixel in the opaqueness adjustable region is individually adjustable creating a visual effect comprising a grating or a lens flare;

instructions for determining a gesture from the image based on the visual effect; and

instructions for, further responsive to the command, capturing the image with the light sensor while the target portion of the opaqueness adjustable region is substantially transparent.

23. A mobile device comprising:

a transparent screen comprising a plurality of pixels, wherein opaqueness of each pixel in the plurality of pixels is individually adjustable;

an opaqueness adjustable display underneath the transparent screen, wherein the opaqueness adjustable display is capable of turning at least a region thereon substantially transparent, wherein the opaqueness of each pixel in the opaqueness adjustable display is individually adjustable creating a visual effect comprising a grating or a lens flare;

a processor to determine a gesture from an image based on the visual effect; and

a camera disposed underneath the transparent screen and at least a portion of the opaqueness adjustable display, wherein the camera is configured to capture the image while a target portion of the opaqueness adjustable display is substantially transparent.

24. The mobile device of claim 23, wherein the opaqueness adjustable display is an organic light emitting diode (OLED) display or a liquid crystal display (LCD).

25. The mobile device of claim 23, further comprising a touch sensing panel coupled to the transparent screen, wherein the touch sensing panel is substantially transparent.

26. A method comprising:

displaying a user interface on at least a portion of a display over a light sensor, the display comprising a plurality of pixels, wherein opaqueness of each pixel in the plurality of pixels is individually adjustable;

identifying, by a processor, a command to capture an image with the light sensor;

responsive to the command, adjusting, by the processor, opaqueness of at least a target portion of an opaqueness adjustable region of the display directly over the light sensor, wherein the opaqueness of each pixel in the opaqueness adjustable region is individually adjustable creating a visual effect comprising a grating or a lens flare;

based on the visual effect, determining, by the processor, a gesture from the image; and

further responsive to the command, the processor capturing the image with the light sensor while the target portion of the opaqueness adjustable region is at least partially transparent.