Head mounted display device and processing method of head mounted display device

• US 10,672,149 B2
• Filed: 06/22/2016
• Issued: 06/02/2020
• Est. Priority Date: 06/22/2016
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Chen, Xin, Gao, Wenmei
• Primary Examiner(s): Wu, Yanna
• Application Number:
  US16/311,311
• Publication Number:
  US 20190206093A1
• Time in Patent Office: 1,441 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information

Abstract

This application discloses a head mounted display device for extending a display area of an electronic device and a processing method of the head mounted display device. A display of the electronic device displays first content of an image. The image further includes second content that is not displayed on the display of the electronic device. The head mounted display device obtains, by using an image sensor, the first content displayed on the display of the electronic device, performs projection transformation on the second content, and displays the obtained first content and the second content obtained after the projection transformation on the display of the head mounted display device.

Claims

1. A method, applied to a head mounted display device with a first display and an image sensor, wherein the method comprises:

establishing a connection to an electronic device, wherein the electronic device has a second display, a defined three-dimensional coordinate system, and a feature point that has known three-dimensional coordinates in the three-dimensional coordinate system, wherein the second display displays first content of an image, wherein image resolution of the image is greater than display resolution of the second display, and wherein the image further comprises second content that exceeds a range of the second display and that is not displayed on the second display;

receiving data sent by the electronic device, wherein the data comprises a grayscale value of each pixel in the second content;

performing projection transformation on the second content according to the three-dimensional coordinates of the feature point and the grayscale value;

obtaining, by using the image sensor, the first content displayed on the second display; and

displaying, on the first display of the head mounted display device, the obtained first content and the second content obtained after the projection transformation.

2. The method according to claim 1, further comprising, before the first content and the second content are displayed on the first display of the head mounted display device:

tracking the feature point; and

calculating a posture of the image sensor relative to the second display.

3. The method according to claim 1, wherein the image comprises a document and a virtual keyboard used to edit the document.

4. The method according to claim 1, wherein the image comprises a static image or a dynamic image.

5. The method according to claim 1, wherein the image is a web page and the electronic device comprises a smartwatch.

6. A head-mounted display device, comprising:

a first display;

an image sensor;

a memory comprising instructions; and

at least one processor coupled to the memory, the instructions being executed by the at least one processor to cause the head-mounted display device to be configured to;

establish a connection to an electronic device, wherein the electronic device has a second display, a defined three-dimensional coordinate system, and a feature point that has known three-dimensional coordinates in the three-dimensional coordinate system, wherein the second display displays first content of an image, wherein image resolution of the image is greater than display resolution of the second display, and wherein the image further comprises second content that exceeds a range of the second display and that is not displayed on the second display;

receive data sent by the electronic device, wherein the data comprises a grayscale value of each pixel in the second content;

perform projection transformation on the second content according to the three- dimensional coordinates of the feature point and the grayscale value;

obtain, by using the image sensor, the first content displayed on the second display; and

display, on the first display of the head mounted display device, the obtained first content and the second content obtained after the projection transformation.

7. The head-mounted display device of claim 6, the instructions further cause the head-mounted display device to:

track the feature point; and

calculate a posture of the image sensor relative to the second display.

8. The head-mounted display device of claim 6, wherein the image comprises a document and a virtual keyboard used to edit the document.

9. The head-mounted display device of claim 6, wherein the image comprises a static image or a dynamic image.

10. The head-mounted display device of claim 6, wherein the image is a web page and the electronic device comprises a smartwatch.