METHOD AND APPARATUS FOR DISPLAYING VIRTUAL ROUTE

• US 20190137294A1
• Filed: 03/13/2018
• Published: 05/09/2019
• Est. Priority Date: 11/09/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: JUNG, Kyungboo, KANG, Nahyup, LEE, KeeChang, HONG, Sunghoon
• Granted Patent: US 10,732,004 B2
• Time in Patent Office: Days
• CPC Class Codes: G01C 21/188 for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3635 Guidance using 3D or perspective road maps
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G01C 21/3819 Road shape data, e.g. outline of a route
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering

Abstract

A method and apparatus for displaying a virtual route estimates a position of a vehicle based on sensing data received asynchronously from sensors, and outputs a three-dimensional (3D) virtual route generated by registering a driving environment model corresponding to a driving environment of the vehicle and the position of the vehicle in map information.

Claims

1. A method of displaying a virtual route, the method comprising:

estimating a position of a vehicle based on sensing data received asynchronously from sensors;

generating a three-dimensional (3D) virtual route by registering a driving environment model corresponding to a driving environment of the vehicle and the position of the vehicle in map information; and

outputting the 3D virtual route.

2. The method of claim 1, wherein the sensors comprise any one or any combination of a global positioning system (GPS) sensor, an inertial measurement unit (IMU) sensor, an on-board diagnostics (OBD) sensor, and a camera sensor.

3. The method of claim 1, wherein the sensing data are sampled separately for each of the sensors.

4. The method of claim 1, wherein the estimating comprises:

inserting, into each piece of the sensing data, information related to a processing time at which the corresponding piece of the sensing data is processed; and

estimating the position of the vehicle based on the sensing data, each including the information related to the processing time.

5. The method of claim 1, wherein the estimating comprises:

estimating a first position of the vehicle based on first sensing data received at a first time from a first sensor of the sensors;

updating the first position of the vehicle to a second position based on second sensing data received at a second time from the first sensor or a second sensor of the sensors; and

updating the second position of the vehicle to a third position based on third sensing data received at a third time from the first sensor, or the second sensor or a third sensor of the sensors.

6. The method of claim 1, further comprising:

measuring an odometry based on image data acquired from a camera sensor among the sensors,

wherein the estimating comprises estimating the position of the vehicle based on the sensing data and the odometry.

7. The method of claim 6, wherein the estimating based on the sensing data and the odometry comprises:

estimating the position of the vehicle based on one piece of the sensing data, except for the image data; and

correcting the position of the vehicle based on the odometry.

8. The method of claim 1, wherein the generating of the 3D virtual route comprises:

generating a segmentation image based on image data acquired from a camera sensor among the sensors;

detecting objects included in the segmentation image;

generating the driving environment model based on depth values of the objects and a driving lane of the vehicle identified from the objects; and

generating the 3D virtual route by registering the driving environment model and the position of the vehicle in the map information.

9. The method of claim 1, wherein the generating of the 3D virtual route comprises:

estimating a lighting and a texture of a region including a driving lane of the vehicle based on image data acquired from a camera sensor among the sensors; and

generating the 3D virtual route by applying the lighting and the texture.

10. The method of claim 1, further comprising:

transforming the 3D virtual route to match to a viewpoint of a driver of the vehicle,

wherein the outputting comprises displaying the transformed 3D virtual route.

11. The method of claim 10, wherein the outputting of the transformed 3D virtual route comprises displaying the transformed 3D virtual route through a head-up display (HUD) of the vehicle.

12. The method of claim 10, wherein the transforming comprises:

tracking the viewpoint of the driver by tracking 3D positions of both eyes of the driver;

predicting a transformation relation between the viewpoint of the driver and a virtual image displayed through the HUD based on the 3D positions of both the eyes of the user; and

transforming the 3D virtual route based on the transformation relation.

13. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

14. An apparatus for displaying a virtual route, the apparatus comprising:

a sensor module comprising sensors configured to sense sensing data including an image of a driving environment of a vehicle, a posture of the vehicle, and a position of the vehicle;

a memory configured to store map information;

a processor configured to estimate the position of the vehicle based on the sensing data received asynchronously from the sensors, and to generate a three-dimensional (3D) virtual route by registering a driving environment model corresponding to the driving environment of the vehicle and the position of the vehicle in the map information; and

a display configured to display the 3D virtual route.

15. The apparatus of claim 14, wherein the sensors comprise at least two or more of:

a global positioning system (GPS) sensor configured to measure an absolute route of the vehicle;

an inertial measurement unit (IMU) sensor configured to measure a relative route of the vehicle;

an on-board diagnostics (OBD) sensor configured to measure a driving distance of the vehicle; and

a camera sensor configured to capture image data including the driving environment of the vehicle.

16. The apparatus of claim 15, wherein the sensing data are sampled separately for each of the sensors.

17. The apparatus of claim 14, wherein the processor is further configured to insert, into each piece of the sensing data, information related to a processing time at which the corresponding piece of the sensing data is processed, and to estimate the position of the vehicle based on the sensing data, each including the information related to the processing time.

18. The apparatus of claim 14, wherein the processor is further configured to estimate a first position of the vehicle based on first sensing data received at a first time from a first sensor of the sensors, to update the first position of the vehicle to a second position based on second sensing data received at a second time from the first sensor or a second sensor of the sensors, and to update the second position of the vehicle to a third position based on third sensing data received at a third time from the first sensor, or the second sensor or a third sensor of the sensors.

19. The apparatus of claim 14, wherein the processor is further configured to measure an odometry based on image data acquired from a camera sensor among the sensors, and to estimate the position of the vehicle based on the sensing data and the odometry.

20. The apparatus of claim 19, wherein the processor is further configured to estimate the position of the vehicle based on one piece of the sensing data, except for the image data, and to correct the position of the vehicle based on the odometry.

21. The apparatus of claim 14, wherein the processor is further configured to generate a segmentation image based on image data acquired from a camera sensor among the sensors, to detect objects included in the segmentation image, to generate the driving environment model based on depth values of the objects and a driving lane of the vehicle identified from the objects, and to generate the 3D virtual route by registering the driving environment model and the position of the vehicle in the map information.

22. The apparatus of claim 14, wherein the processor is further configured to estimate a lighting and a texture of a region including a driving lane of the vehicle based on image data acquired from a camera sensor among the sensors, and transform the 3D virtual route based on the lighting and the texture.

23. The apparatus of claim 14, further comprising:

a second camera sensor configured to track a viewpoint of a driver of the vehicle,

wherein the processor is further configured to transform the 3D virtual route to match the viewpoint of the driver of the vehicle, and to output the transformed 3D virtual route through the display.

24. The apparatus of claim 23, wherein the display comprises a head-up display (HUD).

25. The apparatus of claim 24, wherein the processor is further configured to track the viewpoint of the driver of the vehicle by tracking 3D positions of both eyes of the driver, to predict a transformation relation between the viewpoint of the driver and a virtual image displayed through the HUD based on the 3D positions of both the eyes of the user, and to transform the 3D virtual route based on the transformation relation.

26. The apparatus of claim 15, wherein each of the sensors separately sense the position or posture of the vehicle and the sensing speed of the each of the sensors are not synchronized.

27. A virtual route display apparatus comprising:

a sensor module comprising sensors configured to asynchronously generate sensing data including an image of a driving environment of a vehicle, a viewpoint of a driver of the vehicle based on tracking 3D positions of both eyes of a driver of the vehicle, a posture of the vehicle, and a position of the vehicle;

a head-up display (HUD);

a memory configured to store map information and instructions; and

a processor configured to execute the instructions to

estimate the position of the vehicle based on the sensing data received asynchronously from the sensors,

generate a three-dimensional (3D) virtual route by registering a driving environment model corresponding to the driving environment of the vehicle and the position of the vehicle in the map information,

transform the 3D virtual route to match the viewpoint of the driver of the vehicle, and

output the transformed 3D virtual route through the HUD.

28. The apparatus of claim 27, wherein the memory is further configured to store the sensing data generated by the sensor module and the 3D virtual route generated by the processor.