SYSTEMS AND METHODS FOR AUGMENTED REALITY IN A HEAD-UP DISPLAY

• US 20150175068A1
• Filed: 12/20/2013
• Published: 06/25/2015
• Est. Priority Date: 12/20/2013
• Status: Abandoned Application

• Current Assignee: Intel Corporation
• Original Assignee: Alexandra C. Zafiroglu, Dalila Szostak, Jose K. Sia Jr, Victoria S. Fang
• Inventors: Sia, Jose K. Jr., Fang, Victoria S., Zafiroglu, Alexandra C., Healey, Jennifer A., Szostak, Dalila, Fox, Sarah E., Correa, Juan I., Abreu, Alejandro, Saba Dos Reis, Maria Paula
• Application Number:
  US14/361,188
• Publication Number:
  US 20150175068A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 2360/179 Distances to obstacles or vehicles
• CPC Class Codes: B60K 2360/186 Displaying information according to relevancy
• CPC Class Codes: B60K 2360/785 on or in relation to the windshield or windows
• CPC Class Codes: B60K 2370/1529 Head-up displays
• CPC Class Codes: B60K 2370/177 Augmented reality
• CPC Class Codes: B60K 2370/179 Distances to obstacles or vehicles
• CPC Class Codes: B60K 2370/186 Displaying Information according to relevancy
• CPC Class Codes: B60K 2370/194 by directing line of sight
• CPC Class Codes: B60K 2370/785 on or in relation to the windshield or windows
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/285 for improving awareness by directing driver's gaze direction or eye points
• CPC Class Codes: B60K 35/29 Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
• CPC Class Codes: B60K 35/60 Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/01 Head-up displays

Abstract

Disclosed are systems and methods for augmenting reality in a head-up display implemented using a windshield of a vehicle. Image data of an operator of the vehicle is captured and a gaze tracker processes the operator image data to determine a direction of the gaze of the operator. Image data of the environment ahead of the vehicle is captured. An environment analyzer processes the environment image data. Augmented reality (“AR”) data is received from an external network. The AR data is associated with an object ahead of the vehicle and within the current area of central vision of the operator. A projection system presents AR data on the windshield to appear, to the operator of the vehicle, to be associated with the object.

Claims

1-25. -25. (canceled)

26. A system for presenting augmented reality data in a head-up display of a vehicle, the system comprising:

a gaze tracker to process operator image data of an operator of the vehicle to determine a current area of central vision of the operator;

an environment analyzer to process environment image data of an environment visible to the operator through a windshield of the vehicle; and

\

a projection system to present augmented reality data on a windshield of the vehicle, the projection system configured to present the augmented reality data that is associated with an object that is in the environment ahead of the vehicle and that is within the current area of central vision of the operator, wherein the augmented reality data is displayed on the windshield of the vehicle within the current area of central vision of the operator and displayed to appear, to the operator of the vehicle, to be associated with the object.

27. The system of claim 26, further comprising a network interface to form a wireless data connection with a wireless network access point disposed externally to the vehicle and to receive, via the wireless data connection, augmented reality data pertinent to the environment visible to the operator through the windshield of the vehicle.

28. The system of claim 26, further comprising an internal facing image capture system to capture operator image data of the operator of the vehicle for processing by the gaze tracker.

29. The system of claim 26, further comprising an external facing image capture system to capture environment image data of an environment in front of the vehicle for processing by the environment analyzer.

30. The system of claim 26, further comprising an operator identifier to receive sensor data associated with the operator of the vehicle obtained by a plurality of sensors.

31. The system of claim 26, wherein the projection system is configured to display the augmented reality data to appear, to the operator of the vehicle, to be superimposed on an object in the environment ahead of the vehicle.

32. The system of claim 26, wherein the system is configured to exclude from display on the windshield a portion of the augmented reality data that is associated with an object that is in the environment ahead of the vehicle and that is without the current area of central vision of the operator of the vehicle.

33. The system of claim 26, wherein the gaze tracker is configured to determine a line of sight of a current gaze of the operator of the vehicle, to determine a visual field of the operator based on the line of sight of the current gaze of the operator, and to determine the current area of central vision of the operator within the visual field.

34. The system of claim 26, wherein the wireless network access point is coupled to a cloudlet of a cloud-based distributed computing network.

35. The system of claim 26, wherein the projection system is configured to present the augmented reality data at any area of the windshield, including adjacent all edges of the windshield, based on the current area of central vision of the operator.

36. A method of presenting augmented reality information to an operator of a vehicle, the method comprising:

receiving environment image data from an external facing image capture system mounted to the vehicle, the environment image data including image data for an environment visible to the operator through a windshield of the vehicle;

receiving data indicating a line of sight of a current gaze of the operator through a windshield of the vehicle;

receiving augmented reality data pertinent to the environment visible to the operator; and

displaying a portion of the augmented reality data on the windshield of the vehicle based on the environment image data and based on the line of sight of the current gaze of the operator, wherein the portion of the augmented reality data is associated with an object that is in the environment ahead of the vehicle and in a direction within the field of view corresponding to a direction of the line of sight of the operator, wherein the portion of the augmented reality data is displayed on the windshield of the vehicle corresponding to the line of sight of the operator and displayed to appear, to the operator of the vehicle, to be associated with the object.

37. The method of claim 36, further comprising determining a current area of central vision of the operator, based on the line of sight of the current gaze of the operator,

wherein the portion of the augmented reality data is associated with an object that is in the environment ahead of the vehicle and that is within the central vision of the operator, and

wherein the portion of the augmented reality data is displayed on the windshield of the vehicle within the central vision of the operator.

38. The method of claim 37, wherein determining the current area of central vision of the operator includes:

determining a visual field of the operator based on the line of sight of the current gaze of the operator; and

determining the current area of central vision of the operator within the visual field.

39. The method of claim 36, wherein receiving augmented reality data comprises forming a wireless data connection with a wireless network access point.

40. The method of claim 36, wherein the line of sight of the current gaze of the operator is determined by processing operator image data including the operator'"'"'s face, the operator image data captured by an internal facing image capture system.

41. The method of claim 36, wherein receiving data specifying the line of sight of the current gaze of the operator comprises:

receiving operator head height data;

receiving operator image data from an internal facing image capture system mounted to the vehicle, the operator image data including image data of eyes of the operator; and

processing the operator image data to determine a line of sight of the current gaze of the operator based on the operator head height data.

42. The method of claim 41, wherein receiving operator head height data comprises:

receiving sensor data associated with the operator of the vehicle, the sensor data obtained by a plurality of sensors;

processing the sensor data to determine an identity of the operator of the vehicle; and

retrieving head height data corresponding to the identity of the operator of the vehicle.

43. The method of claim 36, wherein displaying a portion of the augmented reality data on the windshield of the vehicle comprises the augmented reality data adjacent any edge of the windshield according to the line of sight of the current gaze of the operator.

44. The method of claim 36, further comprising occluding a portion of the augmented reality data that is associated with an object that is in the environment ahead of the vehicle and that is without central vision of the operator of the vehicle.

45. A vehicle that presents augmented reality in a head-up display, the vehicle comprising:

a windshield;

an internal facing image capture system to capture operator image data of an operator of the vehicle;

an external facing image capture system to capture environment image data of an environment in front of the vehicle;

a gaze tracker to process operator image data to determine a line of sight of a current gaze of the operator of the vehicle;

a network interface to form a wireless data connection with a wireless network access point disposed externally to the vehicle and to receive, via the wireless data connection, augmented reality data pertinent to the environment visible to the operator through the windshield of the vehicle;

an environment analyzer to process environment image data captured by the external facing image capture system and correlate augmented reality data with one or more objects in the environment visible to the operator through the windshield of the vehicle; and

a projection system to present augmented reality data on a windshield of the vehicle, the projection system configured to present a portion of augmented reality data that is associated with an object that is in the environment ahead of the vehicle and that is likely within the central vision of the operator, based on the line of sight of the current gaze of the operator, wherein the augmented reality data is displayed on the windshield of the vehicle corresponding to the line of sight of the operator and displayed to appear, to the operator of the vehicle, to be associated with the object.

46. A non-transitory computer readable storage medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

receiving environment image data from an external facing image capture system mounted to the vehicle, the environment image data including image data for an environment visible to the operator through a windshield of the vehicle;

receiving data indicating a line of sight of a current gaze of the operator through a windshield of the vehicle;

receiving augmented reality data pertinent to the environment visible to the operator; and

displaying a portion of the augmented reality data on the windshield of the vehicle based on the environment image data and based on the line of sight of the current gaze of the operator, wherein the portion of the augmented reality data is associated with an object that is in the environment ahead of the vehicle and in a direction within the field of view corresponding to a direction of the line of sight of the operator, wherein the portion of the augmented reality data is displayed on the windshield of the vehicle corresponding to the line of sight of the operator and displayed to appear, to the operator of the vehicle, to be associated with the object.

47. The computer readable storage medium of claim 46, further having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

determining a current area of central vision of the operator, based on the line of sight of the current gaze of the operator,

wherein the portion of the augmented reality data is associated with an object that is in the environment ahead of the vehicle and that is within the central vision of the operator, and

wherein the portion of the augmented reality data is displayed on the windshield of the vehicle within the central vision of the operator.

48. The computer readable storage medium of claim 46, wherein receiving augmented reality data comprises forming a wireless data connection with a wireless network access point.

49. The computer readable storage medium of claim 46, further having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

excluding a portion of the augmented reality data that is associated with an object that is in the environment ahead of the vehicle and that is without central vision of the operator of the vehicle.

50. A system for presenting augmented reality data in a head-up display of a vehicle, the system comprising:

means for tracking a current gaze of an operator, wherein the gaze tracking means process operator image data of an operator of the vehicle to determine a current area of central vision of the operator;

means for analyzing an environment visible to the operator through a windshield of the vehicle, the environment analyzing means configured to process environment image data of the environment visible to the operator through the windshield of the vehicle; and

means for projecting augmented reality data on a windshield of the vehicle, the projecting means configured to present the augmented reality data that is associated with an object that is in the environment ahead of the vehicle and that is within the current area of central vision of the operator, wherein the augmented reality data is displayed on the windshield of the vehicle within the current area of central vision of the operator and displayed to appear, to the operator of the vehicle, to be associated with the object.