SYSTEM AND METHOD OF OBJECT-BASED NAVIGATION

• US 20190325746A1
• Filed: 04/24/2018
• Published: 10/24/2019
• Est. Priority Date: 04/24/2018
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: LEWIS, M Anthony, CALLE, Richard Anthony, BERGAN, Charles
• Granted Patent: US 11,282,385 B2
• Time in Patent Office: Days
• CPC Class Codes: G01C 21/3453 Special cost functions, i.e. other than distance or default speed limit of road segments
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3644 Landmark guidance, e.g. using POIs or conspicuous other objects
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G08G 1/096816 where the complete route is transmitted to the vehicle at once
• CPC Class Codes: G08G 1/096872 where instructions are given per voice

Abstract

In an aspect of the disclosure, a method, a computer-readable medium, and apparatus for operating a computational network are provided. The apparatus may obtain a set of navigational instructions describing a route to a destination. The apparatus may obtain first image data through a first camera that is oriented toward the route, the first image data depicting a first scene associated with the route. The apparatus may determine a first field of view associated with a user that is navigating the route to the destination based on a first sensor that is oriented toward the user. The apparatus may identify at least one salient object represented in the first scene based on the first field of view. The apparatus may output instructional information describing a first navigational instruction of the set of navigational instructions with reference to the at least one salient object.

Claims

1. A method for providing navigational instructions using at least one neural network, the method comprising:

obtaining a set of navigational instructions describing a route to a destination;

obtaining first image data through a first camera that is oriented toward the route, the first image data depicting a first scene associated with the route;

determining a first field of view associated with a user that is navigating the route to the destination based on a first sensor that is oriented toward the user;

identifying at least one salient object represented in the first scene based on the first field of view; and

outputting instructional information describing a first navigational instruction of the set of navigational instructions with reference to the at least one salient object.

2. The method of claim 1, wherein the first sensor comprises a second camera that is oriented toward the user.

3. The method of claim 2, further comprising:

obtaining second image data that is captured through the second camera;

determining a second field of view associated with the user based on the second image data; and

storing field-of-view information based on the second field of view,

wherein the determining the first field of view associated with the user that is navigating the route is based on the field-of-view information.

4. The method of claim 3, wherein the storing the field-of-view information based on the second field of view comprises:

storing the field-of-view information based on the second field of view before the obtaining the set of navigational instructions describing the route to the destination.

5. The method of claim 1, further comprising:

processing the first image data using a first deep neural network (DNN) for identification of a set of objects represented in the first scene,

wherein the identifying the at least one salient object represented in the first scene based on the first field of view is further based on the set of objects represented in the first scene.

6. The method of claim 5, wherein the identifying the at least one salient object represented in the first scene based on the first field of view comprises:

processing the set of objects using a second DNN for identification of the at least one salient object that is estimated to be at least partially within the first field of view.

7. The method of claim 6, further comprising:

adjusting one or more weights associated with the second DNN based on the first field of view.

8. The method of claim 7, wherein the one or more weights are adjusted based at least in part on a profile associated with the user, the profile being based on at least one of an age associated with the user, a sex associated with the user, a driving experience level associated with the user, a preference associated with the user, historical information associated with the user, or a location associated with the user.

9. The method of claim 1, wherein the instructional information is configured to be output through at least one of audio presentation or visual presentation.

10. The method of claim 1, wherein the first image data depicts the first scene over a plurality of time steps, and the at least one salient object is in motion over the plurality of time steps.

11. The method of claim 10, further comprising:

determining the at least one salient object is in motion,

wherein the instructional information describing the first navigational instruction of the set of navigational instructions with reference to the at least one salient object is based on the motion of the at least one salient object.

12. An apparatus for providing navigational instructions using at least one neural network, the apparatus comprising:

means for obtaining a set of navigational instructions describing a route to a destination;

means for obtaining first image data through a first camera that is oriented toward the route, the first image data depicting a first scene associated with the route;

means for determining a first field of view associated with a user that is navigating the route to the destination based on a first sensor that is oriented toward the user;

means for identifying at least one salient object represented in the first scene based on the first field of view; and

means for outputting instructional information describing a first navigational instruction of the set of navigational instructions with reference to the at least one salient object.

13. The apparatus of claim 12, wherein the first sensor comprises a second camera that is oriented toward the user.

14. The apparatus of claim 13, further comprising:

means for obtaining second image data that is captured through the second camera;

means for determining a second field of view associated with the user based on the second image data; and

means for storing field-of-view information based on the second field of view,

wherein the determining the first field of view associated with the user that is navigating the route is based on the field-of-view information.

15. The apparatus of claim 14, wherein the means for storing the field-of-view information based on the second field of view is configured to store the field-of-view information based on the second field of view before the obtaining the set of navigational instructions describing the route to the destination.

16. The apparatus of claim 12, further comprising:

means for processing the first image data using a first deep neural network (DNN) for identification of a set of objects represented in the first scene,

wherein the identifying the at least one salient object represented in the first scene based on the first field of view is further based on the set of objects represented in the first scene.

17. The apparatus of claim 16, wherein the means for identifying the at least one salient object represented in the first scene based on the first field of view is configured to process the set of objects using a second DNN for identification of the at least one salient object that is estimated to be at least partially within the first field of view.

18. The apparatus of claim 17, further comprising:

means for adjusting one or more weights associated with the second DNN based on the first field of view.

19. The apparatus of claim 18, wherein the one or more weights are adjusted based at least in part on a profile associated with the user, the profile being based on at least one of an age associated with the user, a sex associated with the user, a driving experience level associated with the user, a preference associated with the user, historical information associated with the user, or a location associated with the user.

20. The apparatus of claim 12, wherein the instructional information is configured to be output through at least one of audio presentation or visual presentation.

21. The apparatus of claim 12, wherein the first image data depicts the first scene over a plurality of time steps, and the at least one salient object is in motion over the plurality of time steps.

22. The apparatus of claim 21, further comprising:

means for determining the at least one salient object is in motion,

wherein the instructional information describing the first navigational instruction of the set of navigational instructions with reference to the at least one salient object is based on the motion of the at least one salient object.

23. An apparatus for providing navigational instructions using at least one neural network, the apparatus comprising:

a memory; and

at least one processor coupled to the memory and configured to;

obtain a set of navigational instructions describing a route to a destination;

obtain first image data through a first camera that is oriented toward the route, the first image data depicting a first scene associated with the route;

determine a first field of view associated with a user that is navigating the route to the destination based on a first sensor that is oriented toward the user;

identify at least one salient object represented in the first scene based on the first field of view; and

output instructional information describing a first navigational instruction of the set of navigational instructions with reference to the at least one salient object.

24. The apparatus of claim 23, wherein the at least one processor is further configured to process the first image data using a first deep neural network (DNN) for identification of a set of objects represented in the first scene,

wherein the identifying the at least one salient object represented in the first scene based on the first field of view is further based on the set of objects represented in the first scene.

25. The apparatus of claim 24, wherein the identification of the at least one salient object represented in the first scene based on the first field of view comprises to process the set of objects using a second DNN for identification of the at least one salient object that is estimated to be at least partially within the first field of view.

26. The apparatus of claim 25, wherein the at least one processor is further configured to adjust one or more weights associated with the second DNN based on the first field of view.

27. The apparatus of claim 23, wherein the instructional information is configured to be output through at least one of audio presentation or visual presentation.

28. The apparatus of claim 23, wherein the first image data depicts the first scene over a plurality of time steps, and the at least one salient object is in motion over the plurality of time steps.

29. The apparatus of claim 28, wherein the at least one processor is further configured to determine the at least one salient object is in motion,

wherein the instructional information describing the first navigational instruction of the set of navigational instructions with reference to the at least one salient object is based on the motion of the at least one salient object.

30. A computer-readable medium storing computer-executable code for providing navigational instructions using at least one neural network, include code to:

obtain a set of navigational instructions describing a route to a destination;

obtain first image data through a first camera that is oriented toward the route, the first image data depicting a first scene associated with the route;

determine a first field of view associated with a user that is navigating the route to the destination based on a first sensor that is oriented toward the user;

identify at least one salient object represented in the first scene based on the first field of view; and

output instructional information describing a first navigational instruction of the set of navigational instructions with reference to the at least one salient object.