Depth sensing techniques for virtual, augmented, and mixed reality systems

• US 10,455,153 B2
• Filed: 03/19/2018
• Issued: 10/22/2019
• Est. Priority Date: 03/21/2017
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Smith, Brian Keith, Shee, Koon Keong, Link, Gregory Michael
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US15/925,577
• Publication Number:
  US 20180278843A1
• Time in Patent Office: 582 Days
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/959 by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics

Abstract

A system and method for operating a sensor which has at least two modes of operation. The sensor may be provided with a sequence of common operation steps which are included in both a first sequence of operation steps which define a first mode of operation and a second sequence of operation steps which define a second mode of operation. The sensor may also be provided with one or more dummy operation steps which relate to the difference between the first mode of operation and the second mode of operation. The sensor can be operated in the first mode of operation by causing it to execute at least the common operation steps and it can be operated in the second mode of operation by causing it to execute the common operation steps and at least one dummy operation step.

Claims

1. A method comprising:

providing a sensor with a sequence of common operation steps which are included in both a first sequence of operation steps which define a first mode of operation and a second sequence of operation steps which define a second mode of operation;

providing the sensor with one or more dummy operation steps which relate to a difference between the first mode of operation and the second mode of operation;

operating the sensor in the first mode of operation by causing it to execute at least the common operation steps; and

operating the sensor in the second mode of operation by causing it to execute the common operation steps and at least one dummy operation step.

2. The method of claim 1, wherein the first mode of operation comprises executing at least the common operation steps at a first rate, and wherein the second mode of operation comprises performing the common operation steps and at least one dummy operation step at a second rate which is slower than the first rate.

3. The method of claim 2, wherein at least one of the one or more dummy operation steps comprises a delay.

4. The method of claim 1, wherein providing the sensor with the sequence of common operation steps and the one or more dummy operation steps comprises storing those operation steps in a sensor memory.

5. The method of claim 4, wherein switching the sensor between the first and second modes of operation does not require any additional act to store an operation step in the sensor memory.

6. The method of claim 1, wherein the sensor comprises a depth sensor.

7. The method of claim 6, wherein the depth sensor comprises a time-of-flight camera.

8. The method of claim 6, wherein the first mode of operation comprises a depth sensing mode with a first frame rate and the second mode of operation comprises a depth sensing mode with a second frame rate which is slower than the first frame rate.

9. The method of claim 6, further comprising providing depth information from the depth sensor to a virtual reality, augmented reality, or mixed reality display system.

10. A system comprising:

a processor configured to execute a method comprising;

providing a sensor with a sequence of common operation steps which are included in both a first sequence of operation steps which define a first mode of operation and a second sequence of operation steps which define a second mode of operation;

providing the sensor with one or more dummy operation steps which relate to a difference between the first mode of operation and the second mode of operation;

operating the sensor in the first mode of operation by causing it to execute at least the common operation steps; and

operating the sensor in the second mode of operation by causing it to execute the common operation steps and at least one dummy operation step.

11. The system of claim 10, wherein the first mode of operation comprises executing at least the common operation steps at a first rate, and wherein the second mode of operation comprises performing the common operation steps and at least one dummy operation step at a second rate which is slower than the first rate.

12. The system of claim 11, wherein at least one of the one or more dummy operation steps comprises a delay.

13. The system of claim 10, wherein providing the sensor with the sequence of common operation steps and the one or more dummy operation steps comprises storing those operation steps in a sensor memory.

14. The system of claim 13, wherein switching the sensor between the first and second modes of operation does not require any additional act to store an operation step in the sensor memory.

15. The system of claim 10, wherein the sensor comprises a depth sensor.

16. The system of claim 15, wherein the depth sensor comprises a time-of-flight camera.

17. The system of claim 15, wherein the first mode of operation comprises a depth sensing mode with a first frame rate and the second mode of operation comprises a depth sensing mode with a second frame rate which is slower than the first frame rate.

18. The system of claim 15, wherein the system is integrated in a virtual reality, augmented reality, or mixed reality display system.

19. The system of claim 10, wherein the processor comprises a state machine.

20. The system of claim 10, further comprising an arbiter to receive requests to operate the sensor in the first mode or the second mode, wherein the arbiter is configured to schedule and prioritize the requests.

21. The system of claim 10, further comprising the sensor.

22. A method comprising:

receiving a first request for a first type of depth measurement;

receiving a second request for a second type of depth measurement;

assigning a first priority to the first request;

assigning a second priority to the second request; and

causing a depth sensor to first obtain the first type of depth measurement if the first priority is higher than the second priority, or causing the depth sensor to first obtain the second type of depth measurement if the second priority is higher than the first priority.

23. The method of claim 22, wherein the first priority is assigned based on a priority of a first application requesting the first type of depth measurement, and the second priority is assigned based on a priority of a second application requesting the second type of depth measurement.

24. A system comprising:

an arbiter configured to receive a first request for a first type of depth measurement and a second request for a second type of depth measurement, the arbiter being configured to assign a first priority to the first request and a second priority to the second request; and

a processor configured to cause a depth sensor to first obtain the first type of depth measurement if the first priority is higher than the second priority, or to cause the depth sensor to first obtain the second type of depth measurement if the second priority is higher than the first priority.

25. The system of claim 24, wherein the arbiter is configured to assign the first priority based on a priority of a first application requesting the first type of depth measurement, and to assign the second priority based on a priority of a second application requesting the second type of depth measurement.

26. The system of claim 24, wherein the system is integrated in a virtual, augmented, or mixed reality display system.

27. A method comprising:

performing a configuration operation for a depth sensor, the configuration operation comprising;

storing a first sequence of operation steps which define a first depth sensing mode of operation in a memory of the depth sensor; and

storing a second sequence of operation steps which define a second depth sensing mode of operation in the memory of the depth sensor;

receiving a first request for depth measurements according to the first depth sensing mode of operation;

in response to the first request, operating the depth sensor in the first mode of operation by causing it to execute the first sequence of operation steps;

receiving a second request for depth measurements according to the second depth sensing mode of operation; and

in response to the second request, and without performing an additional configuration operation, operating the depth sensor in the second mode of operation by causing it to execute the second sequence of operation steps.

28. The method of claim 27, wherein the depth sensor comprises a time-of-flight camera.

29. The method of claim 27, wherein the first depth sensing mode of operation corresponds to a first distance range of measurements and the second depth sensing mode of operation corresponds to a second distance range of measurements which is different from the first distance range of measurements.

30. The method of claim 27, wherein the first depth sensing mode of operation corresponds to a first frame rate and the second depth sensing mode of operation corresponds to a second frame rate which is slower than the first frame rate.

31. The method of claim 27, further comprising providing depth information from the depth sensor to a virtual reality, augmented reality, or mixed reality display system.

32. A system comprising:

a processor configured to execute a method comprising;

performing a configuration operation for a depth sensor, the configuration operation comprising storing a first sequence of operation steps which define a first depth sensing mode of operation in a memory of the depth sensor, and storing a second sequence of operation steps which define a second depth sensing mode of operation in the memory of the depth sensor;

receiving a first request for depth measurements according to the first depth sensing mode of operation;

in response to the first request, operating the depth sensor in the first mode of operation by causing it to execute the first sequence of operation steps;

receiving a second request for depth measurements according to the second depth sensing mode of operation; and

in response to the second request, and without performing an additional configuration operation, operating the depth sensor in the second mode of operation by causing it to execute the second sequence of operation steps.

33. The system of claim 32, wherein the depth sensor comprises a time-of-flight camera.

34. The system of claim 32, wherein the first depth sensing mode of operation corresponds to a first distance range of measurements and the second depth sensing mode of operation corresponds to a second distance range of measurements which is different from the first distance range of measurements.

35. The system of claim 32, wherein the first depth sensing mode of operation corresponds to a first frame rate and the second depth sensing mode of operation corresponds to a second frame rate which is slower than the first frame rate.

36. The system of claim 32, wherein the system is integrated in a virtual, augmented, or mixed reality display system.