Systems and methods for machine control

• US 10,691,219 B2
• Filed: 09/05/2017
• Issued: 06/23/2020
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Inventors: Holz, David S.
• Primary Examiner(s): Khan, Ibrahim A
• Application Number:
  US15/696,086
• Publication Number:
  US 20180046256A1
• Time in Patent Office: 1,022 Days
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

A region of space may be monitored for the presence or absence of one or more control objects, and object attributes and changes thereto may be interpreted as control information provided as input to a machine or application. In some embodiments, the region is monitored using a combination of scanning and image-based sensing.

Claims

1. A computer implemented method for conducting machine control, the method comprising:

monitoring a region of space for a presence of an object, the monitoring including;

scanning the region of space using a first sensing modality, the region of space being scanned at a first rate of time; and

detecting the presence of the object in the region of space;

responsive to detecting the presence of the object in the region of space, performing operations including;

imaging the region of space including the object for at least two cycles using a second sensing modality, the region of space being imaged at a second rate of time, for each cycle, that is longer than the first rate of time, such that each cycle imaged at the second rate of time takes longer than the scanning performed at the first rate of time; and

determining a first set of object attributes of the object for a first imaging cycle of the at least two cycles; and

determining a second set of object attributes of the object for a second imaging cycle of the at least two cycles;

comparing the first set of object attributes and the second set of object attributes to determine control information; and

responding to the determined control information according to response criteria.

2. The computer implemented method according to claim 1, wherein the scanning of the region of space comprises:

directing at least two emission cycles to form at least two scan patterns from an emission region of the first sensing modality to the region of space, one scan pattern of the at least two scan patterns being different from at least one other scan pattern of the at least two scan patterns;

detecting a reflectance of the at least two scan patterns; and

determining that the detected reflectance indicates the presence of the object in the region of space.

3. The computer implemented method according to claim 2, wherein:

one scan pattern of the at least two scan patterns is configured to provide sequential emissions from emitters of the emission region in a binary manner such that each of the emitters is turned on and off one at a time; and

at least one other scan pattern of the at least two scan patterns is configured to provide simultaneously overlapping emissions from the emitters.

4. The computer implemented method according to claim 1, wherein the response criteria include determining whether to respond to the control information.

5. The computer implemented method according to claim 1, wherein the determining of the first set of object attributes and the determining of the second set of object attributes include determining control-portion attributes based at least in part on captured surface detail about the object.

6. The computer implemented method according to claim 1, wherein the determining of the first set of object attributes includes identifying a presence of a hand in the region of space.

7. The computer implemented method according to claim 1, wherein:

the scanning of the region of space using the first sensing modality provides imaging the region of space at the first rate, which provides low resolution imaging of the region of space; and

the imaging of the region of space using the second sensing modality at the second rate provides high resolution imaging of the region of space.

8. The computer implemented method according to claim 1, wherein the first sensing modality and the second sensing modality have overlapping fields of view.

9. The computer implemented method according to claim 8, wherein the second sensing modality includes at least two imaging sensors having overlapping fields of view.

10. The computer implemented method according to claim 1, wherein the second sensing modality includes at least two imaging sensors having overlapping fields of view.

11. The method of claim 1, wherein the scanning of the region of space pre-scans the region of space to detect the presence of the object before the imaging of the region of space.

12. A non-transitory computer readable storage medium impressed with computer program instructions for conducting machine control, the instructions, when executed on a processor, implement a method comprising:

monitoring a region of space for a presence of an object, the monitoring including;

scanning the region of space using a first sensing modality, the region of space being scanned at a first rate of time; and

detecting the presence of the object in the region of space;

responsive to detecting the presence of the object in the region of space, performing operations including;

imaging the region of space including the object for at least two cycles using a second sensing modality, the region of space being imaged at a second rate of time, for each cycle, that is longer than the first rate of time, such that each cycle imaged at the second rate of time takes longer than the scanning performed at the first rate of time; and

determining a first set of object attributes of the object for a first imaging cycle of the at least two cycles; and

determining a second set of object attributes of the object for a second imaging cycle of the at least two cycles;

comparing the first set of object attributes and the second set of object attributes to determine control information; and

responding to the determined control information according to response criteria.

13. The non-transitory computer readable storage medium according to claim 12, wherein the scanning of the region of space comprises:

directing at least two emission cycles to form at least two scan patterns from an emission region of the first sensing modality to the region of space, one scan pattern of the at least two scan patterns being different from at least one other scan pattern of the at least two scan patterns;

detecting a reflectance of the at least two scan patterns; and

determining that the detected reflectance indicates the presence of the object in the region of space.

14. The non-transitory computer readable storage medium according to claim 13, wherein:

one scan pattern of the at least two scan patterns is configured to provide sequential emissions from emitters of the emission region in a binary manner such that each of the emitters is turned on and off one at a time; and

at least one other scan pattern of the at least two scan patterns is configured to provide simultaneously overlapping emissions from the emitters.

15. The non-transitory computer readable storage medium according to claim 12, wherein the response criteria includes determining whether to respond to the control information.

16. The non-transitory computer readable storage medium according to claim 12, wherein the determining of the first set of object attributes and the determining of the second set of object attributes include determining control-portion attributes based at least in part on captured surface detail about the object.

17. A system including one or more processors coupled to memory, the memory loaded with computer instructions for conducting machine control, the instructions, when executed on the processors, implement actions comprising:

monitoring a region of space for a presence of an object, the monitoring including;

scanning the region of space using a first sensing modality, the region of space being scanned at a first rate of time; and

detecting the presence of the object in the region of space;

responsive to detecting the presence of the object in the region of space, performing operations including;

imaging the region of space including the object for at least two cycles using a second sensing modality, the region of space being imaged at a second rate of time, for each cycle, that is longer than the first rate of time, such that each cycle imaged at the second rate of time takes longer than the scanning performed at the first rate of time; and

determining a first set of object attributes of the object for a first imaging cycle of the at least two cycles; and

determining a second set of object attributes of the object for a second imaging cycle of the at least two cycles;

comparing the first set of object attributes and the second set of object attributes to determine control information; and

responding to the determined control information according to response criteria.

18. The system according to claim 17, wherein the scanning of the region of space comprises:

directing at least two emission cycles to form at least two scan patterns from an emission region of the first sensing modality to the region of space, one scan pattern of the at least two scan patterns being different from at least one other scan pattern of the at least two scan patterns;

detecting a reflectance of the at least two scan patterns; and

determining that the detected reflectance indicates the presence of the object in the region of space.

19. The system according to claim 18, wherein:

one scan pattern of the at least two scan patterns is configured to provide sequential emissions from emitters of the emission region in a binary manner such that each of the emitters is turned on and off one at a time; and

at least one other scan pattern of the at least two scan patterns is configured to provide simultaneously overlapping emissions from the emitters.

20. The system according to claim 17, wherein the determining of the first set of object attributes and the determining of the second set of object attributes include determining control-portion attributes based at least in part on captured surface detail about the object.