Systems and methods for a dual modality sensor system

• US 9,437,002 B2
• Filed: 09/25/2014
• Issued: 09/06/2016
• Est. Priority Date: 09/25/2014
• Status: Expired due to Fees

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Cheatham, Jesse R. III, Hyde, Roderick A., Ishikawa, Muriel Y., Kare, Jordin T., Mundie, Craig J., Myhrvold, Nathan P., Petroski, Robert C., Rudder, Eric D., Tan, Desney S., Tegreene, Clarence T., Whitmer, Charles, Wilson, Andrew, Wing, Jeannette M., Wood, Lowell L. Jr., Wood, Victoria Y. H.
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US14/496,037
• Publication Number:
  US 20160093051A1
• Time in Patent Office: 712 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G01S 13/862 Combination of radar systems with sonar systems
• CPC Class Codes: G01S 15/003 Bistatic sonar systems; Multistatic sonar systems
• CPC Class Codes: G01S 15/58 Velocity or trajectory determination systems; Sense-of-movement determination systems velocity measurement in imaging systems G01S15/8979
• CPC Class Codes: G01S 15/74 Systems using reradiation of acoustic waves, e.g. IFF, i.e. identification of friend or foe
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/87 Combinations of sonar systems
• CPC Class Codes: G01S 15/89 for mapping or imaging
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10132 Ultrasound image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 : Static hand or arm

Abstract

The present disclosure provides systems and methods for using two imaging modalities for imaging an object at two different resolutions. For example, the system may utilize a first modality (e.g., ultrasound or electromagnetic radiation) to generate image data at a first resolution. The system may then utilize the other modality to generate image data of portions of interest at a second resolution that is higher than the first resolution. In another embodiment, one imaging modality may be used to resolve an ambiguity, such as ghost images, in image data generated using another imaging modality.

Claims

1. A method for generating an image of an object within a region, comprising:

transmitting ultrasound, via a first ultrasonic transmitter of an ultrasonic transmission system, into the region;

receiving, via a first ultrasonic receiver of an ultrasonic receiver system, ultrasonic reflections of the transmitted ultrasound from a plurality of sites on the object within the region, the ultrasonic reflections comprising sound waves within a first frequency band, the ultrasonic reflections including direct ultrasonic reflection and rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object;

generating, via a processor, coarse image data of the object at a first resolution based on the received ultrasonic reflections;

identifying a portion of interest on the object based on the coarse image data;

receiving electromagnetic radiation from the identified portion of interest on the object, the electromagnetic radiation comprising electromagnetic waves within a second frequency band, the second frequency band being different than the first frequency band;

generating fine image data of the portion of interest on the object at a second resolution based on the received electromagnetic radiation, wherein the second resolution is greater than the first resolution;

determining a kinematic value associated with the portion of interest on the object based on at least one of the received electromagnetic radiation and the received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and

modifying the state of an entertainment device based on the determined kinematic value associated with the portion of interest on the object.

2. The method of claim 1, wherein the portion of interest is identified based at least in part on a state of the entertainment device.

3. The method of claim 1, further comprising transmitting electromagnetic energy into the region via an electromagnetic transmitter, and

wherein receiving electromagnetic radiation from the identified portion of interest on the object comprises receiving a reflected portion of the transmitted electromagnetic radiation.

4. The method of claim 1, wherein at least one of the transmitted ultrasound and the received reflected ultrasound are rebounded from an auxiliary ultrasonic reflector.

5. The method of claim 1, wherein calculating the kinematic value associated with the portion of interest on the object comprises a Doppler velocity calculation in which the first velocity (V_dop) is a function of:

a calculated shift (Δ

_s), the frequency of the transmitted ultrasound (f₀), an arrival velocity of the ultrasound (V_in), and a departure velocity of the ultrasound (V_out).

6. The method of claim 1, wherein detecting a shift and calculating the first velocity component are performed using a derivation or equivalent equation to the following equation:

7. The method of claim 1, further comprising:

receiving, via a second ultrasonic receiver of the ultrasonic receiver system that is physically separated from the first ultrasonic receiver, an ultrasonic reflection from the object;

detecting a shift of the ultrasonic reflection received by the second ultrasonic receiver; and

calculating, via the processor, a second kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the second ultrasonic receiver.

8. The method of claim 7, further comprising:

receiving, via a third ultrasonic receiver of the ultrasonic receiver system that is physically separated from the first ultrasonic receiver, an ultrasonic reflection from the object;

detecting a shift of the ultrasonic reflection received by the third ultrasonic receiver; and

calculating, via the processor, a third kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the third ultrasonic receiver.

9. The method of claim 8, wherein the first, second and third ultrasonic receivers are non-collinear with respect to one another.

10. The method of claim 1, wherein a plurality of kinematic values are calculated for a corresponding plurality of sites on the object.

11. The method of claim 10, wherein the plurality of kinematic values are used to determine a translational velocity of the object.

12. The method of claim 1, wherein the portion of interest on the object comprises a portion of a person selected from the group consisting of a hand, finger, arm, leg, foot, toe, torso, neck, head, mouth, lip, and eye.

13. The method of claim 1, wherein determining the kinematic value of the portion of interest on the object comprises:

transmitting ultrasound, via the ultrasound transmission system, into the region, wherein the region is bounded by a first surface;

receiving, via the ultrasonic receiver system, the direct ultrasonic reflection from the object;

generating direct positional data associated with the object based on the direct ultrasonic reflection;

receiving the rebounded ultrasonic reflection from the object, wherein the rebounded ultrasonic reflection comprises ultrasound reflected by the object and the first surface prior to being received by the ultrasonic receiver;

generating rebounded positional data using the rebounded ultrasonic reflection of the object from the first surface; and

generating enhanced positional data by combining the direct positional data and the rebounded positional data.

14. The method of claim 1, wherein determining the kinematic value of the portion of interest on the object comprises:

transmitting ultrasound into the region, wherein the region is bounded by a first surface;

receiving the direct ultrasonic reflection from the object;

generating direct positional data associated with the object based on the direct ultrasonic reflection;

receiving a rebounded ultrasonic reflection from the object, wherein the rebounded ultrasonic reflection comprises ultrasound reflected by the object and the first surface prior to being received;

generating rebounded positional data using the rebounded ultrasonic reflection of the object from the first surface; and

generating enhanced positional data by combining the direct positional data and the rebounded positional data.

15. The method of claim 14, further comprising:

receiving an additional rebounded ultrasonic reflection from the object,

wherein the additional rebounded ultrasonic reflection comprises ultrasound reflected by the object and a second surface bounding the region prior to being received;

generating additional rebounded positional data using the additional rebounded ultrasonic reflection of the object from the second surface; and

supplementing the enhanced positional data with the additional rebounded positional data.

16. The method of claim 14, wherein transmitting the ultrasound comprises:

transmitting a first ultrasonic pulse that is received as the direct ultrasonic reflection; and

transmitting a second ultrasonic pulse that is received as the rebounded ultrasonic reflection.

31. The method of claim 1, wherein the first frequency band is between 20 kilohertz (kHz) and 250 kHz.

32. The method of claim 1, wherein the first frequency band is between 35 kHz and 45 kHz.

17. A method for generating an image of an object within a region, comprising:

receiving electromagnetic radiation from a plurality of sites on an object within a region, the electromagnetic radiation comprising electromagnetic waves within a first frequency band;

identifying a portion of interest on the object based on the received electromagnetic radiation;

transmitting ultrasound, via a first ultrasonic transmitter of an ultrasonic transmission system, into the region;

receiving, via a first ultrasonic receiver of an ultrasonic receiver system, ultrasonic reflections of the transmitted ultrasound from the portion of interest on the object within the region, the ultrasonic reflections comprising sound waves within a second frequency band, the second frequency band being different than the first frequency band, the ultrasonic reflections including direct ultrasonic reflection and rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the portion of interest on the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the portion of interest on the object;

generating, via a processor, image data of the portion of interest on the object at a first resolution based on the received ultrasonic reflections;

determining a kinematic value associated with the portion of interest based on at least one of the received electromagnetic radiation and the received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and

modifying the state of an entertainment device based on the determined kinematic value associated with the portion of interest on the object.

18. The method of claim 17, further comprising generating image data of the object at a second resolution based on the received electromagnetic radiation, wherein the second resolution is less than the first resolution.

19. The method of claim 17, further comprising transmitting electromagnetic radiation into the region.

33. The method of claim 17, wherein the second frequency band is between 20 kilohertz (kHz) and 250 kHz.

34. The method of claim 18, wherein the second frequency band is between 35 kHz and 45 kHz.

20. A method for generating an image of an object within a region, comprising:

transmitting ultrasound, via an ultrasonic transmission system, into the region;

receiving, via an ultrasonic receiver system, ultrasonic reflections of the transmitted ultrasound from a plurality of sites on the object within the region, the ultrasonic reflections comprising sound waves within a first frequency band, the ultrasonic reflections including direct ultrasonic reflection and rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object;

generating, via a processor, image data of the object at a first resolution based on the received ultrasonic reflections, wherein the image data at the first resolution includes at least one ambiguity;

receiving electromagnetic radiation reflected by the object that is sufficient to resolve the at least one ambiguity, the electromagnetic radiation comprising electromagnetic waves within a second frequency band, the second frequency band being different than the first frequency band;

generating enhanced image data resolving the at least one ambiguity based on the received electromagnetic radiation;

determining a kinematic value associated with the object based on at least one of the received electromagnetic radiation and the received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and

modifying the state of an entertainment device based on the determined kinematic value associated with the portion of interest on the object.

21. The method of claim 20, wherein resolving the at least one ambiguity comprises determining which of a plurality of images associated with the image data is a ghost image.

22. The method of claim 20, further comprising directing electromagnetic radiation towards a site on the object associated with the at least one ambiguity.

35. The method of claim 20, wherein the first frequency band is between 20 kilohertz (kHz) and 250 kHz.

36. The method of claim 20, wherein the first frequency band is between 35 kHz and 45 kHz.

23. A system for generating an image of an object within a region, comprising:

an ultrasonic transmitter configured to transmit ultrasound into the region;

an ultrasonic receiver configured to receive an ultrasonic reflection of the transmitted ultrasound from a plurality of sites on the object within the region, the ultrasonic reflections comprising sound waves within a first frequency band, the ultrasonic reflection including a direct ultrasonic reflection and a rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object;

a first imaging module configured to generate coarse image data of the object at a first resolution based on the received ultrasonic reflections;

an identification module configured to identify a portion of interest on the object based on the coarse image data;

an electromagnetic radiation receiver configured to receive electromagnetic radiation from the identified portion of interest on the object, the electromagnetic radiation comprising electromagnetic waves within a second frequency band, the second frequency band being different than the first frequency band;

a second imaging module configured to generate fine image data of the portion of interest on the object at a second resolution based on electromagnetic radiation received by the electromagnetic radiation receiver, wherein the second resolution is greater than the first resolution;

a kinematic determination module configured to determine a kinematic value associated with the portion of interest on the object based on at least one of received electromagnetic radiation and received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and

a modification module configured to modify the state of an entertainment device based on the kinematic value associated with the portion of interest on the object.

24. The system of claim 23, wherein the kinematic value of the object comprises a velocity of the object.

25. The system of claim 23, wherein at least one of the transmitted ultrasound and the reflected ultrasound are rebounded from an auxiliary ultrasonic reflector.

26. The system of claim 23, further comprising:

a second ultrasonic receiver that is physically separated from the first ultrasonic receiver, wherein the second ultrasonic receiver is configured to receive an ultrasonic reflection from the site;

wherein a shift module is configured to detect a shift of the ultrasonic reflection received by the second ultrasonic receiver, and

wherein the kinematic value module is configured to calculate a second kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the second ultrasonic receiver.

27. The system of claim 26, further comprising:

a third ultrasonic receiver that is physically separated from the first and second ultrasonic receivers, wherein the third ultrasonic receiver is configured to receive an ultrasonic reflection from the site,

wherein the shift module is configured to detect a shift of the ultrasonic reflection received by the third ultrasonic receiver, and

wherein the kinematic value module is configured to calculate a third kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the third ultrasonic receiver.

28. The system of claim 27, further comprising:

a prediction module configured to predict a relative position of the object at a future time based on (1) the first kinematic value associated with the portion of interest on the object, (2) the second kinematic value associated with the portion of interest on the object, and (3) the third kinematic value associated with the portion of interest on the object.

37. The system of claim 23, wherein the first frequency band is between 20 kilohertz (kHz) and 250 kHz.

38. The system of claim 23, wherein the first frequency band is between 35 kHz and 45 kHz.

29. A system for generating an image of an object within a region, comprising:

an electromagnetic radiation receiver configured to receive electromagnetic radiation from a plurality of sites on an object within a region, the electromagnetic radiation comprising electromagnetic waves within a first frequency band;

an identification module configured to identify a portion of interest on the object based on the received electromagnetic radiation;

an ultrasonic transmitter configured to transmit ultrasound into the region;

an ultrasonic receiver configured to receive an ultrasonic reflection of the transmitted ultrasound from the portion of interest on the object within the region, the ultrasonic reflections comprising sound waves within a second frequency band, the second frequency band being different than the first frequency band, the ultrasonic reflection including a direct ultrasonic reflection and a rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object;

a first imaging module configured to generate image data of the portion of interest on the object at a first resolution based on the received ultrasonic reflections;

a kinematic determination module configured to determine a kinematic value associated with the portion of interest on the object based on at least one of received electromagnetic radiation and received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and

a modification module configured to modify the state of an entertainment device based on the kinematic value associated with the portion of interest on the object.

30. The system of claim 29, wherein the kinematic value of the object comprises a position of the object.

39. The system of claim 29, wherein the second frequency band is between 20 kilohertz (kHz) and 250 kHz.

40. The system of claim 29, wherein the second frequency band is between 35 kHz and 45 kHz.