SYSTEMS AND METHODS FOR A DUAL MODALITY SENSOR SYSTEM

• US 20170011521A1
• Filed: 09/06/2016
• Published: 01/12/2017
• Est. Priority Date: 09/25/2014
• Status: Abandoned Application

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Cheatham, Jesse R. III, Hyde, Roderick A, Ishikawa, Muriel Y., Kare, Jordin T., Mundie, Craig J., Myhrvold, Nathan P., Petroski, Robert C., Rudder, Eric D., Tan, Desney S., Tegreene, Clarence T., Whitmer, Charles, Wilson, Andrew, Wing, Jeannette M., Wood, Lowell L. Jr., Wood, Victoria Y.H.
• Application Number:
  US15/257,749
• Publication Number:
  US 20170011521A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G01S 13/862 Combination of radar systems with sonar systems
• CPC Class Codes: G01S 15/003 Bistatic sonar systems; Multistatic sonar systems
• CPC Class Codes: G01S 15/58 Velocity or trajectory determination systems; Sense-of-movement determination systems velocity measurement in imaging systems G01S15/8979
• CPC Class Codes: G01S 15/74 Systems using reradiation of acoustic waves, e.g. IFF, i.e. identification of friend or foe
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/87 Combinations of sonar systems
• CPC Class Codes: G01S 15/89 for mapping or imaging
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10132 Ultrasound image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 : Static hand or arm

Abstract

The present disclosure provides systems and methods for using two imaging modalities for imaging an object at two different resolutions. For example, the system may utilize a first modality (e.g., ultrasound or electromagnetic radiation) to generate image data at a first resolution. The system may then utilize the other modality to generate image data of portions of interest at a second resolution that is higher than the first resolution. In another embodiment, one imaging modality may be used to resolve an ambiguity, such as ghost images, in image data generated using another imaging modality.

Claims

1. A method for generating an image of an object within a region, comprising:

receiving, via a first ultrasonic receiver of an ultrasonic receiver system, ultrasonic reflections from the object within the region, the ultrasonic reflections comprising sound waves within a first frequency band;

generating, via a processor, coarse image data of the object at a first resolution based on the received ultrasonic reflections;

identifying a portion of interest on the object;

receiving electromagnetic radiation from the identified portion of interest on the object, the electromagnetic radiation comprising electromagnetic waves within a second frequency band that is different than the first frequency band;

generating fine image data of the portion of interest on the object at a second resolution based on the received electromagnetic radiation, wherein the second resolution is greater than the first resolution; and

modifying a state of an entertainment device based on the fine image data of the portion of interest on the object.

2. The method of claim 1, wherein the ultrasonic reflections include direct ultrasonic reflections and rebounded ultrasonic reflections.

3. The method of claim 2, wherein the direct ultrasonic reflections are reflected from a first portion of the object and the rebounded ultrasonic reflection are reflected from a second, different portion of the object

4. The method of claim 1, wherein the portion of interest is identified based at least in part on a state of the entertainment device.

5. The method of claim 1, further comprising transmitting electromagnetic energy into the region via an electromagnetic transmitter, and

wherein receiving electromagnetic radiation from the identified portion of interest on the object comprises receiving a reflected portion of the transmitted electromagnetic radiation.

6. The method of claim 1, wherein at least one of the transmitted ultrasound and the received reflected ultrasound are rebounded from an auxiliary ultrasonic reflector.

7. The method of claim 1, further comprising calculating the kinematic value associated with the portion of interest on the object using a Doppler velocity calculation in which the first velocity (V_dop) is a function of:

a calculated shift (Δ

_s), the frequency of the transmitted ultrasound (f_o), an arrival velocity of the ultrasound (V_in), and a departure velocity of the ultrasound (V_out).

8. The method of claim 7, wherein detecting a shift and calculating the first velocity component are performed using a derivation or equivalent equation to the following equation:

9. The method of claim 1, further comprising:

receiving, via a second ultrasonic receiver of the ultrasonic receiver system that is physically separated from the first ultrasonic receiver, an ultrasonic reflection from the object;

detecting a shift of the ultrasonic reflection received by the second ultrasonic receiver; and

calculating, via the processor, a kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the second ultrasonic receiver.

10. The method of claim 9, further comprising:

receiving, via a third ultrasonic receiver of the ultrasonic receiver system that is physically separated from the first ultrasonic receiver, an ultrasonic reflection from the object;

detecting a shift of the ultrasonic reflection received by the third ultrasonic receiver; and

calculating, via the processor, a second kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the third ultrasonic receiver.

11. The method of claim 10, wherein the first, second and third ultrasonic receivers are non-collinear with respect to one another.

12. The method of claim 1, wherein a plurality of kinematic values are calculated for a corresponding plurality of sites on the object.

13. The method of claim 12, wherein the plurality of kinematic values are used to determine a translational velocity of the object.

14. The method of claim 1, wherein the portion of interest on the object comprises a portion of a person selected from the group consisting of a hand, finger, arm, leg, foot, toe, torso, neck, head, mouth, lip, and eye.

15. The method of claim 1, further comprising determining a kinematic value of the portion of interest on the object, and wherein determining the kinematic value of the portion of interest on the object comprises:

transmitting ultrasound, via the ultrasound transmission system, into the region, wherein the region is bounded by a first surface;

receiving, via the ultrasonic receiver system, the direct ultrasonic reflection from the object;

generating direct positional data associated with the object based on the direct ultrasonic reflection;

receiving the rebounded ultrasonic reflection from the object, wherein the rebounded ultrasonic reflection comprises ultrasound reflected by the object and the first surface prior to being received by the ultrasonic receiver;

generating rebounded positional data using the rebounded ultrasonic reflection of the object from the first surface; and

generating enhanced positional data by combining the direct positional data and the rebounded positional data.

16. The method of claim 1, further comprising determining a kinematic value of the portion of interest on the object, wherein determining the kinematic value of the portion of interest on the object comprises:

transmitting ultrasound into the region, wherein the region is bounded by a first surface;

receiving the direct ultrasonic reflection from the object;

generating direct positional data associated with the object based on the direct ultrasonic reflection;

receiving a rebounded ultrasonic reflection from the object, wherein the rebounded ultrasonic reflection comprises ultrasound reflected by the object and the first surface prior to being received;

generating rebounded positional data using the rebounded ultrasonic reflection of the object from the first surface; and

generating enhanced positional data by combining the direct positional data and the rebounded positional data.

17. The method of claim 16, further comprising:

receiving an additional rebounded ultrasonic reflection from the object, wherein the additional rebounded ultrasonic reflection comprises ultrasound reflected by the object and a second surface bounding the region prior to being received;

generating additional rebounded positional data using the additional rebounded ultrasonic reflection of the object from the second surface; and

supplementing the enhanced positional data with the additional rebounded positional data.

18. The method of claim 16, wherein transmitting the ultrasound comprises:

transmitting a first ultrasonic pulse that is received as the direct ultrasonic reflection; and

transmitting a second ultrasonic pulse that is received as the rebounded ultrasonic reflection.

19. A method for generating an image of an object within a region, comprising:

receiving electromagnetic radiation from an object within a region, the electromagnetic radiation comprising electromagnetic waves within a first frequency band;

identifying a portion of interest on the object based on the received electromagnetic radiation;

receiving, via a first ultrasonic receiver of an ultrasonic receiver system, ultrasonic reflections from the portion of interest on the object within the region, the ultrasonic reflections comprising sound waves within a second frequency band, the second frequency band being different than the first frequency band, the ultrasonic reflections including direct ultrasonic reflection and rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the portion of interest on the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the portion of interest on the object;

generating, via a processor, image data of the portion of interest on the object at a first resolution based on the received ultrasonic reflections;

determining positional data based on at least one of the direct ultrasonic reflection and the rebounded ultrasonic reflection; and

modifying the state of an entertainment device based on the positional data associated with the portion of interest on the object.

20. The method of claim 19, further comprising transmitting ultrasound into the region in the direction of the identified portion of interest based on positional data determined using the received electromagnetic radiation.

21. The method of claim 19, further comprising generating image data of the object at a second resolution based on the received electromagnetic radiation, wherein the second resolution is less than the first resolution.

22. The method of claim 19, further comprising transmitting electromagnetic radiation into the region.

23. The method of claim 19, further comprising transmitting ultrasound into the region.

24. A method for generating an image of an object within a region, comprising:

receiving, via an ultrasonic receiver system, ultrasonic reflections of transmitted ultrasound from the object within the region, the ultrasonic reflections comprising sound waves within a first frequency band, the ultrasonic reflections including direct ultrasonic reflection and rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object;

generating, via a processor, image data of the object at a first resolution based on the received ultrasonic reflections, wherein the image data at the first resolution includes at least one ambiguity;

receiving electromagnetic radiation reflected by the object that is sufficient to resolve the at least one ambiguity, the electromagnetic radiation comprising electromagnetic waves within a second frequency band, the second frequency band being different than the first frequency band; and

generating enhanced image data resolving the at least one ambiguity based on the received electromagnetic radiation.

25. The method of claim 24, wherein resolving the at least one ambiguity comprises determining which of a plurality of images associated with the image data is a ghost image.

26. The method of claim 24, further comprising directing electromagnetic radiation toward a site on the object associated with the at least one ambiguity.

27. A system for generating an image of an object within a region, comprising:

an ultrasonic receiver configured to receive an ultrasonic reflection from the object within the region, the ultrasonic reflections comprising sound waves within a first frequency band, the ultrasonic reflection including a direct ultrasonic reflection and a rebounded ultrasonic reflection;

a first imaging module configured to generate coarse image data of the object at a first resolution based on the received ultrasonic reflections;

an identification module configured to identify a portion of interest on the object based on the coarse image data;

an electromagnetic radiation receiver configured to receive electromagnetic radiation from the identified portion of interest on the object, the electromagnetic radiation comprising electromagnetic waves within a second frequency band, the second frequency band being different than the first frequency band;

a second imaging module configured to generate fine image data of the portion of interest on the object at a second resolution based on electromagnetic radiation received by the electromagnetic radiation receiver, wherein the second resolution is greater than the first resolution; and

a modification module configured to modify the state of an entertainment device based on at least one of the fine image data and the course image data.

28. The system of claim 27, further comprising determining a kinematic value of the object using at least one of the fine image data and the course image data, and wherein the kinematic value of the object comprises a velocity of the object.

29. The system of claim 27, wherein the reflected ultrasound has been rebounded from an auxiliary ultrasonic reflector.

30. The system of claim 27, further comprising:

a second ultrasonic receiver that is physically separated from the first ultrasonic receiver, wherein the second ultrasonic receiver is configured to receive an ultrasonic reflection from the site;

wherein a shift module is configured to detect a shift of the ultrasonic reflection received by the second ultrasonic receiver, and

wherein a kinematic value module is configured to calculate a kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the second ultrasonic receiver.

31. The system of claim 30, further comprising:

a third ultrasonic receiver that is physically separated from the first and second ultrasonic receivers, wherein the third ultrasonic receiver is configured to receive an ultrasonic reflection from the site,

wherein the shift module is configured to detect a shift of the ultrasonic reflection received by the third ultrasonic receiver, and

wherein the kinematic value module is configured to calculate a second kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the third ultrasonic receiver.

32. The system of claim 31, further comprising:

a prediction module configured to predict a relative position of the object at a future time based on (1) the first kinematic value associated with the portion of interest on the object and (2) the second kinematic value associated with the portion of interest on the object.

33. A system for generating an image of an object within a region, comprising:

an electromagnetic radiation receiver configured to receive electromagnetic radiation from an object within a region, the electromagnetic radiation comprising electromagnetic waves within a first frequency band;

an identification module configured to identify a portion of interest on the object based on the received electromagnetic radiation;

an ultrasonic receiver configured to receive ultrasonic reflections from the portion of interest on the object within the region, the ultrasonic reflections comprising sound waves within a second frequency band;

a first imaging module configured to generate image data of the portion of interest on the object at a first resolution based on the received ultrasonic reflections;

a kinematic determination module configured to determine a kinematic value associated with the portion of interest on the object based on at least one of received electromagnetic radiation and received ultrasonic reflections; and

a modification module configured to modify the state of an entertainment device based on the kinematic value associated with the portion of interest on the object.

34. The system of claim 33, wherein the ultrasonic reflections include a direct ultrasonic reflection and a rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object.