Methods and systems for detecting objects by non-visible radio frequencies and displaying associated augmented reality effects

• US 10,318,811 B1
• Filed: 11/25/2018
• Issued: 06/11/2019
• Est. Priority Date: 04/22/2018
• Status: Active Grant

• Current Assignee: Cyberknight International LLC
• Original Assignee: Bubbler International LLC
• Inventors: Gold, Robert J., Bueno, Joan
• Primary Examiner(s): Lhymn, Sarah
• Application Number:
  US16/199,179
• Time in Patent Office: 198 Days
• Field of Search: 345619, 345631, 345632
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Methods and systems for detecting and identifying one or more objects in the vicinity using radio frequency signals, excluding visible spectrum frequencies, (the “non-visible RF signals”) and displaying an AR image with an AR element associated with each of the one or more identified objects are disclosed. An exemplary method includes the steps of: detecting and registering one or more objects in the vicinity of a mobile device by non-visible RF signals; tracking the registered objects; rendering the AR elements associated with the registered objects; and displaying an AR image with the AR elements anchored to the associated objects in the visual scan area of the mobile device. This embodiment may further include a step of rendering a special kind of AR elements, called LARI AR elements, that includes a boundary and a cut-out area, and the associated AR effect rendered is limited to the area in the boundary but excluding the cut-out area. As such, in this embodiment, the object with a LARI AR element would appear entirely or partially framed by the AR element in the AR image.

Claims

1. A method for rendering an Augmented Reality (AR) image on a display of a mobile device with one or more objects detected by radio frequency signals, excluding visible spectrum frequencies, (“

non-visible RF signals”

), the method comprising the steps of;

(1) repeatedly detecting and registering the objects located within a vicinity of the mobile device, by one or more sensors of the mobile device communicating with the objects by the non-visible RF signals;

(2) repeatedly tracking a relative location of each of the objects located within a vicinity of the mobile device, relative to the mobile device;

(3) generating in real time an AR element of each of the objects with an AR type as Large-AR-Image (LARI) and located inside a visual scan area by;

(a) determining a boundary of the AR element, the boundary configured to surround a real-world image of the object;

(b) determining a cut-out area within the boundary, the cut-out area configured to show a predetermined portion of the real-world image of the object, and

(c) rendering an AR effect within the boundary, excluding the cut-out area, the AR effect scaled and oriented based on the relative location and an orientation of the object; and

(4) displaying the AR image rendered with the AR element anchored to each of the objects located inside the visual scan area; and

wherein the cut-out area is determined based on;

a distance between the mobile device and each of multiple transmission broadcast units (“

TBUs”

) locate along a peripheral of a projected cut-out area, the distance determined based on a response time for each TBU to respond to the mobile device via the non-visible RF signals and in a sequence of the TBUs; and

a distance between each pair of consecutive TBUs.

2. The method of claim 1, wherein the step of detecting and registering the one or more objects further comprises the steps of:

(1) simultaneously scanning and registering a batch of the one or more objects located within the vicinity of the mobile device, by the one or more sensors;

(a) broadcasting a single request of a unique identification (ID) by the non-visible RF signals,

(b) receiving an ID of each of the one or more objects by the non-visible RF signals, and

(c) if the ID of an object is not registered in a storage component of the mobile device, including the object as a member object of the batch until reaching a predetermined size of the batch; and

(2) register each member object of the batch by saving the ID and the relative location of the member object to the storage component.

3. The method of claim 2, wherein if a reference object selected from the objects, other than a member object in the batch, exists, the object detecting module is further configured to repeatedly determine the relative location of the member object by performing triangulation with a reference to the reference object based on:

(1) a distance between the mobile device and the member object determined in real time based on a response time for the member object to respond to the mobile device via the non-visible RF signals;

(2) a distance between the mobile device and the reference object determined in real time based on a response time for the reference object to respond to the mobile device via the non-visible RF signals; and

(3) a distance between the reference object and the member object, provided by the reference object, determined in real time based on a response time for the member object to respond to the reference object via the non-visible RF signals.

4. The method of claim 1, wherein the tracking module is further configured to remove a registered object from the storage component if the relative location of the registered object has been outside the visual scan area for at least a predetermined period of time.

5. The method of claim 1, wherein the AR element of each of the objects is generated and displayed only if one or more attributes of each of the objects, retrieved directly from each of the objects via the non-visible RF signals, meet one or more user input criteria.

6. The method of claim 1, wherein the AR element of each of the objects in the visual scan area is generated based on one or more attributes of each of the objects, retrieved from a source selected from each of the objects, one or more social media profiles associated with each of the objects, and combinations thereof.

7. The method of claim 6, wherein the AR element of each of the objects in the visual scan area is generated further based on a timing associated with the attributes of each of the objects.

8. The method of claim 1, wherein the AR element of each of the objects in the visual scan area is anchored to each of the objects based on one or more attributes of each of the objects.

9. The method of claim 1, wherein the AR element of each of the objects is scaled and oriented based on the relative location and an orientation of each of the objects.

10. The method of claim 1, wherein the visual scan area is determined based on a field of view of a camera of the mobile device and the non-visible RF signals.

19. The method of claim 6, wherein the AR element of each of the objects in the visual scan area is generated further based on a timing associated with the attributes of each of the objects.

11. A system for rendering an Augmented Reality (AR) image, comprising a mobile device including:

(1) a display;

(2) one or more sensors of radio frequency signals, excluding visible spectrum frequencies (“

non-visible RF signals”

);

(3) an object registering module configured to repeatedly detect and register one or more objects located within a vicinity of the mobile device, by the sensors communicating with the objects by the non-visible RF signals;

(4) an object tracking module configured repeatedly track a relative location of each of the objects located within a vicinity of the mobile device; and

(5) an AR rendering module configured to;

(a) generate in real time an AR element of each of the objects with an AR type as Large-AR-Image (LARI) and located inside a visual scan area by;

A. determining a boundary of the AR element, the boundary configured to surround a real-world image of the object;

B. determining a cut-out area within the boundary, the cut-out area configured to show a predetermined portion of the real-world image of the object, and

C. rendering an AR effect within the boundary, excluding the cut-out area, the AR effect scaled and oriented based on the relative location and an orientation of the object, and

(b) display the AR image rendered with the AR element anchored to each of the objects located inside the visual scan area, and

wherein the cut-out area is determined based on;

a distance between the mobile device and each of multiple transmission broadcast units (“

TBUs”

) locate along a peripheral of a projected cut-out area, the distance determined based on a response time for each TBU to respond to the mobile device via the non-visible RF signals and in a sequence of the TBUs; and

a distance between each pair of consecutive TBUs.

12. The system of claim 11, wherein the AR rendering module is further configured to generate and display the AR element of each of the objects only if one or more attributes of each of the objects, retrieved directly from each of the objects via the non-visible RF signals, meet one or more user input criteria.

13. A method for rendering an Augmented Reality (AR) image on a display of a mobile device with one or more objects detected by radio frequency signals, excluding visible spectrum frequencies, (“

non-visible RF signals”

), the method comprising the steps of;

(1) repeatedly detecting and registering the objects located within a vicinity of the mobile device, by one or more sensors of the mobile device communicating with the objects by the non-visible RF signals;

(2) repeatedly tracking a relative location of each of the objects located within a vicinity of the mobile device, relative to the mobile device;

(3) generating in real time an AR element of each of the objects with an AR type as Large-AR-Image (LARI) and located inside a visual scan area by;

(a) determining a boundary of the AR element, the boundary configured to surround a real-world image of the object;

(b) determining a cut-out area within the boundary, the cut-out area configured to show a predetermined portion of the real-world image of the object, and

(c) rendering an AR effect within the boundary, excluding the cut-out area, the AR effect scaled and oriented based on the relative location and an orientation of the object; and

(4) displaying the AR image rendered with the AR element anchored to each of the objects located inside the visual scan area; and

wherein the cut-out area is determined based on;

a predetermined shape of the cut-out area;

predetermined locations of multiple transmission broadcast units (“

TBUs”

) relative to a projected cut-out area;

a distance between the mobile device and each TBU determined based on a response time for each TBU to respond to the mobile device via the non-visible RF signals; and

a distance between each pair of the TBUs.

14. The method of claim 13, wherein the step of detecting and registering the one or more objects further comprises the steps of:

(1) simultaneously scanning and registering a batch of the one or more objects located within the vicinity of the mobile device, by the one or more sensors;

(a) broadcasting a single request of a unique identification (ID) by the non-visible RF signals,

(b) receiving an ID of each of the one or more objects by the non-visible RF signals, and

(c) if the ID of an object is not registered in a storage component of the mobile device, including the object as a member object of the batch until reaching a predetermined size of the batch; and

(2) register each member object of the batch by saving the ID and the relative location of the member object to the storage component.

15. The method of claim 14, wherein if a reference object selected from the objects, other than a member object in the batch, exists, the object detecting module is further configured to repeatedly determine the relative location of the member object by performing triangulation with a reference to the reference object based on:

(1) a distance between the mobile device and the member object determined in real time based on a response time for the member object to respond to the mobile device via the non-visible RF signals;

(2) a distance between the mobile device and the reference object determined in real time based on a response time for the reference object to respond to the mobile device via the non-visible RF signals; and

(3) a distance between the reference object and the member object, provided by the reference object, determined in real time based on a response time for the member object to respond to the reference object via the non-visible RF signals.

16. The method of claim 13, wherein the tracking module is further configured to remove a registered object from the storage component if the relative location of the registered object has been outside the visual scan area for at least a predetermined period of time.

17. The method of claim 13, wherein the AR element of each of the objects is generated and displayed only if one or more attributes of each of the objects, retrieved directly from each of the objects via the non-visible RF signals, meet one or more user input criteria.

18. The method of claim 13, wherein the AR element of each of the objects in the visual scan area is generated based on one or more attributes of each of the objects, retrieved from a source selected from each of the objects, one or more social media profiles associated with each of the objects, and combinations thereof.

20. The method of claim 13, wherein the AR element of each of the objects in the visual scan area is anchored to each of the objects based on one or more attributes of each of the objects.

21. The method of claim 13, wherein the AR element of each of the objects is scaled and oriented based on the relative location and an orientation of each of the objects.

22. The method of claim 13, wherein the visual scan area is determined based on a field of view of a camera of the mobile device and the non-visible RF signals.

23. A method for rendering an Augmented Reality (AR) image on a display of a mobile device with one or more objects detected by radio frequency signals, excluding visible spectrum frequencies, (“

non-visible RF signals”

), the method comprising the steps of;

(1) repeatedly detecting and registering the objects located within a vicinity of the mobile device, by one or more sensors of the mobile device communicating with the objects by the non-visible RF signals;

(2) repeatedly tracking a relative location of each of the objects located within a vicinity of the mobile device, relative to the mobile device;

(3) generating in real time an AR element of each of the objects with an AR type as Large-AR-Image (LARI) and located inside a visual scan area by;

(a) determining a boundary of the AR element, the boundary configured to surround a real-world image of the object;

(b) determining a cut-out area within the boundary, the cut-out area configured to show a predetermined portion of the real-world image of the object, and

(c) rendering an AR effect within the boundary, excluding the cut-out area, the AR effect scaled and oriented based on the relative location and an orientation of the object; and

(4) displaying the AR image rendered with the AR element anchored to each of the objects located inside the visual scan area; and

wherein the boundary is determined based on;

a predetermined shape of the boundary;

predetermined locations of multiple transmission broadcast units (“

TBUs”

) relative to a projected boundary;

a distance between the mobile device and each TBUs determined in real time based on a response time for each TBU to respond to the mobile device via the non-visible RF signals; and

a distance between each pair of the TBUs.

24. The method of claim 23, wherein the step of detecting and registering the one or more objects further comprises the steps of:

(1) simultaneously scanning and registering a batch of the one or more objects located within the vicinity of the mobile device, by the one or more sensors;

(a) broadcasting a single request of a unique identification (ID) by the non-visible RF signals,

(b) receiving an ID of each of the one or more objects by the non-visible RF signals, and

(c) if the ID of an object is not registered in a storage component of the mobile device, including the object as a member object of the batch until reaching a predetermined size of the batch; and

(2) register each member object of the batch by saving the ID and the relative location of the member object to the storage component.

25. The method of claim 24, wherein if a reference object selected from the objects, other than a member object in the batch, exists, the object detecting module is further configured to repeatedly determine the relative location of the member object by performing triangulation with a reference to the reference object based on:

(1) a distance between the mobile device and the member object determined in real time based on a response time for the member object to respond to the mobile device via the non-visible RF signals;

(2) a distance between the mobile device and the reference object determined in real time based on a response time for the reference object to respond to the mobile device via the non-visible RF signals; and

(3) a distance between the reference object and the member object, provided by the reference object, determined in real time based on a response time for the member object to respond to the reference object via the non-visible RF signals.

26. The method of claim 23, wherein the tracking module is further configured to remove a registered object from the storage component if the relative location of the registered object has been outside the visual scan area for at least a predetermined period of time.

27. The method of claim 23, wherein the AR element of each of the objects is generated and displayed only if one or more attributes of each of the objects, retrieved directly from each of the objects via the non-visible RF signals, meet one or more user input criteria.

28. The method of claim 23, wherein the AR element of each of the objects in the visual scan area is generated based on one or more attributes of each of the objects, retrieved from a source selected from each of the objects, one or more social media profiles associated with each of the objects, and combinations thereof.

29. The method of claim 28, wherein the AR element of each of the objects in the visual scan area is generated further based on a timing associated with the attributes of each of the objects.

30. The method of claim 23, wherein the AR element of each of the objects in the visual scan area is anchored to each of the objects based on one or more attributes of each of the objects.

31. The method of claim 23, wherein the AR element of each of the objects is scaled and oriented based on the relative location and an orientation of each of the objects.

32. The method of claim 23, wherein the visual scan area is determined based on a field of view of a camera of the mobile device and the non-visible RF signals.

33. A method for rendering an Augmented Reality (AR) image on a display of a mobile device with one or more objects detected by radio frequency signals, excluding visible spectrum frequencies, (“

non-visible RF signals”

), the method comprising the steps of;

(1) repeatedly detecting and registering the objects located within a vicinity of the mobile device, by one or more sensors of the mobile device communicating with the objects by the non-visible RF signals;

(2) repeatedly tracking a relative location of each of the objects located within a vicinity of the mobile device, relative to the mobile device;

(3) generating in real time an AR element of each of the objects with an AR type as Large-AR-Image (LARI) and located inside a visual scan area by;

(a) determining a boundary of the AR element, the boundary configured to surround a real-world image of the object;

(b) determining a cut-out area within the boundary, the cut-out area configured to show a predetermined portion of the real-world image of the object, and

(c) rendering an AR effect within the boundary, excluding the cut-out area, the AR effect scaled and oriented based on the relative location and an orientation of the object; and

(4) displaying the AR image rendered with the AR element anchored to each of the objects located inside the visual scan area; and

wherein the boundary is determined based on;

a predetermined shape of the boundary;

locations of multiple transmission broadcast units (“

TBUs”

) relative to a projected boundary;

a distance between the mobile device and each TBUs determined in real time based on a response time for each TBU to respond to the mobile device via the non-visible RF signals; and

a distance between each pair of the TBUs.

34. The method of claim 33, wherein the step of detecting and registering the one or more objects further comprises the steps of:

(1) simultaneously scanning and registering a batch of the one or more objects located within the vicinity of the mobile device, by the one or more sensors;

(a) broadcasting a single request of a unique identification (ID) by the non-visible RF signals,

(b) receiving an ID of each of the one or more objects by the non-visible RF signals, and

(c) if the ID of an object is not registered in a storage component of the mobile device, including the object as a member object of the batch until reaching a predetermined size of the batch; and

(2) register each member object of the batch by saving the ID and the relative location of the member object to the storage component.

35. The method of claim 34, wherein if a reference object selected from the objects, other than a member object in the batch, exists, the object detecting module is further configured to repeatedly determine the relative location of the member object by performing triangulation with a reference to the reference object based on:

(1) a distance between the mobile device and the member object determined in real time based on a response time for the member object to respond to the mobile device via the non-visible RF signals;

(2) a distance between the mobile device and the reference object determined in real time based on a response time for the reference object to respond to the mobile device via the non-visible RF signals; and

(3) a distance between the reference object and the member object, provided by the reference object, determined in real time based on a response time for the member object to respond to the reference object via the non-visible RF signals.

36. The method of claim 33, wherein the tracking module is further configured to remove a registered object from the storage component if the relative location of the registered object has been outside the visual scan area for at least a predetermined period of time.

37. The method of claim 33, wherein the AR element of each of the objects is generated and displayed only if one or more attributes of each of the objects, retrieved directly from each of the objects via the non-visible RF signals, meet one or more user input criteria.

38. The method of claim 33, wherein the AR element of each of the objects in the visual scan area is generated based on one or more attributes of each of the objects, retrieved from a source selected from each of the objects, one or more social media profiles associated with each of the objects, and combinations thereof.

39. The method of claim 38, wherein the AR element of each of the objects in the visual scan area is generated further based on a timing associated with the attributes of each of the objects.

40. The method of claim 33, wherein the AR element of each of the objects in the visual scan area is anchored to each of the objects based on one or more attributes of each of the objects.

41. The method of claim 33, wherein the AR element of each of the objects is scaled and oriented based on the relative location and an orientation of each of the objects.

42. The method of claim 33, wherein the visual scan area is determined based on a field of view of a camera of the mobile device and the non-visible RF signals.

43. A system for rendering an Augmented Reality (AR) image, comprising a mobile device including:

(1) a display;

(2) one or more sensors of radio frequency signals, excluding visible spectrum frequencies (“

non-visible RF signals”

);

(3) an object registering module configured to repeatedly detect and register one or more objects located within a vicinity of the mobile device, by the sensors communicating with the objects by the non-visible RF signals;

(4) an object tracking module configured repeatedly track a relative location of each of the objects located within a vicinity of the mobile device; and

(5) an AR rendering module configured to;

(a) generate in real time an AR element of each of the objects with an AR type as Large-AR-Image (LARI) and located inside a visual scan area by;

A. determining a boundary of the AR element, the boundary configured to surround a real-world image of the object;

B. determining a cut-out area within the boundary, the cut-out area configured to show a predetermined portion of the real-world image of the object, and

C. rendering an AR effect within the boundary, excluding the cut-out area, the AR effect scaled and oriented based on the relative location and an orientation of the object, and

(b) display the AR image rendered with the AR element anchored to each of the objects located inside the visual scan area, and

wherein the cut-out area is determined based on;

a predetermined shape of the cut-out area;

predetermined locations of multiple transmission broadcast units (“

TBUs”

) relative to a projected cut-out area;

a distance between the mobile device and each TBU determined based on a response time for each TBU to respond to the mobile device via the non-visible RF signals; and

a distance between each pair of the TBUs.

44. The system of claim 43, wherein the AR rendering module is further configured to generate and display the AR element of each of the objects only if one or more attributes of each of the objects, retrieved directly from each of the objects via the non-visible RF signals, meet one or more user input criteria.

45. A system for rendering an Augmented Reality (AR) image, comprising a mobile device including:

(1) a display;

(2) one or more sensors of radio frequency signals, excluding visible spectrum frequencies (“

non-visible RF signals”

);

(3) an object registering module configured to repeatedly detect and register one or more objects located within a vicinity of the mobile device, by the sensors communicating with the objects by the non-visible RF signals;

(4) an object tracking module configured repeatedly track a relative location of each of the objects located within a vicinity of the mobile device; and

(5) an AR rendering module configured to;

(a) generate in real time an AR element of each of the objects with an AR type as Large-AR-Image (LARI) and located inside a visual scan area by;

A. determining a boundary of the AR element, the boundary configured to surround a real-world image of the object;

B. determining a cut-out area within the boundary, the cut-out area configured to show a predetermined portion of the real-world image of the object, and

C. rendering an AR effect within the boundary, excluding the cut-out area, the AR effect scaled and oriented based on the relative location and an orientation of the object, and

(b) display the AR image rendered with the AR element anchored to each of the objects located inside the visual scan area, and

wherein the boundary is determined based on;

(6) a predetermined shape of the boundary;

predetermined locations of multiple transmission broadcast units (“

TBUs”

) relative to a projected boundary;

a distance between the mobile device and each TBUs determined in real time based on a response time for each TBU to respond to the mobile device via the non-visible RF signals; and

a distance between each pair of the TBUs.

46. The system of claim 45, wherein the AR rendering module is further configured to generate and display the AR element of each of the objects only if one or more attributes of each of the objects, retrieved directly from each of the objects via the non-visible RF signals, meet one or more user input criteria.

47. A system for rendering an Augmented Reality (AR) image, comprising a mobile device including:

(1) a display;

(2) one or more sensors of radio frequency signals, excluding visible spectrum frequencies (“

non-visible RF signals”

);

(3) an object registering module configured to repeatedly detect and register one or more objects located within a vicinity of the mobile device, by the sensors communicating with the objects by the non-visible RF signals;

(4) an object tracking module configured repeatedly track a relative location of each of the objects located within a vicinity of the mobile device; and

(5) an AR rendering module configured to;

(a) generate in real time an AR element of each of the objects with an AR type as Large-AR-Image (LARI) and located inside a visual scan area by;

A. determining a boundary of the AR element, the boundary configured to surround a real-world image of the object;

B. determining a cut-out area within the boundary, the cut-out area configured to show a predetermined portion of the real-world image of the object, and

C. rendering an AR effect within the boundary, excluding the cut-out area, the AR effect scaled and oriented based on the relative location and an orientation of the object, and

(b) display the AR image rendered with the AR element anchored to each of the objects located inside the visual scan area, and

wherein the boundary is determined based on;

a predetermined shape of the boundary;

locations of multiple transmission broadcast units (“

TBUs”

) relative to a projected boundary;

a distance between the mobile device and each TBUs determined in real time based on a response time for each TBU to respond to the mobile device via the non-visible RF signals; and

a distance between each pair of the TBUs.

48. The system of claim 47, wherein the AR rendering module is further configured to generate and display the AR element of each of the objects only if one or more attributes of each of the objects, retrieved directly from each of the objects via the non-visible RF signals, meet one or more user input criteria.