Mobile augmented reality for managing enclosed areas

• US 9,858,482 B2
• Filed: 05/28/2013
• Issued: 01/02/2018
• Est. Priority Date: 05/28/2013
• Status: Active Grant

• Current Assignee: Ent Services Development Corporation LP (DXC Technology Company)
• Original Assignee: Ent Services Development Corporation LP (DXC Technology Company)
• Inventors: Perez, Taciano Dreckmann, de Lima, Diogo Strube, Haas, Carlos Costa, Schuh, Saulo Saraiva
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US14/759,956
• Publication Number:
  US 20150363647A1
• Time in Patent Office: 1,680 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Example embodiments relate to providing mobile augmented reality for an enclosed area. In example embodiments, controller device receives a fixed video stream from a fixed camera and a mobile video stream of a current field of view of a mobile user device. The mobile user device comprises a reality augmentation module to project information on the current field of view. Further, the controller device includes a tracking module to identify a position and orientation of a mobile user of the mobile user device based on image processing of the fixed video stream and a fuzzy map module to use a fuzzy map of the enclosed area and the position and orientation of the mobile user to identify items of interest in the current field of view of the mobile user device, where the fuzzy map is generated based on a floor plan of the enclosed area.

Claims

1. A system for mobile augmented reality for managing enclosed areas, comprising:

a fixed camera to obtain a fixed video stream of an enclosed area; and

a controller device comprising;

a device interface to receive the fixed video stream from the fixed camera;

a mobile device interface to receive a mobile video stream of a current field of view of a mobile user device, wherein the mobile user device comprises a reality augmentation module to project information on the current field of view;

a tracker running on at least one processor of the controller device to determine a position and orientation of a mobile user of the mobile user device based on image processing of the fixed video stream and based on a distance perimeter calculated using signal strength between the mobile user device and a wireless adapter, wherein the tracker is further to provide user instruction to return to a check-in location when the tracker experiences tracking difficulty; and

a fuzzy mapper running on at least one processor of the controller device to use a fuzzy map of the enclosed area and the position and orientation of the mobile user to identify items of interest in the current field of view of the mobile user device, wherein the fuzzy map is generated based on a floor plan of the enclosed area.

2. The system of claim 1, wherein:

the tracker is further to;

track a movement of the mobile user device as movement data; and

determine a predicted movement of the mobile user device based on the movement data; and

the fuzzy mapper is further to update the items of interest in the current field of view based on the predicted movement of the mobile user device.

3. The system of claim 1, further comprising:

a depth sensor to obtain depth data of the enclosed area, wherein the identification of the position and orientation of the mobile user is enhanced using the depth data.

4. The system of claim 1, wherein the tracker is further to:

detect the mobile user at the check-in location, wherein the check-in location is in the enclosed area, and wherein the check-in location is a designated location for initiating the mobile augmented reality in the enclosed area; and

associate the mobile user device with the mobile user based on a predetermined QR code that is provided by the mobile user device.

5. The system of claim 1, wherein the tracker identifies the position and orientation of the mobile user by performing blob tracking of the mobile user in the fixed video stream, and wherein the tracker is further to:

determine a device position of the mobile user device based on positioning data received from the mobile user device; and

associate the mobile user device with the mobile user based on the device position of the mobile user device and the position and orientation of the mobile user.

6. The system of claim 5, further comprising:

a plurality of wireless adapters, including the wireless adapter, wherein each of the plurality of wireless adapters is used to transmit a radio frequency (RF) signal to the mobile user device for determining a relative position of the mobile user device, wherein the relative position of each of the plurality of wireless adapters is used to determine the device position of the mobile user device.

7. The system of claim 1, wherein the enclosed area is a datacenter and the fuzzy map further comprises status information for server equipment in the datacenter.

8. The system of claim 1, wherein a field of view level is determined based on the determined position and orientation of the mobile user.

9. The system of claim 8, further wherein the field of view level displays high level status information about a server based on the determined position and orientation of the mobile user.

10. The system of claim 8, further wherein the field of view level displays component level information about a server based on the determined position and orientation of the mobile user.

11. A method for mobile augmented reality for managing enclosed areas, comprising:

obtaining a fixed video stream of a datacenter from a fixed camera;

obtaining a mobile video stream of a current field of view of a mobile user device from the mobile user device, wherein the mobile user device comprises a reality augmentation module to project information on the current field of view;

determining a position and orientation of a mobile user of the mobile user device based on image processing of the fixed video stream and based on distance perimeter calculated using signal strength between the mobile user device and a wireless adapter, wherein user instruction to return to a check-in location is provided when tracking difficulty is experienced; and

using a fuzzy map of the datacenter and the position and orientation of the mobile user to identify server equipment in the current field of view of the mobile user device, wherein the fuzzy map is generated based on a floor plan of the datacenter.

12. The method of claim 11, further comprising:

tracking a movement of the mobile user device as movement data;

determining a predicted movement of the mobile user device based on the movement data; and

updating the server equipment in the current field of view based on the predicted movement of the mobile user device.

13. The method of claim 11, further comprising:

detecting the mobile user at the check-in location, wherein the check-in location is in the datacenter, and wherein the check-in location is a designated location for initiating the mobile augmented reality in the datacenter; and

associating the mobile user device with the mobile user based on a predetermined QR code that is provided by the mobile user device.

14. The method of claim 11, further comprising:

determining a device position of the mobile user device based on positioning data received from the mobile user device; and

associating the mobile user device with the mobile user based on the device position of the mobile user device and the position and orientation of the mobile user.

15. A non-transitory machine-readable storage medium encoded with instructions executable by a processor, the machine-readable storage medium comprising:

instructions to obtain a fixed video stream of a datacenter from a fixed camera;

instructions to obtain a mobile video stream of a current field of view of a mobile user device from the mobile user device, wherein the mobile user device comprises a reality augmentation module to project information on the current field of view;

instructions to determine a position and orientation of a mobile user of the mobile user device based on image processing of the fixed video stream and based on distance perimeter calculated using signal strength between the mobile user device and a wireless adapter, wherein user instruction to return to a check-in location is provided when tracking difficulty is experienced; and

instructions to use a fuzzy map of the datacenter and the position and orientation of the mobile user to identify server equipment in the current field of view of the mobile user device, wherein the fuzzy map is generated based on a floor plan of the datacenter.

16. The storage medium of claim 15, further comprising instructions to:

track a movement of the mobile user device as movement data;

determine a predicted movement of the mobile user device based on the movement data; and

update the server equipment in the current field of view based on the predicted movement of the mobile user device.

17. The storage medium of claim 15, further comprising instructions to:

detect the mobile user at the check-in location, wherein the check-in location is in the datacenter, and wherein the check-in location is a designated location for initiating the mobile augmented reality in the datacenter; and

associate the mobile user device with the mobile user based on a predetermined QR code that is provided by the mobile user device.

18. The storage medium of claim 15, further comprising instructions to:

determine a device position of the mobile user device based on positioning data received from the mobile user device; and

associate the mobile user device with the mobile user based on the device position of the mobile user device and the position and orientation of the mobile user.