An Explainable AI Approach to Process Data in Mixed Reality Environments for Field Service Operations

Digital Twins is a concept that describes how physical objects can be represented and connected to the virtual world, the main goal of a Digital Twin is to centralise all the available information of an object of interest in a single virtual model. The Digital Twin consist of three main components: the physical object, a virtual representation of that object (typically a 3D model), and a real-time connection between both objects so that any change can be communicated to the other part. The possibility of understanding, visualising, and interacting with physical objects through a virtual environment is, at a very high level, the main benefit of using Digital Twins. The adoption of this concept has grown a lot in the recent years in industries such as the manufacturing, construction, health, and energy. Utility companies in the telecommunication industry, water services, and gas services are still falling behind in the adoption of these new concepts. The potential benefit for these sectors is huge where some of these benefits are real-time remote monitoring, predictive maintenance, scenario and risk assessment, better collaboration between stakeholders (internal and external), and better documentation. Existing Mixed Reality, Virtual Reality and Augmented Reality technologies can help with the interaction and visualisation of the virtual twin. The different levels of reality in combination with the digital twins will help with different tasks, for example, Virtual Reality is useful for remote tasks were most of the interaction happens with the virtual twin and Augmented Reality will help bringing the virtual twin and all its information to onsite tasks to help field engineers. However, there are different challenges when trying to connect all the different components and some of these challenges did slow down the adoption of these technologies by the utility companies. The research work in this thesis will focus on two main challenges: the cost of creating these digital twins from existing sources of information and the lack of an explainable AI approach that can be used as an enabler for the interaction between human and Digital Twin in the mixed reality environment. To address the challenge of automating the creation of digital representations at a low cost, two interval type-2 Fuzzy Rule-based Systems are presented as the best explainable AI alternatives to the opaque AI models for processing images and extracting information of the objects of interest. One of them was used to extract information about trees in a satellite image and generate a 3D representation of the geographic area combined with terrain data. This will be used for remote scenario and risk assessment and prediction of the telecommunication equipment getting damaged by natural elements like trees. The proposed approach achieved an 86.90% of accuracy, 3.5% better than the type-1 but 3.0% worse than the opaque Multilayer Perceptron model. The second interval type-2 Fuzzy Rule-based System is an explainable AI model that incorporates the use of context information in its rule to process 2D floor plan images, identify elements of interest and create a 3D digital representation of the building floors. This will benefit the telecommunication company by automating, at a low cost, the process of creating a more detailed in-building map with the telecommunication assets and improve the collaboration with external stakeholders like contractors for maintenance tasks or construction companies for any works in the building. The proposed method achieved a 97.5% Intersection over Union metric value which was comparable to the 99.3% Intersection over Union of the opaque Convolutional Neural Network model, however our proposed solution is highly interpretable and augmentable by human experts which cannot be achieved via opaque box AI models. Additionally, another interval type-2 Fuzzy Rule-based System for hand gesture classification is also presented in this thesis. This rule-based system achieved a 96.4% accuracy, and it is an easily adjustable model that can be modified to include more hand gestures, the opaque model alternative, a K-Nearest Neighbour algorithm achieved a 98.9% accuracy, however, this model cannot be easily modified by a human expert and re-training is needed which results in a cost of time. This hand gesture recognition model, alongside another fuzzy rule-based system, will help to address the challenge of the interaction between human and digital twin objects in Mixed Reality environments.