A major telecommunications company would like to further automate its processes and has initiated a joint project between HR and IT to develop an AI system to evaluate job applications. This system aims to screen candidates by assessing various suitability indicators (e.g. open-mindedness, communication skills) on the basis of short video segments submitted by the applicants. Management wants to make sure that the AI system evaluates fairly, both to produce a healthy shortlist as well as to avoid reputational damage on allegations of prejudice.

Management commissions an independent audit, including a data quality assessment, an analysis of the pre-processing, an evaluation of the metadata model and a black-box test. One such test reveals a general discrimination against women. Deloitte’s investigation of the training data quality traces the bias back to an imbalanced and sparse training set. This is unearthed combining the innovative bias detection tool Model Guardian with an image recognition engine to extract visual features from the videos. The tool’s broad spectrum of metrics aid interpretation and analysis of these features.

The test results show that while facial features indicative of gender are indeed removed, the model deduces gender from proxy features, for example clothing such as headscarves. The background imagery in the applicant video also plays a significant role in classifications made by the system. On the basis of our audit findings, the company changes the AI system to reduce the risk of unfair (and inadvertent) discrimination and to ensure all applicants are treated fairly.

A university hospital is collaborating with a tech startup to develop an image recognition software that characterizes a tumor as benign or malignant based solely on the CT scan. This AI system is considered particularly critical for the hospital, as false diagnoses may not only cause harm to patient health, but also lead to lawsuits and reputational damage. After delivering favorable results in a test environment, the hospital management agrees to roll out the software for use with patients. As one last cautionary measure, management needs assurance that predictions made by the system remain within an acceptable margin of error. The system should generally outperform human diagnosis, but in no case exceed a human margin of error. As the first AI-based tool in use at the hospital, the stakes are especially high.

To address the hospital’s concerns, Deloitte’s experts test the robustness of the AI system, using a collection of techniques ranging from stress-testing to white-box attacks, as well as benchmarking against so-called challenger models. The tests reveal one false assumption affecting the relevance of training images: as the quality of a CT scan varies from machine to machine in practice, the system must scale images in order to allow an AI model trained on them to deliver reliable predictions or any predictions at all. Auditing the testing process itself exposes the quality and thoroughness of model validation, as well as the quality of documentation. The stress tests determine whether the model is able to provide reliable results even in atypical situations.

The results of the stress test reveal that the model exhibits insufficient and at times random behavior under stressful conditions. Deloitte communicates this and other findings to the hospital management, who tasks the development team with implementing the recommended adjustments. After another test run, the deficiencies are demonstrably resolved in the new AI model, and the hospital may confidently put the system into operation.

Some of the major shareholders in a medium-sized vacation rental company have become aware of media reports concerning court rulings and penalties against companies that fail to comply with data protection regulations. They ask senior management to verify that the AI systems in use at the company (e.g., for pricing) comply with relevant data protection rules. As the company only has a vague idea about data privacy, the management finds it difficult to assess the risk.

Deloitte conducts an independent audit of the company’s compliance with the General Data Protection Regulation (GDPR), not only to expose potential deficiencies but also to sensitize the management to the impact these regulations can have on its business. The broad experience of the Deloitte Center for Data Privacy helps tailor the audit specifically to the company’s needs. The audit team identifies which regulations and laws apply and then evaluate the current system on that basis. Both audit access privileges and the underlying data are within the audit scope, to avoid risk of granting access to non-essential employees. The Deloitte team inspects the underlying data for customer consent and for sufficient anonymization where consent is not granted. The team attempts to “de-anonymize” customer data in order to assess re-identification risk. The client is given access to the Deloitte GDPR Navigator in order to increase awareness and strengthen governance around data protection: users search current regulations or requirements for specific aspects, e.g., the relevant regulatory agency, the objective or the date of passage, and obtain the specific information the company needs.

Our investigation finds the anonymization process to be inadequate: the audit team is able to identify the names connected to the data with only modest effort. In our audit of the rental process, we find the system does not uniformly fulfill foreign regulations and standards. The management begins to address the issues methodically, aided by its newfound awareness for data protection. GDPR-compliant AI-based solutions are within its reach.

A major logistics company uses a proprietary AI-enabled tool to analyze and forecast incoming orders. The tool contains highly sensitive, system-critical data for the company relating to customers, order volume and terms of offer. The security of the system is paramount. It is also showing its age, vulnerable to a new generation of cyber-attacks. The company lacks its own cyber-security team and is therefore unable to fully appreciate the extent of its cyber risk.

The company commissions Deloitte to audit the tool against the standards of the Trustworthy AI Framework. A penetration test (or “PenTest”) constitutes a major part of the audit, as it scans the company’s entire network. Experienced licensed specialists from Deloitte’s PenTest Lab launch a targeted, simulated attack of pre-defined vulnerabilities. We also run AI-specific tests alongside more generic software-related tests to determine whether the AI system is potentially vulnerable to data-poisoning or backdoor attacks.

The audit discovers a vulnerability within one customization of the cloud configuration that provides would-be attackers backdoor access. Once identified, Deloitte experts draft a list of recommendations and a sequential plan to close these vulnerabilities. The investigation and recommendations increase the security of the existing tool, giving the company peace of mind with regard to cyber-attacks and the freedom to once again focus on analyzing and forecasting incoming orders.

A bank is designing an AI-enabled risk model to assess the creditworthiness of loan applicants. Because the bank sees itself as a responsible lender and a customer-focused financial institution, it wants to explain the decision-making process in a way that is understandable for customers.

Deloitte provides extensive support for the bank’s plan to provide a customer experience fulfilling both the functional needs and high standards that customers expect of the institution. We implement a feedback loop toward customers that makes the decision-making process more transparent and provides instant access to the relevant decision drivers. If customers report a negative experience, the bank can use the integrated feedback loop to establish direct contact with the responsible member of staff.

The bank has an enterprise-wide control system in place to ensure error-free operation of all processes and an AI monitoring system to ensure all of the decisions are transparent and documented. For ethical reasons, the bank will also inform loan applicants that their applications have been processed using an AI-enabled system, which will allow the bank to continue to use its innovative risk classification system in good conscience.

As part of its loan business, a large German bank seeks to improve its credit risk forecasts and offer customers more personalized services. The AI system developed by the bank for this purpose (based on an IRB approach) uses customer data to calculate and issue credit ratings as well as a total credit limits for customers. The system relies on an underlying AI model that has been trained with historic data and requires an annual risk-based evaluation of all aspects of the IRB systems. As this is a substantial new system (and underlying technology), the bank engages an external firm to conduct an audit.

Deloitte has long-standing experience in audits such as these. We expose the drivers behind a model using state-of-the-art techniques and tools, such as the aiStudio’s Lucid [ML] to explain the inner workings of the overall model as well as individual decisions in an intuitive manner. Additional measures form an important part of these audits, such as interviewing a sample of customers to determine whether the decisions made by the AI system were clearly articulated. In order to comply with regulatory standards, the audit must also identify which algorithms the system uses and establish whether the results of the model can be fully disclosed.

The explanations constitute multiple components including feature importance analyses, surrogate models and counterfactual explanations. Our audit of the bank’s AI system finds some parameters required to articulate the decisions to be missing, meaning it cannot provide full disclosure. In addition, the parameters being used remain opaque; so even though users receive the correct numbers, they cannot properly interpret them. We present our audit findings to the bank, and they initiate steps to correct any potential deficiencies. After these adjustments, we issue a positive audit report and submit it to the financial authority BaFin. The bank can now rest assured that the results of its AI system will satisfy regulatory transparency requirements.