AI governance in banking: building trust and minimizing risks

Artificial intelligence is opening up new opportunities for banks, such as automating processes, supporting risk analysis, personalizing services, or detecting fraud. At the same time, it also brings dangers associated with possible wrong decisions made by algorithms and threats posed to cybersecurity. With regulations such as the EU AI Act coming into force, financial institutions must take action to effectively manage AI systems, ensuring their transparency and reliability, in order to build trust with customers and regulators.

This article will discuss the importance of AI governance, a comprehensive approach to the supervision of AI systems, including regulations, principles of ethical operations, and risk control mechanisms. It will also attempt to answer the questions of how AI governance  affects trust-building and what risks associated with the application of AI systems should be monitored in the banking sector.

The fundamentals of trust in AI systems

The widespread use and rapid development of artificial intelligence systems are being met with concerns from those who are subjected to the use of such tools. Therefore, it is important to ensure that the data processed by AI systems is properly secured, and that the processes are transparent and understandable.

According to the Ethics Guidelines for Trustworthy AI published by the European Commission, the fundamental elements that build trust in artificial intelligence systems lie in four ethical principles:

• Respect for human autonomy. AI systems must not deceive, control, or manipulate humans, and interactions with such systems must ensure that humans are fully capable of self-determination and participation in the democratic process, leaving them with meaningful opportunities for human choice. AI systems should also enhance human cognitive, social, and cultural skills in the course of such interactions, and support humans in the work environment.
• Prevention of harm. Artificial intelligence systems should not have an adverse effect on humans, ensuring the protection of their dignity and mental and physical integrity. This principle also includes the requirement to ensure that such systems are technically reliable, as well as to preclude their vulnerability to use for purposes that are harmful to both humans and the environment. Additionally, it draws attention to potential situations in which the use of AI may lead to or exacerbate asymmetries of power or information, such as in the relationship between businesses and consumers.
• Fairness. AI systems must be developed, deployed, and used in a way that is fair in both substantive and procedural terms, where just distribution of benefits and costs should be ensured and efforts should be made to strengthen societal fairness. AI systems should be used in a manner that balances competing interests. Any decision-making processes supported by AI must be explainable and provide an opportunity for those subject to them to contest their results, with care taken to identify the entity responsible for those decisions. This principle also emphasizes that the use of AI systems should never result in deception or limitation to end users’ freedom of choice.
• Explicability. Artificial intelligence systems must provide clear and transparent information about their goals and functioning, while the results of their operations must be as explainable as possible to those affected by decisions made by the system.

The same document lists seven key factors required to achieve trustworthy artificial intelligence, which require systematic evaluation throughout the lifecycle of a given AI system:

• Human agency and oversight. AI systems should support people in making informed decisions without restricting their choices, and user autonomy must play a guiding role in the operation of any such system. Even before a system is implemented, an analysis should be conducted examining its impact on basic human rights—both positive and negative, including through the use of subconscious processes, such as manipulation or conditioning, that could threaten individual autonomy. In doing so, people should have access to the tools and knowledge enabling them to understand how the system works, as well as have the ability to evaluate or challenge decisions made by the system. Moreover, the lower the possibility of human control over a system, the more rigorous the tests and procedures for managing that system must be.
• Technical robustness and safety. AI systems should be reliable in order to minimize possible errors and damage, both intentional and unexpected. In this way, their objective is to ensure the mental and physical integrity of those using them. In addition, AI systems must be adequately protected against technological risks and equipped with mechanisms for responding to failures and emergencies. The operation of the system should also be characterized by the reliability and reproducibility of results.
• Privacy and data governance. It is crucial to ensure the quality, integrity, and relevance of data to the domain of deployment in question. In doing so, data must be secured at all stages. Data sets must also be free of social biases, errors, and inaccuracies, and it is necessary to ensure that mistakes be eliminated before AI models are trained. In addition, all organizations processing personal data should implement protocols governing access to the data collected and processed.
• Transparency. This term refers to the transparency of the data, the systems, and the business model. The way data is collected and processed, the algorithms used, and the decisions made by AI systems should be documented in a way that ensures their traceability, which in turn supports the explainability of technical processes. Importantly, AI systems must be recognizable as such, and the user has the right to know that they are interacting with an AI system.
• Diversity, non-discrimination, and fairness. Trustworthy artificial intelligence must incorporate the principles of inclusion, diversity, and equity at every stage of its lifecycle, ensuring equal access and equal treatment for all users. To do so, systems must avoid using unfairly biased data. They should also be designed with accessibility and diversity in mind, allowing all users, including people with disabilities, equal access to the technology by incorporating “universal design” principles.
• Social and environmental wellbeing. AI systems should take into account the interests of society, the environment, and sentient beings, promote sustainability, and influence the general public in ways that benefit current and future generations. While AI systems can contribute to the wellbeing of society, they can also negatively affect people’s mental and physical health, and their impact should therefore be carefully monitored.
• Accountability. This requirement entails putting in place mechanisms to control, minimize, and report negative outcomes and ensure adequate redress both before and after the implementation of AI systems. Information on business models and intellectual property related to the AI system in question must be widely available.

Opportunities and risks associated with the use of artificial intelligence

There are many new opportunities associated with the use of artificial intelligence in the banking sector. AI-based solutions allow banks to increase the efficiency of tasks and services by reducing the time required for their execution. Complex tasks can be performed in an efficient and systematic manner even using unstructured data, which, if implemented without AI support, could involve time-consuming, error-prone procedures. In this way, the skillful and efficient use of AI in banking services helps optimize operating costs.

At the same time, it is important to recognize the dangers associated with the use of new technologies. The following classification of potential risks is based on the April 2025 summary of the activities of the AI working group at the Banking Technologies Forum and the source publication by IBM.

Preexisting traditional risks

Traditional risks are risks that occur with both traditional modeling methods and generative artificial intelligence. They include:

• data usage restrictions, where regulations may prohibit the use of certain data for AI model training, which may lead to the generation of unrepresentative results,

• inclusion of sensitive personal information and personally identifiable information in the data used to train the model, which may result in unwanted disclosure of that information,
• reidentification meaning a situation in which it may be possible to identify individuals even after sensitive and identifiable personal information is removed from the dataset used to train the model,
• data poisoning,
• decision bias, where a certain group may be unfairly favored as a result of an AI model’s decision due to the use of biased data or biased model training,
• lack of system transparency, including insufficient documentation, which makes it difficult to understand the impact of model-generated results on the functionality of a system or application.

Risks amplified by AI

Amplified risks refer to risks intensified by generative artificial intelligence, which can also occur with traditional modeling methods. They include:

• lack of training data transparency, which can result in difficulties in determining the representativeness of the data,
• uncertain data provenance associated with the lack of standardized methods for verifying the origin of data, which can lead to undesirable results due to the risk of using unethically collected, manipulated, or falsified data,
• data acquisition and usage rights restrictions due to laws and regulations, which may result in insufficient availability of the data needed to train the model and generate representative results,
• data bias (in the form of data based on historical and societal biases), which may lead to the generation of results that discriminate against individuals or groups of individuals,
• improper data curation, both in terms of the collection and preparation of data for training the model, such as using contradictory or erroneous information,
• evasion attacks attempting to make the targeted model produce incorrect results by distorting the input data sent to the trained model,
• poor model accuracy meaning that a model’s performance is inadequate for the task it was designed to perform—as a result, end users or systems that rely on the results generated by the model may be adversely affected,
• improper usage, which is using a model for a purpose that it was not originally designed to fulfill, which may result in its unexpected and undesirable behavior,
• over- or under-reliance on the results generated by the model,
• data leakage,
• unexplainable output meaning difficulties in explaining decisions behind the results generated by the model,
• challenges in determining who is responsible for a given AI model, associated with the lack of adequate documentation and governance procedures,
• AI’s impact on jobs, the environment, and human agency in terms of independent decision-making, as well as the risk of exploitation of workers training AI models.

New risks

Emerging new risks are typically related to generative artificial intelligence. They include:

• prompt injection, where the user manipulates the structure, instructions, or information contained in a prompt, the execution of which by the model is expected to generate results in favor of the attacker,
• the inclusion of personal information and confidential data in prompts, which may result in their inclusion in the results generated by the model, further leading to their storage and use for purposes such as re-training the model if this data is not effectively removed from the database,
• generating content that is similar or identical to existing copyrighted works or works covered by an open source license agreement,
• nonconsensual use, where generative artificial intelligence models are used to imitate individuals without their consent by generating deepfakes in the form of sound recordings, videos, or images, which can result in the dissemination of disinformation about that person,
• failure to disclose that the content in question has been generated by AI, something that individuals interacting with it absolutely must be clearly informed about,
• toxic output (such as offensive content),
• harmful output,
• spreading disinformation, which can negatively affect people’s ability to make informed decisions,
• output bias resulting in the unfair representation of individuals or groups, which can reinforce existing prejudices and discriminatory behaviors,
• uncertainty about intellectual property rights in case of AI-generated content,
• homogenization of culture and thoughts resulting from the disproportionate representation of certain majority cultures in the results generated, which can negatively affect cultural diversity,
• negative impact on education associated with the ease with which existing works can be plagiarized and students can bypass the learning process.

Security of LLM-based AI systems and associated unique risks

Here, it is worth taking a look at a recent overview of possible threats common to AI systems based on large language models (LLMs) prepared by the OWASP (Open Worldwide Application Security Project), a global nonprofit organization that seeks to improve software security by providing a variety of resources to help organizations and individuals build and maintain more secure applications. The report lists the following:

1. Prompt injection – the risk associated with prompts that can cause an uncontrolled change in the operation of the LLM leading to effects unintended by the developer, such as bypassing security or performing unauthorized actions.
2. Sensitive information disclosure – the risk of an LLM disclosing confidential or sensitive information.
3. Supply chain – the risk of exploiting the vulnerability of external components, libraries, or data to manipulation.
4. Data and model poisoning – data poisoning involving the deliberate introduction of malicious or biased data to affect the performance of a model in the course of its training.
5. Improper output handling – improper validation and skipping the process of cleaning the LLM’s output before it is made available to the user or integrated with other systems.
6. Excessive agency – an LLM’s excessive authority to perform tasks in response to a prompt, which can result in unauthorized operations.
7. System prompt leakage – the risk associated with the disclosure of system commands containing an entity’s protected data or sensitive information about the system architecture.
8. Vector and embedding weakness – risks associated with vulnerability to manipulation in vector and embedding mechanisms used in RAG (retrieval-augmented generation).
9. Misinformation – the risk of LLMs generating false information that appears to be credible.
10. Unbounded consumption – inefficient use of computing resources resulting in increased costs or potential system interruptions.

AI governance in the banking sector

AI governance refers to a set of processes and tools designed to ensure the responsible and effective management of AI systems—particularly in light of emerging regulatory frameworks such as the EU AI Act. These mechanisms draw on the expertise of cross-functional teams spanning legal, compliance, IT, data science, and specific business units. Through this interdisciplinary collaboration, organizations aim to align AI initiatives with business objectives while mitigating risks, supporting regulatory compliance, and maximizing the value derived from AI technologies.

The collaboration of business and technical stakeholders in the area of AI governance focuses on:

• managing the entire lifecycle of AI systems, involving documenting, monitoring, and managing the performance of these solutions,
• risk management, involving identifying, addressing, and managing all risks associated with the use of AI systems,
• ensuring regulatory compliance by verifying the compliance of processes with the provisions of applicable laws, ethical principles, and recommended best practices, as well as internal procedures established by the organization.

The responsibilities of an AI governance team in banking

Ensuring efficient AI governance processes in the banking sector requires interdisciplinary cooperation at the intersection of law, technology, business, and risk management to establish and enforce company-wide policies on processes for managing data, risk, model lifecycles, and ethical use of AI technologies. For this reason, organizations should establish appropriate AI governance teams with a wide range of competencies. As recommended by the AI working group at the Banking Technologies Forum, an exemplary team should include:

• AI Officer: a person who coordinates the deployment of AI technologies in an organization and is responsible for ensuring compliance with relevant regulations, implementing the principles of AI governance, and promoting the ethical use of artificial intelligence.
• AIOps Engineer: a person who provides technical support for operational compliance, responsible for maintaining the AI system, monitoring model performance, detecting anomalies, and responding to related problems.
• Data Scientist/AI Engineer: a technical person whose responsibilities include building an AI system, ensuring its efficiency and explainability, and controlling its quality. This role is also responsible for preparing data, running tests, and ensuring the readiness of the system for project environments.

• Data Steward: a person responsible for the data used by the AI system in terms of its quality, consistency, correctness, and compliance with regulations and internal policies of the organization.
• Architect: a person responsible for designing and overseeing the architecture of the AI system. Their job is to ensure the scalability and security of the system and to support its integration with existing IT infrastructure.
• Product Owner: a person who determines the functional scope of the AI system by defining the business requirements, from the scope of data used to the expected results produced by the system.
• Risk and Compliance Manager: a person responsible for ensuring compliance with applicable laws, rules of ethics, and internal policies of the organization.
• Security Expert: a person responsible for identifying AI system security risks and minimizing them. Their role includes activities associated with data protection and cybersecurity.

AI system security

AI guardrails are a set of technical, operational, and ethical preventive mechanisms designed to ensure that artificial intelligence systems operate in accordance with a given organization’s values, regulations, and goals, minimizing the risk of generating harmful or inaccurate content. Their purpose is not only to eliminate errors, but also to protect user privacy, prevent discrimination, and ensure compliance with applicable regulations. These dynamic systems monitor, analyze, and correct AI performance by classifying content, detecting personal data, checking facts, and analyzing semantics, allowing human intervention if necessary. This allows artificial intelligence to develop in a controlled manner, without the risk of unwanted consequences.

Based on the classification proposed by McKinsey, we can distinguish the following types of guardrails:

• Appropriateness guardrails are mechanisms seeking to ensure that artificial intelligence systems do not generate harmful, biased, or discriminatory content, protecting the right to privacy, preventing abuse, and supporting the principles of fairness and human dignity,
• Regulatory compliance guardrails are intended to ensure that AI systems operate in compliance with applicable laws and industry regulations (e.g., GDPR), protecting data privacy and meeting security standards,
• Hallucination guardrails are designed to prevent AI from generating content that contains erroneous or misleading information, ensuring that its output is factually correct,
• Alignment guardrails ensure that AI-generated content is in line with user expectations and does not deviate from its intended purpose—for example, by helping maintain brand consistency, 
• Validation guardrails are technical safeguards established to prevent unintended or harmful actions by AI, including monitoring the process to detect anomalies and errors, and possibly submitting the results for human evaluation.

Guardrails work in stages: first, the generated content is scanned for errors and inconsistencies. If there is a problem, the system then attempts to correct the content. This is followed by another check, and the whole process repeats until all detected inconsistencies have been corrected. The system also makes a decision whether the generated content can be approved, whether it should be rejected, or whether it should be submitted to a human for verification. Therefore, in addition to AI guardrails, an organization should also implement other systems and procedures for controlling AI-based solutions and not rely solely on an automated system.

An example of the application of AI guardrails in the banking sector could be customer service bots equipped with appropriate regulatory safeguards to ensure that their actions comply with applicable laws and prevent unethical sales practices.

Examples of AI governance good practices in banking

Effective AI governance in the banking sector requires clearly defined principles that ensure regulatory compliance, model transparency, and risk control. Below are key best practices that help financial institutions implement AI in a safe, ethical, and effective manner.

1. Establish a governance framework. This step includes forming a multidisciplinary team to oversee the development of AI in the organization (in compliance, risk, legal, ethics, data science), and then defining rules for data collection, consent, model development, fairness testing, and documentation.
2. Maintain a model inventory with risk classification. This involves keeping track of such details as each model’s function, owner, data used, and risk level. Control mechanisms should also be adjusted accordingly to the assigned risk scores.
3. Ensure model explainability. Whenever possible, it is recommended that interpretable models be used. It is also important to ensure that the results are understandable to non-technical stakeholders.
4. Monitor performance. This step involves tracking model drift, bias shifts, and data changes, and implementing alerts, dashboards, and regular retraining and audits.
5. Treat models like software. All model updates, retraining events, and key performance milestones should be documented. For pre-deployment testing of changes, staging environments ought to be used.
6. Define data and environment constraints. In order to reduce unexpected behaviors, it is recommended that training data quality be controlled and operational environments be limited.
7. Promote ethical oversight. Even before launching AI solutions, their societal and ethical impact should be evaluated using internal benchmarks of fairness and accuracy.
8. Educate continuously. Legal literacy support for tech teams and AI literacy support for compliance/legal roles should be provided. It is recommended that governance playbooks be refreshed annually.

Building trust through responsible AI governance in banks

Managing AI systems in banking is based not only on the correct implementation of new technologies, but most importantly on ensuring regulatory compliance, transparency, and security. It becomes crucial to build trust by responsibly implementing and overseeing AI systems to maximize benefits while minimizing potential risks. For this reason, implementing the provisions of the AI Act requires an interdisciplinary approach that takes into account dynamic regulatory changes as well as the need for constant monitoring of risks and their efficient mitigation.