Optimising compliance and risk advisory with GenAI

To ensure global and local regulations and policies are covered, our client operates a Centre of Excellence with hundreds of Risk Stewards providing advice on compliance-related enquiries.

Recognising Generative AI (GenAI) as an opportunity to improve speed and efficiency in the way that queries are handled, the bank initiated an ‘Advisory Optimisation’ transformation programme.

The aim was to create an automated advisory solution that would provide a self-serve model for raising queries. In doing so, the bank would be able to support Risk Stewards in moving away from basic (and sometimes unnecessary) queries towards more complex ones and provide the necessary insights to drive quality and consistent guidance.

GenAI-based solutions were at nascent stages within the bank, and this presented some notable risks and considerations:

• Data preparation – the policies, procedures and FAQs for the knowledge base need to be from golden sources, refined and curated and kept up to date in real-time, to ensure a good quality knowledge base
• Model risk – the Large Language Model (LLM) may produce misleading answers which look genuine, requiring the following measures to be implemented:
  •  Human in the loop – ensuring a subject matter expert (SME) reviews the answers before providing them to the user
  • Query classifier – classifying the queries based on complexity, so that users may pay more attention to highly complex/sensitive queries
  • Response scoring – evaluating the response using another model and providing the AI Score along with the response
• User experience and trust – GenAI may be perceived as a threat to the SME’s job, and a higher percentage of inaccurate / insufficient GenAI answers may lead to more work for the Risk Steward, ultimately impacting the credibility of the GenAI solution
• Model evaluation and monitoring – a key input for model monitoring is to obtain user feedback on the GenAI response, which requires careful design: the more dimensions captured in the feedback, the better it is for evaluating and fine-tuning the model, but this must be balanced with costs (using LLM to evaluate the responses is also highly recommended)

The system needed to automate suggested responses to user queries, and
formulate the response dynamically, based on the context of the question and
parameters like location and query type. It also needed to be able to reference and
cite a wide range of knowledge artefacts, such as policies, procedures and FAQ
documents.

Icon has prior knowledge of the customer landscape and experience supporting
financial institutions to effectively leverage GenAI, in addition to technical
experience gained through developing our own GenAI tool for architects and
consultants.

To design the solution architecture, Icon took the following approach:

• Contributed to the bank’s group-wide GenAI strategy to ensure alignment with the overall architecture
• Aligned with the Risk and Compliance technology platform strategy, including the GenAI platform, Data Platform and Machine Learning platform

• Collaborated with the Model Risk team to understand the Governance process and Model Risk management

• Developed an optimised user journey that meets the business outcomes while also mitigating the model risks

• Developed a solution architecture that leverages the strategic data assets as a knowledge base, and aligns with the target GenAI architecture

The design has now completed three rounds of Proof of Concept (PoC), meeting
the required success criteria with more than 70% of queries answered correctly by
the GenAI. As a result, the project has now moved to the Pilot phase, where it will
focus on queries related to high-volume risk type. Once this provides confidence to
the business and reaches the desired level of maturity, it will be extended to other
query types and wider stakeholders.