By Icon Solutions ·
The challenge
The Client is a global Tier 1 bank, managing $3 trillion in assets and serving
more than 40 million customers in 60+ countries.
Being a global bank, the client operates a Risk and Compliance Centre of Excellence with hundreds of Risk Stewards to provide advice on compliance-related queries across the bank, covering various global and local regulations and policies.
Advisory Optimisation is a programme designed to transform how compliance queries are handled by Risk Stewards. It enables a self-serve model for raising queries, and delivers improved efficiencies and economies of scale by leveraging automation of high-frequency, lower complexity requests. This shifts Risk Stewards away from basic and often unnecessary queries and equips them with the insights needed to deliver high-quality, consistent guidance on more complex cases where human input is required.
The key requirement for the client was to create a system capable of enabling Generative AI (GenAI) suggested responses to user queries, significantly reducing the time and effort required by Risk Stewards. The system needed to dynamically formulate responses based on the context of each query and relevant parameters, such as location and query type. It also had to draw from a wide range of knowledge base artefacts, including policies, procedures and FAQ documents, when generating each response.
The approach
Following the successful completion of the GenAI proof of concept (PoC), as outlined in our previous case study, the client engaged Icon to define the production rollout strategy and target state-design for this GenAI use case.
Transitioning from PoC to production required detailed planning and careful consideration of critical factors, including model risk, regulatory compliance, security, and scalability. Icon adopted a structured and collaborative approach to ensure a successful rollout.
The scope for the initial pilot and the phased roadmap was defined jointly with the Business, Analytics, and Technology teams. Model Risk Management was engaged early to understand and embed required controls, while alignment with the bank's ‘Responsible, Safe, and Secure’ AI principles ensured ethical and compliant deployment. The solution was also reviewed in close collaboration with Security and Cyber teams to identify and mitigate potential risks.
End-user adoption was prioritised through targeted training and ongoing engagement to optimise user experience. Clear model monitoring metrics were established, with defined ownership for tracking and reporting. A structured feedback loop was implemented to support continuous monitoring and improvement. Finally, well-defined, measurable Key Performance Indicators (KPIs) were established to evaluate success and inform future enhancements.
The key challenge was evaluating the Model Risk Management and Governance process to ensure the proposed rollout strategy and solution had all the controls in place to obtain necessary approvals.
Some of the key controls include:
- Content Moderation: This control is introduced before the query is sent to the Large Language Model (LLM). This prevents jail breaking, detects any harmful content present in the input query, prevents any bad actors with malicious intent to circumvent policies or commit fraud,removes any Personally Identifiable (PII) Data, etc.
- Human in the Loop: This ensures humans review the answers suggested by GenAI, provide feedback, make any necessary amendments, before sharing a response with the end users.
- LLM as a Judge: Evaluate the LLM response in real time using another LLM as a judge and calculate key metrics such as completeness, faithfulness and safety.
- Confidence Scoring: Additional Natural Language Processing (NLP) models were developed to assess the user query, the GenAI response,and the supporting information retrieved to generate that response.These models provide a red, amber, or green confidence rating that gives human reviewers an overall confidence score for each response.
- Model Monitoring: A comprehensive, continuous process for monitoring and evaluating the model’s performance against ground truth and capturing metrics that detect ongoing model drift.
Solutions and benefits
Icon consultants implemented a set of deliverables, including defining the necessary controls to achieve a safe and responsible AI solution, which includes Content Moderation, Risk Assessments and Confidence Scoring. In addition, Icon developed a comprehensive model monitoring design to evaluate performance both in real time and through periodic reviews, mitigating the risk of model drift and ensuring sustained accuracy.
To support operational integrity, Icon implemented a real-time knowledge management solution, ensuring policies and procedures remain up-to-date and aligned with regulatory requirements. This approach ensured that the AI system could provide reliable responses while adapting to evolving compliance standards.
The initial launch delivered measurable benefits. The average turnaround time (TAT) for queries was reduced by 50%, dropping from two days to less than one. Response accuracy was significantly improved, with Right First Time (RFT) scores, indicating compliance queries answered correctly on the first attempt, increasing from 59% to 78%. The effort for Risk Stewards responding to queries was reduced, although a human-in-the-loop process remains in place to review and provide feedback on GenAI-suggested responses. This is expected to decrease further as confidence in AIgenerated
answers increases.
This engagement highlights Icon’s ability to deliver both strategic consultancy and robust architecture, backed by our proven expertise in applying GenAI across real-world use cases. By delivering solutions that reduced query turnaround time, improved response accuracy, and freed up Risk Stewards to focus on higher-value tasks, we enabled the client to adopt AI responsibly and at scale.”
Our unique contribution
With consultants that are highly-skilled in the application of AI appropriately to real-world use cases, Icon’s expertise was instrumental in delivering a comprehensive solution suited to the client’s requirements.
Icon also leveraged its deep knowledge of Generative AI and prior experience in developing 'Ask ADA', an internal solution designed to provide responses to structured queries from Icon’s internal knowledge assets. This hands-on expertise enabled the team to apply proven design principles and accelerate delivery. In addition, Icon’s knowledge on similar GenAI-based engagements for other financial institutions provided valuable insights into regulatory considerations and best practices, ensuring a solution tailored to the client’s needs.
