Building our own Generative AI solution to deliver high quality insights to our clients

As a leading Financial Services solutions company, Icon has built up a large body of knowledge and array of collateral over the years. This large collection of material
means that it can be difficult and time-consuming for Icon’s consultants to search through and find what they are looking for. With information spread across many documents and Confluence pages, it has been a laborious process to explore a given topic and summarise it appropriately.

Icon Solutions have developed “Ask Ada”, a Generative AI solution that enables
Consultants to use natural language queries to search our vast knowledge store. As a result, Icon’s Consultants can deliver high quality insights, that are built upon Icon’s in-depth experience of Financial Services, rapidly to clients.

Ask Ada is named as a tribute to Ada Lovelace, who is seen by many as the first
Computer Programmer.

To deliver Ask Ada, Icon ensured that it started with a clear business requirement,
that the data was complete, accurate and unbiased, that the underlying infrastructure was scalable to ensure that colleagues benefitted from rapid response times and that the solution was developed to produce high-quality results.

1. Dataset – recognising that Generative AI is dependent upon high-quality data: We cleaned the Confluence data and documents to ensure that noise and irrelevant metadata content is removed and then normalised the data into a consistent format for efficient processing. We associated the data with domains such as Payments, Cloud, AI, Data, Financial Crime, etc. as appropriate and we ensured that relevant metadata such as date modified was maintained. To ensure that the dataset is always kept up to date, we have created an automated overnight loading process.

2. Models – selecting the appropriate language model: We performed research and analysis into pre-trained open-source models such as Meta Llama and sentence transformers as well as exploring paid models like OpenAI’s GTP-4o. The main things we had to understand were the speed and quality of the response, ease of
   migrating to a different model as required, data privacy/security considerations as well as the total cost per query (including infrastructure costs).
3. Embeddings – how to generate high-dimensional vector representations: Embeddings capture the semantic meaning of the text and are generated as part of the text processing. We had to determine the appropriate embedding model based on the model size, retrieval average, number of dimensions and max tokens. We also had to run multiple proof of concepts with different models as well as dimension and token sizes to ensure that we had selected the most appropriate for the dataset based on two main metrics: embedding latency and quality of responses.
4. Vector Database – how to store the vector representations: Analyse which vector database to use, set up environments as appropriate and write code to store the embeddings in the vector database. Latency is a concern we wanted to address here so we also had to ensure that retrieval from the vector database was fast despite potentially being hosted in a different region due to vendor limitations.
5. Response – how do we generate the response quickly and efficiently? We want to reduce the Time to First Byte (TTFB) for the response, enabling what feels like realtime to the end user. To help achieve this, we investigated response streaming and determined that this is the most appropriate solution. This helps keep engagement levels high.
6. User Interface (UI) – how does the user interact with the tool? We had to ensure that the experience was as frictionless as possible. We explored creating a new login portal before deciding that we would enable OAuth2 integration to prevent the user from having additional credentials and the additional overhead management processes. We then had to address how the user asked a query and received a response. After conducting analysis into front-end libraries/frameworks, we opted to stick with JavaScript. The front-end makes a secure POST request to a back-end Python API which verifies the request before streaming the response back to the front-end in chunks. The chunks are then joined together and displayed with Markdown HTML formatting applied in real-time.
7. Query – how to word the query: We have two considerations when it comes to the query; 1) user and 2) system. On the user side, we have conducted research into what makes a good query/prompt and have provided training to consultants what language they should use and what language they should avoid. On the system side, we have created a prompt template which ensures that responses are in UK English, questions can only be asked about the data we have loaded and that no hallucinating should take place. It is better for everyone that if we don’t know, based on the given context, that we simply advise that instead of hallucinating something that could be believed, which may be factually incorrect.
8. Technology Architecture – how to deliver performance: We performed analysis and testing of various architecture components to ensure that we were able to deliver appropriate levels of performance whilst controlling costs. We also conducted analysis into deployment strategies to ensure that the solution was resilient and scalable, taking advantage of cloud capabilities including alerting and monitoring.

Through this approach, Icon was able to deliver an AI tool which allows our consultants to ask questions related to Icon’s vast knowledge base and receive responses, with links to the source material for additional reading.

This resulted in the following benefits:

Icon’s subject matter expertise in AI, architecture and technology was a key
success factor. We were able to make use of our strategic thinking to assess
options appropriately and determine the best route forward given the requirements.

Using our experience of Cloud-based architecture, we were able to ensure that
the solution was scalable and resilient. This also ensured that Ask Ada was
available to deliver high performance when it was rolled out to all of Icon’s
consultants.

Given the large number of horror stories of Generative AI hallucinating, from
systems giving incorrect advice to making biased recommendations, our
combination of prompt engineering and data governance ensured that Ask Ada
doesn’t try to make things up. As with many other aspects this was achieved
through a combination of ensuring data quality and robust engineering. 

By understanding the data that underpinned Ask Ada and ensuring that testing was thorough, we were able to ensure that high-quality results were returned. To do this, it was necessary to develop thousands of lines of Python code to improve answers, including developing solutions to handle acronyms, tabular data and the right metadata. Our experience was that high-level APIs did not provide the flexibility to deliver the quality that we required. This could only be achieved by adopting Icon’s approach of understanding the fundamentals of how the technology worked.