Unlocking Scientific Knowledge: EKT’s AWS-Based AI Retrieval System

The National Documentation Center (EKT) of Greece is a public organization under the Ministry of Digital Governance, dedicated to making scientific knowledge widely accessible for research, education, and innovation. It manages a vast repository of research data, scientific publications, and services to support Open Science and knowledge dissemination. EKT’s initiatives include platforms for open-access publications, data repositories, and digital tools for researchers and academic institutions worldwide.

As a national leader in Open Science, EKT’s mission to foster knowledge sharing relied on ensuring its vast collection of over 25,000 Greek scientific articles was both accessible and relevant to researchers globally. However, traditional keyword-based search systems presented significant limitations:

• Contextual Search Limitations: Traditional search systems struggled with nuanced, context-dependent queries, making it difficult for researchers to locate information efficiently, especially when searching across scientific fields or in multiple languages.
• Fragmented User Experience: Without an intuitive search interface, users often needed to conduct multiple searches or rely on external tools to refine their results, slowing down the research process.
• Source Attribution and Integrity: EKT needed to ensure that search results included accurate citations for source material, reinforcing trust and academic integrity in research outputs.
• Scalability and Language Support: The knowledge base was expected to grow rapidly, requiring a system capable of scaling to meet future data demands while offering robust support for both Greek and English queries.

Additionally, EKT aimed to create a centralized platform that could support complex information retrieval while remaining simple for non-technical users to navigate.

To address these challenges, EKT partnered with PCG to implement a Retrieval-Augmented Generation (RAG) system built using AWS Cloud Services. The solution was designed to optimize information retrieval and integrate seamlessly with EKT’s existing infrastructure.

Data Storage and Embedding Generation

• Amazon S3 was used as a secure, scalable data source to store the knowledge base files.
• AWS Bedrock enabled embedding generation using Amazon Titan and text generation using Anthropic Claude 3, empowering EKT to harness state-of-the-art Large Language Models (LLMs).

High-Performance Search

• Amazon OpenSearch Service facilitated vector-based similarity search for complex queries. The service’s scalability ensured consistent performance across large data volumes.

Automated Resource Management

• Custom workflows using AWS Lambda ensured automated tagging and syncing of knowledge base updates.
• Cross-lingual support enabled users to retrieve information seamlessly in Greek and English.

This approach provided a cloud-based, scalable solution that maintained low operational costs while improving user experience.

The technical benefits of EKT’s new system addressed the root challenges posed by traditional search methods. By leveraging AWS’s advanced machine learning capabilities and high-performance search architecture, the platform enhanced retrieval precision, ensured real-time performance, and provided the scalability required to handle an expanding repository of scientific articles.

• Enhanced Retrieval Accuracy: The RAG system’s context-aware approach improved the precision of search results, reducing irrelevant results and improving the experience for users conducting complex, domain-specific queries.
• Cross-Lingual Functionality: The solution supported both Greek and English queries, ensuring broader accessibility for international research communities.
• Real-Time Performance: The integration of Amazon OpenSearch Service enabled fast similarity searches across large datasets, even as the repository grew in size.
• Reliable Source Attribution: The system ensured that generated outputs included citations and attributions, providing transparent and verifiable results.
• Scalability and Automation: The architecture was designed to scale automatically as more documents were added to the repository, while workflows using AWS Lambda streamlined updates to the knowledge base.

Business Benefits

By addressing technical limitations, the platform generated substantial business value, helping EKT fulfill its vision of making scientific knowledge more accessible and impactful. The improved system not only empowered researchers but also reinforced EKT’s position as a trusted leader in Open Science.

• Improved Research Efficiency: Researchers could now find relevant information more quickly, improving productivity and reducing the time spent refining search queries.
• Increased Trust in Outputs: By including proper source attribution, the platform reinforced academic integrity and encouraged wider adoption by the research community.
• Broader Accessibility: With cross-lingual support and a user-friendly interface, the platform extended EKT’s reach, making Greek scientific knowledge accessible to a global audience.
• Future-Proof Infrastructure: The system’s ability to scale ensured that EKT could continue to support Open Science initiatives as the volume of scientific data grows.

EKT’s forward-thinking use of AWS technologies has not only solved a complex challenge but also laid the groundwork for future innovation. With an intelligent, scalable platform in place, EKT is well-positioned to lead the way in Open Science across Europe and beyond.

Public Cloud Group (PCG) supports companies in their digital transformation through the use of public cloud solutions.

With a product portfolio designed to accompany organisations of all sizes in their cloud journey and competence that is a synonym for highly qualified staff that clients and partners like to work with, PCG is positioned as a reliable and trustworthy partner for the hyperscalers, relevant and with repeatedly validated competence and credibility.

We have the highest partnership status with the three relevant hyperscalers: Amazon Web Services (AWS), Google, and Microsoft. As experienced providers, we advise our customers independently with cloud implementation, application development, and managed services.