AWS has released an open-source framework that aims to solve one of enterprise AI’s biggest limitations: searching large collections of documents for complex answers.
Most enterprise chatbots use retrieval-augmented generation, or RAG, which searches a knowledge base each time a user asks a question and sends the most relevant document fragments to the AI model. That approach works well for straightforward lookups but often struggles when answers require connecting information scattered across hundreds of files.
AWS is proposing a different approach called task-aware knowledge compression, according to a company blog post. Instead of repeatedly searching the original documents, the system analyzes an organization’s knowledge base in advance and creates compressed versions tailored to specific business tasks, such as financial analysis, regulatory compliance or legal review.
When users ask questions later, the AI works from those pre-built representations rather than searching the full document collection every time. AWS has released an open-source implementation that customers can deploy in their own accounts.
The technique could reduce both AI inference costs and response times because models process far fewer tokens for routine questions. AWS says the compressed representations can reduce the amount of context sent to a model by 8 to 64 times, while preserving information most relevant to a particular task. The system automatically selects different compression levels depending on the complexity of each query.
The tradeoff is flexibility. Because the compressed knowledge is optimized for a specific purpose, organizations must regenerate it when business priorities or source documents change. AWS says the approach is best suited for relatively stable knowledge bases that are queried repeatedly, such as financial due diligence, legal analysis and compliance reviews, rather than information that changes constantly.