AI-Powered Document Question Answering System Using Agentic RAG and Local Language Models

Abstract: The rapid growth of digital information and unstructured data has transformed the way users search, analyze, and interact with documents. However, traditional search systems often lack contextual understanding, intelligent reasoning, and the ability to generate precise answers from multiple sources. This paper presents DocAgent, an AI- powered Agentic Retrieval-Augmented Generation (RAG) system designed to provide accurate and context-aware responses from user-provided documents and web content. The proposed system integrates document ingestion, semantic search, and local large language model processing to enable efficient knowledge retrieval. Documents and URLs are processed into vector embeddings and stored in a vector database, allowing the system to perform similarity-based retrieval. An agent-based workflow dynamically decides whether to retrieve relevant context, refine the query, or directly generate responses, improving accuracy and reducing irrelevant outputs. The system utilizes a locally hosted language model through Ollama, ensuring privacy, cost efficiency, and reduced dependency on external APIs. The application is evaluated using real-time query scenarios, demonstrating improved response relevance, reduced hallucinations, and enhanced user interaction compared to traditional document search systems.

Keywords: Artificial Intelligence, Retrieval-Augmented Generation, Agentic AI, Document Question Answering, Ollama, Vector Database, Semantic Search, Web-Based System