Dai Wei, Zhu Xingce, Song Yang, Yang Xiao, Geng Xueyu, Ma Jingdong
[Objective] To compare the performance of various reasoning methods based on Large Language Model (LLM) in policy question answering (Q&A) for public health emergencies. [Methods] Using DeepSeek-R1 as the base model, we implemented five reasoning settings: retrieval-augmented generation (RAG), knowledge graph (KG) collaboration, fine-tuning, web search, and reasoning without external data. All reasoning methods were evaluated against a manually annotated Q&A benchmark. For the setting without external data, we also included the performance of Qwen-QwQ and GPT-4o for comparison. [Results] The approach integrating RAG, KG collaboration, and fine-tuning achieved the best performance in the automatic evaluation, with BLEU-4 and ROUGE-L scores of 0.259 and 0.494, respectively. Web search ranked second, achieving BLEU-4 and ROUGE-L scores of 0.225 and 0.465, respectively. In the manual evaluation, the approach integrating RAG, KG collaboration, and fine-tuning achieved the highest content accuracy score (3.560). Large-scale models without external data performed better in terms of sentence fluency, answer completeness, answer usability, and answer credibility. [Limitations] The experimental data were derived from publicly available online policy texts and did not include private data from internal settings or multimedia formats. Therefore, reasoning performance in multimodal scenarios could not be verified. [Conclusions] With the support of local data, various reasoning methods can significantly mitigate hallucinations and improve the accuracy in policy Q&A, demonstrating their strong applicability to vertical domains such as public health emergencies. While external data improves the accuracy of LLM-generated content, self-contained reasoning without external data tends to yield more complete and credible results.