PMoE：在P-tuning中引入混合專家的參數(shù)高效微調(diào)框架

打開文本圖片集

關(guān)鍵詞：大語言模型；參數(shù)高效微調(diào)；P-tuning；混合專家；多任務(wù)學(xué)習(xí)中圖分類號(hào)：TP18 文獻(xiàn)標(biāo)志碼：A 文章編號(hào)：1001-3695（2025）07-005-1956-08doi：10.19734/j.issn.1001-3695.2024.11.0484

Abstract：Large language model （LLM）has significantly improved performanceinreasoning and generation tasks.However， existing open-sourceLLMstillackssuffcientdomain-specificknowledgeandrequiresfine-tunngforspecializedtasks.Traditionalfine-tuningmethodsstruggletobalancelowcostandhigheficiencyinmuli-taskleaing.Toaddressthisisue，hispaperproposedaparameter-effcientfine-tuning framework namedPMoE.BasedontheP-tuning method，this framework introducedamixture-of-expertsmechanism toenhancemulti-task proessingwhilemaintaininglow-costtuning.Ineach Transformer modulelayer，PMoE constructed trainable expert modules toreplace the prompt modules in P-tuning and utilizedarouting mechanism todynamicallyalocatetasksbasedoninput task features.Aditionally，itdesignedtheexpert modulesinMoEto bedetachable，enabling modelreuseacrossdferent task scenariosandfurtherreducingcomputationalcosts.Experimentalresults demonstrate that PMoE achieves a 6.24% performance improvement over P-tuning on a Chinese medical dataset and exhibitssuperiorcapabilities inmulti-taskprocessngandtransferlearning，verifying itseficiencyandbroadapplicability.

Key words：large language model；parameter-effcient fine-tuning；P-tuning；mixture of experts；multi-task learning

0 引言

隨著大語言模型（largelanguagemodel，LLM）的不斷迭代更新，這些模型在推理和文本生成方面的能力得到了顯著增強(qiáng)。（剩余21086字）