面向視覺-語言模型的遞進(jìn)互提示學(xué)習(xí)

打開文本圖片集

doi： 10.19734/j. issn. 1001-3695.2024.10.0446

ProgCoPL： progressive co-prompting learning for vision-language models

Tao Junjie1，Zhang Weifeng1，2+，Wang Yuxia3，Miao Yi1 ，Xu Ling1 （1.Schoolofuece&o（lflellgee），ZgSUesit，g;2. Schoolfee&niUitinZ;i Institute，Jiaxing Zhejiang 31400o，China）

Abstract：Thelarge-scalepre-trainedvision-language modelCLIPaligns imagesandtexts inasharedsemanticspace，demonstratingrobust generalizationcapabilitiesacrossdiversedownstream tasks.However，existing promptlearning methodsoftenindependently insert learnable prompt vectors intoeach layerofCLIP's visualand text encoders.This appoach results in limitedcross-modalinteraction，withindependentpromptsacrosslayersfailing toefectivelyguidetheencoders incapturing taskrelevant information.Toaddress these isses，thispaper proposedProgCoPL.This method introduced text-guided promptvectorsintothevisualencoderlayersandvision-guidedpromptvectorsintothetextencoderlayers，therebyenhancingcro-modal interactionandalignment.Furthermore，ProgCoPL incorporated informationtransmissionchannelsbetweenpromptvectors acrosslayers，enablinghierarchicalandprogressiveintegrationof taskspecificinformation.Experimentson11datasetsshow thatProgCoPLeficientlyadaptsCLIPtodownstreamtasks，significantlyimprovingitscros-datasetgeneralizationability. ProgCoPLoutperforms existing methods in multiplegeneralization tests，particularlyachieving notable advancements incrossdataset scenarios.

Key Words：multimodal；prompt learning；vision-language model； Transformer encoder

0 引言

大規(guī)模視覺-語言模型（visual languagemodel，V-L Model）已經(jīng)成為當(dāng)今計算機(jī)跨模態(tài)智能領(lǐng)域的核心技術(shù)之一。（剩余19086字）