CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning

Modern automated audio captioning systems paira frozen audio encoder with a large language model (LLM)via a trainable projector, incurring the encoder’s inference costand bottlenecking the model through its fixe acoustic features.We present CARD, an encoder-free audio captioning model thatremoves the encoder at inference: a 13.2M projector feeds afrozen LLM with merged LoRA adapters, while the teacher usedto train it is discarded. CARD distills a pretrained audio teacher(CLAP-HTSAT) into the model, but rather than injecting it intothe LLM alone, it routes the teacher’s representations acrosscomponents: perceptual stages to the projector and semanticstages to the LLM. This placement improves CIDEr-D by +12.18over an LLM-only distilled model on AudioCaps and by +5.21 onClotho, reaching 55.4 against a 66.4 encoder-kept upper boundwith no encoder at inference, showing that where a teacher’sknowledge is placed matters as much as its presence.