Multimedia Papers

ALT Unified multimodal models combine image understanding and generation within one system, creating an opportunity to improve both capabilities through paired image–text supervision. We examine the outputs of a pretrained unified model in both directions and observe a loss of specificity: generated images can contain the requested objects with incorrect attributes, while captions identify a scene without expressing its distinguishing details. These failures call for supervision that specifies both the visual content to realize and the descriptive differences to distinguish. We propose UniMore, a post-training method that derives these complementary learning tasks from the fine-grained content of image–text pairs. The generation route predicts grounded crops from their corresponding phrases, matching the scope of the condition and visual target while using the native visual objective. The understanding route combines positive-caption fitting with ranking against captions that omit or alter

ALT Audio-driven 3D head animation has made considerable progress in generation realism, yet generating coordinated behavior for two-person dialogue remains difficult, because current dialogue head generation methods overlook the modeling of non-verbal behavior semantics between the two participants. To address this limitation, we present DiaHeads, a framework that generates the behavior of both participants jointly. The two audio streams are concatenated and decoded together, so each participant's motion is conditioned on the other, which makes the interaction part of the generation itself. We further introduce four discrete non-verbal cues, gaze-on, head shaking, nodding, and smiling, and fuse them with the continuous parameters in a shared latent space through a product-of-experts posterior. These cues carry the interaction semantics and constrain how the continuous motion is generated. A flow-based prior over the shared latent variable replaces the standard Gaussian prior and makes the

ALT Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners. Turning them into virtual experiments requires not only reproducing the observed apparatus and state changes, but also enforcing the preconditions governing each operation. We introduce Lab2Sim, a benchmark of 218 science experiment videos with evidence-grounded specifications and 11,881 rubric items. To address this task, we propose EduVideo2Code, which extracts experimental procedures hierarchically, encodes them as explicit state machines to guide code generation, and refines the resulting pages through execution-based verification. Experiments show that existing multimodal models readily generate runnable pages but struggle to faithfully reconstruct experimental apparatus. EduVideo2Code consistently improves all four evaluation dimensions across two backbone models, with apparatus fidelity gains of 24.15 points.

ALT Music score files are more difficult to compare than their text-based equivalents, as their content is more complex and organised into hierarchical structures. This explains the lack of tools for processing music scores similar to the widely used old Unix diff command. We propose a new algorithm and tool for comparing digital music scores in various encodings, such as MusicXML or MEI, based on a graph-structured abstract representation called Tree Score Model (TSM). Given two scores, after conversion into TSM, our procedure calculates jointly: A list of differences, localised by time positions in the input scores; A time distance, cumulated duration of time-spans where scores differ; An edit distance, called surface distance, based on notational attributes not related to time. The two later measures are normalised and combined into a similarity distance between scores. Our procedure has been implemented in C++ and is available as a Python module. Our first experiments have shown a very

ALT Editing multi-camera concerts requires coordinated decisions about musical relevance, visual quality and cut timing. We present a modular framework that combines instrument activity from multitrack audio, visible instruments and image sharpness with beat-based timing and probabilistic camera selection. An interpretable scoring model links these components to the choice of views over complete shot intervals. Component tests and an exploratory expert study on a four-camera concert yield mixed assessments across the editing variants. The findings motivate extending local audiovisual relevance with contextual and sequence-level criteria. The architecture provides a basis for testing these extensions and studying how component behaviour affects the quality of complete edits.

ALT Multimodal misinformation detection methods often rely on different preprocessing pipelines and auxiliary resources, making experimental comparisons difficult to interpret. We present MMFact, a collection of shared resources and tools that simplifies this research workflow. It brings together 47 detector implementations and baselines, three multimodal datasets, and prepared materials ranging from semantic features to OCR, image descriptions, and external evidence. Compatible methods can use the same materials, reducing variation introduced by preprocessing and supporting fairer comparisons. A modular training and evaluation framework preserves method-specific designs while providing common experimental procedures. By treating prepared materials as a shared research resource, MMFact helps researchers focus on developing and evaluating detection methods.

ALT Advertisers routinely create multiple duration variants of ads to meet marketing budgets and viewer preferences, a labor-intensive and costly process. While multimodal large language models (MLLMs) excel at general video understanding, their ability to perform specialized editorial tasks remains unexplored. We introduce AdShot, a benchmark for evaluating MLLMs on shot selection for ad clipping. It contains 823 professionally edited ad pairs (30-sec sources and 15-sec edits) across 194 brands and 17 industries, annotated with shot-level boundaries and content features (action, emotion, information, imagery). We evaluate 13 state-of-the-art video-only MLLMs and 2 audio-visual MLLMs (7B-72B parameters) on shot selection accuracy (Precision, Recall, F1), duration constraints (15s), and content preservation, including an audio ablation. We find: (1) model architecture matters more than scale, with smaller models often matching or exceeding larger ones; performance is strongest for target ad

ALT This study develops an audio-visual emotion recognition system that integrates automatic utterance segmentation with multimodal fusion. Unlike conventional methods that rely on manually annotated utterance boundaries, the proposed system automatically generates candidate segments from continuous dialogue and extracts the corresponding audio and visual features for emotion classification. To address the challenges introduced by automatic segmentation, the proposed AGF-Net integrates Audio-Guided Fusion (AGF), Multimodal Negative Learning (MNL), and Predictive Dynamic Fusion (PDF) to improve cross-modal fusion, modality balance, and robustness to varying modality quality. Experimental results show that AGF-Net achieves 64.92% Accuracy and 65.47% F1-score under clean conditions, 62.13% Accuracy and 63.08% F1-score under moderate noise, and 58.19% Accuracy and 59.41% F1-score under higher noise. Overall, the proposed framework reduces reliance on manually annotated utterance boundaries and

ALT Reliable video perception over bandwidth-constrained wireless links requires transmitting information that the receiver cannot already recover from its evolving scene representation. This paper introduces WorldGT, a world-model-guided semantic packet transmission framework that prioritizes information according to its contribution to the receiver's perception. A shared feature-space world model predicts the receiver's scene state through affine motion compensation and learned residual displacement, while sender-side verification identifies reusable predictions. An acknowledgment-synchronized receiver twin estimates task-impact scores through two counterfactual evaluations, guiding packet selection and unequal error protection over LDPC-coded wireless channels. A compact 3-bit feature representation increases spatial coverage under stringent bandwidth constraints, while predictive fallback sustains perception during packet outages. Experiments on KITTI and VisDrone under Rayleigh fading

ALT Multimodal reasoning systems increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding. We show that this assumption fails in referring audio-visual segmentation (Ref-AVS): simple queries can be harmed by excessive reasoning, creating an "overthinking trap" associated with phrase and localization drift, while genuinely ambiguous queries still benefit from multi-step reasoning. To systematically study this, we construct counterfactual reasoning-budget labels under a five-fold out-of-fold evaluation protocol. Our results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem: before the first generated token, the generation-onset state contains linearly decodable information predictive of whether additional reasoning will improve segmentation. A linear probe predicts binary reasoning need with 70.1% accuracy, substantially outperforming text-only features and uncertainty-based proxy signals. As a router, the onset-state controlle

ALT The optical electric network frequency, the faint mains-driven flicker imprinted on rolling-shutter video, is increasingly proposed for verifying where footage was captured and for flagging manipulated or generated video. This letter shows that reading the mains frequency from a single short clip is confounded by the camera frame rate: at the rates in common use, a harmonic of that rate lands in the same narrow band as the true flicker, so the recovered grid is set by the frame rate, not the supply, and the estimator returns a confident value instead of abstaining. The underlying aliasing is known; its forensic consequence is not, and is the contribution here. On the 804 clips with a measurable in-band line the flicker feature does not separate genuine from generated video (area under the curve 0.509), while the recovered grid tracks the frame rate exactly for the generators, and a controlled synthesis confirms the mechanism against ground truth for genuine grids. Single-clip grid, and

ALT Pretrained unimodal models offer a flexible route to multimodal prediction through collaboration, but this collaboration can become less accurate than a surviving unimodal model when modalities are missing. We study this setting using fixed surviving-model anchors and find that collaborative gains can weaken or reverse as inputs disappear, even when the anchor's own input is unchanged. Our diagnostics further reveal reduced recovery of modality-specific information after aggregation and state-dependent collaborative representations. We propose Lluna, which combines a direct private prediction path from adapted unimodal features with a common graph-based collaboration branch that provides a residual correction. Progressive Collaboration Consistency organizes richer-to-poorer guidance along single-modality removals, aligning normalized common representations while task supervision adapts predictions to the available evidence. Across eight datasets spanning affective computing, scene unde

ALT Continuous multimodal sentiment analysis estimates polarity and intensity from linguistic, acoustic, and visual cues that may disagree. Hard judgments compress sentiment alternatives, while learned fusion interfaces tie task adaptation to model representations. We introduce Shared Sentiment Evidence (SSE), an output-space interface with fixed view and sentiment-option semantics across frozen backbones. Four queries yield five-option distributions whose probabilities, entropy, and cross-view relations form a 28-dimensional vector for CatBoost regression. On CH-SIMS-v2 and CMU-MOSI, the Qwen-based interface improves on supervised joint-only readout. Matched Ridge and CatBoost controls assess the contributions of additional views, deterministic summaries, and nonlinear readout. With the same interface specification and fixed readout settings, separately fitted heads also improve on joint-only prediction for three additional frozen backbones. These results support learning continuous senti

ALT Uniform frame sampling for vision-language model (VLM)-based video analysis often floods multimodal backbones with redundant content, while existing keyframe selection methods are typically optimized for human viewing rather than downstream extraction accuracy. We propose AdaFrame, a content-adaptive preprocessing cascade that reduces visual redundancy without sacrificing extraction fidelity. Three progressively finer filters, perceptual hashing, learned perceptual image patch similarity (LPIPS), and CLIP-based semantic clustering, efficiently eliminate redundant frames before costly cross-modal inference. Per-video frame budgets are determined using Intrinsic Semantic Dimensionality (ISD), which estimates each video's semantic complexity through singular value decomposition of its frame embeddings. Experiments on 500 video advertisements from the Pitt Ads Dataset show that AdaFrame reduces the frame count by 48 percent and VLM inference cost by 75 percent compared with uniform samplin

ALT Understanding human emotion is essential for multimodal systems that interpret both the content of an interaction and the speaker's expressive state. However, connecting paralinguistic cues to generated answers remains challenging, particularly when vocal expression diverges from visual appearance. We propose DualAffect, a dual-path architecture that integrates visually conditioned acoustic features and temporally organized acoustic tokens into a shared language decoder. The two pathways reuse the same acoustic encoding. The fusion pathway associates sound with visual content through visual queries, while the direct pathway compresses the valid acoustic sequence into ordered tokens through an independent projector. This design gives the decoder access to both joint audiovisual context and acoustic evidence organized along its own timeline. Mixed-task adaptation with task-dependent output preservation supports emotion recognition and general audiovisual question answering within the sam

ALT Open-vocabulary audio-visual event localization aims to infer explicit segment-level categories for both seen and unseen audio-visual events. Existing solutions rely heavily on hand-crafted prompt templates, whose predictions are highly sensitive to subtle wording variations. Learnable prompts remove this manual tuning burden, but when trained solely on seen events, they tend to overfit to seen categories and distort the inter-event semantic topology required for generalization to unseen categories, a failure we term semantic topology drift. In this paper, we propose SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams. Specifically, our SIMPLE combines Learnable Continuous Prompt Tuning (LCPT) with Semantic Topology Consistency Regularization (STCR). The LCPT automates prompt engineering through data-driven optimization, while the STCR module a

ALT Large Music Models (LMMs) can synthesize high-fidelity audio and support increasingly flexible prompt control, yet modeling the subjective preferences of an individual listener remains underexplored. We present PAG-Agent, a zero-shot multimodal framework that uses an explicit Personal Aesthetic Profile to bridge user preference inference and music generation control. A multimodal LLM distills favorite-song audio and contextual user information into this structured representation, which a planning LLM translates into generation instructions for existing music synthesizers. We further introduce PA-Eval, which evaluates personalization through the Aesthetic Consistency Score (ACS) for target-user alignment and the Aesthetic Specificity Index (ASI) for cross-user specificity. Experiments with Personal-100 on multiple MLLM and music-generation backbones show consistent improvements over generic and direct conditioning. A 50-participant follow-up study further shows a correlation r=0.64 betw

ALT AI-generated image collections combine visual content with heterogeneous provenance such as prompts, model parameters, and workflow graphs. Managing these collections locally requires coordinated metadata extraction, responsive browsing, and retrieval without mandatory remote inference. We present Berry AI Studio, an edge-first asset management system built with Vue, Tauri, Rust, SQLite, and ONNX Runtime. Berry integrates format-specific metadata parsing with raw-payload retention, generation-aware cancellation of queued thumbnail work, a persistent multi-tier thumbnail manifest, cursor pagination, and local image-text retrieval. We evaluate these mechanisms in five independent repetitions on a pinned source revision. With 50,000 catalog records, the median warm first-page database call was 285.3 ms; at offset 40,000, keyset pagination reduced median raw-query latency from 57.9 to 0.46 ms. A controlled browser harness recorded a 0.3 ms median-of-run P95 update time during virtualized s

ALT The continuous evolution of cloud gaming has enabled users to play high-quality cloud games on thin clients, such as smartphones and tablets. However, how to effectively transmit the gameplay frames with high resolution and low latency, especially over wireless networks, remains a significant challenge. To address this challenge, we propose a Deep Reinforcement Learning (DRL)-based Interest-Aware Adaptive Bitrate (ABR) scheme, called DIA, that guarantees the quality of perceptually key objects in each frame, while dynamically allocating lower resolutions for non-key objects to conserve bandwidth under constrained network conditions. By leveraging an off-the-shelf semantic segmentation model, the proposed DIA scheme can adjust the bitrate at the object-level. We also design a transformer-based feature extractor and an interest-aware reward function to enable the proposed DIA scheme to identify key objects and dynamically adjust their bitrates. This adaptation is guided by users' in-game

ALT Multimodal emotion preference learning uses pairwise comparisons to identify which open-ended emotion descriptions align more closely with human judgments. However, existing methods primarily focus on the final preference decision and provide little explicit supervision for localizing and correcting erroneous content. To address this limitation, we propose EmoGavel, a Grounding-Aware reinforcement learning framework with Verifiable rewards for multimodal Emotion preference Learning. We first introduce grounded supervised fine-tuning (Grounded SFT), which exploits controlled-error construction information to jointly learn preference prediction, error localization, repair, and missing-evidence recovery. We then optimize these verifiable outcomes with grounding-aware group relative policy optimization (GA-GRPO), combining preference-gated grounding rewards with dynamic zero-variance rescue to prioritize preference correctness and recover learning signals from all-wrong groups. Experiments