Semantic profile construction for explainable and semantically consistent multimodal fashion recommendation
Year: 2026
Type: article
Abstract: Fashion recommendation requires heterogeneous visual and textual product content to be transformed into representations that are both retrievable and interpretable. We present FACTOR (Feature-Aware Contextual Optimization and Refinement), a framework that constructs six-field semantic profiles comprising category, subcategory, color, material, pattern, and cross-dataset style information, and combines them with image and multimodal retrieval. FACTOR integrates hierarchy-constrained category–subcategory decoding, cosine-based semantic retrieval, and late fusion, and is evaluated using both profile-derived consistency metrics and independent human judgments. FACTOR achieves category accuracy of 0.93, subcategory accuracy of 0.72, and semantic similarity of 0.85–0.87 for color, material, and pattern. Profile-based retrieval reaches weighted attribute consistency of approximately 0.99, exceeding image-only retrieval in semantic-profile agreement. In a 125-image in-domain audit, FACTOR achieved category/subcategory accuracy of 0.78/0.50 and multi-label F1 of 0.39, 0.22, and 0.28 for color, material, and pattern. LLaMA 3 metadata extraction achieved 0.64 precision (95% CI [0.58, 0.69]), identifying unsupported and missing labels as the main upstream risks. Hierarchy-constrained decoding reduced invalid category–subcategory pairs from 3.4% to 0% without new joint errors. In a matched ablation, the full FACTOR profile improved Human nDCG@5 over the base probe by 0.05 (95% CI [0.02, 0.09], Holm-adjusted p = 0.02), primarily through auxiliary refinement. On an independently judged 50-query benchmark, image-only FashionCLIP achieved the highest Human nDCG@5 (0.781), followed by native multimodal retrieval (0.707), rank fusion (0.674), and text-only retrieval (0.467). These findings show that structured semantic profiles provide strong semantic consistency and traceability, while visual representations remain essential for human-perceived relevance.
Authors: Shao-Yu Huang, Mohammad Masum
Institution: San Jose State University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,190
OpenAlex ID: W7225456924