Current fabric recognition techniques often struggle with complex real-world environments due to their limited adaptability and generalization capabilities. To address these challenges, this paper introduces an innovative data processing framework that applies three distinct augmentation methods to training and testing datasets. This approach enhances the model's ability to recognize fabrics in diverse conditions. An optimized MobileNetV3 model is proposed, integrating Early Neural Attention (ENA) to focus on essential image features at an early stage. Furthermore, the conventional bottleneck structure is modified, and the ReLU activation function is replaced with SELU to improve robustness and convergence speed. Comparative experiments validate the improved model's effectiveness, demonstrating notable enhancements in accuracy, precision, recall, and F1-score. These results confirm the model's ability to perform reliably in dynamic and challenging fabric recognition scenarios.