AI for Textiles — Convolutional Neural Network Based Fabric Structure Classifier
Today, deep learning is used in a wide variety of artificial intelligence applications including facial recognition, natural language processing and so on. It is possible to find a number of applications of deep learning in the field of textile engineering as well, and computer vision has widely been used in this context. This article describes the approach used in developing a convolutional neural network for identifying fabric structures from input images of fabric surfaces. The developed model is capable of successfully distinguishing between knitted and woven fabric structures.
Knitted and woven structures can easily be distinguished due to their structural differences. The loop structure of knitted fabrics and the interlacing warp and weft yarns on woven fabrics allow the easy identification of the two structures. If a neural network can be trained to learn these features that are inherent to the fabric structures, by showing a set of labelled knitted and woven fabric images, then the neural network would be able to correctly distinguish between knitted and woven fabric images, that it has never seen before. In order to implement this, a convolutional neural network (CNN) architecture was decided to be used, as CNNs are capable of effectively extracting features from images.
The model was developed using python with the TensorFlow framework and Keras API. To obtain a dataset for training the neural network, an open source database of images available on, https://ibug.doc.ic.ac.uk/resources/fabrics/ was used, which was originally prepared for a research (C. Kampouris, S. Zafeiriou, A. Ghosh, S. Malassiotis, Fine-grained material classification using micro-geometry and reflectance, 14th European Conference on Computer Vision, Amsterdam, 2016). The fabric images in this original data set were labelled according to the type of material (i.e. nylon, polyester, cotton, etc.). Therefore, before training, a total of 4300 images were selected from this original data set and manually labelled them according to the fabric structure (i.e. Knitted and Woven). Out of the 4300 images, 4200 were used as training data, while the remaining 100 were used as validation data. (Even though the validation data set was too small, majority of the images were used for training to avoid overfitting). Both the training and validation data sets consisted of an equal number of knitted and woven fabric images.
Initially, the transfer learning technique was decided to be used. Therefore, the VGG16 architecture (https://arxiv.org/abs/1409.1556) was used with pre-trained weights. Only the final output layer was changed to be a softmax layer with two units. Using transfer learning, the final output layer was trained, keeping the weights of the other layers frozen, and after 100 epochs, the training and validation accuracy reached 88% and 83% respectively.
To improve the model, the final three dense layers of the original VGG16 architecture were removed, and replaced by a couple of slightly modified dense layers. Using transfer learning, these newly added layers were trained while keeping the weights of the remaining layers frozen. The model reached a maximum training accuracy of 99.81% and a validation accuracy of 91%. The model was now clearly overfitting to the training data.
To overcome the overfitting problem, again the final dense layers of the model were trained with a dropout layer added between the last two dense layers, along with data augmentation. However, after 20 epochs, the model reached a training accuracy of 84.55% and a validation accuracy of 84% and didn’t seem to be further improving. The overfitting problem was overcome, but now the model was having high bias.
Finally, it was decided to train the entire model, instead of using transfer learning. However, since the amount of training data available was limited, it was decided to reduce the complexity of the original VGG16 architecture. Hence, the fifth convolutional block of the original VGG16 architecture was removed and an average pooling layer was added, followed by the two dense layers. To avoid overfitting, data augmentation was used with several augmentation techniques such as rotating, vertical flipping, zooming and different brightness levels (https://keras.io/api/preprocessing/image/). Rotation of the input images is important as it allows the model to identify the wales of knitted fabric images and warp and weft yarns in woven fabric images, that are oriented in different directions, due to the variations that occur when capturing the images. Zooming in on the images allows the model to clearly identify the loop structure of knitted fabrics and the interlacing pattern of woven fabrics.
import numpy as np;
import keras;
from keras.layers import AveragePooling2D;
from keras. layers.core import Dense, Flatten;
from keras.optimizers import Adam;
from keras.metrics import binary_crossentropy;
from keras.preprocessing.image import ImageDataGenerator;
from keras.models import Model;
from keras.applications import imagenet_utils;
from keras.callbacks import ModelCheckpoint;train_data_path = '/content/drive/My Drive/fabric_data/Train';
test_data_path = '/content/drive/My Drive/fabric_data/Test';train_data = ImageDataGenerator(rescale = 1.0/255,
rotation_range = 180,
vertical_flip = True,
horizontal_flip = True,
brightness_range = [0.5, 1.5],
zoom_range = [1, 1.5]);train_generator = train_data.flow_from_directory(directory = train_data_path,
target_size = (224,224),
classes = ['Woven','Knitted'],
batch_size = 70,
shuffle = True);test_data = ImageDataGenerator(rescale = 1.0/255);test_generator = test_data.flow_from_directory(directory = test_data_path, target_size = (224,224), classes = ['Woven', 'Knitted'], batch_size = 50, shuffle = False);vgg16_model = keras.applications.VGG16();
x = vgg16_model.layers[-9].output;x = AveragePooling2D(pool_size = (2,2))(x);
x = Flatten(name="flatten")(x);
x = Dense(128, activation = 'relu')(x);
x = Dense(2, activation = 'softmax')(x);model = Model(inputs = vgg16_model.input, outputs = x);model.compile(optimizer = Adam(lr=0.00001, clipvalue = 0.5, clipnorm = 1), loss = 'binary_crossentropy', metrics = ['accuracy']);print("\nTraining.....");checkpoint = ModelCheckpoint(filepath = '/content/drive/My Drive/new_fab_model.h5', monitor='val_accuracy', verbose=1, save_best_only=True, mode='max');history = model.fit_generator(generator = train_generator,
steps_per_epoch = 60,
validation_data = test_generator,
validation_steps = 2,
epochs = 250,
verbose = 1,
callbacks = [checkpoint]);
The entire model was trained from scratch, using the Adam optimizer, at a learning rate of 0.00001. After 50 epochs of training, the model achieved a training accuracy of 98% and a validation accuracy of 97%.
Since the validation data set used was too small (only 100 images), in order to further validate the model’s performance in the real world, a different set of 100 fabric images were tested using the trained model. The model predicted 97 of those images correctly. The significance of this new test sample is that, the images were taken from a completely different distribution to the original training and validation data. One set of images were downloaded off the internet (3D knitted fabric images). The other set of images were scanned using a scanner and the images were zoomed in by 50%, cropped out and resized into 224x224 pixels to feed the neural network. The fabric images of the original training and validation data sets had been captured using a photometric stereo sensor (C. Kampouris, S. Zafeiriou, A. Ghosh, S. Malassiotis, Fine-grained material classification using micro-geometry and reflectance, 14th European Conference on Computer Vision, Amsterdam, 2016).
It should be noted that the training data for the model consisted of weft knitted fabrics only. Only technical front images of single jersey knitted structures were available and no 3D knitted structures were included. However, the trained model was capable of correctly predicting 3D cable knitted structures and it correctly predicted some of the single jersey technical back images as well. Most of the woven fabric images in the training set consisted of plain and twill structures.
Intermediate activations of the trained model were visualized to understand how the convolutions learn features from the fabric images. A knitted fabric image is fed to the model as the input and the corresponding layer activations are shown below. Please note that only some of the convolutions of a few layers are shown here.
The initial layers of the model seem to be identifying the most basic features of the image, such as horizontal and vertical edges. Some of the convolutions have identified the edges of the wales on the knitted fabric surface. In the middle layers, the convolutions begin to extract much finer details such as the shape of the knitted loops and the max pooling layers are highlighting these features. The activations of the deepest layers are difficult to interpret visually, as they are encoding information specific to the fabric structure, according to what the model has learned during training.
It should be noted that this model was developed for academic purposes only. The model is capable of distinguishing between two main fabric structures (i.e. Knitted and Woven) only. Distinguishing between several fabric structure variations such as single jersey, rib and interlock would be a more interesting task, but due to the unavailability of a large data set of such different types of fabric structures, the model was only limited to distinguishing between knitted and woven fabric structures. However, with sufficient data, a model can be trained for achieving such a task as well. It should further be noted that it may be possible to improve this model further, by using a different neural network architecture and more data.
