Introduction to Convolutional Neural Networks#

CSC/DSC 340 Week 10 Slides

Author: Dr. Julie Butler

Date Created: October 21, 2023

Last Modified: October 21, 2023

Introduction and Termoinology#

  • This week we will cover convolutional neural networks (CNNs)

    • Image analysis, video analysis, object detection

  • Note that we will not be going through any mathematics this week as the mathematics of CNNs is quite complicated but there are many good resources (including your textbook) if you are interested

  • Convolution refers to the mathematical combination of two functions to produce a third function

    • It merges two sets of information

    • The convolution is performed on the input data with the use of a filter or kernel (these terms are used interchangeably) to then produce a feature map.

CNNs

Image Source

Introduction to the MNIST Data Set#

  • Photos (28 pixels by 28 pixels) of handwritten numeric digits as the input

  • Number shown in the photo as an output

  • Challenges: 2D data structure (images), variations in handwriting, low quality images, size of data set (60,000 images in the training set, 10,000 in the test set)

Import With Tensorflow#

  • Also avaliable through Scikit-Learn, but does not come automatically split into a training and test set

import tensorflow as tf

# Load MNIST data
mnist = tf.keras.datasets.mnist
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

# Normalize the data
train_images, test_images = train_images / 255.0, test_images / 255.0

# Print the shape of the data
print("Train images shape:", train_images.shape)
print("Train labels shape:", train_labels.shape)
print("Test images shape:", test_images.shape)
print("Test labels shape:", test_labels.shape)

Train images shape: (60000, 28, 28)
Train labels shape: (60000,)
Test images shape: (10000, 28, 28)
Test labels shape: (10000,)

  • Let’s display some of the images

import matplotlib.pyplot as plt
# Display a small number of images
num_images = 5
plt.figure(figsize=(10, 3))
for i in range(num_images):
    plt.subplot(1, num_images, i + 1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(train_images[i], cmap=plt.cm.binary)
    plt.xlabel(train_labels[i])
plt.show()
_images/ed81f33164a680f126bb6c3ae3e04f9efcc72f9214ac0b22619c19eb14cce87b.png

  • 60k is a lot of images to have in a training set (though may be needed for large neural networks)

  • Let’s randomly select 5k images to use for training instead of 60k so networks train faster

  • Can use a smaller data set in the construction of your neural network (hyperparameter tuning process) but then use a larger sample to train the final network

import numpy as np
random_indices = np.random.choice(60000, 5000, replace=False)
train_images_smaller = train_images[random_indices]
train_labels_smaller = train_labels[random_indices]

  • Classification can suffer from class imbalances. Let’s make sure our data is relatively evenly distributed.

# Create a histogram of the labels
plt.figure(figsize=(8, 5))
plt.hist(train_labels_smaller, bins=range(11), align='left', rwidth=0.8, color='skyblue', edgecolor='black')
plt.title('Histogram of Labels in MNIST Training Set')
plt.xlabel('Label')
plt.ylabel('Frequency')
plt.xticks(range(10))
plt.grid(axis='y', alpha=0.75)
plt.show()
_images/e4de6ca1579e7d11244296be505ab015534311f687d6f0da6c0fe19917a9aebe.png

Classification with a Regular Neural Network#

  • We can clasify the MNIST data with a regular neural network, but due to its architecture we have to flatten the data before it can reach the dense layers

  • Neural networks can be used to successfully classify images, but flattening the images can remove important patterns

  • Create a model that flattens the data (28x28 pixel images). We then have one hidden layer with 128 neurons and a Relu activation function, and an output layer with 10 neurons (10 possible outputs) and a softmax activation function since this is a classification.

from tensorflow.keras import layers, models
# Build the model
model = models.Sequential([
    layers.Flatten(input_shape=(28, 28)),
    layers.Dense(128, activation='relu'),
    layers.Dense(10, activation='softmax')
])

  • For compiling the model we will use the Adam optimizer, our metric of success will be accuracy, and our loss function is sparse categorical cross-entropy

    • Sparse categorical cross-entropy is similiar to categorical cross-entropy but while categorical cross-entropy requires the data to be one-hot encoded prior to training the model, sparse categorical cross-entropy does not

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

  • Train the model and the determine the accuracy

# Train the model
model.fit(train_images, train_labels, epochs=5, verbose=1)

# Evaluate the model
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print('\nTest accuracy:', test_acc)

Epoch 1/5

   1/1875 [..............................] - ETA: 6:06 - loss: 2.4967 - accuracy: 0.0938


  63/1875 [>.............................] - ETA: 1s - loss: 1.1396 - accuracy: 0.6811  


 131/1875 [=>............................] - ETA: 1s - loss: 0.7930 - accuracy: 0.7765


 199/1875 [==>...........................] - ETA: 1s - loss: 0.6493 - accuracy: 0.8183


 267/1875 [===>..........................] - ETA: 1s - loss: 0.5709 - accuracy: 0.8407


 335/1875 [====>.........................] - ETA: 1s - loss: 0.5201 - accuracy: 0.8543


 403/1875 [=====>........................] - ETA: 1s - loss: 0.4821 - accuracy: 0.8635


 472/1875 [======>.......................] - ETA: 1s - loss: 0.4538 - accuracy: 0.8712


 541/1875 [=======>......................] - ETA: 0s - loss: 0.4293 - accuracy: 0.8781


 606/1875 [========>.....................] - ETA: 0s - loss: 0.4084 - accuracy: 0.8837


 675/1875 [=========>....................] - ETA: 0s - loss: 0.3900 - accuracy: 0.8885


 744/1875 [==========>...................] - ETA: 0s - loss: 0.3747 - accuracy: 0.8932


 814/1875 [============>.................] - ETA: 0s - loss: 0.3603 - accuracy: 0.8973


 884/1875 [=============>................] - ETA: 0s - loss: 0.3503 - accuracy: 0.8999


 953/1875 [==============>...............] - ETA: 0s - loss: 0.3400 - accuracy: 0.9028


1023/1875 [===============>..............] - ETA: 0s - loss: 0.3295 - accuracy: 0.9061


1092/1875 [================>.............] - ETA: 0s - loss: 0.3201 - accuracy: 0.9085


1162/1875 [=================>............] - ETA: 0s - loss: 0.3129 - accuracy: 0.9106


1231/1875 [==================>...........] - ETA: 0s - loss: 0.3062 - accuracy: 0.9127


1300/1875 [===================>..........] - ETA: 0s - loss: 0.2998 - accuracy: 0.9144


1370/1875 [====================>.........] - ETA: 0s - loss: 0.2936 - accuracy: 0.9164


1442/1875 [======================>.......] - ETA: 0s - loss: 0.2872 - accuracy: 0.9181


1510/1875 [=======================>......] - ETA: 0s - loss: 0.2817 - accuracy: 0.9196


1565/1875 [========================>.....] - ETA: 0s - loss: 0.2776 - accuracy: 0.9208


1620/1875 [========================>.....] - ETA: 0s - loss: 0.2736 - accuracy: 0.9219


1686/1875 [=========================>....] - ETA: 0s - loss: 0.2692 - accuracy: 0.9233


1750/1875 [===========================>..] - ETA: 0s - loss: 0.2647 - accuracy: 0.9244


1819/1875 [============================>.] - ETA: 0s - loss: 0.2614 - accuracy: 0.9252


1875/1875 [==============================] - 2s 750us/step - loss: 0.2581 - accuracy: 0.9262

Epoch 2/5

   1/1875 [..............................] - ETA: 2s - loss: 0.0811 - accuracy: 0.9688


  68/1875 [>.............................] - ETA: 1s - loss: 0.1593 - accuracy: 0.9559


 139/1875 [=>............................] - ETA: 1s - loss: 0.1508 - accuracy: 0.9584


 208/1875 [==>...........................] - ETA: 1s - loss: 0.1419 - accuracy: 0.9602


 278/1875 [===>..........................] - ETA: 1s - loss: 0.1344 - accuracy: 0.9608


 347/1875 [====>.........................] - ETA: 1s - loss: 0.1296 - accuracy: 0.9623


 416/1875 [=====>........................] - ETA: 1s - loss: 0.1287 - accuracy: 0.9621


 488/1875 [======>.......................] - ETA: 1s - loss: 0.1287 - accuracy: 0.9623


 557/1875 [=======>......................] - ETA: 0s - loss: 0.1279 - accuracy: 0.9625


 627/1875 [=========>....................] - ETA: 0s - loss: 0.1267 - accuracy: 0.9629


 696/1875 [==========>...................] - ETA: 0s - loss: 0.1238 - accuracy: 0.9639


 765/1875 [===========>..................] - ETA: 0s - loss: 0.1217 - accuracy: 0.9644


 835/1875 [============>.................] - ETA: 0s - loss: 0.1217 - accuracy: 0.9647


 906/1875 [=============>................] - ETA: 0s - loss: 0.1209 - accuracy: 0.9649


 976/1875 [==============>...............] - ETA: 0s - loss: 0.1200 - accuracy: 0.9651


1045/1875 [===============>..............] - ETA: 0s - loss: 0.1211 - accuracy: 0.9650


1117/1875 [================>.............] - ETA: 0s - loss: 0.1192 - accuracy: 0.9656


1187/1875 [=================>............] - ETA: 0s - loss: 0.1189 - accuracy: 0.9655


1258/1875 [===================>..........] - ETA: 0s - loss: 0.1189 - accuracy: 0.9654


1329/1875 [====================>.........] - ETA: 0s - loss: 0.1182 - accuracy: 0.9656


1399/1875 [=====================>........] - ETA: 0s - loss: 0.1171 - accuracy: 0.9659


1469/1875 [======================>.......] - ETA: 0s - loss: 0.1166 - accuracy: 0.9659


1538/1875 [=======================>......] - ETA: 0s - loss: 0.1153 - accuracy: 0.9663


1607/1875 [========================>.....] - ETA: 0s - loss: 0.1143 - accuracy: 0.9665


1677/1875 [=========================>....] - ETA: 0s - loss: 0.1145 - accuracy: 0.9666


1747/1875 [==========================>...] - ETA: 0s - loss: 0.1148 - accuracy: 0.9664


1817/1875 [============================>.] - ETA: 0s - loss: 0.1138 - accuracy: 0.9666


1875/1875 [==============================] - 1s 725us/step - loss: 0.1126 - accuracy: 0.9670

Epoch 3/5

   1/1875 [..............................] - ETA: 3s - loss: 0.0950 - accuracy: 0.9688


  63/1875 [>.............................] - ETA: 1s - loss: 0.0768 - accuracy: 0.9816


 132/1875 [=>............................] - ETA: 1s - loss: 0.0758 - accuracy: 0.9754


 201/1875 [==>...........................] - ETA: 1s - loss: 0.0779 - accuracy: 0.9759


 270/1875 [===>..........................] - ETA: 1s - loss: 0.0775 - accuracy: 0.9763


 339/1875 [====>.........................] - ETA: 1s - loss: 0.0761 - accuracy: 0.9771


 408/1875 [=====>........................] - ETA: 1s - loss: 0.0767 - accuracy: 0.9772


 478/1875 [======>.......................] - ETA: 1s - loss: 0.0783 - accuracy: 0.9765


 547/1875 [=======>......................] - ETA: 0s - loss: 0.0780 - accuracy: 0.9767


 616/1875 [========>.....................] - ETA: 0s - loss: 0.0778 - accuracy: 0.9771


 685/1875 [=========>....................] - ETA: 0s - loss: 0.0787 - accuracy: 0.9767


 754/1875 [===========>..................] - ETA: 0s - loss: 0.0782 - accuracy: 0.9767


 824/1875 [============>.................] - ETA: 0s - loss: 0.0787 - accuracy: 0.9767


 894/1875 [=============>................] - ETA: 0s - loss: 0.0780 - accuracy: 0.9770


 964/1875 [==============>...............] - ETA: 0s - loss: 0.0775 - accuracy: 0.9770


1033/1875 [===============>..............] - ETA: 0s - loss: 0.0776 - accuracy: 0.9770


1103/1875 [================>.............] - ETA: 0s - loss: 0.0786 - accuracy: 0.9767


1172/1875 [=================>............] - ETA: 0s - loss: 0.0788 - accuracy: 0.9766


1242/1875 [==================>...........] - ETA: 0s - loss: 0.0780 - accuracy: 0.9767


1305/1875 [===================>..........] - ETA: 0s - loss: 0.0785 - accuracy: 0.9765


1351/1875 [====================>.........] - ETA: 0s - loss: 0.0788 - accuracy: 0.9766


1405/1875 [=====================>........] - ETA: 0s - loss: 0.0790 - accuracy: 0.9765


1471/1875 [======================>.......] - ETA: 0s - loss: 0.0791 - accuracy: 0.9765


1537/1875 [=======================>......] - ETA: 0s - loss: 0.0790 - accuracy: 0.9765


1602/1875 [========================>.....] - ETA: 0s - loss: 0.0789 - accuracy: 0.9766


1668/1875 [=========================>....] - ETA: 0s - loss: 0.0789 - accuracy: 0.9767


1737/1875 [==========================>...] - ETA: 0s - loss: 0.0786 - accuracy: 0.9768


1805/1875 [===========================>..] - ETA: 0s - loss: 0.0784 - accuracy: 0.9768


1875/1875 [==============================] - ETA: 0s - loss: 0.0782 - accuracy: 0.9768


1875/1875 [==============================] - 1s 754us/step - loss: 0.0782 - accuracy: 0.9768

Epoch 4/5

   1/1875 [..............................] - ETA: 2s - loss: 0.0111 - accuracy: 1.0000


  69/1875 [>.............................] - ETA: 1s - loss: 0.0488 - accuracy: 0.9851


 139/1875 [=>............................] - ETA: 1s - loss: 0.0487 - accuracy: 0.9854


 201/1875 [==>...........................] - ETA: 1s - loss: 0.0546 - accuracy: 0.9831


 267/1875 [===>..........................] - ETA: 1s - loss: 0.0542 - accuracy: 0.9831


 336/1875 [====>.........................] - ETA: 1s - loss: 0.0547 - accuracy: 0.9828


 405/1875 [=====>........................] - ETA: 1s - loss: 0.0558 - accuracy: 0.9830


 475/1875 [======>.......................] - ETA: 1s - loss: 0.0565 - accuracy: 0.9832


 544/1875 [=======>......................] - ETA: 0s - loss: 0.0555 - accuracy: 0.9837


 614/1875 [========>.....................] - ETA: 0s - loss: 0.0558 - accuracy: 0.9838


 683/1875 [=========>....................] - ETA: 0s - loss: 0.0561 - accuracy: 0.9836


 753/1875 [===========>..................] - ETA: 0s - loss: 0.0573 - accuracy: 0.9832


 822/1875 [============>.................] - ETA: 0s - loss: 0.0589 - accuracy: 0.9827


 891/1875 [=============>................] - ETA: 0s - loss: 0.0583 - accuracy: 0.9830


 960/1875 [==============>...............] - ETA: 0s - loss: 0.0582 - accuracy: 0.9831


1030/1875 [===============>..............] - ETA: 0s - loss: 0.0581 - accuracy: 0.9832


1100/1875 [================>.............] - ETA: 0s - loss: 0.0582 - accuracy: 0.9833


1170/1875 [=================>............] - ETA: 0s - loss: 0.0590 - accuracy: 0.9830


1240/1875 [==================>...........] - ETA: 0s - loss: 0.0594 - accuracy: 0.9828


1308/1875 [===================>..........] - ETA: 0s - loss: 0.0594 - accuracy: 0.9827


1377/1875 [=====================>........] - ETA: 0s - loss: 0.0594 - accuracy: 0.9826


1446/1875 [======================>.......] - ETA: 0s - loss: 0.0589 - accuracy: 0.9829


1514/1875 [=======================>......] - ETA: 0s - loss: 0.0586 - accuracy: 0.9829


1583/1875 [========================>.....] - ETA: 0s - loss: 0.0590 - accuracy: 0.9827


1653/1875 [=========================>....] - ETA: 0s - loss: 0.0591 - accuracy: 0.9828


1724/1875 [==========================>...] - ETA: 0s - loss: 0.0588 - accuracy: 0.9827


1795/1875 [===========================>..] - ETA: 0s - loss: 0.0589 - accuracy: 0.9826


1865/1875 [============================>.] - ETA: 0s - loss: 0.0589 - accuracy: 0.9826


1875/1875 [==============================] - 1s 731us/step - loss: 0.0589 - accuracy: 0.9826

Epoch 5/5

   1/1875 [..............................] - ETA: 2s - loss: 0.0651 - accuracy: 0.9688


  70/1875 [>.............................] - ETA: 1s - loss: 0.0346 - accuracy: 0.9884


 137/1875 [=>............................] - ETA: 1s - loss: 0.0376 - accuracy: 0.9893


 206/1875 [==>...........................] - ETA: 1s - loss: 0.0391 - accuracy: 0.9877


 275/1875 [===>..........................] - ETA: 1s - loss: 0.0395 - accuracy: 0.9882


 344/1875 [====>.........................] - ETA: 1s - loss: 0.0391 - accuracy: 0.9880


 411/1875 [=====>........................] - ETA: 1s - loss: 0.0385 - accuracy: 0.9880


 478/1875 [======>.......................] - ETA: 1s - loss: 0.0410 - accuracy: 0.9875


 544/1875 [=======>......................] - ETA: 0s - loss: 0.0416 - accuracy: 0.9874


 610/1875 [========>.....................] - ETA: 0s - loss: 0.0422 - accuracy: 0.9872


 677/1875 [=========>....................] - ETA: 0s - loss: 0.0419 - accuracy: 0.9872


 744/1875 [==========>...................] - ETA: 0s - loss: 0.0415 - accuracy: 0.9873


 795/1875 [===========>..................] - ETA: 0s - loss: 0.0418 - accuracy: 0.9871


 859/1875 [============>.................] - ETA: 0s - loss: 0.0418 - accuracy: 0.9870


 924/1875 [=============>................] - ETA: 0s - loss: 0.0426 - accuracy: 0.9865


 990/1875 [==============>...............] - ETA: 0s - loss: 0.0421 - accuracy: 0.9869


1056/1875 [===============>..............] - ETA: 0s - loss: 0.0426 - accuracy: 0.9867


1122/1875 [================>.............] - ETA: 0s - loss: 0.0431 - accuracy: 0.9866


1187/1875 [=================>............] - ETA: 0s - loss: 0.0431 - accuracy: 0.9867


1252/1875 [===================>..........] - ETA: 0s - loss: 0.0428 - accuracy: 0.9867


1319/1875 [====================>.........] - ETA: 0s - loss: 0.0430 - accuracy: 0.9867


1385/1875 [=====================>........] - ETA: 0s - loss: 0.0434 - accuracy: 0.9865


1450/1875 [======================>.......] - ETA: 0s - loss: 0.0440 - accuracy: 0.9862


1515/1875 [=======================>......] - ETA: 0s - loss: 0.0443 - accuracy: 0.9861


1580/1875 [========================>.....] - ETA: 0s - loss: 0.0447 - accuracy: 0.9859


1647/1875 [=========================>....] - ETA: 0s - loss: 0.0450 - accuracy: 0.9859


1716/1875 [==========================>...] - ETA: 0s - loss: 0.0451 - accuracy: 0.9857


1785/1875 [===========================>..] - ETA: 0s - loss: 0.0454 - accuracy: 0.9858


1850/1875 [============================>.] - ETA: 0s - loss: 0.0454 - accuracy: 0.9858


1875/1875 [==============================] - 1s 763us/step - loss: 0.0456 - accuracy: 0.9857

313/313 - 0s - loss: 0.0863 - accuracy: 0.9753 - 229ms/epoch - 732us/step

Test accuracy: 0.9753000140190125

  • When we perform classification without one-hot encoding, the outputs of the model are not the class labels, but rather the probability that the input belongs to each class

# Predict the test set
y_pred = model.predict(test_images)
print(y_pred[0])

  1/313 [..............................] - ETA: 13s


101/313 [========>.....................] - ETA: 0s 


208/313 [==================>...........] - ETA: 0s


313/313 [==============================] - 0s 478us/step

[2.3810202e-08 2.7606712e-09 4.0897237e-07 1.5288603e-04 6.8588052e-12
 3.6298051e-08 4.4512565e-13 9.9984586e-01 3.8440454e-08 6.5594935e-07]

  • Before performing further anaysis on our neural network outputs, we need to convert the data from probabilities to the actual class label

# Predict the test set
y_pred = np.argmax(model.predict(test_images), axis=-1)

print(y_pred[0])

  1/313 [..............................] - ETA: 2s


104/313 [========>.....................] - ETA: 0s


211/313 [===================>..........] - ETA: 0s


313/313 [==============================] - 0s 479us/step

7

  • Now let’s create a confusion matrix to see which points are being misclassified

from sklearn.metrics import confusion_matrix
import seaborn as sns

# Create confusion matrix
cm = confusion_matrix(test_labels, y_pred)

# Plot confusion matrix
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', cbar=False)
plt.xlabel('Predicted labels')
plt.ylabel('True labels')
plt.title('Confusion Matrix for MNIST Classification')
plt.show()
_images/d801af5b115bccea6e386641e5545f661ee98d2169efaa9314b2f9d0972b4798.png

Classification with a Convolutional Neural Network#

Types of Layers and Keras Syntax#

  • Conv2D

    • Documentation

    • The first number is the number of filters (dimensionality of the output)

    • The tuple is the stride (the width and height of the section of pixels to be reduced down)

    • Activation has the same function here as in regular neural networks

    • Convolutional layers apply learnable filters to the data that creates a feature map to a higher or lower dimensional space

    • Used to detect important features and patterns in the data

  • MaxPooling2D

    • Documentation

    • Tuple is the pool size (the size block over which to find the maximum)

    • Pooling layers reduce the dimensionality of the data while keeping the most important features

    • Two types of pooling layers: max pooling and average pooling

  • Stacking convolutional and pooling layers allows CNNS to learn in a heirarchical manner

    • First the networks learns basic featutes of the data (like edges and textures) and then more complicated features

    • This heirarchical learning is what makes CNNS so effective at image analysis

  • Note that not every convolutional layer has to be followed by a pooling layer, too many pooling layers can be bad

Full Neural Network for Classification#

  • Have two pairs of convolutional layers/pooling layers of different sizes followed by a lone convolutional layer

  • The Flatten() layer is needed to take the 2D data down to one dimension for the dense layers

  • The first dense layer (a hidden layer) does some post-processing on the data that comes from the CNN layers, the second dense layer is the output layer

# Build the model
model = models.Sequential([
    layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dense(10, activation='softmax')
])

  • Model is compiled, trained, and tested in the same way as the regular neural network

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(train_images, train_labels, epochs=5, batch_size=64, validation_split=0.1)

# Evaluate the model
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print('\nTest accuracy:', test_acc)

Epoch 1/5

  1/844 [..............................] - ETA: 3:13 - loss: 2.2898 - accuracy: 0.1562


  6/844 [..............................] - ETA: 9s - loss: 2.2472 - accuracy: 0.1927  


 11/844 [..............................] - ETA: 8s - loss: 2.1747 - accuracy: 0.3054


 16/844 [..............................] - ETA: 8s - loss: 2.0425 - accuracy: 0.3818


 21/844 [..............................] - ETA: 8s - loss: 1.8709 - accuracy: 0.4479


 26/844 [..............................] - ETA: 8s - loss: 1.6806 - accuracy: 0.5084


 31/844 [>.............................] - ETA: 8s - loss: 1.5200 - accuracy: 0.5539


 36/844 [>.............................] - ETA: 8s - loss: 1.3950 - accuracy: 0.5855


 41/844 [>.............................] - ETA: 8s - loss: 1.3017 - accuracy: 0.6117


 46/844 [>.............................] - ETA: 8s - loss: 1.2267 - accuracy: 0.6345


 51/844 [>.............................] - ETA: 8s - loss: 1.1531 - accuracy: 0.6547


 56/844 [>.............................] - ETA: 8s - loss: 1.0930 - accuracy: 0.6749


 60/844 [=>............................] - ETA: 8s - loss: 1.0533 - accuracy: 0.6859


 65/844 [=>............................] - ETA: 8s - loss: 1.0032 - accuracy: 0.6993


 70/844 [=>............................] - ETA: 8s - loss: 0.9595 - accuracy: 0.7116


 75/844 [=>............................] - ETA: 8s - loss: 0.9194 - accuracy: 0.7244


 80/844 [=>............................] - ETA: 8s - loss: 0.8840 - accuracy: 0.7346


 85/844 [==>...........................] - ETA: 8s - loss: 0.8493 - accuracy: 0.7441


 90/844 [==>...........................] - ETA: 8s - loss: 0.8184 - accuracy: 0.7524


 95/844 [==>...........................] - ETA: 8s - loss: 0.7915 - accuracy: 0.7610


100/844 [==>...........................] - ETA: 8s - loss: 0.7675 - accuracy: 0.7684


105/844 [==>...........................] - ETA: 8s - loss: 0.7458 - accuracy: 0.7749


110/844 [==>...........................] - ETA: 8s - loss: 0.7230 - accuracy: 0.7815


115/844 [===>..........................] - ETA: 7s - loss: 0.7034 - accuracy: 0.7871


120/844 [===>..........................] - ETA: 7s - loss: 0.6805 - accuracy: 0.7947


125/844 [===>..........................] - ETA: 7s - loss: 0.6606 - accuracy: 0.8005


130/844 [===>..........................] - ETA: 7s - loss: 0.6444 - accuracy: 0.8049


135/844 [===>..........................] - ETA: 7s - loss: 0.6270 - accuracy: 0.8101


140/844 [===>..........................] - ETA: 7s - loss: 0.6145 - accuracy: 0.8137


145/844 [====>.........................] - ETA: 7s - loss: 0.6044 - accuracy: 0.8172


150/844 [====>.........................] - ETA: 7s - loss: 0.5932 - accuracy: 0.8213


155/844 [====>.........................] - ETA: 7s - loss: 0.5797 - accuracy: 0.8251


160/844 [====>.........................] - ETA: 7s - loss: 0.5668 - accuracy: 0.8288


165/844 [====>.........................] - ETA: 7s - loss: 0.5552 - accuracy: 0.8322


170/844 [=====>........................] - ETA: 7s - loss: 0.5446 - accuracy: 0.8353


174/844 [=====>........................] - ETA: 7s - loss: 0.5355 - accuracy: 0.8382


179/844 [=====>........................] - ETA: 7s - loss: 0.5237 - accuracy: 0.8417


184/844 [=====>........................] - ETA: 7s - loss: 0.5160 - accuracy: 0.8442


189/844 [=====>........................] - ETA: 7s - loss: 0.5056 - accuracy: 0.8471


194/844 [=====>........................] - ETA: 7s - loss: 0.4950 - accuracy: 0.8504


199/844 [======>.......................] - ETA: 7s - loss: 0.4849 - accuracy: 0.8536


203/844 [======>.......................] - ETA: 7s - loss: 0.4779 - accuracy: 0.8554


208/844 [======>.......................] - ETA: 7s - loss: 0.4701 - accuracy: 0.8578


213/844 [======>.......................] - ETA: 7s - loss: 0.4634 - accuracy: 0.8597


217/844 [======>.......................] - ETA: 7s - loss: 0.4583 - accuracy: 0.8612


221/844 [======>.......................] - ETA: 7s - loss: 0.4522 - accuracy: 0.8627


225/844 [======>.......................] - ETA: 7s - loss: 0.4468 - accuracy: 0.8642


228/844 [=======>......................] - ETA: 7s - loss: 0.4424 - accuracy: 0.8656


231/844 [=======>......................] - ETA: 7s - loss: 0.4388 - accuracy: 0.8665


233/844 [=======>......................] - ETA: 7s - loss: 0.4358 - accuracy: 0.8673


236/844 [=======>......................] - ETA: 7s - loss: 0.4321 - accuracy: 0.8684


239/844 [=======>......................] - ETA: 7s - loss: 0.4281 - accuracy: 0.8694


242/844 [=======>......................] - ETA: 7s - loss: 0.4248 - accuracy: 0.8705


245/844 [=======>......................] - ETA: 7s - loss: 0.4210 - accuracy: 0.8716


248/844 [=======>......................] - ETA: 7s - loss: 0.4185 - accuracy: 0.8724


251/844 [=======>......................] - ETA: 7s - loss: 0.4150 - accuracy: 0.8734


254/844 [========>.....................] - ETA: 7s - loss: 0.4133 - accuracy: 0.8741


257/844 [========>.....................] - ETA: 7s - loss: 0.4103 - accuracy: 0.8750


260/844 [========>.....................] - ETA: 7s - loss: 0.4070 - accuracy: 0.8760


263/844 [========>.....................] - ETA: 7s - loss: 0.4039 - accuracy: 0.8770


266/844 [========>.....................] - ETA: 7s - loss: 0.4010 - accuracy: 0.8779


269/844 [========>.....................] - ETA: 7s - loss: 0.3981 - accuracy: 0.8789


272/844 [========>.....................] - ETA: 7s - loss: 0.3952 - accuracy: 0.8798


275/844 [========>.....................] - ETA: 7s - loss: 0.3918 - accuracy: 0.8807


278/844 [========>.....................] - ETA: 7s - loss: 0.3888 - accuracy: 0.8817


281/844 [========>.....................] - ETA: 7s - loss: 0.3864 - accuracy: 0.8824


284/844 [=========>....................] - ETA: 7s - loss: 0.3836 - accuracy: 0.8832


287/844 [=========>....................] - ETA: 7s - loss: 0.3814 - accuracy: 0.8839


290/844 [=========>....................] - ETA: 7s - loss: 0.3780 - accuracy: 0.8850


293/844 [=========>....................] - ETA: 7s - loss: 0.3754 - accuracy: 0.8858


296/844 [=========>....................] - ETA: 7s - loss: 0.3743 - accuracy: 0.8863


299/844 [=========>....................] - ETA: 7s - loss: 0.3716 - accuracy: 0.8872


302/844 [=========>....................] - ETA: 7s - loss: 0.3698 - accuracy: 0.8878


305/844 [=========>....................] - ETA: 7s - loss: 0.3672 - accuracy: 0.8886


308/844 [=========>....................] - ETA: 7s - loss: 0.3645 - accuracy: 0.8895


311/844 [==========>...................] - ETA: 7s - loss: 0.3623 - accuracy: 0.8903


314/844 [==========>...................] - ETA: 7s - loss: 0.3598 - accuracy: 0.8910


317/844 [==========>...................] - ETA: 7s - loss: 0.3572 - accuracy: 0.8918


320/844 [==========>...................] - ETA: 7s - loss: 0.3545 - accuracy: 0.8925


323/844 [==========>...................] - ETA: 7s - loss: 0.3521 - accuracy: 0.8933


325/844 [==========>...................] - ETA: 7s - loss: 0.3505 - accuracy: 0.8938


328/844 [==========>...................] - ETA: 7s - loss: 0.3480 - accuracy: 0.8944


331/844 [==========>...................] - ETA: 7s - loss: 0.3466 - accuracy: 0.8950


334/844 [==========>...................] - ETA: 7s - loss: 0.3447 - accuracy: 0.8955


337/844 [==========>...................] - ETA: 7s - loss: 0.3424 - accuracy: 0.8962


339/844 [===========>..................] - ETA: 7s - loss: 0.3413 - accuracy: 0.8967


341/844 [===========>..................] - ETA: 7s - loss: 0.3399 - accuracy: 0.8971


344/844 [===========>..................] - ETA: 7s - loss: 0.3383 - accuracy: 0.8975


347/844 [===========>..................] - ETA: 7s - loss: 0.3369 - accuracy: 0.8979


350/844 [===========>..................] - ETA: 7s - loss: 0.3359 - accuracy: 0.8983


353/844 [===========>..................] - ETA: 7s - loss: 0.3340 - accuracy: 0.8988


356/844 [===========>..................] - ETA: 7s - loss: 0.3320 - accuracy: 0.8993


359/844 [===========>..................] - ETA: 7s - loss: 0.3303 - accuracy: 0.8998


362/844 [===========>..................] - ETA: 7s - loss: 0.3286 - accuracy: 0.9003


365/844 [===========>..................] - ETA: 7s - loss: 0.3272 - accuracy: 0.9007


368/844 [============>.................] - ETA: 7s - loss: 0.3252 - accuracy: 0.9013


371/844 [============>.................] - ETA: 7s - loss: 0.3234 - accuracy: 0.9019


374/844 [============>.................] - ETA: 7s - loss: 0.3216 - accuracy: 0.9026


377/844 [============>.................] - ETA: 7s - loss: 0.3197 - accuracy: 0.9031


379/844 [============>.................] - ETA: 7s - loss: 0.3185 - accuracy: 0.9035


382/844 [============>.................] - ETA: 7s - loss: 0.3169 - accuracy: 0.9040


385/844 [============>.................] - ETA: 7s - loss: 0.3148 - accuracy: 0.9047


388/844 [============>.................] - ETA: 7s - loss: 0.3134 - accuracy: 0.9052


391/844 [============>.................] - ETA: 7s - loss: 0.3120 - accuracy: 0.9056


394/844 [=============>................] - ETA: 7s - loss: 0.3106 - accuracy: 0.9060


397/844 [=============>................] - ETA: 6s - loss: 0.3093 - accuracy: 0.9064


400/844 [=============>................] - ETA: 6s - loss: 0.3078 - accuracy: 0.9069


403/844 [=============>................] - ETA: 6s - loss: 0.3062 - accuracy: 0.9075


406/844 [=============>................] - ETA: 6s - loss: 0.3047 - accuracy: 0.9078


409/844 [=============>................] - ETA: 6s - loss: 0.3031 - accuracy: 0.9082


412/844 [=============>................] - ETA: 6s - loss: 0.3019 - accuracy: 0.9085


415/844 [=============>................] - ETA: 6s - loss: 0.3009 - accuracy: 0.9089


418/844 [=============>................] - ETA: 6s - loss: 0.2994 - accuracy: 0.9094


420/844 [=============>................] - ETA: 6s - loss: 0.2984 - accuracy: 0.9096


423/844 [==============>...............] - ETA: 6s - loss: 0.2969 - accuracy: 0.9102


426/844 [==============>...............] - ETA: 6s - loss: 0.2955 - accuracy: 0.9106


428/844 [==============>...............] - ETA: 6s - loss: 0.2944 - accuracy: 0.9109


431/844 [==============>...............] - ETA: 6s - loss: 0.2927 - accuracy: 0.9114


434/844 [==============>...............] - ETA: 6s - loss: 0.2913 - accuracy: 0.9118


437/844 [==============>...............] - ETA: 6s - loss: 0.2899 - accuracy: 0.9122


440/844 [==============>...............] - ETA: 6s - loss: 0.2887 - accuracy: 0.9125


443/844 [==============>...............] - ETA: 6s - loss: 0.2872 - accuracy: 0.9130


446/844 [==============>...............] - ETA: 6s - loss: 0.2859 - accuracy: 0.9133


449/844 [==============>...............] - ETA: 6s - loss: 0.2849 - accuracy: 0.9137


452/844 [===============>..............] - ETA: 6s - loss: 0.2836 - accuracy: 0.9142


455/844 [===============>..............] - ETA: 6s - loss: 0.2823 - accuracy: 0.9145


458/844 [===============>..............] - ETA: 6s - loss: 0.2812 - accuracy: 0.9148


461/844 [===============>..............] - ETA: 6s - loss: 0.2797 - accuracy: 0.9153


463/844 [===============>..............] - ETA: 6s - loss: 0.2788 - accuracy: 0.9156


466/844 [===============>..............] - ETA: 6s - loss: 0.2772 - accuracy: 0.9162


469/844 [===============>..............] - ETA: 6s - loss: 0.2763 - accuracy: 0.9164


472/844 [===============>..............] - ETA: 6s - loss: 0.2750 - accuracy: 0.9168


475/844 [===============>..............] - ETA: 6s - loss: 0.2736 - accuracy: 0.9172


478/844 [===============>..............] - ETA: 6s - loss: 0.2723 - accuracy: 0.9176


480/844 [================>.............] - ETA: 6s - loss: 0.2716 - accuracy: 0.9178


483/844 [================>.............] - ETA: 6s - loss: 0.2706 - accuracy: 0.9182


486/844 [================>.............] - ETA: 5s - loss: 0.2694 - accuracy: 0.9185


489/844 [================>.............] - ETA: 5s - loss: 0.2684 - accuracy: 0.9188


492/844 [================>.............] - ETA: 5s - loss: 0.2672 - accuracy: 0.9191


495/844 [================>.............] - ETA: 5s - loss: 0.2659 - accuracy: 0.9195


498/844 [================>.............] - ETA: 5s - loss: 0.2647 - accuracy: 0.9199


501/844 [================>.............] - ETA: 5s - loss: 0.2639 - accuracy: 0.9201


504/844 [================>.............] - ETA: 5s - loss: 0.2628 - accuracy: 0.9205


507/844 [=================>............] - ETA: 5s - loss: 0.2619 - accuracy: 0.9207


510/844 [=================>............] - ETA: 5s - loss: 0.2612 - accuracy: 0.9209


513/844 [=================>............] - ETA: 5s - loss: 0.2602 - accuracy: 0.9212


516/844 [=================>............] - ETA: 5s - loss: 0.2590 - accuracy: 0.9216


519/844 [=================>............] - ETA: 5s - loss: 0.2582 - accuracy: 0.9218


522/844 [=================>............] - ETA: 5s - loss: 0.2573 - accuracy: 0.9220


525/844 [=================>............] - ETA: 5s - loss: 0.2562 - accuracy: 0.9223


528/844 [=================>............] - ETA: 5s - loss: 0.2550 - accuracy: 0.9227


530/844 [=================>............] - ETA: 5s - loss: 0.2546 - accuracy: 0.9228


533/844 [=================>............] - ETA: 5s - loss: 0.2535 - accuracy: 0.9231


536/844 [==================>...........] - ETA: 5s - loss: 0.2525 - accuracy: 0.9234


539/844 [==================>...........] - ETA: 5s - loss: 0.2514 - accuracy: 0.9238


542/844 [==================>...........] - ETA: 5s - loss: 0.2506 - accuracy: 0.9240


545/844 [==================>...........] - ETA: 5s - loss: 0.2498 - accuracy: 0.9242


547/844 [==================>...........] - ETA: 5s - loss: 0.2492 - accuracy: 0.9244


550/844 [==================>...........] - ETA: 5s - loss: 0.2482 - accuracy: 0.9247


553/844 [==================>...........] - ETA: 5s - loss: 0.2471 - accuracy: 0.9251


556/844 [==================>...........] - ETA: 5s - loss: 0.2461 - accuracy: 0.9254


559/844 [==================>...........] - ETA: 4s - loss: 0.2453 - accuracy: 0.9255


562/844 [==================>...........] - ETA: 4s - loss: 0.2444 - accuracy: 0.9258


565/844 [===================>..........] - ETA: 4s - loss: 0.2435 - accuracy: 0.9260


568/844 [===================>..........] - ETA: 4s - loss: 0.2425 - accuracy: 0.9263


571/844 [===================>..........] - ETA: 4s - loss: 0.2415 - accuracy: 0.9266


574/844 [===================>..........] - ETA: 4s - loss: 0.2408 - accuracy: 0.9268


577/844 [===================>..........] - ETA: 4s - loss: 0.2400 - accuracy: 0.9271


580/844 [===================>..........] - ETA: 4s - loss: 0.2391 - accuracy: 0.9273


583/844 [===================>..........] - ETA: 4s - loss: 0.2383 - accuracy: 0.9276


586/844 [===================>..........] - ETA: 4s - loss: 0.2374 - accuracy: 0.9278


589/844 [===================>..........] - ETA: 4s - loss: 0.2368 - accuracy: 0.9280


592/844 [====================>.........] - ETA: 4s - loss: 0.2360 - accuracy: 0.9283


595/844 [====================>.........] - ETA: 4s - loss: 0.2352 - accuracy: 0.9285


598/844 [====================>.........] - ETA: 4s - loss: 0.2344 - accuracy: 0.9288


601/844 [====================>.........] - ETA: 4s - loss: 0.2337 - accuracy: 0.9290


604/844 [====================>.........] - ETA: 4s - loss: 0.2332 - accuracy: 0.9292


607/844 [====================>.........] - ETA: 4s - loss: 0.2324 - accuracy: 0.9293


610/844 [====================>.........] - ETA: 4s - loss: 0.2316 - accuracy: 0.9296


613/844 [====================>.........] - ETA: 4s - loss: 0.2309 - accuracy: 0.9298


615/844 [====================>.........] - ETA: 4s - loss: 0.2305 - accuracy: 0.9300


618/844 [====================>.........] - ETA: 4s - loss: 0.2298 - accuracy: 0.9302


621/844 [=====================>........] - ETA: 3s - loss: 0.2288 - accuracy: 0.9305


624/844 [=====================>........] - ETA: 3s - loss: 0.2282 - accuracy: 0.9306


627/844 [=====================>........] - ETA: 3s - loss: 0.2274 - accuracy: 0.9308


629/844 [=====================>........] - ETA: 3s - loss: 0.2268 - accuracy: 0.9310


632/844 [=====================>........] - ETA: 3s - loss: 0.2260 - accuracy: 0.9313


635/844 [=====================>........] - ETA: 3s - loss: 0.2251 - accuracy: 0.9315


638/844 [=====================>........] - ETA: 3s - loss: 0.2245 - accuracy: 0.9316


641/844 [=====================>........] - ETA: 3s - loss: 0.2238 - accuracy: 0.9318


643/844 [=====================>........] - ETA: 3s - loss: 0.2234 - accuracy: 0.9320


645/844 [=====================>........] - ETA: 3s - loss: 0.2228 - accuracy: 0.9322


648/844 [======================>.......] - ETA: 3s - loss: 0.2220 - accuracy: 0.9324


650/844 [======================>.......] - ETA: 3s - loss: 0.2215 - accuracy: 0.9326


652/844 [======================>.......] - ETA: 3s - loss: 0.2210 - accuracy: 0.9327


655/844 [======================>.......] - ETA: 3s - loss: 0.2202 - accuracy: 0.9330


658/844 [======================>.......] - ETA: 3s - loss: 0.2195 - accuracy: 0.9332


660/844 [======================>.......] - ETA: 3s - loss: 0.2191 - accuracy: 0.9333


662/844 [======================>.......] - ETA: 3s - loss: 0.2186 - accuracy: 0.9335


665/844 [======================>.......] - ETA: 3s - loss: 0.2179 - accuracy: 0.9336


667/844 [======================>.......] - ETA: 3s - loss: 0.2175 - accuracy: 0.9338


670/844 [======================>.......] - ETA: 3s - loss: 0.2168 - accuracy: 0.9340


673/844 [======================>.......] - ETA: 3s - loss: 0.2163 - accuracy: 0.9341


675/844 [======================>.......] - ETA: 3s - loss: 0.2158 - accuracy: 0.9343


677/844 [=======================>......] - ETA: 3s - loss: 0.2153 - accuracy: 0.9344


680/844 [=======================>......] - ETA: 3s - loss: 0.2149 - accuracy: 0.9345


683/844 [=======================>......] - ETA: 2s - loss: 0.2144 - accuracy: 0.9347


686/844 [=======================>......] - ETA: 2s - loss: 0.2137 - accuracy: 0.9349


689/844 [=======================>......] - ETA: 2s - loss: 0.2131 - accuracy: 0.9351


692/844 [=======================>......] - ETA: 2s - loss: 0.2124 - accuracy: 0.9353


694/844 [=======================>......] - ETA: 2s - loss: 0.2121 - accuracy: 0.9354


696/844 [=======================>......] - ETA: 2s - loss: 0.2116 - accuracy: 0.9355


699/844 [=======================>......] - ETA: 2s - loss: 0.2112 - accuracy: 0.9357


702/844 [=======================>......] - ETA: 2s - loss: 0.2106 - accuracy: 0.9359


704/844 [========================>.....] - ETA: 2s - loss: 0.2103 - accuracy: 0.9359


707/844 [========================>.....] - ETA: 2s - loss: 0.2097 - accuracy: 0.9361


710/844 [========================>.....] - ETA: 2s - loss: 0.2090 - accuracy: 0.9363


713/844 [========================>.....] - ETA: 2s - loss: 0.2084 - accuracy: 0.9365


716/844 [========================>.....] - ETA: 2s - loss: 0.2078 - accuracy: 0.9366


719/844 [========================>.....] - ETA: 2s - loss: 0.2074 - accuracy: 0.9368


722/844 [========================>.....] - ETA: 2s - loss: 0.2068 - accuracy: 0.9369


725/844 [========================>.....] - ETA: 2s - loss: 0.2061 - accuracy: 0.9371


728/844 [========================>.....] - ETA: 2s - loss: 0.2056 - accuracy: 0.9372


731/844 [========================>.....] - ETA: 2s - loss: 0.2052 - accuracy: 0.9374


734/844 [=========================>....] - ETA: 2s - loss: 0.2045 - accuracy: 0.9375


737/844 [=========================>....] - ETA: 1s - loss: 0.2040 - accuracy: 0.9377


740/844 [=========================>....] - ETA: 1s - loss: 0.2034 - accuracy: 0.9379


743/844 [=========================>....] - ETA: 1s - loss: 0.2028 - accuracy: 0.9380


746/844 [=========================>....] - ETA: 1s - loss: 0.2026 - accuracy: 0.9381


749/844 [=========================>....] - ETA: 1s - loss: 0.2022 - accuracy: 0.9382


752/844 [=========================>....] - ETA: 1s - loss: 0.2016 - accuracy: 0.9384


755/844 [=========================>....] - ETA: 1s - loss: 0.2012 - accuracy: 0.9386


758/844 [=========================>....] - ETA: 1s - loss: 0.2007 - accuracy: 0.9387


761/844 [==========================>...] - ETA: 1s - loss: 0.2001 - accuracy: 0.9389


764/844 [==========================>...] - ETA: 1s - loss: 0.1994 - accuracy: 0.9391


767/844 [==========================>...] - ETA: 1s - loss: 0.1990 - accuracy: 0.9392


770/844 [==========================>...] - ETA: 1s - loss: 0.1988 - accuracy: 0.9393


773/844 [==========================>...] - ETA: 1s - loss: 0.1984 - accuracy: 0.9395


775/844 [==========================>...] - ETA: 1s - loss: 0.1981 - accuracy: 0.9396


778/844 [==========================>...] - ETA: 1s - loss: 0.1979 - accuracy: 0.9396


781/844 [==========================>...] - ETA: 1s - loss: 0.1975 - accuracy: 0.9398


784/844 [==========================>...] - ETA: 1s - loss: 0.1969 - accuracy: 0.9399


787/844 [==========================>...] - ETA: 1s - loss: 0.1964 - accuracy: 0.9400


789/844 [===========================>..] - ETA: 1s - loss: 0.1961 - accuracy: 0.9401


791/844 [===========================>..] - ETA: 1s - loss: 0.1957 - accuracy: 0.9402


794/844 [===========================>..] - ETA: 0s - loss: 0.1950 - accuracy: 0.9405


797/844 [===========================>..] - ETA: 0s - loss: 0.1945 - accuracy: 0.9406


800/844 [===========================>..] - ETA: 0s - loss: 0.1940 - accuracy: 0.9408


803/844 [===========================>..] - ETA: 0s - loss: 0.1936 - accuracy: 0.9409


806/844 [===========================>..] - ETA: 0s - loss: 0.1931 - accuracy: 0.9411


809/844 [===========================>..] - ETA: 0s - loss: 0.1925 - accuracy: 0.9412


812/844 [===========================>..] - ETA: 0s - loss: 0.1923 - accuracy: 0.9413


815/844 [===========================>..] - ETA: 0s - loss: 0.1918 - accuracy: 0.9415


818/844 [============================>.] - ETA: 0s - loss: 0.1913 - accuracy: 0.9417


821/844 [============================>.] - ETA: 0s - loss: 0.1909 - accuracy: 0.9418


824/844 [============================>.] - ETA: 0s - loss: 0.1905 - accuracy: 0.9419


827/844 [============================>.] - ETA: 0s - loss: 0.1903 - accuracy: 0.9420


830/844 [============================>.] - ETA: 0s - loss: 0.1900 - accuracy: 0.9421


833/844 [============================>.] - ETA: 0s - loss: 0.1893 - accuracy: 0.9423


836/844 [============================>.] - ETA: 0s - loss: 0.1889 - accuracy: 0.9424


839/844 [============================>.] - ETA: 0s - loss: 0.1883 - accuracy: 0.9426


841/844 [============================>.] - ETA: 0s - loss: 0.1881 - accuracy: 0.9427


844/844 [==============================] - ETA: 0s - loss: 0.1876 - accuracy: 0.9428


844/844 [==============================] - 17s 20ms/step - loss: 0.1876 - accuracy: 0.9428 - val_loss: 0.0517 - val_accuracy: 0.9852

Epoch 2/5

  1/844 [..............................] - ETA: 19s - loss: 0.0567 - accuracy: 0.9688


  4/844 [..............................] - ETA: 16s - loss: 0.0341 - accuracy: 0.9883


  7/844 [..............................] - ETA: 16s - loss: 0.0634 - accuracy: 0.9821


 10/844 [..............................] - ETA: 16s - loss: 0.0562 - accuracy: 0.9844


 13/844 [..............................] - ETA: 16s - loss: 0.0594 - accuracy: 0.9844


 16/844 [..............................] - ETA: 16s - loss: 0.0556 - accuracy: 0.9844


 19/844 [..............................] - ETA: 17s - loss: 0.0532 - accuracy: 0.9844


 22/844 [..............................] - ETA: 17s - loss: 0.0550 - accuracy: 0.9830


 25/844 [..............................] - ETA: 17s - loss: 0.0514 - accuracy: 0.9837


 28/844 [..............................] - ETA: 16s - loss: 0.0526 - accuracy: 0.9821


 31/844 [>.............................] - ETA: 16s - loss: 0.0522 - accuracy: 0.9829


 34/844 [>.............................] - ETA: 16s - loss: 0.0528 - accuracy: 0.9825


 37/844 [>.............................] - ETA: 16s - loss: 0.0521 - accuracy: 0.9831


 40/844 [>.............................] - ETA: 16s - loss: 0.0515 - accuracy: 0.9832


 43/844 [>.............................] - ETA: 16s - loss: 0.0544 - accuracy: 0.9822


 46/844 [>.............................] - ETA: 16s - loss: 0.0521 - accuracy: 0.9834


 49/844 [>.............................] - ETA: 16s - loss: 0.0512 - accuracy: 0.9837


 52/844 [>.............................] - ETA: 16s - loss: 0.0500 - accuracy: 0.9844


 55/844 [>.............................] - ETA: 16s - loss: 0.0485 - accuracy: 0.9849


 58/844 [=>............................] - ETA: 16s - loss: 0.0515 - accuracy: 0.9841


 61/844 [=>............................] - ETA: 16s - loss: 0.0516 - accuracy: 0.9841


 64/844 [=>............................] - ETA: 16s - loss: 0.0512 - accuracy: 0.9839


 67/844 [=>............................] - ETA: 16s - loss: 0.0526 - accuracy: 0.9832


 70/844 [=>............................] - ETA: 16s - loss: 0.0535 - accuracy: 0.9830


 73/844 [=>............................] - ETA: 16s - loss: 0.0537 - accuracy: 0.9829


 76/844 [=>............................] - ETA: 15s - loss: 0.0528 - accuracy: 0.9831


 79/844 [=>............................] - ETA: 15s - loss: 0.0515 - accuracy: 0.9836


 82/844 [=>............................] - ETA: 15s - loss: 0.0509 - accuracy: 0.9836


 84/844 [=>............................] - ETA: 15s - loss: 0.0509 - accuracy: 0.9836


 87/844 [==>...........................] - ETA: 15s - loss: 0.0503 - accuracy: 0.9838


 90/844 [==>...........................] - ETA: 15s - loss: 0.0502 - accuracy: 0.9840


 93/844 [==>...........................] - ETA: 15s - loss: 0.0530 - accuracy: 0.9837


 96/844 [==>...........................] - ETA: 15s - loss: 0.0533 - accuracy: 0.9837


 98/844 [==>...........................] - ETA: 15s - loss: 0.0527 - accuracy: 0.9839


101/844 [==>...........................] - ETA: 15s - loss: 0.0529 - accuracy: 0.9834


104/844 [==>...........................] - ETA: 15s - loss: 0.0518 - accuracy: 0.9838


106/844 [==>...........................] - ETA: 15s - loss: 0.0521 - accuracy: 0.9835


109/844 [==>...........................] - ETA: 15s - loss: 0.0521 - accuracy: 0.9832


112/844 [==>...........................] - ETA: 15s - loss: 0.0514 - accuracy: 0.9834


115/844 [===>..........................] - ETA: 15s - loss: 0.0511 - accuracy: 0.9836


118/844 [===>..........................] - ETA: 15s - loss: 0.0515 - accuracy: 0.9834


121/844 [===>..........................] - ETA: 15s - loss: 0.0515 - accuracy: 0.9833


124/844 [===>..........................] - ETA: 15s - loss: 0.0521 - accuracy: 0.9830


127/844 [===>..........................] - ETA: 15s - loss: 0.0521 - accuracy: 0.9830


130/844 [===>..........................] - ETA: 15s - loss: 0.0518 - accuracy: 0.9832


133/844 [===>..........................] - ETA: 15s - loss: 0.0512 - accuracy: 0.9834


136/844 [===>..........................] - ETA: 14s - loss: 0.0508 - accuracy: 0.9837


139/844 [===>..........................] - ETA: 14s - loss: 0.0511 - accuracy: 0.9836


142/844 [====>.........................] - ETA: 14s - loss: 0.0511 - accuracy: 0.9835


145/844 [====>.........................] - ETA: 14s - loss: 0.0516 - accuracy: 0.9834


148/844 [====>.........................] - ETA: 14s - loss: 0.0520 - accuracy: 0.9835


151/844 [====>.........................] - ETA: 14s - loss: 0.0514 - accuracy: 0.9837


154/844 [====>.........................] - ETA: 14s - loss: 0.0515 - accuracy: 0.9836


157/844 [====>.........................] - ETA: 14s - loss: 0.0515 - accuracy: 0.9835


160/844 [====>.........................] - ETA: 14s - loss: 0.0514 - accuracy: 0.9836


163/844 [====>.........................] - ETA: 14s - loss: 0.0509 - accuracy: 0.9838


165/844 [====>.........................] - ETA: 14s - loss: 0.0519 - accuracy: 0.9835


168/844 [====>.........................] - ETA: 14s - loss: 0.0520 - accuracy: 0.9835


171/844 [=====>........................] - ETA: 14s - loss: 0.0529 - accuracy: 0.9831


174/844 [=====>........................] - ETA: 14s - loss: 0.0524 - accuracy: 0.9832


177/844 [=====>........................] - ETA: 14s - loss: 0.0522 - accuracy: 0.9831


180/844 [=====>........................] - ETA: 14s - loss: 0.0524 - accuracy: 0.9832


183/844 [=====>........................] - ETA: 13s - loss: 0.0520 - accuracy: 0.9834


186/844 [=====>........................] - ETA: 13s - loss: 0.0519 - accuracy: 0.9835


189/844 [=====>........................] - ETA: 13s - loss: 0.0515 - accuracy: 0.9836


192/844 [=====>........................] - ETA: 13s - loss: 0.0516 - accuracy: 0.9836


195/844 [=====>........................] - ETA: 13s - loss: 0.0518 - accuracy: 0.9837


198/844 [======>.......................] - ETA: 13s - loss: 0.0521 - accuracy: 0.9833


201/844 [======>.......................] - ETA: 13s - loss: 0.0529 - accuracy: 0.9831


204/844 [======>.......................] - ETA: 13s - loss: 0.0527 - accuracy: 0.9832


207/844 [======>.......................] - ETA: 13s - loss: 0.0533 - accuracy: 0.9829


210/844 [======>.......................] - ETA: 13s - loss: 0.0535 - accuracy: 0.9828


213/844 [======>.......................] - ETA: 13s - loss: 0.0539 - accuracy: 0.9826


216/844 [======>.......................] - ETA: 13s - loss: 0.0540 - accuracy: 0.9825


219/844 [======>.......................] - ETA: 13s - loss: 0.0542 - accuracy: 0.9824


222/844 [======>.......................] - ETA: 13s - loss: 0.0544 - accuracy: 0.9824


224/844 [======>.......................] - ETA: 13s - loss: 0.0540 - accuracy: 0.9826


227/844 [=======>......................] - ETA: 13s - loss: 0.0537 - accuracy: 0.9827


230/844 [=======>......................] - ETA: 12s - loss: 0.0534 - accuracy: 0.9828


233/844 [=======>......................] - ETA: 12s - loss: 0.0536 - accuracy: 0.9827


235/844 [=======>......................] - ETA: 12s - loss: 0.0535 - accuracy: 0.9828


238/844 [=======>......................] - ETA: 12s - loss: 0.0540 - accuracy: 0.9827


241/844 [=======>......................] - ETA: 12s - loss: 0.0538 - accuracy: 0.9828


244/844 [=======>......................] - ETA: 12s - loss: 0.0535 - accuracy: 0.9829


247/844 [=======>......................] - ETA: 12s - loss: 0.0536 - accuracy: 0.9829


250/844 [=======>......................] - ETA: 12s - loss: 0.0533 - accuracy: 0.9830


253/844 [=======>......................] - ETA: 12s - loss: 0.0528 - accuracy: 0.9832


256/844 [========>.....................] - ETA: 12s - loss: 0.0524 - accuracy: 0.9833


259/844 [========>.....................] - ETA: 12s - loss: 0.0526 - accuracy: 0.9833


262/844 [========>.....................] - ETA: 12s - loss: 0.0522 - accuracy: 0.9834


265/844 [========>.....................] - ETA: 12s - loss: 0.0524 - accuracy: 0.9834


268/844 [========>.....................] - ETA: 12s - loss: 0.0520 - accuracy: 0.9836


271/844 [========>.....................] - ETA: 12s - loss: 0.0524 - accuracy: 0.9834


274/844 [========>.....................] - ETA: 12s - loss: 0.0520 - accuracy: 0.9835


277/844 [========>.....................] - ETA: 11s - loss: 0.0516 - accuracy: 0.9836


280/844 [========>.....................] - ETA: 11s - loss: 0.0515 - accuracy: 0.9836


283/844 [=========>....................] - ETA: 11s - loss: 0.0512 - accuracy: 0.9837


285/844 [=========>....................] - ETA: 11s - loss: 0.0514 - accuracy: 0.9836


288/844 [=========>....................] - ETA: 11s - loss: 0.0515 - accuracy: 0.9835


291/844 [=========>....................] - ETA: 11s - loss: 0.0517 - accuracy: 0.9835


294/844 [=========>....................] - ETA: 11s - loss: 0.0518 - accuracy: 0.9834


297/844 [=========>....................] - ETA: 11s - loss: 0.0521 - accuracy: 0.9832


300/844 [=========>....................] - ETA: 11s - loss: 0.0521 - accuracy: 0.9831


303/844 [=========>....................] - ETA: 11s - loss: 0.0519 - accuracy: 0.9832


306/844 [=========>....................] - ETA: 11s - loss: 0.0523 - accuracy: 0.9831


309/844 [=========>....................] - ETA: 11s - loss: 0.0520 - accuracy: 0.9832


312/844 [==========>...................] - ETA: 11s - loss: 0.0527 - accuracy: 0.9831


315/844 [==========>...................] - ETA: 11s - loss: 0.0529 - accuracy: 0.9831


318/844 [==========>...................] - ETA: 11s - loss: 0.0528 - accuracy: 0.9832


321/844 [==========>...................] - ETA: 11s - loss: 0.0530 - accuracy: 0.9833


324/844 [==========>...................] - ETA: 10s - loss: 0.0533 - accuracy: 0.9833


327/844 [==========>...................] - ETA: 10s - loss: 0.0534 - accuracy: 0.9832


330/844 [==========>...................] - ETA: 10s - loss: 0.0534 - accuracy: 0.9832


333/844 [==========>...................] - ETA: 10s - loss: 0.0530 - accuracy: 0.9833


336/844 [==========>...................] - ETA: 10s - loss: 0.0531 - accuracy: 0.9834


339/844 [===========>..................] - ETA: 10s - loss: 0.0529 - accuracy: 0.9835


342/844 [===========>..................] - ETA: 10s - loss: 0.0530 - accuracy: 0.9834


345/844 [===========>..................] - ETA: 10s - loss: 0.0528 - accuracy: 0.9833


348/844 [===========>..................] - ETA: 10s - loss: 0.0528 - accuracy: 0.9833


351/844 [===========>..................] - ETA: 10s - loss: 0.0532 - accuracy: 0.9831


354/844 [===========>..................] - ETA: 10s - loss: 0.0533 - accuracy: 0.9831


357/844 [===========>..................] - ETA: 10s - loss: 0.0533 - accuracy: 0.9830


360/844 [===========>..................] - ETA: 10s - loss: 0.0535 - accuracy: 0.9829


363/844 [===========>..................] - ETA: 10s - loss: 0.0539 - accuracy: 0.9828


366/844 [============>.................] - ETA: 10s - loss: 0.0538 - accuracy: 0.9829


369/844 [============>.................] - ETA: 9s - loss: 0.0541 - accuracy: 0.9827 


372/844 [============>.................] - ETA: 9s - loss: 0.0540 - accuracy: 0.9827


375/844 [============>.................] - ETA: 9s - loss: 0.0541 - accuracy: 0.9827


377/844 [============>.................] - ETA: 9s - loss: 0.0541 - accuracy: 0.9827


380/844 [============>.................] - ETA: 9s - loss: 0.0543 - accuracy: 0.9826


383/844 [============>.................] - ETA: 9s - loss: 0.0542 - accuracy: 0.9826


386/844 [============>.................] - ETA: 9s - loss: 0.0541 - accuracy: 0.9826


389/844 [============>.................] - ETA: 9s - loss: 0.0542 - accuracy: 0.9825


392/844 [============>.................] - ETA: 9s - loss: 0.0543 - accuracy: 0.9825


395/844 [=============>................] - ETA: 9s - loss: 0.0541 - accuracy: 0.9826


398/844 [=============>................] - ETA: 9s - loss: 0.0539 - accuracy: 0.9827


401/844 [=============>................] - ETA: 9s - loss: 0.0539 - accuracy: 0.9826


404/844 [=============>................] - ETA: 9s - loss: 0.0536 - accuracy: 0.9826


407/844 [=============>................] - ETA: 9s - loss: 0.0535 - accuracy: 0.9827


410/844 [=============>................] - ETA: 9s - loss: 0.0538 - accuracy: 0.9827


413/844 [=============>................] - ETA: 9s - loss: 0.0537 - accuracy: 0.9828


416/844 [=============>................] - ETA: 9s - loss: 0.0537 - accuracy: 0.9828


419/844 [=============>................] - ETA: 8s - loss: 0.0539 - accuracy: 0.9827


422/844 [==============>...............] - ETA: 8s - loss: 0.0538 - accuracy: 0.9827


425/844 [==============>...............] - ETA: 8s - loss: 0.0537 - accuracy: 0.9828


428/844 [==============>...............] - ETA: 8s - loss: 0.0537 - accuracy: 0.9827


431/844 [==============>...............] - ETA: 8s - loss: 0.0538 - accuracy: 0.9826


433/844 [==============>...............] - ETA: 8s - loss: 0.0540 - accuracy: 0.9826


436/844 [==============>...............] - ETA: 8s - loss: 0.0540 - accuracy: 0.9826


439/844 [==============>...............] - ETA: 8s - loss: 0.0540 - accuracy: 0.9826


441/844 [==============>...............] - ETA: 8s - loss: 0.0538 - accuracy: 0.9826


444/844 [==============>...............] - ETA: 8s - loss: 0.0538 - accuracy: 0.9826


447/844 [==============>...............] - ETA: 8s - loss: 0.0539 - accuracy: 0.9826


450/844 [==============>...............] - ETA: 8s - loss: 0.0538 - accuracy: 0.9826


453/844 [===============>..............] - ETA: 8s - loss: 0.0537 - accuracy: 0.9826


456/844 [===============>..............] - ETA: 8s - loss: 0.0536 - accuracy: 0.9827


459/844 [===============>..............] - ETA: 8s - loss: 0.0536 - accuracy: 0.9827


462/844 [===============>..............] - ETA: 8s - loss: 0.0541 - accuracy: 0.9825


465/844 [===============>..............] - ETA: 8s - loss: 0.0543 - accuracy: 0.9825


468/844 [===============>..............] - ETA: 7s - loss: 0.0544 - accuracy: 0.9824


471/844 [===============>..............] - ETA: 7s - loss: 0.0544 - accuracy: 0.9824


474/844 [===============>..............] - ETA: 7s - loss: 0.0546 - accuracy: 0.9824


477/844 [===============>..............] - ETA: 7s - loss: 0.0546 - accuracy: 0.9824


480/844 [================>.............] - ETA: 7s - loss: 0.0548 - accuracy: 0.9823


482/844 [================>.............] - ETA: 7s - loss: 0.0548 - accuracy: 0.9823


484/844 [================>.............] - ETA: 7s - loss: 0.0549 - accuracy: 0.9822


486/844 [================>.............] - ETA: 7s - loss: 0.0550 - accuracy: 0.9822


489/844 [================>.............] - ETA: 7s - loss: 0.0550 - accuracy: 0.9822


492/844 [================>.............] - ETA: 7s - loss: 0.0550 - accuracy: 0.9823


495/844 [================>.............] - ETA: 7s - loss: 0.0548 - accuracy: 0.9823


497/844 [================>.............] - ETA: 7s - loss: 0.0549 - accuracy: 0.9823


500/844 [================>.............] - ETA: 7s - loss: 0.0547 - accuracy: 0.9824


503/844 [================>.............] - ETA: 7s - loss: 0.0547 - accuracy: 0.9824


506/844 [================>.............] - ETA: 7s - loss: 0.0546 - accuracy: 0.9825


509/844 [=================>............] - ETA: 7s - loss: 0.0544 - accuracy: 0.9825


512/844 [=================>............] - ETA: 7s - loss: 0.0546 - accuracy: 0.9825


515/844 [=================>............] - ETA: 7s - loss: 0.0545 - accuracy: 0.9825


518/844 [=================>............] - ETA: 6s - loss: 0.0545 - accuracy: 0.9825


521/844 [=================>............] - ETA: 6s - loss: 0.0544 - accuracy: 0.9825


524/844 [=================>............] - ETA: 6s - loss: 0.0543 - accuracy: 0.9826


527/844 [=================>............] - ETA: 6s - loss: 0.0544 - accuracy: 0.9825


530/844 [=================>............] - ETA: 6s - loss: 0.0545 - accuracy: 0.9825


533/844 [=================>............] - ETA: 6s - loss: 0.0548 - accuracy: 0.9824


536/844 [==================>...........] - ETA: 6s - loss: 0.0546 - accuracy: 0.9825


539/844 [==================>...........] - ETA: 6s - loss: 0.0545 - accuracy: 0.9825


542/844 [==================>...........] - ETA: 6s - loss: 0.0545 - accuracy: 0.9825


544/844 [==================>...........] - ETA: 6s - loss: 0.0544 - accuracy: 0.9825


546/844 [==================>...........] - ETA: 6s - loss: 0.0544 - accuracy: 0.9825


549/844 [==================>...........] - ETA: 6s - loss: 0.0543 - accuracy: 0.9826


552/844 [==================>...........] - ETA: 6s - loss: 0.0543 - accuracy: 0.9826


555/844 [==================>...........] - ETA: 6s - loss: 0.0547 - accuracy: 0.9825


558/844 [==================>...........] - ETA: 6s - loss: 0.0548 - accuracy: 0.9825


561/844 [==================>...........] - ETA: 6s - loss: 0.0551 - accuracy: 0.9825


564/844 [===================>..........] - ETA: 5s - loss: 0.0549 - accuracy: 0.9825


567/844 [===================>..........] - ETA: 5s - loss: 0.0551 - accuracy: 0.9826


570/844 [===================>..........] - ETA: 5s - loss: 0.0551 - accuracy: 0.9825


573/844 [===================>..........] - ETA: 5s - loss: 0.0551 - accuracy: 0.9825


576/844 [===================>..........] - ETA: 5s - loss: 0.0550 - accuracy: 0.9825


579/844 [===================>..........] - ETA: 5s - loss: 0.0549 - accuracy: 0.9825


581/844 [===================>..........] - ETA: 5s - loss: 0.0549 - accuracy: 0.9825


583/844 [===================>..........] - ETA: 5s - loss: 0.0549 - accuracy: 0.9825

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In [13], line 2
      1 # Train the model
----> 2 model.fit(train_images, train_labels, epochs=5, batch_size=64, validation_split=0.1)
      4 # Evaluate the model
      5 test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)

File ~/Library/Python/3.9/lib/python/site-packages/keras/src/utils/traceback_utils.py:65, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     63 filtered_tb = None
     64 try:
---> 65     return fn(*args, **kwargs)
     66 except Exception as e:
     67     filtered_tb = _process_traceback_frames(e.__traceback__)

File ~/Library/Python/3.9/lib/python/site-packages/keras/src/engine/training.py:1783, in Model.fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1775 with tf.profiler.experimental.Trace(
   1776     "train",
   1777     epoch_num=epoch,
   (...)
   1780     _r=1,
   1781 ):
   1782     callbacks.on_train_batch_begin(step)
-> 1783     tmp_logs = self.train_function(iterator)
   1784     if data_handler.should_sync:
   1785         context.async_wait()

File ~/Library/Python/3.9/lib/python/site-packages/tensorflow/python/util/traceback_utils.py:150, in filter_traceback.<locals>.error_handler(*args, **kwargs)
    148 filtered_tb = None
    149 try:
--> 150   return fn(*args, **kwargs)
    151 except Exception as e:
    152   filtered_tb = _process_traceback_frames(e.__traceback__)

File ~/Library/Python/3.9/lib/python/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:831, in Function.__call__(self, *args, **kwds)
    828 compiler = "xla" if self._jit_compile else "nonXla"
    830 with OptionalXlaContext(self._jit_compile):
--> 831   result = self._call(*args, **kwds)
    833 new_tracing_count = self.experimental_get_tracing_count()
    834 without_tracing = (tracing_count == new_tracing_count)

File ~/Library/Python/3.9/lib/python/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:867, in Function._call(self, *args, **kwds)
    864   self._lock.release()
    865   # In this case we have created variables on the first call, so we run the
    866   # defunned version which is guaranteed to never create variables.
--> 867   return tracing_compilation.call_function(
    868       args, kwds, self._no_variable_creation_config
    869   )
    870 elif self._variable_creation_config is not None:
    871   # Release the lock early so that multiple threads can perform the call
    872   # in parallel.
    873   self._lock.release()

File ~/Library/Python/3.9/lib/python/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py:139, in call_function(args, kwargs, tracing_options)
    137 bound_args = function.function_type.bind(*args, **kwargs)
    138 flat_inputs = function.function_type.unpack_inputs(bound_args)
--> 139 return function._call_flat(  # pylint: disable=protected-access
    140     flat_inputs, captured_inputs=function.captured_inputs
    141 )

File ~/Library/Python/3.9/lib/python/site-packages/tensorflow/python/eager/polymorphic_function/concrete_function.py:1264, in ConcreteFunction._call_flat(self, tensor_inputs, captured_inputs)
   1260 possible_gradient_type = gradients_util.PossibleTapeGradientTypes(args)
   1261 if (possible_gradient_type == gradients_util.POSSIBLE_GRADIENT_TYPES_NONE
   1262     and executing_eagerly):
   1263   # No tape is watching; skip to running the function.
-> 1264   return self._inference_function.flat_call(args)
   1265 forward_backward = self._select_forward_and_backward_functions(
   1266     args,
   1267     possible_gradient_type,
   1268     executing_eagerly)
   1269 forward_function, args_with_tangents = forward_backward.forward()

File ~/Library/Python/3.9/lib/python/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py:217, in AtomicFunction.flat_call(self, args)
    215 def flat_call(self, args: Sequence[core.Tensor]) -> Any:
    216   """Calls with tensor inputs and returns the structured output."""
--> 217   flat_outputs = self(*args)
    218   return self.function_type.pack_output(flat_outputs)

File ~/Library/Python/3.9/lib/python/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py:252, in AtomicFunction.__call__(self, *args)
    250 with record.stop_recording():
    251   if self._bound_context.executing_eagerly():
--> 252     outputs = self._bound_context.call_function(
    253         self.name,
    254         list(args),
    255         len(self.function_type.flat_outputs),
    256     )
    257   else:
    258     outputs = make_call_op_in_graph(
    259         self,
    260         list(args),
    261         self._bound_context.function_call_options.as_attrs(),
    262     )

File ~/Library/Python/3.9/lib/python/site-packages/tensorflow/python/eager/context.py:1479, in Context.call_function(self, name, tensor_inputs, num_outputs)
   1477 cancellation_context = cancellation.context()
   1478 if cancellation_context is None:
-> 1479   outputs = execute.execute(
   1480       name.decode("utf-8"),
   1481       num_outputs=num_outputs,
   1482       inputs=tensor_inputs,
   1483       attrs=attrs,
   1484       ctx=self,
   1485   )
   1486 else:
   1487   outputs = execute.execute_with_cancellation(
   1488       name.decode("utf-8"),
   1489       num_outputs=num_outputs,
   (...)
   1493       cancellation_manager=cancellation_context,
   1494   )

File ~/Library/Python/3.9/lib/python/site-packages/tensorflow/python/eager/execute.py:60, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     53   # Convert any objects of type core_types.Tensor to Tensor.
     54   inputs = [
     55       tensor_conversion_registry.convert(t)
     56       if isinstance(t, core_types.Tensor)
     57       else t
     58       for t in inputs
     59   ]
---> 60   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     61                                       inputs, attrs, num_outputs)
     62 except core._NotOkStatusException as e:
     63   if name is not None:

KeyboardInterrupt: 

# Predict the test set
y_pred = np.argmax(model.predict(test_images), axis=-1)

# Create confusion matrix
cm = confusion_matrix(test_labels, y_pred)

# Plot confusion matrix
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', cbar=False)
plt.xlabel('Predicted labels')
plt.ylabel('True labels')
plt.title('Confusion Matrix for MNIST Classification')
plt.show()

313/313 [==============================] - 1s 4ms/step
_images/1d147771fac5cb2628e33ce8c371a3623954b40df1f42b5fc252f0562214044a.png

Hyperparameter Tuning for Convolutional Neural Networks#

  • You can use the same strategies learned for tuning regular neural networks

  • First, reduce the number of layers

# Build the model
model = models.Sequential([
    layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    layers.MaxPooling2D((2, 2)),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dense(10, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(train_images, train_labels, epochs=5, batch_size=64, validation_split=0.1)

# Evaluate the model
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print('\nTest accuracy:', test_acc)

Epoch 1/5
844/844 [==============================] - 7s 8ms/step - loss: 0.2286 - accuracy: 0.9323 - val_loss: 0.0831 - val_accuracy: 0.9780
Epoch 2/5
844/844 [==============================] - 7s 9ms/step - loss: 0.0740 - accuracy: 0.9781 - val_loss: 0.0658 - val_accuracy: 0.9823
Epoch 3/5
844/844 [==============================] - 7s 9ms/step - loss: 0.0500 - accuracy: 0.9850 - val_loss: 0.0613 - val_accuracy: 0.9817
Epoch 4/5
844/844 [==============================] - 7s 9ms/step - loss: 0.0380 - accuracy: 0.9884 - val_loss: 0.0544 - val_accuracy: 0.9847
Epoch 5/5
844/844 [==============================] - 8s 9ms/step - loss: 0.0289 - accuracy: 0.9910 - val_loss: 0.0522 - val_accuracy: 0.9872
313/313 - 0s - loss: 0.0443 - accuracy: 0.9861 - 417ms/epoch - 1ms/step

Test accuracy: 0.9861000180244446

  • Remove the post-processing Dense layer

# Build the model
model = models.Sequential([
    layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.Flatten(),
    layers.Dense(10, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(train_images, train_labels, epochs=5, batch_size=64, validation_split=0.1)

# Evaluate the model
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print('\nTest accuracy:', test_acc)

Epoch 1/5
844/844 [==============================] - 16s 19ms/step - loss: 0.1937 - accuracy: 0.9404 - val_loss: 0.0603 - val_accuracy: 0.9838
Epoch 2/5
844/844 [==============================] - 18s 21ms/step - loss: 0.0571 - accuracy: 0.9822 - val_loss: 0.0502 - val_accuracy: 0.9843
Epoch 3/5
844/844 [==============================] - 17s 20ms/step - loss: 0.0388 - accuracy: 0.9875 - val_loss: 0.0350 - val_accuracy: 0.9903
Epoch 4/5
844/844 [==============================] - 17s 20ms/step - loss: 0.0309 - accuracy: 0.9903 - val_loss: 0.0359 - val_accuracy: 0.9907
Epoch 5/5
844/844 [==============================] - 16s 19ms/step - loss: 0.0240 - accuracy: 0.9925 - val_loss: 0.0351 - val_accuracy: 0.9910
313/313 - 1s - loss: 0.0283 - accuracy: 0.9919 - 1s/epoch - 3ms/step

Test accuracy: 0.9919000267982483

  • Have more filters in the convolutional layers

# Build the model
model = models.Sequential([
    layers.Conv2D(64, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(128, (3, 3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(128, (3, 3), activation='relu'),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dense(10, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(train_images, train_labels, epochs=5, batch_size=64, validation_split=0.1)

# Evaluate the model
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print('\nTest accuracy:', test_acc)

Epoch 1/5
844/844 [==============================] - 42s 50ms/step - loss: 0.1557 - accuracy: 0.9517 - val_loss: 0.0495 - val_accuracy: 0.9835
Epoch 2/5
844/844 [==============================] - 40s 48ms/step - loss: 0.0441 - accuracy: 0.9860 - val_loss: 0.0405 - val_accuracy: 0.9887
Epoch 3/5
844/844 [==============================] - 41s 49ms/step - loss: 0.0315 - accuracy: 0.9904 - val_loss: 0.0467 - val_accuracy: 0.9850
Epoch 4/5
844/844 [==============================] - 42s 50ms/step - loss: 0.0241 - accuracy: 0.9924 - val_loss: 0.0350 - val_accuracy: 0.9892
Epoch 5/5
844/844 [==============================] - 57s 67ms/step - loss: 0.0178 - accuracy: 0.9942 - val_loss: 0.0279 - val_accuracy: 0.9925
313/313 - 3s - loss: 0.0258 - accuracy: 0.9913 - 3s/epoch - 10ms/step

Test accuracy: 0.9912999868392944

  • Add an extra pooling layer

# Build the model
# Build the model
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])
# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(train_images, train_labels, epochs=5, batch_size=64, validation_split=0.1)

# Evaluate the model
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print('\nTest accuracy:', test_acc)

Epoch 1/5
844/844 [==============================] - 17s 19ms/step - loss: 0.2846 - accuracy: 0.9132 - val_loss: 0.0936 - val_accuracy: 0.9725
Epoch 2/5
844/844 [==============================] - 19s 23ms/step - loss: 0.0905 - accuracy: 0.9717 - val_loss: 0.0706 - val_accuracy: 0.9782
Epoch 3/5
844/844 [==============================] - 16s 19ms/step - loss: 0.0678 - accuracy: 0.9789 - val_loss: 0.0559 - val_accuracy: 0.9857
Epoch 4/5
844/844 [==============================] - 18s 21ms/step - loss: 0.0510 - accuracy: 0.9840 - val_loss: 0.0545 - val_accuracy: 0.9847
Epoch 5/5
844/844 [==============================] - 19s 22ms/step - loss: 0.0423 - accuracy: 0.9865 - val_loss: 0.0732 - val_accuracy: 0.9795
313/313 - 1s - loss: 0.0792 - accuracy: 0.9759 - 1s/epoch - 3ms/step

Test accuracy: 0.9758999943733215

  • Remove some pooling layers

# Build the model
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1), padding='same'),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])
# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(train_images, train_labels, epochs=5, batch_size=64, validation_split=0.1)

# Evaluate the model
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print('\nTest accuracy:', test_acc)

Epoch 1/5
844/844 [==============================] - 41s 49ms/step - loss: 0.1689 - accuracy: 0.9474 - val_loss: 0.0542 - val_accuracy: 0.9855
Epoch 2/5
844/844 [==============================] - 47s 56ms/step - loss: 0.0477 - accuracy: 0.9851 - val_loss: 0.0436 - val_accuracy: 0.9897
Epoch 3/5
154/844 [====>.........................] - ETA: 34s - loss: 0.0348 - accuracy: 0.9891

A Few Warnings#

  • Because of the downsampling/dimensionality reduction performed by the convolutional and pooling layers, you can reduce the information passed onto the next layer to nothing (ValueError)

  • Tips to fix these errors:

    • Reduce the number of layers, especially the number of pooling layers

    • Reduce the pool size (minimum is (2,2))

    • Use the padding="same" for the Conv2D layers

  • These networks can take a very long time to train, especially with many filters, many layers, or a large amount of data