{"id":1531,"date":"2023-09-26T21:06:35","date_gmt":"2023-09-26T19:06:35","guid":{"rendered":"http:\/\/blog.schmoigl-online.de\/?p=1531"},"modified":"2023-09-26T23:21:22","modified_gmt":"2023-09-26T21:21:22","slug":"tic-tac-toe-and-ai-and-what-about-the-winning-move","status":"publish","type":"post","link":"http:\/\/blog.schmoigl-online.de\/?p=1531","title":{"rendered":"Tic-Tac-Toe and AI: And what about the Winning Move? (Part 4)"},"content":{"rendered":"\n<p>After having implemented neural networks for determining whether a <a href=\"http:\/\/blog.schmoigl-online.de\/?p=1493\">Tic-Tac-Toe board has a winner or not<\/a>, and <a href=\"http:\/\/blog.schmoigl-online.de\/?p=1526\">which player the winner is<\/a>, it is now time to have a look at which is the winning move.<\/p>\n\n\n\n<p>Looking at this case, you will notice that there are only three possible cases:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The winner is clear after the third move (of either &#8220;X&#8221; or &#8220;O&#8221;),<\/li>\n\n\n\n<li>the winner is determined after the fourth move (of either &#8220;X&#8221; or &#8220;O&#8221;), or<\/li>\n\n\n\n<li>the winner is determined after the fifth move (only &#8220;X&#8221; may achieve that state, as there is an odd number of fields available on the board).<\/li>\n<\/ul>\n\n\n\n<p>Additionally, there is the special case that no winner exists, which we deserved the special identifier of &#8220;nine moves&#8221; (being the last single digit value). So, in total there are four possible states. There can only be one single state valid at a time (&#8220;one-hot case&#8221;).<\/p>\n\n\n\n<p>However, this also means that we need to adjust our data preparation: <\/p>\n\n\n\n<!--more-->\n\n\n\n<pre class=\"wp-block-code\"><code>moveAllDataFrame = pd.DataFrame(list(zip(&#091;x.vector&#091;0] for x in validtttRecordsList], \n                                     &#091;x.vector&#091;1] for x in validtttRecordsList],\n                                     &#091;x.vector&#091;2] for x in validtttRecordsList],\n                                     &#091;x.vector&#091;3] for x in validtttRecordsList],\n                                     &#091;x.vector&#091;4] for x in validtttRecordsList],\n                                     &#091;x.vector&#091;5] for x in validtttRecordsList],\n                                     &#091;x.vector&#091;6] for x in validtttRecordsList],\n                                     &#091;x.vector&#091;7] for x in validtttRecordsList],\n                                     &#091;x.vector&#091;8] for x in validtttRecordsList],\n                                     &#091;x.move == 3 for x in validtttRecordsList],\n                                     &#091;x.move == 4 for x in validtttRecordsList],\n                                     &#091;x.move == 5 for x in validtttRecordsList],\n                                     &#091;x.move == 9 for x in validtttRecordsList],\n                                        )), \n             columns =&#091;'pos1', 'pos2', 'pos3','pos4','pos5','pos6','pos7','pos8','pos9', 'move3', 'move4', 'move5', 'move9'])\n\nprint(moveAllDataFrame.tail())\nmove_train_dataset = moveAllDataFrame.sample(frac=0.8, random_state=42)\nmove_test_dataset = moveAllDataFrame.drop(move_train_dataset.index)\n\nprint(moveAllDataFrame.shape, move_train_dataset.shape, move_test_dataset.shape)\nprint(move_train_dataset.describe().transpose())\n\n# split features from labels\nmove_train_features = move_train_dataset.copy()\nmove_test_features = move_test_dataset.copy()\n\nmoveColumns = &#091;'move3', 'move4', 'move5', 'move9']\n\nmove_train_labels = move_train_features&#091;moveColumns].copy()\nmove_train_features = move_train_features.drop(moveColumns, axis=1)\nmove_test_labels = move_test_features&#091;moveColumns].copy()\nmove_test_features = move_test_features.drop(moveColumns, axis=1)<\/code><\/pre>\n\n\n\n<p>and our model we typically used previously before. For example, the last layer must have four output values:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>move_model = keras.models.Sequential(&#091;\n    move_normalizer,\n    layers.Dense(units=64, activation='relu'), #1\n    layers.Dense(units=64, activation='relu'), #2\n    layers.Dense(units=128, activation='relu'), #3 \n    layers.Dense(units=4)\n])<\/code><\/pre>\n\n\n\n<p>and also the loss function needs to be able to handle multiple outputs:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>move_model.compile(loss=keras.losses.CategoricalCrossentropy(from_logits=True),\n              optimizer=keras.optimizers.Adam(learning_rate=0.01), \n              metrics = &#091;\"accuracy\"])<\/code><\/pre>\n\n\n\n<p>The normalization layer may still remain the old one.<\/p>\n\n\n\n<p>The training shows that after 25 epochs, we get a model with a test accuracy of 1.0:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Model: \"sequential_12\"\n_________________________________________________________________\n Layer (type)                Output Shape              Param #   \n=================================================================\n normalization_3 (Normaliza  (None, 9)                 19        \n tion)                                                           \n                                                                 \n dense_35 (Dense)            (None, 64)                640       \n                                                                 \n dense_36 (Dense)            (None, 64)                4160      \n                                                                 \n dense_37 (Dense)            (None, 128)               8320      \n                                                                 \n dense_38 (Dense)            (None, 4)                 516       \n                                                                 \n=================================================================\nTotal params: 13655 (53.34 KB)\nTrainable params: 13636 (53.27 KB)\nNon-trainable params: 19 (80.00 Byte)\n_________________________________________________________________\nNone\nEpoch 1\/50\n567\/567 &#091;==============================] - 9s 12ms\/step - loss: 0.7764 - accuracy: 0.6625 - lr: 0.0100\nEpoch 2\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.4873 - accuracy: 0.7995 - lr: 0.0100\nEpoch 3\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.3017 - accuracy: 0.8853 - lr: 0.0100\nEpoch 4\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.1976 - accuracy: 0.9307 - lr: 0.0100\nEpoch 5\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.1389 - accuracy: 0.9522 - lr: 0.0100\nEpoch 6\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.1256 - accuracy: 0.9593 - lr: 0.0100\nEpoch 7\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.1020 - accuracy: 0.9658 - lr: 0.0100\nEpoch 8\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0795 - accuracy: 0.9731 - lr: 0.0100\nEpoch 9\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.0938 - accuracy: 0.9691 - lr: 0.0100\nEpoch 10\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.0746 - accuracy: 0.9755 - lr: 0.0100\nEpoch 11\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0585 - accuracy: 0.9821 - lr: 0.0100\nEpoch 12\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0744 - accuracy: 0.9761 - lr: 0.0100\nEpoch 13\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0464 - accuracy: 0.9845 - lr: 0.0100\nEpoch 14\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0677 - accuracy: 0.9792 - lr: 0.0100\nEpoch 15\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0563 - accuracy: 0.9829 - lr: 0.0100\nEpoch 16\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0089 - accuracy: 0.9978 - lr: 0.0020\nEpoch 17\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0046 - accuracy: 0.9992 - lr: 0.0020\nEpoch 18\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.0035 - accuracy: 0.9995 - lr: 0.0020\nEpoch 19\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.0027 - accuracy: 0.9997 - lr: 0.0020\nEpoch 20\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.0020 - accuracy: 0.9998 - lr: 0.0020\nEpoch 21\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0016 - accuracy: 0.9999 - lr: 0.0020\nEpoch 22\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 0.0012 - accuracy: 0.9999 - lr: 0.0020\nEpoch 23\/50\n567\/567 &#091;==============================] - 3s 6ms\/step - loss: 9.4242e-04 - accuracy: 0.9999 - lr: 0.0020\nEpoch 24\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 0.0089 - accuracy: 0.9975 - lr: 0.0020\nEpoch 25\/50\n567\/567 &#091;==============================] - 3s 5ms\/step - loss: 8.9431e-04 - accuracy: 1.0000 - lr: 0.0020<\/code><\/pre>\n\n\n\n<p>Like before, we again may try to optimize to find a &#8220;minimal&#8221; neural network configuration such that we still keep test accuracy at 1.0. After a little trial and error, the best model which I could find with a little of fine-tuning (learning rate down to 0.0016) was with two hidden Dense layers: The first one with 80 units and the second one with 128 units. Making it any smaller did not allow me to find a model with test accuracy at 1.0. This &#8220;optimal&#8221; model can be found downloaded here.<\/p>\n\n\n<p><img decoding=\"async\" src=\"http:\/\/blog.schmoigl-online.de\/wp-content\/plugins\/wp-downloadmanager\/images\/ext\/zip.gif\" alt=\"\" title=\"\" style=\"vertical-align: middle;\" \/>&nbsp;&nbsp;<strong><a href=\"http:\/\/blog.schmoigl-online.de\/?dl_id=14\">tic-tac-toe-Move-Model.zip<\/a><\/strong> (121.4 KiB, 814 hits)<\/p>\n","protected":false},"excerpt":{"rendered":"<p>After having implemented neural networks for determining whether a Tic-Tac-Toe board has a winner or not, and which player the winner is, it is now time to have a look at which is the winning move. Looking at this case, you will notice that there are only three possible cases: Additionally, there is the special &#8230;<\/p>\n<p><a href=\"http:\/\/blog.schmoigl-online.de\/?p=1531\" class=\"more-link\">Continue reading &lsquo;Tic-Tac-Toe and AI: And what about the Winning Move? (Part 4)&rsquo; &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[38],"tags":[48,50],"class_list":["post-1531","post","type-post","status-publish","format-standard","hentry","category-machine-learning","tag-tensorflow","tag-tic-tac-toe"],"_links":{"self":[{"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/posts\/1531","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1531"}],"version-history":[{"count":12,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/posts\/1531\/revisions"}],"predecessor-version":[{"id":1583,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/posts\/1531\/revisions\/1583"}],"wp:attachment":[{"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1531"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1531"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1531"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}