The service uses powerful algorithms to create ML models by finding patterns in your existing data. Then, Amazon Machine Learning uses these models to process new data and generate predictions for your application.<\/p><\/blockquote>\n
Source: https:\/\/aws.amazon.com\/aml\/?nc1=h_ls<\/a><\/p>\n But how far does this “learning” go already? If you listen to the news, you could assume that in the future, learning is no longer a task for humans anymore, but soon will only be done by machines. Let’s see how far real life really is. <\/p>\n For doing exactly this, we want to conduct a little experiment with “Amazon Machine Learning”. The service offering permits developers and data scientist to create Machine Learning models (ML models) based on a table typically provided in the CSV file format. The dataset provided (by default) is split up by 70% of data to be used for training the machine learning model, and the rest (30%) of the data is used to verify the quality of the learning process (Amazon calls this an “Evaluation”). Afterwards, Amazon allows to tweak the probability to adjust the likelihood of false positives and negatives. Finally, it is possible to query the model for its prediction either in a real-time fashion (which would be used for single requests where latency is critical), or in batch mode. Pricing is both on training and requesting predictions, whereas the pricing model differs between real-time and batch requests. The most likely simplest thing a computer may “learn” is mathematics: It is logical, highly predictable, and reproducible as often as you may want it to be. So, as Machine Learning gurus claim that their algorithms are capable to master the irrationalness of human decision making, mathematical formulas should be a piece of cake for them.<\/p>\n To test the abilities of Amazon’s machine learning offering I have set up the following test scenario for it:<\/p>\n (continued on next page) The most prominent use case provided by Amazon in their tutorial is a human decision-making process, based on a set of categorizations. The general idea with it is based on these characteristics, a binary decision (such as “will buy a product or not”) is being made. Machine learning shall support (based on a probabilistic approach) the identification of people with buying-will. <\/p>\n In the scenario depicted above, we are simulating that kind of decision: The three variables a<\/strong>, b<\/strong>, and c<\/strong> represent three characteristics based on which a decision could be taken. The decision itself is represented by the target value. Due to the fact that the formula used above always yields the same result, the decision-making process can be considered 100% rational. <\/p>\n Machine learning models require a set of data, which can be used for the learning process. Typically, the number of records for such training data is in the order of thousands (if not millions, cf. Big Data). As the scenario should stay clear, we will only provide a little redundancy to the model. That is why for each combination for each variable ranging from 1 to 7 the corresponding result and the target value is being computed. Note though, that some values with a = 1<\/em> and a = 2<\/em> are left out (we will come to that a little later). Yet, the number of records are not sufficient yet. That is why the same block of records (with exactly the same values) are being repeated five times. Also note that the (intermediate) result is not<\/strong> part of the input data for the ML model (as this would be too easy). You may find the resulting input file for your reference attached to this post. Equipped with the input data as depicted above, we may proceed now with generating the ML Datasource and the ML model, very similarly as described in Amazon’s official tutorial. On creating the datasource, Amazon visualizes the data distribution of the binary target attribute by stating (descriptively) that there are 29 falsy observations and only 11 truthy observations in the data set. As already stated above, automatically, the first 70% of the data is being used for the learning process; the other 30% is being used to assess the learning quality. <\/p>\n After less than 10 minutes processing (wall clock) time, the services returns a model, whose AUC (area under curve) is 1.000, which indicates a “perfect, not-improvable” quality. This superior result is also clearly visible in the “ML model performance analysis”, which Amazon visualizes for this model (the perfect match is indicated by the fact that the two curves do not overlap at all):<\/p>\n ML model performance Analysis<\/p><\/div>\n It is based on the fact that we have created a repetitive set tuples as input data. Due to the 70:30 ratio, the evaluation was based on exactly the same values with which the model was trained before. Therefore, setting the score threshold is quite arbitrary. To be identifiable, we will set it to 0.6.<\/p>\n Querying some sample values like a = 3, b = 3, c = 3<\/strong>, and a = 4, b = 4, c = 4<\/strong> return the expected result. It seems that Amazon’s machine learning did the trick and has “learned” what can be learned…<\/p>\n All this so far looked like that the machine is able to master mathematical computations like nothing: Also conducting a test of all 40 values, which we made available with the input data, showed that the expected result is equal to the actual result (i.e. the model responded truthy, if the input data was truthy and the model responded falsy, if the input data indicated that the target value was falsy).<\/p>\n However, “learning” is not just the activity of reproducing memorized information – the technique of memoization<\/a> is already known to computer science for decades (it came up in the late 1960s) and may be easily made persistent in a relation database management system. Instead, the most important effect is the capability to transfer the learned<\/a>. That is where the missed records mentioned above kick in: Note that, for example, we did not provide the target value a = 1, b = 2, c = 2<\/strong> in the input data! So, querying for that tuple (and such alike) might give an indication on how strong the “real learning effect” really is. See yourself:<\/p>\n The result is not very appealing: Though the response for a = 1, b = 2, c = 2<\/strong> is correct, confidence from the model (the score value) could be higher. Moreover, the other two results provided are simply wrong – and also the model appears to know that (see the low score values)!<\/p>\n An even more interesting result is provided as soon as we leave the pre-defined range for a, b, and c between 1 and 7:<\/p>\n The table above gives an indication to another effect: Observe that for each tuple, where at least one value is used, which is out of the initial range, the target value false<\/strong> is returned! This observation can also be verified with all tuples having the following patterns:<\/p>\n Given the initial data analysis observation that in the training data, there are more falsy values than truthy one’s, defaulting to the value false<\/strong> seems reasonable. However, I would not dare to call “defaulting to a single value whose probability is the highest” to be the same as “learning”.<\/p>\n A complete list of all predictions made by the ML model with (x,y,z)<\/em>, whereas each variable ranges from 1 to 10, can be downloaded here: The Scenario<\/h3>\n
\nThe tutorial<\/a> describes all the necessary steps for this. Pricing can be found at the usual Amazon Pricing pages<\/a>.<\/p>\n\n
\n<\/p>\nHow does this fit to Amazon Machine Learning?<\/h3>\n
Generation of Input Data<\/h3>\n
\n![\"\"](\"http:\/\/blog.schmoigl-online.de\/wp-content\/plugins\/wp-downloadmanager\/images\/ext\/unknown.gif\" "\\"\\"")
input-data-for-amazon-ml.csv<\/a><\/strong> (1.9 KiB, 659 hits)<\/p><\/p>\nGenerating the Machine Learning Model<\/h3>\n
![\"\"](\"http:\/\/blog.schmoigl-online.de\/wp-content\/uploads\/auc1.png\")
<\/a>Looking a bit closer<\/h3>\n
\n
\n a<\/th>\n b<\/th>\n c<\/th>\n intermediate result<\/th>\n expected target<\/th>\n actual target<\/th>\n reported score<\/th>\n<\/tr>\n \n 1<\/td>\n 2<\/td>\n 2<\/td>\n 0<\/td>\n false<\/td>\n false<\/td>\n 0.328229159116745<\/td>\n<\/tr>\n \n 1<\/td>\n 2<\/td>\n 3<\/td>\n 16<\/td>\n true<\/td>\n false<\/td>\n 0.0001738734426908195<\/td>\n<\/tr>\n \n 2<\/td>\n 1<\/td>\n 3<\/td>\n 16<\/td>\n true<\/td>\n false<\/td>\n 0.0001329469378106296<\/td>\n<\/tr>\n<\/table>\n \n
\n a<\/th>\n b<\/th>\n c<\/th>\n intermediate result<\/th>\n expected target<\/th>\n actual target<\/th>\n<\/tr>\n \n 8<\/td>\n 8<\/td>\n 8<\/td>\n 5<\/td>\n false<\/td>\n false<\/td>\n<\/tr>\n \n 8<\/td>\n 8<\/td>\n 9<\/td>\n 4<\/td>\n false<\/td>\n false<\/td>\n<\/tr>\n \n 8<\/td>\n 9<\/td>\n 8<\/td>\n 13<\/td>\n true<\/td>\n false<\/td>\n<\/tr>\n \n 12<\/td>\n 12<\/td>\n 12<\/td>\n 13<\/td>\n true<\/td>\n false<\/td>\n<\/tr>\n \n 44<\/td>\n 1<\/td>\n 1<\/td>\n 9<\/td>\n false<\/td>\n false<\/td>\n<\/tr>\n \n 45<\/td>\n 1<\/td>\n 1<\/td>\n 10<\/td>\n true<\/td>\n false<\/td>\n<\/tr>\n \n 1<\/td>\n 1<\/td>\n 18<\/td>\n 0<\/td>\n false<\/td>\n false<\/td>\n<\/tr>\n \n 1<\/td>\n 1<\/td>\n 19<\/td>\n 16<\/td>\n true<\/td>\n false<\/td>\n<\/tr>\n<\/table>\n \n
Descriptive Analysis of Predictions<\/h3>\n
![\"\"](\"http:\/\/blog.schmoigl-online.de\/wp-content\/uploads\/scorehistogram.png\")
<\/a>