With the increasing popularity of Machine learning today, it is important to focus the testing aspect of ML application. Testing of any ML application will not be same as testing traditional software. It has become a debatable topic. Many literature categorised ML application as non-testable. However, many are now trying to make it testable and coming up many innovative approaches. All are very much technical and normal users without technical knowledge find difficulties. Therefore, here I have tried to explain the difference of testing traditional software and ML application in a simple terminology without going into any technical term.
Say I have taught a person about mathematical number line. After that I like to test him. Now what do I mean by test him. I can check what he knows after this or what he learned. It’s bit tricky. Let us have example.
If I ask him “what will be answer if I add two positive numbers 5 & 6 in number line”. What answer I expects.
1) Obviously 11. Yes this is the correct answer if know the number line and most of us will expect this answer as correct.
Now, if he answer as follows in different language
2) A number value of which is between 10 & 12
3) A number value of which is greater than 8, but less than 12
4) It is equivalent of 8+3
5) …and so on
Are these answers wrong? Absolutely no. All are correct answer. However, if I get such answers where all of them are correct to him, I’ll say he learned the number line/system.
This is the difference of testing traditional Software testing and testing a ML application
In Traditional software we check out exactly the output is as compared to expected. Example the (1) case. We set expectation 11 and I’ll check output is 11 or not. Here the person knows just a rule about addition and that is why answer is 11. However, rest of the examples are testing of ML application, where output is not exactly what I expected, but all are correct and close to my expectation. Hence, in ML application we’ll not test how exact the output is, rather how close the output to correctness.
We need to understand that while you are testing ML application, you are basically testing a software which itself learns, not just sequence of rules. With same data as input, you might get different output in two different run. Therefore, testing ML application requires an entirely different approach and the test team also needs to be elevated with ML skill. Eventually, to adopt ML application, we also need a different culture across organisation to sync our expectation. Even support engineer also needs this skill to handle any incident for ML application. Most important we need to come out from traditional mindset and embrace a new way of thinking. With this basic understanding, I like to encourage reader to read any articles which are available in internet about testing of traditional application vs testing ML application, they will understand better. I'm very much open to have any new idea, new thought on this topic from all of you.