Comments on: Fraud detection – decision trees versus support vector machines (classification) http://www.baqmar.be/?p=1821 the Belgian Association for Quantitative and Qualitative Marketing Research Wed, 18 Aug 2010 15:21:25 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Geert Bilcke http://www.baqmar.be/?p=1821&cpage=1#comment-1856 Geert Bilcke Wed, 25 Nov 2009 20:35:37 +0000 http://www.baqmar.be/?p=1821#comment-1856 Hi, I think it is a bit "unfair" to compare a strong learner (SVM) with a weak learner (DT), especially if you work with a --for data mining standards-- very small minority class (300.000 * 0.06% = only 180 positive targets). With such a dataset strong learners (e.g. logistic regression) will always outperform a single decision tree. Try doing the exercise with DT bagging and I am sure your conclusion will be totally different. Geert Hi,
I think it is a bit “unfair” to compare a strong learner (SVM) with a weak learner (DT), especially if you work with a –for data mining standards– very small minority class (300.000 * 0.06% = only 180 positive targets).
With such a dataset strong learners (e.g. logistic regression) will always outperform a single decision tree. Try doing the exercise with DT bagging and I am sure your conclusion will be totally different.
Geert

]]> By: datalligence http://www.baqmar.be/?p=1821&cpage=1#comment-1855 datalligence Wed, 25 Nov 2009 12:10:04 +0000 http://www.baqmar.be/?p=1821#comment-1855 hi eric, the business requirement was to identify fraudulent transations and the data was at the transaction level. so a customer can have multiple transactions in the data. some businesses require the false positives to be on the lower side, some prefer it to be on the higher side. here, the idea was to deploy the model in real-time so that transactions suspected to be fraudulent can be put on hold or cancelled. as such, a higher FP was ok as long as the model could capture more number of fraudulent transactions. but i guess, i generalized a little too much when i first wrote this! as for the data, sorry, no comments :) hi eric,

the business requirement was to identify fraudulent transations and the data was at the transaction level. so a customer can have multiple transactions in the data.

some businesses require the false positives to be on the lower side, some prefer it to be on the higher side. here, the idea was to deploy the model in real-time so that transactions suspected to be fraudulent can be put on hold or cancelled. as such, a higher FP was ok as long as the model could capture more number of fraudulent transactions.

but i guess, i generalized a little too much when i first wrote this! as for the data, sorry, no comments :)

]]> By: Eric Lecoutre http://www.baqmar.be/?p=1821&cpage=1#comment-1854 Eric Lecoutre Wed, 25 Nov 2009 10:21:29 +0000 http://www.baqmar.be/?p=1821#comment-1854 Hi, I can see a problem with your conclusions. If you look at the confusion matrices, you can see that - for DT there are a total of 72 cases predicted as 1. The 61 correctly classified represent 85% of those predictions. - for SVM, there are a total of 278+79=357 cases predicted as 1. The 79 correctly predicted represent only 22% of those ones. Both learnings methods rely on parameters you may change to adjust what somewhat is the cutpoint of predictions 0/1. If a am a fraud detection organism, I prefer DT: making verifications has a cost. Having the estimation that doing less verifications to have more real frauders is of great interest... Another way to say it is: do verifications for all 127.000 people, you will find all frauders! That's why one introduce the notion cost matrix ; it is not clear to me what the prefered modeling would then be. BTW, where can I find this dataset? There is an modeling approch I would like to test for comparisons and assessment purpose. Kind regards, Eric Hi,

I can see a problem with your conclusions.
If you look at the confusion matrices, you can see that
- for DT there are a total of 72 cases predicted as 1. The 61 correctly classified represent 85% of those predictions.
- for SVM, there are a total of 278+79=357 cases predicted as 1. The 79 correctly predicted represent only 22% of those ones.
Both learnings methods rely on parameters you may change to adjust what somewhat is the cutpoint of predictions 0/1.
If a am a fraud detection organism, I prefer DT: making verifications has a cost. Having the estimation that doing less verifications to have more real frauders is of great interest…
Another way to say it is: do verifications for all 127.000 people, you will find all frauders!
That’s why one introduce the notion cost matrix ; it is not clear to me what the prefered modeling would then be.
BTW, where can I find this dataset? There is an modeling approch I would like to test for comparisons and assessment purpose.
Kind regards,
Eric

]]>