I Need A Statistician! Help!

[Smell the Fear]

I could have helped you about 10 years ago, but I've forgotten most of that now. But it seems some of the other guys remember it.

But you don't seem to have said what information or statistic you are looking for from the data and stats you have provided.

The data is individual monkey test scores.

[crash2006]

var 36.83 32.35

Observations 159. 87 this is just the number of times or events

means -36.83 from the mean and + 36.83 from the mean

Mean 70.35

so it is medium Volatile, high risk.

[DataMonkey]

The data is individual monkey test scores.

I should have guessed

The 2 possible outcomes of a hypothesis test are i) Reject H0; and ii) Fail to reject H0. Note there is no acceptance going on, it's the same principle as court cases, where the null hypothesis is that you are innocent, and, based on the evidence, you are either found 'guilty' or 'not guilty'. Contrary to what the media say, no one is ever found innocent.

With respect to the confidence level, it's best not to try and interpret it. The true interpretation is not what you think, and this has caught out countless statisticians. The confidence level (5% in this case) is the probability of Type I error; or the probability that you will reject H0 when it's actually true. So what you're proposing is to increase your Type I error rate to 25%, which I would suggest is unwise!

The opposite of Type I error is Type II error, i.e. not rejecting H0 when you should. Unfortunately it's not possible to minimise Type I error and Type II error simultaneously, hence the strategy of setting Type I error to a known, acceptable level, then minimising Type II error, as done in the hypothesis test.

[Uriah Heap]

Daedalus/Datamonkey - thanks very much. That confirms what I thought, more or less.

When you say "there is no evidence at all that the means are different" is that a fair comment? Is it not the case that a P value of say 0.25 would indicate a 75% probability that the difference was due to factors other than chance? In scientific terms that would be rejected but in other areas of life it may be regarded as sufficient. Please correct me if this is wrong as I don't want to look like an idiot by stating it!

I think that is a common fallacy, though I see how people get there.

The correct answer is that you need decide on what the probability you are going to accept before you start the analysis. In clinical trials the generally accepted figure is 5%. You reject any data where p exceeds 0.05 - end of story. The way to think of it is that p has to have some value, even if the data are completely random. That doesn't mean that the p value is a measure of significance.

For what its worth, the interpretation I would put on the data you present is that there is no evidence that the two groups differ in any way from the numbers you have crunched.

[crash2006]

I think that is a common fallacy, though I see how people get there.

The correct answer is that you need decide on what the probability you are going to accept before you start the analysis. In clinical trials the generally accepted figure is 5%. You reject any data where p exceeds 0.05 - end of story. The way to think of it is that p has to have some value, even if the data are completely random. That doesn't mean that the p value is a measure of significance.

For what its worth, the interpretation I would put on the data you present is that there is no evidence that the two groups differ in any way from the numbers you have crunched.

Only really works for T and Z test, F-test different method. f test you need to know the DF v1 and v2 to get the critical value to base the F test on.

[Smell the Fear]

I think that is a common fallacy, though I see how people get there.

The correct answer is that you need decide on what the probability you are going to accept before you start the analysis. In clinical trials the generally accepted figure is 5%. You reject any data where p exceeds 0.05 - end of story. The way to think of it is that p has to have some value, even if the data are completely random. That doesn't mean that the p value is a measure of significance.

For what its worth, the interpretation I would put on the data you present is that there is no evidence that the two groups differ in any way from the numbers you have crunched.

So if the value I set at the start was 0.3 (i.e. reject if p>0.3) would I get a different value for p after calculating than if I set a value of 0.05?

I'm suspecting here that everyone believes that only a p<0.05 is significant. Is it not the case that a p of <0.05 is the generally accepted standard for scientific research, i.e. to prove beyond all reasonable doubt? But that setting a lower standard can still provide us with valuable information, albeit at a lower level?

What if I wanted to set a standard of "on the balance of probabilities"? Could I then say that a p of 0.45 proves that the data is different on a balance of probabilities? Would not setting this parameter at the outset in some way affect the calculation of the p value or would I get the same p value as when I set a parameter of reject p<0.05?

[DataMonkey]

Only really works for T and Z test, F-test different method. f test you need to know the DF v1 and v2 to get the critical value to base the F test on.

Eh? You need 2 DF values (which can be calculated from the number of observations in the 2 samples) but the general methodology is the same, you still set the level of significance before carrying out the test.

Edit: evidently can't do Maths and English at the same time.

[Smell the Fear]

I should have guessed

The 2 possible outcomes of a hypothesis test are i) Reject H0; and ii) Fail to reject H0. Note there is no acceptance going on, it's the same principle as court cases, where the null hypothesis is that you are innocent, and, based on the evidence, you are either found 'guilty' or 'not guilty'. Contrary to what the media say, no one is ever found innocent.

With respect to the confidence level, it's best not to try and interpret it. The true interpretation is not what you think, and this has caught out countless statisticians. The confidence level (5% in this case) is the probability of Type I error; or the probability that you will reject H0 when it's actually true. So what you're proposing is to increase your Type I error rate to 25%, which I would suggest is unwise!

The opposite of Type I error is Type II error, i.e. not rejecting H0 when you should. Unfortunately it's not possible to minimise Type I error and Type II error simultaneously, hence the strategy of setting Type I error to a known, acceptable level, then minimising Type II error, as done in the hypothesis test.

As I mentioned in reposnse to another poster, is it not the case that I can interpret a p of 0.45 as meaning that there is a 45% of a type I error (and hence there is a 55% chance that there IS a difference between the 2 means). Guilty/not guilty applies to criminal cases. A civil case is decided on the basis of a balance of probabilities, i.e. is it more likely than not likely. How would this fit with your interpretation?

[DataMonkey]

So if the value I set at the start was 0.3 (i.e. reject if p>0.3) would I get a different value for p after calculating than if I set a value of 0.05?

I'm suspecting here that everyone believes that only a p<0.05 is significant. Is it not the case that a p of <0.05 is the generally accepted standard for scientific research, i.e. to prove beyond all reasonable doubt? But that setting a lower standard can still provide us with valuable information, albeit at a lower level?

What if I wanted to set a standard of "on the balance of probabilities"? Could I then say that a p of 0.45 proves that the data is different on a balance of probabilities? Would not setting this parameter at the outset in some way affect the calculation of the p value or would I get the same p value as when I set a parameter of reject p<0.05?

No, your t statistic (I assume this is what you're referring to) is fixed; it is a function of your data. All you are doing by setting your value up front is defining your reject / don't reject criteria. Changing this is simply moving the goalposts (or more accurately expanding the goal) to increase the likelihood of scoring. Of course, there is always your borderline case (hitting the post) which is the critical value.

I appreciate where you're coming from with the 'balance of probabilities' thing, but it's not really that simple! Of course, you can reduce the significance of your test, but it just casts more doubt on your result. I've never seen anything less significant than 10% used in practice. Using 45% is extremely likely to yield a 'Reject H0' result, but you have a 45% chance of being wrong. That's pretty much equivalent to sacking off the whole statistics thing and flipping a coin to decide whether your means are significantly different!

The problem is that you are testing for a difference in population means using sample means, thus you have the added headache of how your sample selection might affect this.

I'm sorry, I appreciate that's probably not a very good description, I might try and come up with a better one tomorrow after some more thinking. It's not an intuitive concept, and I remember when first seeing it at A Level that it was completely baffling so I do sympathise

[Wad]

Daedalus/Datamonkey got there before me and I agree with what they say.

The absolutely fundamental conclusion you should draw from this statistical output is that the means of the two samples are NOT different at a statistically significant level that would be acceptable to any statistician. The fact that there is slight difference between the actual means you have calculated for the two samples is simply down to random sampling error.

[crash2006]

Eh? You need 2 DF values (which can be calculated from the number of observations in the 2 samples) but the general methodology is the same, you still set the level of significance before carrying out the test.

Edit: evidently can't do Maths and English at the same time.

f test is a bit different you work out the v1 and v2 (numerator and denominator) if you look there are 2 means h0 = means 1 = means 2, h1 doesnt equal.

v2= k-1 v1 = n-k .

clearly you haven't looked at the question and understood how it works.

f test the critical point will always change based on the observations.

[crash2006]

Daedalus/Datamonkey got there before me and I agree with what they say.

The absolutely fundamental conclusion you should draw from this statistical output is that the means of the two samples are NOT different at a statistically significant level that would be acceptable to any statistician. The fact that there is slight difference between the actual means you have calculated for the two samples is simply down to random sampling error.

Doesnt matter if the means are different, that why we use F test for multi values, and for r2.

you can have h0 = 100=99=80, h1 isnt =.

mean1=100

mean2=99

mean3=80

[DataMonkey]

f test is a bit different you work out the v1 and v2 (numerator and denominator) if you look there are 2 means h0 = means 1 = means 2, h1 doesnt equal.

v2= k-1 v1 = n-k .

clearly you haven't looked at the question and understood how it works.

f test the critical point will always change based on the observations.

Ah, I see what you mean. Yes, the critical value will change based on the number of degrees of freedom (function of number observations) but as this is known in this case we can disregard it in our discussion on levels of significance. The same is in fact true of the t test (as n tends to infinity, the t distribution tends to the Normal distribution).

[Cogs]

So if the value I set at the start was 0.3 (i.e. reject if p>0.3) would I get a different value for p after calculating than if I set a value of 0.05?

I'm suspecting here that everyone believes that only a p<0.05 is significant. Is it not the case that a p of <0.05 is the generally accepted standard for scientific research, i.e. to prove beyond all reasonable doubt? But that setting a lower standard can still provide us with valuable information, albeit at a lower level?

What if I wanted to set a standard of "on the balance of probabilities"? Could I then say that a p of 0.45 proves that the data is different on a balance of probabilities? Would not setting this parameter at the outset in some way affect the calculation of the p value or would I get the same p value as when I set a parameter of reject p<0.05?

No, it wouldn't affect the calculation at all. If you do it by hand (which I recommend if you want to learn how it works) you'll see you calculate your t ratio and then in old skool mode, go and look up what your p value is on a table of critical values given the degrees of freedom and whether it was one-tailed or two-tailed.

p = 0.05 is not beyond reasonable doubt, you are still running a 1 in 20 risk that what looks like a significant difference is just attributable to chance.

This is just considered the acceptable level at which it might be worth telling other people what you've found. Replication is expected to follow.

At p = 0.45 you are basically flipping a coin, so no if there is a difference there it is as likely to be illusionary (sampling error) as genuine.

A common problem here (made by journos as well) is to confuse the p value with being some sort of magnitude of effect measure. They are separate things.

I suspect the confusion arises from people using SPSS before they've learned how to grind out these things with pen, paper and calculator.

Either way, you are nowhere near finding a difference.

[Smell the Fear]

Daedalus/Datamonkey got there before me and I agree with what they say.

The absolutely fundamental conclusion you should draw from this statistical output is that the means of the two samples are NOT different at a statistically significant level that would be acceptable to any statistician. The fact that there is slight difference between the actual means you have calculated for the two samples is simply down to random sampling error.

Are you sure?

Is it not the case that there if p=0.45 there is a 45% chance that it is down to a random sampling error?

And hence there is a 55% chance that it is NOT down to a random sampling error?

Or to look at it another way, on a balance of probabilities it is likely that the difference is not due to chance?

This is the key question that I need to answer.

[Smell the Fear]

No, your t statistic (I assume this is what you're referring to) is fixed; it is a function of your data. All you are doing by setting your value up front is defining your reject / don't reject criteria. Changing this is simply moving the goalposts (or more accurately expanding the goal) to increase the likelihood of scoring. Of course, there is always your borderline case (hitting the post) which is the critical value.

I appreciate where you're coming from with the 'balance of probabilities' thing, but it's not really that simple! Of course, you can reduce the significance of your test, but it just casts more doubt on your result. I've never seen anything less significant than 10% used in practice. Using 45% is extremely likely to yield a 'Reject H0' result, but you have a 45% chance of being wrong. That's pretty much equivalent to sacking off the whole statistics thing and flipping a coin to decide whether your means are significantly different!

The problem is that you are testing for a difference in population means using sample means, thus you have the added headache of how your sample selection might affect this.

I'm sorry, I appreciate that's probably not a very good description, I might try and come up with a better one tomorrow after some more thinking. It's not an intuitive concept, and I remember when first seeing it at A Level that it was completely baffling so I do sympathise

If you could find a coin (or to extend it, a method of picking shares) that went in your favour 55% of the time (or even 50.1% of the time) you would have a coin or method which could make you extremely rich in the long run.

I am getting a feeling that people trained in statistics have a knee-jerk reaction that says "reject if p>0.05" or at a stretch "reject if p>0.10". There seems a reluctance to attach any value to a p value above this level.

If a situation was to be judged on a balance of probabilities (in the real world many are, not everything lends itself to a 95% confidence interval and decisions MUST be made) would you say that a p value of 0.45 indicates that the null hypothesis should be rejected? I fail to see why it wouldn't.

I'm guessing many people here are approaching this from an engineering perspective where a great deal of precision is required and possible. What if it isn't? Do statistics have any value?

[Smell the Fear]

At p = 0.45 you are basically flipping a coin, so no if there is a difference there it is as likely to be illusionary (sampling error) as genuine.

Do you not mean that there is a 45% chance that it is illusory, and a 55% chance that it is genuine? Are you saying that 45% is approximately the same as 50%, or do you mean that the p value is meaningless at this level?

[uptherebels]

WTF