InferencePermutation tests
First, let me be upfront with a bit of a warning. This lesson is a bit more technical than the previous. To boot there isn't much material in the main book. To compensate, I add much more background here in the notes than in the first three lessons.
Before describing the new concepts we will introduce this week, let's put the past three weeks in some perspective. The progression: one-sample \( t \)-test, two-sample \( t \)-test, then linear regression was chosen for both practical and pedagogical reasons and were we more ambitious, would have been extended to include multiple regression and ANOVA. These topics naturally come together, as, from some perspective, they are all specializations of the multiple regression model:
\[ Y_i = \beta_0 + \beta_1 x_{1i} + \cdots + \beta_k x_{ki} + \epsilon_i \]
E.g.,
The simple regression model is just an assumption \( \beta_2 = ... = \beta_k = 0 \), that is \( Y_i = \beta_0 + \beta_1 x_i + \epsilon_i \);
the one-sample \( t \) test is basically an assumption that \( \beta_1 = ... = \beta_k = 0 \), or \( Y_i = \mu + \epsilon_i \);
the two-sample \( t \) test takes \( Y \) to be both data sets combined, \( x_{1i} \) to be 1 or 0 depending on which data set the \( i \) th data point is in. Coding like this is how ANOVA is done internally by R.
(The latter can be seen with the following commands you can copy and paste)
x <- rnorm(4)
y <- rnorm(3, 1)
d <- stack(list(x = x, y = y))
oneway.test(values ~ ind, data = d, var.equal = TRUE)
summary(res <- lm(values ~ ind, data = d))
model.matrix(res)[, 2]
For the regression model the coefficients are estimated with the least squares technique, and, more importantly for this discussion, if we assume the error terms, \( \epsilon_i \), are normally distributed and are a random sample then much is known about sampling distribution of various test statistics. These are either \( t \)- or \( F \)-distributions. (Basically, tests of coefficient are \( t \)-tests, and tests comparing a model to a sub-model involve an \( F \) test.)
This lesson is somehow similar, in that some framework is introduced that when specialized can cover many different tests, which at first glance my seem unrelated.
Later, we will talk briefly about R's coin package which covers this
class of tests. The unifying theme here is that an assumption on the
data \( Y \) and \( X \) leads to a class of statistics with known sampling
distributions. With this, forming \( p \)-values is a matter of
reinterpreting what “as or more extreme” means. The key assumption –
though often only stated implicitly – is about independence. This
assumption allows the sampling distribution to be described in terms
of permutations (or combinations) hence the name. A permutation test
is one where the test statistic under the null hypothesis is
“exchangeable”, meaning one can rearrange or permute the data and
the distribution is unchanged. We will see that some applications are
naturally called re-randomization, as that is how the problem is
approached.
In theory, such permutation distributions are known, in practice they may be intractable. Historically this may have been an impediment, however this topic has opened up in recent years due to advances in computation, in particular the ease and ubiquity of simulation. 'Ease', of course, is relative, and the point here is an attempt to 'ease' you in. R is particularly helpful with simulation. We will discuss how to do basic simulations and then apply what we see to one type of computer-based inference: permutation methods for doing significance tests.
To do all this, in the process of learning about permutation tests we also get introduced to the following useful R programming topics:
Writing basic functions
Looping or iterating over values. For this we discuss for,
replicate and the “apply” functions like apply and sapply.
some special-purpose functions for doing tests and a general
framework provided by the coin package.
Our plan: we will start with an example to make things more concrete, then build up some R skills needed for simulations. (These are useful skills for more advanced R use even if simulation isn't your goal.) From there we give several different examples and finally present an R package that can be used to unify the disparate things discussed.
Some extra sources of information on permutation tests are available on line. Here are a few:
T. Lynn Eudey, Joshua D. Kerr, and Bruce E. Trumbo, http://www.amstat.org/publications/jse/v18n1/eudey.pdf – a very nice article on simulating the distributions using R and a good motivation of the material covered here.
Hesterberg, et. al., http://www.timhesterberg.net/bootstrap – here one can find a long, informative chapter on bootstrap methods and permutation tests to accompany an older version of the very popular “Moore and McCabe” textbook on introductory statistics. Examples in S-Plus (basically R) are included.
Cobb, George W., The Introductory Statistics Course: A Ptolemaic Curriculum? Technology Innovations in Statistics Education, 1(1). (Can be fond online, http://escholarship.org/uc/uclastat_cts_tise.) This article makes the case for introducing permutation tests as the motivator of significance tests, as opposed to the \( t \) test, as was done here. It derives in part from an article by Ernst.
Michael Ernst, http://projecteuclid.org/DPubS/Repository/1.0/Disseminate?view=body&id=pdfview_1&handle=euclid.ss/1113832732
Douglas Curran-Everett has this article “Explorations in statistics: permutation methods” in Advances in Physiology Education http://advan.physiology.org/content/36/3/181.full
Example An introductory example
The basic ideas of a permutation test are contained in the following example, which we borrow from the Eudey article listed above.
Corn yields can be effected by many different factors, some – such as drought – are widely reported in the papers. The following data comes from an experiment to see if the presence of weeds significantly effects corn yields. The data found are:Weed Free: 166.7, 172.2, 165.0, 176.9 Many Weeds: 162.8, 142.4, 162.8, 162.4For this data, a basic question would be: does weeding incease yields?
The experimental design: 8 plots were planted. Of these 8, 4 were randomly chosen to be weeded (the weed-free plots). The other 4 were not. At the end of the growing season, the yields were then measured with some standard units.
The statistical approach to this question – do weeds reduce yield? – would be to formulate the problem into a significance test. The null hypothesis would be an assumption of no effect, the alternative would be an effect (one-tailed, weeds reduce yield). If we let \( \mu_1 \) be the population mean for all fields if weed free and \( \mu_2 \) be the population mean for all fields if many weeds, then we have:
\[ H_0: \mu_1 = \mu_2 \quad H_A: \mu_1 > \mu_2 \]
If we further assume that the populations of all potential corn yields are normally distributed with a common variance, then the following \( t \)-test would be appropriate:
weed_free <- c(166.7, 172.2, 165, 176.9)
many_weeds <- c(162.8, 142.4, 162.8, 162.4)
t.test(weed_free, many_weeds, alternative = "greater", var.equal = TRUE)
##
## Two Sample t-test
##
## data: weed_free and many_weeds
## t = 2.192, df = 6, p-value = 0.03542
## alternative hypothesis: true difference in means is greater than 0
## 95 percent confidence interval:
## 1.432 Inf
## sample estimates:
## mean of x mean of y
## 170.2 157.6
A result which is statistically significant. Though this doesn't “prove” weeding plots has an effect, it shows that the data is unusual given the null hypothesis is true.
Now, what are the issues with the above that make us want to possibly consider some other approach? Well:
The \( t \)-test assumes these yields are some sample from a parent population that applies to all possible fields. This may not be realistic. We do know here that what was randomized was the choice of which 4 fields would be weeded, as we had control over that.
Even if we assume some wider parent population, we had to make a leap of faith that the parent populations were normally distributed. Clearly there aren't enough sample data to warrant this assumption based solely on the observed data. Perhaps we have enough experience to say it typically should be. But maybe not. Gossett says in his seminal paper, http://www.york.ac.uk/depts/maths/histstat/student.pdf, “since some law of distribution must he assumed it is better to work with a curve whose area and ordinates are tabled, and whose properties are well known.” But that is more of a rationalization than anything else.
We used the mean to summarize yields. It might – due to an understanding of the distribution of yields – make more sense to consider other statistics when comparing two populations of yields: median yield, the 25th percentile, …
So, let's look at this again.
The null hypothesis is that there is no effect. We translated this into a mathematical statement that \( \mu_1 = \mu_2 \), but we needn't have done that. It was just convenient as the parent population is parameterized by the mean. Instead, we here follow Apple's mantra and “think differently.”
If we assume there is no effect, then the act of randomization of
treatment (weeding) to plot should have had no impact. We could have
done any of the possible assignments of 4 weeded and 4 unweeded
plots. Perhaps it was just the ones that we did do happened to give an
unusual difference of the mean. There are 8 choose 4 ways to have done
this randomization (don't worry if you don't know that). This is 70 =
choose(8,4) different values.
Here is the key point: imagine there really is no effect, then if we had done these 70 assignments the yields by plot would be no different than what we saw – their differences are due to other factors (e.g., seeds, shade, sun, soil, terrain, …), but the yields by weed type would have differed (as we would have chosen different plots as weed free).
Well, this assumption gives us a different way to investigate the difference. We would just have to imagine the 70 different ways to have weeded the fields and compare the differences.
The basic computer framework we use for the problem assumes we number the plots 1 through 8. We then just look at all ways to choose 4 plots from the 8 to be weed free. The actual choice with our coding below being 1,2,3,4. But, for a different choice of plots – say 1,3,4,7 – we get different plots that would have been weed free. Instead we would have seen a difference of:
all_data <- c(weed_free, many_weeds)
cmb <- c(1, 3, 4, 7)
all_data[cmb] ## weed free plots
## [1] 166.7 165.0 176.9 162.8
all_data[-cmb] ## non-weeded plots
## [1] 172.2 162.8 142.4 162.4
mean(all_data[cmb]) - mean(all_data[-cmb]) ## difference for this combination
## [1] 7.9
So instead of the value of 12.6 for the difference we would have gotten here a difference of 7.9.
We now just need to look at the 68 other ways we could have done the assignment of 4 weeded plots and see how extreme 12.6 is. Doing this one by one is, well, tedious. We want to have some way to push off to the computer what it can do really well – repeatedly doing things.
For this we note that R's useful combn function will make our
combinations and store them in a matrix – one column for each
permutation. Here is how we could look at the first two:
m <- combn(8, 4)
dim(m) # A 4 row 70 column matrix
## [1] 4 70
m[, 1:4] # first 4 permutations
## [,1] [,2] [,3] [,4]
## [1,] 1 1 1 1
## [2,] 2 2 2 2
## [3,] 3 3 3 3
## [4,] 4 5 6 7
mean(all_data[m[, 1]]) - mean(all_data[-m[, 1]])
## [1] 12.6
mean(all_data[m[, 2]]) - mean(all_data[-m[, 2]])
## [1] 5.55
Computing this all 70 times requires some way of iterating over the columns of
m (the second index). In R there are many different ways to
iterate. We mention several in this week's lecture, where we also
discuss functions. Putting off the details for later, at this time we
choose the convenient apply function to iterate and define a
function to reduce the data. This function, assigned to f below, will compute the
difference of the means for given choice of weeded plots (passed in as
i):
f <- function(i) {
mean(all_data[i]) - mean(all_data[-i])
}
And here is the command to generate all the numbers:
out <- apply(m, 2, f)
head(out) ## first 6
## [1] 12.60 5.55 -4.65 5.55 5.35 11.50
Now, how unusual was 12.6 (the observed difference in yields)? Well, this is answered with the \( p \)-value which is the probability the value would have been as or more extreme than observed. In this case, would have been 12.6 (the first one) or more:
sum(out >= out[1])/length(out)
## [1] 0.01429
So, very unlikely (we observed the combination with the largest case).
As an aside, to see the data permuted copy and paste the following
code into R and look at m1. (The m2 data is m1 sorted in decreasing order
by the difference. The observed difference is the first.)
all_data <- c(166.7, 172.2, 165, 176.9, 162.8, 142.4, 162.8, 162.4)
cmb <- combn(8, 4)
m <- t(apply(cmb, 2, function(i) c(all_data[i], all_data[-i])))
colnames(m) <- paste("Plot", 1:8)
m1 <- cbind(m, apply(m, 1, function(i) mean(i[1:4])), apply(m, 1, function(i) mean(i[5:8])),
apply(m, 1, function(i) mean(i[1:4]) - mean(i[5:8])))
colnames(m1)[9:11] <- c("Mean weeded plots", "Mean unweeded plots", "Difference")
ind <- order(m1[, 11], decreasing = TRUE)
m2 <- m1[ind, ]
head(m1)
head(m2) ## first 6
In general, we have the following concepts being applied in this example:
A re-interpretation of the NULL hypothesis away from \( \mu_1 = \mu_2 \) to something along the lines of “no effect.”
A sampling distribution that is generated by considering permutations of the data
A means to compute this sampling distribution. Here we could enumerate all the cases and tally up. For larger problems, we may need to approximate this distribution through simulation.
A computation of the \( p \)-value using its basic definition as the probability under the assumption of the null hypothesis of observing what we did or something more extreme.
The above is similar to the Mann-Whitney test, a non-parametric test of independent samples. (cf. http://http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U for some detail). This test is implemented in the The Mann-Whitney test can be described as follows. Imagine two
continuous distributions with the same shape, but possibly different
centers. (The shape need not be bell shaped, but the assumption is the
two are the same. Were the shape normal, the \( t \)-test would apply) If
the null is no effect, then this is the same as saying the two parent
populations are indeed the same. Then one takes both samples, combines
them, then ranks the values from least to smallest. The sum of the
ranks of one of the samples is the observed test statistic. Now, as
there is nothing distinguishing which rank goes with which sample the
distribution of this test statistic can be computed by considering all
possible rearrangements of the ranks that match the sample sizes. This
distribution can be computed as we did above, but under these
assumptions the asymptotic distribution has also been computed. The R
help page indicates this is used if there are more than 50 data
points.
wilcox.test function. What is the \( p \)-value found from the command: wilcox.test(weed_free, many_weeds, alt="greater")?
Read the tab labeled “More background” in the last problem. There you can see why there is an issue with data that is tied. (You should have seen R's warning about this in the last problem.) For the way we tested the data, would you need to issue such a warning?
The use of negative indexing is not typical in programming languages,
but is very convenient. There are some subtleties though. Which of
these commands gives an error when What is the value of
x <- 1:5:
-2:3?
In the weed/no weed problem, the sampling distribution of the test
statistics is discrete and computed when we found the 70 values. The
plot in the graphic tab shows the distribution. What is the best
reason that the graph is symmetric?
hist(out, main = "Distribution of permuations")
Click on the demo tab and you should find embedded a webpage from
http://lock5stat.com that allows us to simulate the sampling
distribution when we permute the values. In our example we computed
this by looking at all of the 70 possible permuations. For larger
problems, looking at all cases becomes problematic and one turns to
approximating the distribution by sampling from it. (This is
simulation.) That is we look at many random permuations to get a
sense of what the population looks like for all possible permutations. After 10 samples the distribution need not look symmetric. Does it look symmetric after 1000 samples? (It may not be exactly symmetric as we are looking at a summary of a random sample.) A video showing how to use
Statkey. The powerpoint referenced is
from http://stat.duke.edu/courses/Spring12/sta101.2/Feb13.pptx
Simulations Computing or approximating permutation distributions
The basic setup of a significance test involves several steps:
A choice of a null and alternative hypotheses
A choice of test statistic. In order for this to work out, this must have a known sampling distribution under the assumption that the null hypothesis is true.
A single observation of the test statistic
A computation of a \( p \)-value which is the probability under the null hypothesis that the test statistic will have a value as or more extreme than the observed value. Here “extreme” is interpreted as supporting the alternative hypothesis.
For the univariate \( t \)-test, we have a null along the lines of \( \mu=5 \)
and then if the population is normally distributed, then the test
statistic has the \( t \) distribution with \( n-1 \) degrees of freedom. The
\( p \)-values are computed using a table to look these up (or the pt
function).
For a permuatation test the sampling distributions are different. The basic null hypothesis is one of “no effect”. This assumption – in certain situations, many of which are described in the following – lead us to a description of the sampling distribution of the test statistic through a look at permutations or combinations of the data.
The example on corn yield allowed us to find the exact sampling distribution by considering all 70 possible combinations of planting 4 of 8 plots in a weed-free manner. With this distribution, finding a \( p \)-value is just a matter of using the distribution, the observed value and the alternative to identify what part of the distribution is used to compute the \( p \)-value.
Had we done 80 plots and planted 40, there would have been \( 10^{23} \) possibilities. This huge number would have precluded us from finding the exact distribution. Instead, we could approximate this distribution. This is done through simulation. A large number of values drawn from the distribution is taken, and based on that we get an idea of the actual distribution.
This might sound new, but really it isn't. The creation of values might be, but we have looked at sample data and made assumptions about whether a normal distribution makes sense or not. Same thing here, we make assumptions about the distribution from a sample – basically that our sample will be large enough the empirical distribution generated by the sample will be fairly close to the true one. With this assumption, then the approximate \( p \)-value is found by looking at the proportion of as-or-more-extreme cases there are in our large simulated sample. This is just using the empirical distribution like the regular sampling distribution. (The empirical distribution is just the one generated by the sample, mathematically \( F_n(x) = \#\{i | x_i \leq x\}/n \).)
A simulation results in data sampled from a population. In the context here, we want to infer something about the population. Statistical inference makes inference about parameters or single values. In finding an approximate \( p \)-value, we use the empirical distribution of the sample to compute the \( p \)-value for that. Other uses have us trying to understand the shape of a distribution or compare to a known distribution. For these questions, a graphical approach is often needed.
There are some statistical tests to compare a distribution to a target
distribution (cf. ?ks.test). We focus here on a graphical
approach. Given a sample from a population how can we compare it to a
target? We use some of the graphics introduced in weeks 1 and 2.
Let's look at some specific samples, one from a normal distribution, one from a \( t \)-distribution with a small number of degrees of freedom and one from a \( t \)-distribution with a large degrees of freedom.
res.norm <- rnorm(2000)
res.t.3 <- rt(2000, df = 3)
res.t.100 <- rt(2000, df = 100)
A natural question is to see how close to the theoretical distribution
is res.norm to the known one. We can compare with a histogram and
layered density plot:
hist(res.norm, probability = TRUE)
d <- density(res.norm)
lines(d$x, d$y, lwd = 2)
(Here we didn't need to adjust the y-limits. Had we need to, we would have computed both the histogram and density values before plotting, and used the computed information to set the y limits:
h <- hist(res.norm, plot = FALSE)
d <- density(res.norm)
ylim <- range(c(0, d$y, h$density))
hist(res.norm, probability = TRUE, ylim = ylim)
with(d, lines(x, y, lwd = 2))
)
The histogram is great for showing center spread and shape, and from this graph we see these match. The histogram for a large sample gives us confidence that the approximate distribution (the empirical distribution) can be a useful replacement for an unknown parent population.
More subtle questions like the length of the tails is not so obvious. The quantile plot does a better job here:
qqnorm(res.norm)
Here we see the values basically align on a line, as expected when the sample comes from the normal distribution.
Will the same be true for the \( t \)-distribution with a large degrees of freedom? A quick check:
qqnorm(res.t.100)
The above compares to a known distribution. Suppose we wanted to compare two samples to see if they come from a similar distribution. This of course is still possible. Comparing histograms is not so good, but we can easily compare with boxplots, density plots or quantile-quantile plots. We illustrate all 3.
The boxplot below shows similar centers and perhaps similar spreads, but completely different tail behavior between the normal and t samples when a small degrees of freedom is given.
boxplot(list(normal = res.norm, t.3 = res.t.3))
The two density plots show the normal and \( t \) for a large degrees of freedom are similarly shaped, though not in total agreement:
d1 <- density(res.norm)
d2 <- density(res.t.100)
xlim <- range(c(d1$x, d2$x))
ylim <- range(c(0, d1$y, d2$y))
plot(d1, xlim = xlim, ylim = ylim)
lines(d2, col = "red", lty = 2) ## red and dashed
The qqplot function compares two samples. Here we see the two \( t \)
samples are quite different, as the data does not fall on a line:
qqplot(res.t.3, res.t.100)
In the qqnorm graphic we compare a theoretical distribution to a
sample. We can do a similar comparison with densities. Make the
following graphic, then answer the question.
samp <- rt(300, df = 250 - 1)
dens <- density(samp)
curve(dnorm(x), from = -3, to = 3, ylim = c(0, max(c(dnorm(0), dens$y)))) ## curve is convenient
lines(density(samp), col = "blue")
Despite the randomness in the blue graph, the two densities show similar symmetry properties, centers, and tail behaviour.
Make the following set of boxplots (well, discuss sapply later):
n <- c(3, 10, 25, 50, 100, 250)
l <- sapply(n, function(i) rt(1000, df = i), simplify = FALSE)
names(l) <- sprintf("df_%s", n)
boxplot(l)
Which of the following seems to be stable for all the values of n?
Functions Using functions in simulating distributions
To simulate data in R we need to learn two things:
how to simulate one value from the sampling distribution
how to repeat something many times to get a large sample from the sampling distribution
For the former, we need to learn about the “r” part of the “d”, “p”, “q” and “r” functions and learn some basics about functions. For the latter, we see that R has many ways to iterate over a collection of items.
The “r” functions in R allow us to generate random numbers from a
named (parameterized) distribution. The sample function allows us to
randomly sample (with or without replacement) from a set of values
with specified probabilities. Both are useful in simulation.
Let's look at a simple case, simulating the \( t \)-distribution with \( 4 \) degrees of freedom. We should be able to do this by taking a random sample of size 5 from a normal population and forming the t-statistic.
The rnorm function is an “r” function (where “r” is appended to R's
name for the normal distribution) to generate random samples from the normal
distribution. The normal distribution has two parameters: \( \mu \), with
default of 0, and \( \sigma \), with default of 1. To get a sample of size
5 from the standard normal is done with:
rnorm(5)
## [1] 0.19556 0.14765 0.03357 0.42664 -2.19635
To get a sample of size 5 from a normal distribution with mean 68 and standard deviation 2.5 one would have either:
rnorm(5, mean = 68, sd = 2.5)
## [1] 72.58 68.37 70.49 71.45 65.09
rnorm(5, 68, 2.5)
## [1] 69.24 72.46 68.36 66.97 68.22
Each produces 5 random numbers from the distribution. The first uses named arguments the second positional arguments. Positional arguments are easier to type, harder to read. (RStudio makes using positional arguments easier, but I prefer naming all but the first argument for clarity.)
Then a single sample from the \( t \) distribution with 4 degrees of freedom can be found with:
x <- rnorm(5)
(mean(x) - 0)/(sd(x)/sqrt(length(x)))
## [1] 0.4012
(Why did I subtract 0 above?)
In this common case there is a built-in function, rt, to do this (as
with rt(1, df=4)), so why bother with the extra fuss? Well, we want
to think about this in general: our description of the sampling
distribution of a test statistic depends on a population assumption
(normal) and the ability to compute the test statistic under the null
(\( \mu=0 \) above). R can make this explicit calculation fairly
straightforward.
In the above example we computed the \( t \)-statistic by hand with:
(mean(x) - 0)/(sd(x)/sqrt(length(x)))
## [1] 0.4012
We could also have used a built-in function to do this, had it been difficult:
t.test(x)$statistic
## t
## 0.4012
However, developing our R muscles to be able to define our own
functions is really useful. This function should be able to take the
data, x, possibly another parameter (the assumed value of \( \mu \)
under the null) and return the value of the \( t \)-statistic. The math
part is already done, it is just a matter of putting into R's syntax
for functions.
Functions have three main things:
their inputs, or arguments
their calculations as to what they do to the inputs
their value that they return.
In R, these are the basic three issues (though R also has a subtle fourth which is the environment where values within the function are found, but that is a more advanced matter).
A function template looks something like:
function_name <- function(arguments) {
function_body ...
return_value
}
The above assigns the function object to a name so that it can be used latter. (One can define one-off anonymous functions without a name, but let's think about that later). The arguments parameterize a function. For our example we want two: one for the data and one for the value of \( \mu \). Here is how such a function might look:
tstat <- function(x, mu = 0) {
out <- (mean(x) - mu)/(sd(x)/sqrt(length(x)))
return(out)
}
We have:
The arguments have positions: x is first, mu is second.
The arguments have names: x and mu
The second argument has a default value: that is, if not
specified, it will be taken as 0. This is indicated by the =0 part
of the definition.
The body uses x and mu to define a value out to return
we explicitly used return to return from the function. This was
unnecessary as the last value evaluated in a function body will be
returned otherwise, but explicitness is not such a bad thing.
To use this function we can do any of these:
x <- rnorm(5)
tstat(x) ## using mu-0
## [1] 0.5403
tstat(x, 0) ## specifying mu by position
## [1] 0.5403
tstat(x, mu = 0) ## specifying my by name
## [1] 0.5403
While it is most convenient from a typing standpoint to avoid using named arguments, for pedagogical reasons I almost always add them except for the first argument.
Here are some sample functions for illustration:
A function to compute the mean of a data set is already done for you
with mean. Here is a simple implementation:
our_mean <- function(x) sum(x)/length(x)
## see it work
x <- c(1, 3, 5, 7, 4)
our_mean(x) ## call like any function
## [1] 4
The our_mean function uses the convention that the last evaluated
expression is returned. Since this is a “one liner” we don't need
braces to separate off multiple commands. We could have, but they
aren't required.
The real mean function has an argument na.rm indicating if you
should remove any NA values before taking the mean. (If you don't
then any expression with NA will return NA, as R doesn't assume it
can make calculations when the data is not available.) We could
implement as follows:
our_mean <- function(x, na.rm = FALSE) {
if (na.rm) {
x <- x[!is.na(x)]
}
sum(x)/length(x)
}
There we use the flow control if expression to do a conditional
removal of NA values. We don't dwell on this point, but were we to
spend much more time talking about functions we would have to come to
grips with if and else constructs.
To see our function in action, we have
x <- c(1, 1, NA, 2, 3, 5)
our_mean(x, na.rm = TRUE)
## [1] 2.4
our_mean(x)
## [1] NA
A video on how to define a function for the sample skew in the code editor of RStudio:
The kurtosis of a distribution is a description of peakedness of the distribution, as compared to a normal distribution. The definition below by Pearson, actually measures longer tails.
We use this definition of the kurtosis: \[ \frac{(1/n)\sum(x_i - \bar{x})^4}{((1/n)\sum(x_i-\bar{x})^2)^2} \quad - \quad 3 \]
(The \( -3 \) makes the normal distribution have 0 kurtosis on average and as the wikipedia page notes, makes the kurtosis of a sum of a random sample depend easily on the kurtosis of a single value.)
An R function to compute the above kurtosis is:
kurtosis <- function(x, na.rm = FALSE) {
if (na.rm)
x <- x[!is.na(x)] ## clear our NA values if asked
n <- length(x)
center <- x - mean(x)
top <- (1/n) * sum(center^4)
bottom <- ((1/n) * sum(center^2))^2
top/bottom - 3
}
For sample data, let's see if there is a difference between the \( t \)-distribution with a small degrees of freedom and a large one (as the degrees of freedom gets larger, the \( t \)-distribution looks more and more like the standard normal which has 0 kurtosis.
kurtosis(rt(1000, df = 3))
## [1] 8.845
kurtosis(rt(1000, df = 300))
## [1] 0.1143
Comparing just one sample against another is generally not possible, unless the difference are as extreme as above. Later we will simulate the distributions by taking many such samples and compare these.
The value passed into a function need not be a data vector or a simple numeric parameter. The following function expects a matrix and computes the row sums of the matrix then applies a function. This function can be passed in as well by the user:
f <- function(x, FUN = mean) {
y <- rowSums(x)
FUN(y)
}
To use this we define a simple matrix with rbind and do a few calculations:
x <- rbind(c(1, 2, 3, 4), c(3, 2, 1, 4), c(5, 3, 2, 4))
x ## 3 by 4
## [,1] [,2] [,3] [,4]
## [1,] 1 2 3 4
## [2,] 3 2 1 4
## [3,] 5 3 2 4
f(x) ## find the mean of the row sums
## [1] 11.33
f(x, var) ## find the variance of the row sums
## [1] 5.333
This function could be used later (though we don't for clarity of exposition). Here we give it to illustrate something about R that is not true for all programming languages: functions are “first class” objects and can be passed as arguments to other functions. In many of the iterating styles that R has available this will be exploited.
When browsing the source of many R functions you will see a mysterious
argument: ... . This is R's way to pass along a variable number of
arguments. Often these are just passed to an internal function
call. The plot functions are good examples of this style.
A classic simulation example is to see how robust the \( t \) statistic is
to deviations from the population assumption of normality. Here is the
\( t \)-statistic where the parent population is passed in as an “r”
function and any parameters specified via ...:
sim_tstat <- function(n, POP = rnorm, mu = 0, ...) {
x <- POP(n, ...)
(mean(x) - mu)/(sd(x)/sqrt(length(x)))
}
And we can use it as follows:
sim_tstat(5, rnorm, mu = 5, mean = 5, sd = 10) ## normal
## [1] -0.5734
sim_tstat(5, rt, df = 4) ## long tailed
## [1] -0.05338
sim_tstat(5, rexp, mu = 2, rate = 1/2)
## [1] -0.9504
Of course there is a lot more to say about functions. We gave a brief
hint of conditional flow control, but could have said much more. We
could have talked about using invisible to return values. We could
have discussed how R finds the values of variables within a function
(for values not passed in, but used within a function). We could have
talked about lazy evaluation. We could have talked about
closures. Well, lots to talk about. If you are interested, you migh
find https://github.com/hadley/devtools/wiki/functions a good
read. But we are here to talk about simulation and functions are just
a piece of that, so let's move on.
Write a function that computes the mean of a random sample from the normal distribution. Your function should take You can get fancy and pass in arguments to parameterize the normal and
give a default value to
n as the size of the sample.
Okay, don't peek here until you have tried. One way is:
f <- function(n) {
x <- rnorm(n)
mean(x)
}
n:f <- function(n = 10, mean = 0, sd = 1) {
x <- rnorm(n, mean = mean, sd = sd)
mean(x)
}
Here the function can return from 3 points. The last expression
evaluated is returned and due to the braces and if-else functions,
this will be -1, 0, or 1. I personally find using a There is much to critique about the “R”-ness of the this function, but
the point is to illustrate some basic control flow using if-else and
in fact using it twice. Try to mimic the above to write an absolute value function, that is if
the argument is negative return its negative, else return the value.
The most basic control-flow statement is the if-else statement. Here we show how one can use it to define the sign function which is -1, 0 or 1 depending on the value of x:
sign_fun <- function(x) {
if (x < 0) {
-1
} else if (x == 0) {
0
} else {
1
}
}
return call
makes this easier to read after the fact.
A simple answer would be:
abs_fun <- function(x) {
if (x < 0) {
-x
} else {
x
}
}
Write a function which plots both the histogram and density at the same time. Here is a go where we are careful to make sure the y-limits are such to accomodate both.
hist_dens <- function(x, ...) {
dens <- density(x)
hst <- histogram(x, plot = FALSE)
ylim <- range(c(dens$y, hst$density)) ## key part to fit, but o/w not needed
histogram(x, probability = TRUE)
lines(dens)
}
Iterations Lather, rinse, repeat, repeat, repeat, …
The sim_tstat function computes 1 sample value from some
distribution. When the POP is normal (rnorm) then we know it has
the \( t \)-distribution with \( n-1 \) degrees of freedom, but don't know
this when POP is rt with some degrees of freedom, say. We can
investigate what it is by taking many such samples and looking at
those to gather insight into the population.
The key being: repeat some process lots of times. The question: How? We'll see there are many answers to how.
The simplest, and likely most familiar, way to repeat something in R
is with a for loop. (See chapter 6.2 in the “UsingR” book for some detail.)
The use is pretty simple:
Here we go:
res <- numeric(0) ## or pre-allocate with numeric(2000)
for (i in 1:2000) {
res[i] <- sim_tstat(5, rt, df = 3)
}
After this, res has 2000 values each a random sample from some
unknown distribution.
Okay, so what did we do? The line res <- numeric(0) makes a numeric
vector of 0-length to be filled in during each loop. For better
performance, we could have pre-allocated the storage of the values
using numeric(2000), rather than have the vector grow each time. The
i in the index takes all the values in the vector 1:2000. For each,
it assigns the \( i \) th component of res to be a single realization
using sim_tstat.
Does res appear to be sampled from the \( t \) distribution with \( 4=5-1 \)
degrees of freedom? A common graph to check is the “quantile-quantile”
graph. Here we plot our sample data against a sample of similar size
from the \( t \)-distribution:
qqplot(res, rt(2000, df = 5 - 1))
Were the graphic more or less on a straight line, we'd conclude the parent populations are similar. Not so in this case.
Another example, might be to see which has smaller variability, the mean or the median. For normal distributions this is something we could compute, but let's simulate instead. You'll do the exponential distribution as a population.
res.norm.mean <- res.norm.median <- numeric(0)
for (i in 1:2000) {
x <- rnorm(10)
res.norm.mean[i] <- mean(x)
res.norm.median[i] <- median(x)
}
boxplot(list(median = res.norm.median, mean = res.norm.mean))
The median is more variable for a normal population.
Using for loops is fairly easy to understand and works just fine. So
why talk any more? Well, some R users find there are more convenient
ways to think about iterating over a collection of items. And R has
spawed a gazillion ways (well a lot anyways) to do this task.
Here are some R idioms for the task of repeating something many times:
for loops, as just illustrated
vectorizing a function with Vectorize or using replicate
applying the function to the objects using one of apply, sapply,
lapply and many other such functions
mapping the function to the values with Map.
Of these, the “apply” family of functions is the most useful to know,
though Map is maybe a more familiar concept throughout computer
programming.
What is the idea of “applying” a function to some collection. Basically we want to call the function for each item in the collection. We use the terms “item” and “collection” to be abstract. To make concrete here are some typical “collections” and their “items”:
A vector as a collection has items which are its components
A data frame as a collection has items which are the variables (columns)
A list as a collection has items which are the components. (A list is a generalized vector, a data frame a specialized list.)
a matrix as a collection has items which are either the row vectors or the column vectors. (Arrays are more general matrices and have dimensions to apply values along.)
As there are many different ideas, there are many different ways to
“apply” a function. The plyr package tries to bring some consistency
to the game, but we talk here of the solutions found in base R. The
apply function is related to the last case, matrices. It has an
extra argument to specify which dimension to apply the function
to. For matrices, 1 is for rows and 2 for columns.
In an earlier example we used the rowSums function to, well, sum the
rows of a matrix. We could have done this with apply:
x <- rbind(c(1, 2, 3, 4), c(3, 4, 5, 6), c(2, 5, 7, 8))
x ## see the values
## [,1] [,2] [,3] [,4]
## [1,] 1 2 3 4
## [2,] 3 4 5 6
## [3,] 2 5 7 8
apply(x, 1, sum)
## [1] 10 18 22
The 1 is the dimension to call apply on. Basically it takes the
matrix x and applies the sum function to x[1,], x[2,] and
finally x[3,]. What is nice is we didn't have to do any bookkeeping:
no worries about defining some accumulator like res in our previous
examples, no worries about counting how many rows before looping.
We could have done column sums with
apply(x, 2, sum)
## [1] 6 11 15 18
The only difference is the dimension to reduce around. (Of course
rowSums and colSums are not redundant. They are much faster than
the above for large matrices.)
The apply function is pretty straightforward, but the simplest case
above can be extended. Consider the following slight modification:
x[1, 1] <- NA
apply(x, 1, sum)
## [1] NA 18 22
Well, if we really wanted the first row summed by ignoring the NA
value we could call sum(x[1,], na.rm=TRUE). So how to pass arguments
to sum? No problem, just do it after the function name:
apply(x, 1, sum, na.rm = TRUE)
## [1] 9 18 22
The apply function uses ... to pass arguments on to the function
(as do the other “apply”-like functions).
Now, for vectors the apply function doesn't really make sense, as we
don't reduce along some dimension. Rather we just iterate over each
item. The simplest function to do this task is sapply. Not only does
it iterate over the items, it also tries to simplify, or combine, the
output and tries to preserve the names attribute of x, if present.
Here is how we could do the simulation of 2000 samples, which we did
above with a for loop:
res <- sapply(1:2000, function(i) sim_tstat(5, rt, df = 3))
Here we used an anonymous function of the index variable i to call
the sim_tstat function lots of times. This is a tad kludgy – we
don't actually use the i value, but not too bad.
For this special case where the value we iterate over isn't used by
the function we can use the replicate function. The above could have
simply been:
res <- replicate(2000, sim_tstat(5, rt, df = 3))
Though sapply makes this task longer, I don't really encourage you
to commit replicate to memory. The sapply style has much more
potential use, whereas replicate is pretty limited.
The sapply idiom works cleanest when the function you call is to be
called on each item in the collection. A perfect example is how to
find the mean or median of a data frame column by column. For a data
frame, the items are the columns, so this will do it:
m <- mtcars[, 1:4] ## shorten the problem
sapply(m, mean)
## mpg cyl disp hp
## 20.091 6.188 230.722 146.688
sapply(m, median)
## mpg cyl disp hp
## 19.2 6.0 196.3 123.0
As you can see, the names attributes are maintained and the function
is applied to each variable (column of the data frame).
Compare doing the above with a for loop:
res <- numeric()
for (j in 1:ncol(m)) {
res[j] <- mean(m[,j])
}
names(res) <- names(m)
I hope you agree that sapply(m, mean) is much clearer.
Some of you may be familiar with the “Map” construct. R is naturally
vectorized in many cases, but other programming languages aren't. Many
have a “Map” function to apply the function to each item of a
collection. R does too. Here is how it can be used. (Unlike sapply
where the collection goes first, Map has the function first.)
Map(mean, m)
## $mpg
## [1] 20.09
##
## $cyl
## [1] 6.188
##
## $disp
## [1] 230.7
##
## $hp
## [1] 146.7
A few comments. First, the output is not simplified the way sapply
does. This is a design choice. The sapply function has a relative lapply
which is basically sapply without the conversion, returning a
list. Second, unlike sapply, Map can apply functions with multiple
arguments. (That is, we could have done Map(mean, m, na.rm=TRUE).)
Okay, with all this we can get serious about simulation. A common way to simulate is to create a matrix of random numbers (all of them at once is faster) then apply a function. This approach can be much faster that looping. Here we create a matrix with 2000 rows and 5 columns of normal random numbers:
nr <- 2000
nc <- 5
m <- matrix(rexp(nr * nc), nrow = nr)
And here is how we can compute the \( t \)-statistic for each, knowing the mean is 1:
tstat <- function(x, mu = 0) (mean(x) - mu)/(sd(x)/sqrt(length(x)))
res <- apply(m, 1, tstat, mu = 1)
qqplot(res, rt(2000, df = nc - 1))
Clearly from the qqplot, the two distributions (the simulated \( t \) with different population) and the \( t \)-distribution are quite different.
The Central Limit Theorem is responsible for most all of the classical
statistics framework. Basically it says that for most cases, the mean
of a random sample will have a sampling distribution that gets closer
and closer to normal as the sample size increases. How big a sample is needed to be close to normal depends on the
population. If the population is normal, then the sample can be of
size 1. What about when the population is uniform? Let's see. Unlike the \( t \)-statistic where we need only know the mean, here we
need to know both the mean and standard deviation. For the uniform
normal these are ½ and standard deviation \( 1/sqrt{12} \). Here are some commands: Verify that for the most part these density plots are basically the same. (One should add colors and it isn't hard, but I passed.) Now, let's see if the same holds true when we have a skewed
distribution, the exponential distribution with mean 1 and standard
deviation 1. Repeat the above modifying the call to You should see the peak shifting over (though the mean is the same!). This is due to the sampling distribution of the statistic becoming more normal, and less skewed. To see if \( n=50 \) is large enough to consider the distribution approximately normal, you can compare with
zstat <- function(x, mu, sd) (mean(x) - mu)/(sd/sqrt(length(x)))
n <- c(2, 5, 10, 15, 20, 30, 50)
out <- lapply(n, function(i) {
replicate(3000, zstat(runif(i), 1/2, 1/sqrt(12)))
})
names(out) <- paste("Size", n)
## visualize with density plots
plot(density(out[[1]]))
sapply(out[-1], function(i) lines(density(i)))
zstat just
slightly and comment on the resulting graphic.
qqnorm.
The \( t \) distribution with \( n-1 \) degrees of freedom was historically computed as the sampling disribution of the scaled mean of a random sample of \( n \) normally distributed numbers. The scaling is done by the standard error, not the standard deviation. It is conventional wisdom that the statistic is resistant to changes
in the underlying assumption of a normal population. We can
check. Here is how when the population is a uniform: Not too bad. This agrees with conventional wisdom that the \( t \) is a
reasonable choice for short-tailed data. Now repeat with No, notice the pronounced curve in the
tstat <- function(x, mu = 0) (mean(x) - mu)/(sd(x)/sqrt(length(x)))
n <- 10
res <- replicate(1000, tstat(runif(n), mu = 1/2))
## compare
qqplot(res, rt(1000, df = n - 1))
rexp(n) in
place of runif and mu=1. This is skewed data. Do you get a similar result?
qqplot:res <- replicate(1000, tstat(rexp(n), mu = 1))
## compare
qqplot(res, rt(1000, df = n - 1))
This is a similar question to the previous one. The chi square distribution describes the chi-squared statistic
asymptotically. The main R function has a disclaimer if any of the
expected numbers are less than 5. Let's compare the simulated
distribution with the chi-squared one. Here are some commands to modify. If you run these, you have each
value of Ours doesn't show this, notice the widening as the statistic gets bigger.
n*p is greater than 5, so the approximation should be
pretty good. Verify that it is. Then change n <- 5 and rerun. Is the
qqplot still showing agreement?p <- c(0.1, 0.1, 0.6, 0.1, 0.1)
n <- 50
B <- 3000
res <- numeric(B)
for (i in 1:B) {
x <- sample(factor(letters[1:5]), n, replace = TRUE, prob = p)
out <- chisq.test(table(x))
res[i] <- out$statistic
}
qqplot(res, rchisq(B, n - 1))
p <- c(0.1, 0.1, 0.6, 0.1, 0.1)
n <- 5
B <- 3000
res <- numeric(B)
for (i in 1:B) {
x <- sample(factor(letters[1:5]), n, replace = TRUE, prob = p)
out <- chisq.test(table(x))
res[i] <- out$statistic
}
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
## Warning: Chi-squared approximation may be incorrect
qqplot(res, rchisq(B, n - 1))
Permutation distributions
Finally, with these R skills discussed to the point where they are very useful, we now illustrate several applications where permutation tests may be employed. The point here isn't to expect you to understand the details of any one of them, but to see the variety of potential applications.
This example comes from the Ernst article, also reworked in the Cobb article. We slightly reword it.
Suppose that a new treatment for postsurgical recovery is being compared to a standard treatment by observing the recovery times (in days) of the patients on each treatment. Of the N subjects available for the study, n are randomly assigned to receive the new treatment, while the remaining m = N − n receive the standard treatment. The null and alternative hypotheses of interest are
H0: There is no difference between the treatments,
H1: The new treatment decreases recovery times.
Denote the recovery times for the standard and new treatments
by X1, X2, … , Xm and Y1, Y2, … , Yn, respectively. To measure
the difference between the treatments, we might calculate the
difference in mean recovery times between the two groups, T = mean(Y)
− mean(X). We wish to determine if this difference is extreme enough
in some reference distribution to suggest that the new treatment
decreases recovery times.
If the null hypothesis is true and there is no difference between the treatments, then the recovery time for each subject will be the same regardless of which treatment is received.
Let the data be:
New Treatment: 19, 22, 25, 26 Standard Treatment: 23, 33, 40
For example, the subject who recovered in 19 days on the new treatment would have recovered in the same amount of time on the standard treatment if there is no treatment effect. So the recovery times are not random (because the subjects were not chosen randomly); only their assignment to the treatments is random.
Therefore, the basis for building a probability distribution for
mean(Y) − mean(X) comes from the randomization of the available
subjects to the treatments. This randomization results in n subjects
getting the new treatment and m subjects getting the standard
treatment, but this is just one of N choose n equally likely
randomizations that could have occurred. A probability distribution
for mean(Y) - mean(X) is called the randomization distribution,
and can be constructed by calculating mean(Y) − mean(X) for each of
the possible randomizations.
The probability under H0 of any one of these randomizations is 1/35.
Okay, this is like the initial example. To compute the left-tailed
\( p \)-value we can explicity calculate the sampling distribution of
T and look at those values less or equal to the observed value:
new_t <- c(19, 22, 25, 26)
std_t <- c(23, 33, 40)
all_data <- c(new_t, std_t)
cmb <- combn(7, 4)
f <- function(i) mean(all_data[i]) - mean(all_data[-i])
T <- apply(cmb, 2, f)
obs <- mean(new_t) - mean(std_t)
p_val <- sum(T <= obs)/length(T) ## left tailed test
p_val
## [1] 0.08571
The small \( p \)-value is strong evidence against the null hypothesis of no effect (that the new treatment is no better than the standard).
As pointed out in Ernst, an alternate test statistic (alternate to the
difference of means) would be the difference of sum of the ranks. This
is the Wilcoxen rank-sum test implemented in R with
wilcox.test. In this case, the test statistic can be found using R's rank
function. We sum the ranks for each combination and take their
difference:
ranked_data <- rank(all_data)
f <- function(i) sum(ranked_data[i]) - sum(ranked_data[-i])
T <- apply(cmb, 2, f)
obs <- sum(ranked_data[1:4]) - sum(ranked_data[5:7])
p_val <- sum(T <= obs)/length(T)
p_val
## [1] 0.1143
The wilcox.test function is called just like t.test. Does it
produce a different \( p \)-value than just computed?
Imagine a scenario where we can make a measurement two different ways, a cheaper way and a more expensive way. The cheaper way is calibrated so that its mean response is the same as the more expensive way. However, we may want to know if the cheaper way has more variability – generally not a better thing. If we assume it does not, then can we make a significance test?
Let's be more concrete. We take a data set from one of R's examples.
## Hollander & Wolfe (1973, p. 86f): Serum iron determination using Hyland
## control sera
ramsay <- c(111, 107, 100, 99, 102, 106, 109, 108, 104, 99, 101, 96, 97, 102,
107, 113, 116, 113, 110, 98)
jung.parekh <- c(107, 108, 106, 98, 105, 103, 110, 105, 104, 100, 96, 108, 103,
104, 114, 114, 113, 108, 106, 99)
The assumption is that these two data sets are random samples from some parent population that differs not in the location parameter, but the scale parameter. Let \( s \) be the ratio of the scales. (This would be like testing samples from a normal with \( \mu_1 \) and \( \sigma_1 \) and \( \mu_2=\mu_1 \) and \( \sigma_2 \) with \( s = \sigma_1/\sigma_2 \), but the parent population need not be normal.)
With this hypothesis, the null of \( s=1 \) makes the two data sets indistinguishable. So one could figure out a distribution of a test statistic under permuations. Historically (http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoms/1177705688) a test statistic involving the ranks of the first sample has been used. We can replicate that here.
rank_stat <- function(x, y) {
m <- length(x)
n <- length(y)
N <- m + n
r <- rank(c(x, y))
STATISTIC <- sum(pmin(r, N - r + 1)[seq_along(x)])
STATISTIC
}
rank_stat(ramsay, jung.parekh)
## [1] 185.5
The pmin bit takes the difference of the rank from 0 or \( N + 1 \) so
large or small ranks are treated symmetrically.
The question is 185.5 an unusual rank sum? As before, we can
simulate. Let's take 3000 replications and use a for loop.
all_data <- c(ramsay, jung.parekh)
res <- numeric(3000)
for (i in 1:3000) {
x <- sample(all_data) ## a permuation
res[i] <- rank_stat(x[1:20], x[21:40])
}
We look at the value for \( X \) being small
sum(res <= rank_stat(ramsay, jung.parekh))/length(res)
## [1] 0.1013
Small, but not significant.
This test is the “Ansari-Bradley” test. Here we see a similar computation:
ansari.test(ramsay, jung.parekh, alt = "greater")
## Warning: cannot compute exact p-value with ties
##
## Ansari-Bradley test
##
## data: ramsay and jung.parekh
## AB = 185.5, p-value = 0.09073
## alternative hypothesis: true ratio of scales is greater than 1
Consider the question of whether two variables, \( x \) and \( y \), are correlated. If we try a significance test where the null hypothesis is there is no correlation and the alternative is that there is a correlation, then under the null hypothesis the correlation of \( x \) and \( y \) should not depend on the order of \( x \). (The order of \( x \) preserves the pairing between measurements, if there is no correlation then there is no real pairing.)
With this insight, a significance test can be formed. Let's look at an
example using the heartrate dataset measuring maximum heart rate by
age.
Our observed correlation is
require(UsingR)
## Loading required package: UsingR
## Loading required package: MASS
## Loading required package: ggplot2
## Loading required package: Hmisc
## Loading required package: survival
## Loading required package: splines
## Hmisc library by Frank E Harrell Jr
##
## Type library(help='Hmisc'), ?Overview, or ?Hmisc.Overview') to see overall
## documentation.
##
## NOTE:Hmisc no longer redefines [.factor to drop unused levels when
## subsetting. To get the old behavior of Hmisc type dropUnusedLevels().
## Attaching package: 'Hmisc'
## The following object is masked from 'package:survival':
##
## untangle.specials
## The following object is masked from 'package:questionr':
##
## label
## The following object is masked from 'package:base':
##
## format.pval, round.POSIXt, trunc.POSIXt, units
## Loading required package: coin
## Loading required package: mvtnorm
## Loading required package: modeltools
## Loading required package: stats4
## Loading required package: quantreg
## Loading required package: SparseM
## Package SparseM (0.99) loaded. To cite, see citation("SparseM")
## Attaching package: 'SparseM'
## The following object is masked from 'package:base':
##
## backsolve
## Attaching package: 'quantreg'
## The following object is masked from 'package:Hmisc':
##
## latex, untangle.specials
## The following object is masked from 'package:survival':
##
## untangle.specials
## Loading required package: HistData
## Loading required package: LearnEDA
## Loading required package: aplpack
## Loading required package: tcltk
## Loading required package: vcd
## Loading required package: grid
## Loading required package: colorspace
## Attaching package: 'LearnEDA'
## The following object is masked from 'package:MASS':
##
## farms
## Attaching package: 'UsingR'
## The following object is masked from 'package:survival':
##
## cancer
## The following object is masked from 'package:ggplot2':
##
## movies
obs <- with(heartrate, cor(age, maxrate))
obs
## [1] -0.9535
There are 15 measurements, so 15! or 1.307674e+12 many different
permuations. Clearly this is too many to try by hand, so we
simulate. That is, let's see how to take 3,000 different random
permutations and look at the 3000 correlations corresponding to
each. Here is how we do one:
ind <- sample(1:15)
with(heartrate, cor(age[ind], maxrate))
## [1] 0.1401
To get 3000 requires us to iterate and store the values:
out <- sapply(1:3000, function(i) {
ind <- sample(1:15)
with(heartrate, cor(age[ind], maxrate))
})
A histogram shows the distribution we found (if you repeat this, you will get a slightly different – but similar pattern):
hist(out)
abline(v = obs, col = "blue", lty = 2)
The \( p \)-value is found by using both tails below. The observed correlation of \( -0.95 \) is clearly significant.
sum(out >= abs(obs) | out <= -abs(obs))/length(obs)
## [1] 0
We finish our illustration of various applications with a study that is similar to the chi-squared test introduced in week 2 where we tested for homogeneity amongst rows or columns in a contingency table.
We again follow an example of Eudey, this time the problem involves a hypothetical rating of 3 recipes by 6 judges. Each judge gives a score to each brand. Our data is:
scores <- rbind(c(7, 15, 20, 16, 15, 18), c(16, 6, 4, 10, 13, 11), c(8, 0, 0,
9, 5, 6))
dimnames(scores) <- list(paste("Brand", LETTERS[1:3]), paste("Judge", 1:6))
scores
## Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 6
## Brand A 7 15 20 16 15 18
## Brand B 16 6 4 10 13 11
## Brand C 8 0 0 9 5 6
While we are here, we generate the ranks for each brand:
ranks <- apply(scores, 2, function(x) 4 - rank(x))
rownames(ranks) <- paste("Brand", LETTERS[1:3])
ranks
## Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 6
## Brand A 3 1 1 1 1 1
## Brand B 1 2 2 2 2 2
## Brand C 2 3 3 3 3 3
And the most favorite:
best <- apply(ranks, 2, function(x) as.numeric(x == 1))
rownames(best) <- paste("Brand", LETTERS[1:3])
best
## Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 6
## Brand A 0 1 1 1 1 1
## Brand B 1 0 0 0 0 0
## Brand C 0 0 0 0 0 0
While scores has the most data, it is easy to envision a scenario
where one only records the best or the ranks by themselves.
Okay, the natural question here is are the brands the same (or different)? The null would be that the brands are all the same (meaning equally likely to be top rated/ranked/scored). The alternative would be that this is not true.
What might be a reasonable test statistic? One that measures when
there are differences from the null. Well, the null is very similar to
what is done in a chi-squared test of similarity of
distributions. There the test statistic is related to \( \sum(observed -
expected)^2/expected \). We could calculate this directly, or instead
use the output of chisq.test. Here is how we can compute, after putting
aside any warnings:
f <- function(x) {
observed <- rowSums(x)
expected <- 2 ## 6 * 1/3
sum((observed - expected)^2/expected)
}
f(best)
## [1] 7
We get 7, but that doesn't answer our question. Rather we want to know – given the null how likely is it to be this extreme or more?
Let's look at the best values. If there is no difference in the
brand, then relabeling them would have no effect in the distribution
under the null. So we should be able to see how extreme our value is
by considering all reorderings of the brands.
The following will apply sample to each column of the best
matrix. Note what sample does on a vector of the form c(1,0,0):
tmp <- c(1, 0, 0)
sample(tmp) ## permutes the values
## [1] 0 0 1
sample(tmp)
## [1] 1 0 0
sample(tmp)
## [1] 0 0 1
That is it permutes the values, exactly what we want to do. A simulation where each judge has their brands switched around can be done with:
apply(best, 2, sample)
## Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 6
## [1,] 1 0 1 0 0 0
## [2,] 0 1 0 0 0 1
## [3,] 0 0 0 1 1 0
And we can repeat this 3000 times, say, with:
out <- sapply(1:3000, function(i) {
x <- apply(best, 2, sample)
f(x)
})
table(out)
## out
## 0 1 3 4 7 12
## 388 1500 618 358 127 9
We see not many are 7 or more. Precisely, we have in our simulation the approximate p-value:
sum(out >= 7)/length(out)
## [1] 0.04533
Turning to the ranks, we can do something similar. A statistic to measure differences might be one measuring the variability of the row sums. (The total of all the ranks for a brand.) If the null is true, we would expect little variability. If not true, then this would be big.
To see we use the var function:
obs <- var(rowSums(ranks))
To do a simulation, we have:
out <- sapply(1:3000, function(i) {
x <- apply(ranks, 2, sample)
var(rowSums(x))
})
The histogram shows the approximate sampling distribution:
hist(out)
abline(v = obs)
And the approximate \( p \)-value is small:
sum(out >= obs)/length(out)
## [1] 0.02967
friedman.test
Finally, we might consider the raw scores. A common critque of the
variance is its sensitivity to values far from the center. The MAD, or
median absolute deviation, also measures variability but using the
more robust median. In R, the mad function implements this. A
measure of variability per brand could then be:
f <- function(x) {
mad(apply(x, 1, median))
}
obs <- f(scores)
obs
## [1] 7.413
Then just as before, we can simulate 3000 times with:
out <- sapply(1:3000, function(i) {
x <- apply(scores, 2, sample)
f(x)
})
The distribution of values is approximated by the following:
x <- table(out)
plot(as.numeric(names(x)), x, type = "h", main = "Distribution of MAD scores")
Our \( p \)-value is then:
sum(out >= obs)/length(out)
## [1] 0.005333
Basically negligible. Again, the data is statistically significant.
Coin package
As mentioned in the introduction, there is a body of theory that
unifies these seemingly different tests. The add-on coin package
brings that theory to the R user. The details of how this is done are
beyond the scope of these notes, but if you are interested, please
read through the vignette accompanying the package. It has some detail
and references for more.
For this discussion, I just wish to point out some of the functionality provided by that package that overlaps with the computerized attempts just done.
The coin package provides functions oneway_test, median_test and
kruskal_test which give various tests for location. The two-sample
\( t \) test is also a test of location – do two normal populations have
the same location parameter. Similarly ANOVA (implemented in R's
oneway.test amongst others) is a location test. Base R has many of
these tests defined, however, the coin package provides indepenpent
implementations using the developed framework and often covers cases
(such as when there are ties in the data) that are not handled by R's
defaults.
As discussed above, one can test as assumption of equal scales
assuming similar parent populations with equal centers, but possibly
different scales. The coin function ansari_test will compare two
samples, whereas fligner_test is used for multiple tests.
In a problem from week 2 we had measurements that we compared with a \( t \)-test. Repeat the same analysis (a two-side test of equivalence of centers)
using Use
bottom <- c(0.43, 0.266, 0.567, 0.531, 0.707, 0.716)
surface <- c(0.415, 0.238, 0.39, 0.41, 0.605, 0.609)
wilcox.test. The \( p \)-value is:
wilcox.test as in wilcox.test(x,y).
boxplot(list(bottom = bottom, surface = surface))
The coin functions differ by an underscore (ansari.test or
ansari_test), do they do much more?
Compare the outputs of these commands:
ramsay <- c(111, 107, 100, 99, 102, 106, 109, 108, 104, 99, 101, 96, 97, 102,
107, 113, 116, 113, 110, 98)
jung.parekh <- c(107, 108, 106, 98, 105, 103, 110, 105, 104, 100, 96, 108, 103,
104, 114, 114, 113, 108, 106, 99)
d <- stack(list(ramsay = ramsay, jung = jung.parekh))
require(coin)
ansari.test(values ~ ind, data = d, alt = "less")
## Warning: cannot compute exact p-value with ties
##
## Ansari-Bradley test
##
## data: values by ind
## AB = 234.5, p-value = 0.09073
## alternative hypothesis: true ratio of scales is less than 1
ansari_test(values ~ ind, data = d, alt = "less")
##
## Asymptotic Ansari-Bradley Test
##
## data: values by ind (jung, ramsay)
## Z = 1.336, p-value = 0.09073
## alternative hypothesis: true mu is greater than 1
Do the \( p \)-values agree?
The http://lock5stat.com video used this data on memory. There are two
treatments done prior to a memory test – getting adequate sleep or
ingesting adequate caffeine. The measured scores on memory are better
if higher. The data was: Test the NULL hypothesis of no difference in the effect against an
alternative that sleep is better. Use the following test statistic the
difference of the medians: Rather than look at over 2 million possible permuations (
sleep <- c(9, 11, 13, 14, 14, 15, 16, 17, 17, 18, 18, 21)
caffeine <- c(6, 7, 10, 10, 12, 12, 13, 14, 14, 15, 16, 18)
test_stat <- function(x, y) median(x) - median(y)
choose(24,
12)), we will do a simulation here:B <- 3000
all_data <- c(sleep, caffeine)
n <- length(sleep)
m <- length(caffeine)
res <- numeric(B)
for (i in 1:B) {
ind <- sample(1:(n + m), n)
res[i] <- test_stat(all_data[ind], all_data[-ind])
}
## p-value
sum(res >= test_stat(sleep, caffeine))/B
## [1] 0.03367
Data on the amount of time spent exercising for two samples, one of
men and one of women are given: Is there statistically significant evidence that men exercise more
than women? Answer this with a permutation test with the stastic being
the difference in third quartiles: From the large \( p \)-value we say no.
men <- c(2, 2, 2, 3, 3, 3, 8, 8, 10, 10, 14, 14, 14, 15, 19, 20, 20, 24, 27,
30)
women <- c(0, 1, 1, 2, 2, 2, 10, 5, 3, 10, 3, 10, 10, 10, 10, 3, 8, 7, 10, 6,
10, 12, 12, 12, 14, 15, 20, 17, 23, 34)
test_stat <- function(x, y) quantile(x, 0.75) - quantile(y, 0.75)
We use a simulation. This is very similar to the previous answer.
B <- 3000
all_data <- c(men, women)
n <- length(men)
m <- length(women)
res <- numeric(B)
for (i in 1:B) {
ind <- sample(1:(n + m), n)
res[i] <- test_stat(all_data[ind], all_data[-ind])
}
## p-value
sum(res >= test_stat(sleep, caffeine))/B
## [1] 0.178
In the The Compare the above to the output of We use the The result is similar, though there is a different test statistic. The permuation approach can be simulated and produces a similarly small \( p \)-value: The Spearman test is another “rank” test. Before computers became
ubiquitous, a common trick was to take ranks which made things easier
to compute. We mentioned the wilcoxen rank-sum test, the Ansari test
and now this Spearman test where ranks are used in comparison. The ranking allows one to make tables for small values and carry out
asymptotic results when there are many values. The
cor.test example page we find this data:## Hollander & Wolfe (1973), p. 187f. Assessment of tuna quality. We
## compare the Hunter L measure of lightness to the averages of consumer
## panel scores (recoded as integer values from 1 to 6 and averaged over
## 80 such values) in 9 lots of canned tuna.
x <- c(44.4, 45.9, 41.9, 53.3, 44.7, 44.1, 50.7, 45.2, 60.1)
y <- c(2.6, 3.1, 2.5, 5, 3.6, 4, 5.2, 2.8, 3.8)
cor.test function has the argument method. We use
method="spearman" to test the correlation:cor.test(x, y, method = "spearman")
##
## Spearman's rank correlation rho
##
## data: x and y
## S = 48, p-value = 0.0968
## alternative hypothesis: true rho is not equal to 0
## sample estimates:
## rho
## 0.6
coin's spearman_test and then
repeat a test of correlation by permuting the x values.
spearman_test with a formula interface:require(coin)
spearman_test(y ~ x)
##
## Asymptotic Spearman Correlation Test
##
## data: y by x
## Z = 1.697, p-value = 0.08969
## alternative hypothesis: true mu is not equal to 0
res <- replicate(3000, cor(sample(x), y)^2)
sum(res >= cor(x, y)^2)/length(res)
## [1] 0.09533
coin functions
basically have 3 options for each test: do an exact test, do a simulation, or use an
asymptotic result.
Data on the amount of time spent exercising for two samples, one of
men and one of women are given: Is there statistically significant evidence that men exercise more
than women? Answer this with a permutation test with the stastic being
the difference in third quartiles: Here we see a small \( p \)-value: Notice we combined two statements in the block of From: https://onlinecourses.science.psu.edu/stat464/node/51 (Ingots Example) A sample of 16 pound ingots were taken from two different distributors, A and B. We know they are typically the same weight but what is equally important is that there is as little variability as possible. Let's assume they have the same parent population upto a scale parameter. Let \( s \) be the ratio of these two parameters. Then test if \( s=0 \) against a two sided alternative. The Repeat this test using The only trick to using the Still a statistically significant difference. A graphic to test the assumptions of similarly shaped (though not
scaled) parent populations might be two density plots: It is hard to say the assumptions are satisfied, but the samples are
small so one can't be too sure. The question writer has this comment: What to do? Well, their next link is to the Ansari-Bradley test. For fun, you could do the simulation investigation hinted at above to see if the comment is apt.
men <- c(2, 2, 2, 3, 3, 3, 8, 8, 10, 10, 14, 14, 14, 15, 19, 20, 20, 24, 27,
30)
women <- c(0, 1, 1, 2, 2, 2, 10, 5, 3, 10, 3, 10, 10, 10, 10, 3, 8, 7, 10, 6,
10, 12, 12, 12, 14, 15, 20, 17, 23, 34)
test_stat <- function(x, y) quantile(x, 0.75) - quantile(y, 0.75)
men <- c(2, 2, 2, 3, 3, 3, 8, 8, 10, 10, 14, 14, 14, 15, 19, 20, 20, 24, 27,
30)
women <- c(0, 1, 1, 2, 2, 2, 10, 5, 3, 10, 3, 10, 10, 10, 10, 3, 8, 7, 10, 6,
10, 12, 12, 12, 14, 15, 20, 17, 23, 34)
test_stat <- function(x, y) quantile(x, 0.75) - quantile(y, 0.75)
all_data <- c(men, women)
res <- replicate(3000, {
ind <- sample(1:length(all_data), length(men))
test_stat(all_data[ind], all_data[-ind])
})
sum(res > test_stat(men, women))/length(res)
## [1] 0.029
replicate. The last value computed is the one assigned into res. One other minor point: the construct 1:length(all_data) is generally programmed as seq_along(all_data). This handles the case where the length is 0 more gracefully, as 1:0 is not empty.
var.test function will do this and compute a \( p \)-value under the
further assumption that the parent populations are normal with a
common mean:A <- c(15.7, 16.1, 15.9, 16.2, 15.9, 16, 15.8, 16.1, 16.3, 16.5, 15.5)
B <- c(15.4, 16, 15.6, 15.7, 16.6, 16.3, 16.4, 16.8, 15.2, 16.9, 15.1)
var.test(A, B)
##
## F test to compare two variances
##
## data: A and B
## F = 0.1942, num df = 10, denom df = 10, p-value = 0.01607
## alternative hypothesis: true ratio of variances is not equal to 1
## 95 percent confidence interval:
## 0.05224 0.72171
## sample estimates:
## ratio of variances
## 0.1942
coin's ansari_test or the built-in ansari.test.
coin function is getting the data into a data frame in stacked format:d <- stack(list(A = A, B = B))
ansari_test(values ~ ind, data = d)
##
## Asymptotic Ansari-Bradley Test
##
## data: values by ind (A, B)
## Z = 2.586, p-value = 0.009721
## alternative hypothesis: true mu is not equal to 1
dA <- density(A)
dB <- density(B)
plot(dA, xlim = range(c(dA$x, dB$x)), ylim = range(c(dA$y, dB$y)))
lines(dB, col = "blue")
It turns out that F is horrible if you violate the Normal
assumption. It is even horrible if the distribution is not Normal but
you have symmetry. By simulations, it is seen that its performs well
when the data are from Normal distributions but if you have any other
distribution and the test is bad. So, what do we do?