Use of generalized linear mixed models in analyzing mutant frequency data from the transgenic mouse assay.

The transgenic mouse assay is now widely used for the study of mutagenesis in diverse rodent tissues and to test chemicals for genotoxic potential. This kind of assay generally involves nested observations at several levels of sampling, e.g., animals, packaging reactions, and plates. Due to the common origin, the mutant frequency (MF) in tissues from the same animal are likely to be positively correlated, inducing extra variation relative to the common binomial variation. In this article, a generalized linear mixed model is used to analyze the overdispersed binomial data on mutant frequency from the transgenic mouse assay, with a random effect for each level of the sampling hierarchy. This is a comprehensive framework within which different sources of variation in the data can be evaluated in nested factorial experiments and treatment effects can be assessed simultaneously. It avoids the current practice of repeated testing for excess binomial variability at each level of the sampling hierarchy and aggregating data up the levels, but fits the data with one single model. Parameters associated with the fixed effects, particularly dose, and the variance components for the random effects (e.g., animals, packages, and plates) can be estimated and tested for significance. Data previously reported in the literature involving the lacl gene from the Big Blue mouse are used to illustrate the proposed method.