Skip to main content

Quantitative assessment of the use of modified nucleoside triphosphates in expression profiling: differential effects on signal intensities and impacts on expression ratios

Abstract

Background

The power of DNA microarrays derives from their ability to monitor the expression levels of many genes in parallel. One of the limitations of such powerful analytical tools is the inability to detect certain transcripts in the target sample because of artifacts caused by background noise or poor hybridization kinetics. The use of base-modified analogs of nucleoside triphosphates has been shown to increase complementary duplex stability in other applications, and here we attempted to enhance microarray hybridization signal across a wide range of sequences and expression levels by incorporating these nucleotides into labeled cRNA targets.

Results

RNA samples containing 2-aminoadenosine showed increases in signal intensity for a majority of the sequences. These results were similar, and additive, to those seen with an increase in the hybridization time. In contrast, 5-methyluridine and 5-methylcytidine decreased signal intensities. Hybridization specificity, as assessed by mismatch controls, was dependent on both target sequence and extent of substitution with the modified nucleotide. Concurrent incorporation of modified and unmodified ATP in a 1:1 ratio resulted in significantly greater numbers of above-threshold ratio calls across tissues, while preserving ratio integrity and reproducibility.

Conclusions

Incorporation of 2-aminoadenosine triphosphate into cRNA targets is a promising method for increasing signal detection in microarrays. Furthermore, this approach can be optimized to minimize impact on yield of amplified material and to increase the number of expression changes that can be detected.

Background

DNA microarrays have been widely adopted in genomics because of the their ability to simultaneously examine the expression levels of thousands of genes. As a result, the scope of applications for microarrays has broadened rapidly, from drug discovery [1], to classification of cancers [2–4] and analysis of splice variants [5]. Novel analytical tools have been constructed to address every component of the microarray experiment and optimize performance [6]. However, ideal systems with maximized sensitivity and data reproducibility have not been achieved. One approach to enhance sensitivity in microarrays, using a novel signal amplification technique, has recently been reported [7]. Another approach is to increase the affinity of a probe (nucleic acid present on the array) for its target through modifications to the length [8], chemistry [9], or physical structure [10] of the probe.

Naturally occurring analogs of purine and pyrimidine bases have been examined extensively for their ability to increase the thermodynamic stability of DNA:DNA and DNA:RNA duplexes [11–15]. Among these are 2-aminoadenine also known as diaminopurine (DAP), which is found in S-2L cyanophage DNA [16], 5-methyl uracil (MeU), and 5-methyl cytidine (MeC). Previous studies have shown the ribonucleoside triphosphate derivatives of modified bases to be effectively incorporated by RNA polymerases [17, 18], which make them an excellent substrate for use in microarray sample preparations requiring amplification through in vitro transcriptions (IVTs).

Due to their effects on duplex stability and secondary structures and their lack of replication by DNA polymerases, modified nucleotides have recently been exploited in a variety of technologies in molecular biology and genomics. For example, 2'-O-methyl ribonucleotides and 5-(1-propynyl)pyrimidines have been chemically incorporated into oligonucleotides which were used to detect telomeric repeat sequences in fluorescence in situ hybridization (FISH) assays [19]. Furthermore, chimeric primers containing deoxynucleotides and 2'-O-methyl ribonucleotides have been used to eliminate artifacts, produced by exponential amplification of minor side-products, in cycle sequencing [20].

Recently, studies employing DNA arrays have also examined the use of modified nucleotides such as DAP, 5-bromodeoxyuridine, and 2'-O-methylthymidine in the probe [21, 22]. An alternative method to increase the probe-target affinity is to incorporate the modified ribonucleotide, as the triphosphate derivative, during the IVT process so that the cRNA produced has the desired level of substitution of the corresponding unmodified nucleotide. Such an approach has been demonstrated in a study which applied these modified cRNA products to high density oligonucleotide arrays [23]. However, that study measured the signal when amplifying a specific gene rather than a genome-wide approach. A genome-wide amplification, coupled with the incorporation of modified nucleotides, permits measurement of every transcript in the sample. To our knowledge, no systematic studies have been performed examining the incorporation of modified ribonucleotide triphosphates into cRNA, yield of amplified material, effects on hybridization intensities and reproducibility, and the impact on the differential expression ratios. It will not be possible to gauge the full potential, advantages, and disadvantages until such studies have been completed.

In this study, we investigated the possibility that incorporation of modified nucleotides into complementary RNA (cRNA) target samples could increase signal intensity on the Motorola Codelinkâ„¢ Expression microarray platform. The ratios of modified to unmodified nucleoside triphosphates (NTPs) were varied in each target synthesis in order to measure the range of effects on cRNA yield, specific activity of target sample, hybridization signals, and differential expression ratios. Our results suggest that incorporation of 2-aminoadenosine (DAP) triphosphate into target cRNA samples may increase hybridization signal intensity for a wide variety of RNA:DNA hybrid duplexes. In contrast, 5-methyluridine and 5-methylcytidine had detrimental effects on signal intensities. Hybridization specificity was dependent on both target sequence and extent of substitution with the modified nucleotide. Concurrent incorporation of modified and unmodified ATP in a 1:1 ratio resulted in significantly greater numbers of above-threshold ratio calls across tissues, while preserving ratio integrity and reproducibility.

Results

Effect of modified nucleotides on cRNA yield and assessment of their incorporation and biotin-11-UTP incorporation

Although DAP, MeC, and MeU were tolerated by T7 RNA polymerase, incorporation of these analogs into our IVT reaction cocktails reduced the yields of amplified cRNA. The magnitude of the decrease was 20–30% and depended on RNA tissue source and the NTP modification. However, within the context of our pre-established assay conditions (a single color, single sample per array system using ten micrograms of cRNA), enough cRNA was generated from each IVT reaction to perform hybridizations in triplicate. Thus, we were able to normalize input amounts of cRNA for each condition tested in order to quantitatively compare relative changes due to each respective modification. In these experiments, we generated cRNA from five micrograms of total RNA and did not explore the impact on yield when smaller or larger amounts of input total RNA are used.

We next determined how well the modified NTPs were incorporated by the T7 RNA polymerase and whether incorporation of biotinylated UTP was altered due to the presence of these modified NTPs. To address these questions, we used an analytical method developed in our laboratory and described in a previous study [24]. Briefly, the complex cRNA is digested with P1 nuclease and calf intestinal phosphatase and applied to a high performance liquid chromatography (HPLC) column to separate the nucleosides, followed by an absorbance measurement at 260 nm. As seen in Figure 1A, when only the unmodified NTPs are incorporated during the IVT, there is good separation of the individual nucleosides. Furthermore, as reported earlier [24], using the extinction coefficients and integrating the area under these peaks, there are approximately equal amounts of each of the four nucleosides. When DAP was added at a 1:1 molar ratio to the adenosine, the chromatogram showed incorporation of DAP into the cRNA, and this level of incorporation appeared to be equivalent to the level of incorporation of adenosine (Figure 1B). Moreover, when the adenosine was fully substituted by the DAP in the IVT reaction cocktail, only a peak corresponding to the DAP was detected (Figure 1C), and this peak was approximately equal in area to that of the adenosine in the control situation. Similarly, when MeC was fully substituted for cytidine in the IVT reaction cocktail, only a peak corresponding to the MeC was detected (Figure 1D), and this peak was approximately equal in area to that of the cytidine in the control situation. In contrast, we were unable to examine incorporation levels of the MeU because the MeU peak was eluted at approximately the same time as the guanosine peak (data not shown). It is important to note that these analyses used the complex cRNA and represent a global, average view of the incorporation. Incorporation may differ somewhat depending on the sequence, structure, or expression level of the nascent RNA transcript. We therefore incorporated each of these modified nucleotides into a unique bacterial transcript, generated by run off transcription from a plasmid into which the gene was cloned. When this transcript was digested and applied to the HPLC, we found similar results as seen with the complex cRNA (data not shown). We conclude that these modified nucleotides are incorporated during the IVT reaction at ratios which are reflective of the input ratio to the unmodified counterpart.

Figure 1
figure 1

HPLC analysis of digested cRNA demonstrates incorporation of modified nucleotides and their resolution from unmodified counterparts. Absorbance profiles at 260 nm are shown for (A) control, (B) 1:1 DAP:A, (C) fully substituted DAP, and (D) fully substituted MeC conditions. Proportions of each nucleoside were calculated using peak areas and extinction coefficients. The peaks for cytosine, uridine, guanosine, and adenosine show up in the unmodified control sample (A) at approximately 5.0, 7.2, 12.6, and 15.6 min, respectively.

Because the sensitivity and reproducibility of microarrays depends on the specific activity of target cRNA, we also wanted to determine if adding modified NTPs to the IVT changed the incorporation rate of biotin-11-UTP. This biotinylated nucleotide is used in our biotin-streptavidin-Alexa647 conjugate detection system. We found that the level of biotinylated uridine, detected at 294 nm after digestion and application to the HPLC, was unaffected by the presence of any of the modified NTPs in the IVT reaction cocktail (data not shown). Moreover, analysis of individual transcripts, which were enzymatically digested into mononucleosides, showed equivalent incorporation rates of biotin-11-UTP for both control samples and target samples containing modified ATP.

Lastly, we examined what effect, if any, incorporation of modified NTPs had on the length of the amplified cRNA. Transcript size was determined by running samples on an Agilent 2100 BioAnalyzer. No difference was observed for either individual transcripts or complex samples in modified versus unmodified ATP samples (data not shown).

Effect of modified nucleotides on hybridization signals and specificity

We next determined the effect of these modified NTPs on hybridization intensities. After hybridization, scatter plots of hybridization intensities were generated comparing those intensities generated from unmodified cRNA (control) to either the duplicate control hybridization or to intensities generated from cRNA where one of the modified NTPs was incorporated. The comparison of control versus control (Figure 2A) illustrates that, if hybridization intensities are equal in the two conditions, the points should lie on the diagonal. Complete substitution of unmodified ATP with DAP (Figure 2B) produced greater increases overall in individual probe signals than the target sample containing a 1:1 ratio of DAP to unmodified ATP (Figure 2C), as would be expected if DAP incorporation increased the DNA:RNA hybrid duplex stability. A quantitative estimation of assay performance increase can be derived from the median probe signal on each microarray. Although Figure 2B and 2C show that increases in signal intensity are not linear for all probe sequences, median slide intensity across sets of duplicate hybridizations for two different tissues (human embryonic kidney and Burkitt's lymphoma) increased an average of 40% +/- 7% and 99% +/- 6% for the 1:1 DAP:A and all DAP conditions, respectively, over the unmodified control sample. However, Figure 2B also shows that full substitution of adenosine by DAP reduced the hybridization signals for many of the probes, generating a bowing towards the unmodified control condition and suggesting a duplex destabilizing effect of complete DAP substitution in certain sequence contexts [11, 23]. On a global scale, such a high incidence of probes with reduced hybridization signals may limit the usefulness of the complete DAP substitution. Of particular interest is the observation that the number of probes which have increased signal and the degree to which their signals are increased are highest at the lower end of the signal range. In contrast, both of the modified pyrimidine triphosphates (MeC and MeU) tested resulted in a decrease in overall signal intensities for nearly all of the probes tested when compared against unmodified control samples (Figures 2D &2E). However, data points on the right of the diagonal in figures 2D and 2E also show that for a fractional set of target-probe pairs, substitution with MeC and MeU increased hybridization signal relative to the unmodified control. Those probes whose signal intensities were increased at least two-fold are summarized in Table 1. As Table 1 shows, for a number of transcripts substitution with MeC or MeU may potentially augment hybridization signals more so than even DAP. This may prove useful for target samples containing very low concentrations of these particular transcripts. Nevertheless, because our goal was to utilize modifications that would increase hybridization signals for the vast majority of target-probe duplexes, we subsequently focused our efforts on the samples containing DAP.

Figure 2
figure 2

Modified purines and pyrimidines show differential effects on hybridization intensities when incorporated into cRNA targets. Modified and unmodified (Control) samples were normalized for concentration, fragmented, and hybridized onto Human Uniset I microarrays in duplicate, except for 5-methyl CTP samples, which were hybridized in duplicate onto arrays with 1100 probes spotted in 6-fold redundancy. Average signal intensities after background subtraction were plotted for all probes of the Human Uniset I array for the control samples versus (A) control, (B) fully DAP-substituted, (C) 1:1 substituted DAP, (D) fully meC-substituted, and (E) fully MeU-substituted.

Table 1 Percent signal change due to nucleotide substitution.

We next determined whether the increases in signal intensity observed with DAP substitution were similar in magnitude to those observed when the hybridization time is increased. We therefore hybridized two micrograms of cRNA with no modifications or two micrograms with DAP at a 1:1 molar ratio to adenosine for 18 hours and hybridized two micrograms of cRNA with no modifications for 42 hours. We generated two scatterplot and overplayed these scatterplot on each other (Figure 3A). The first scatterplot compares the intensities of the 18 hour control with those of the 42 hour control (orange signals). The second scatterplot compares the intensities of the 18 hour control with those of the DAP-modified cRNA (blue signals). The longer hybridization time increases the median signal intensity by 47% +/- 2%. The increase is more pronounced for the medium and high signals. As shown before, the DAP-modified cRNA generated hybridization signals which were also increased relative to the 18 hour control hybridization. However, the increase seemed more pronounced for the low and medium signals. We conclude that both DAP modification and an increased hybridization time can affect the hybridization reaction but in very different ways.

Figure 3
figure 3

Effects of DAP substitution are similar, and additive, to those seen with an increased hybridization time. (A) Comparison of intensities obtained with unmodified cRNA hybridized for 18 hours versus partially DAP-modified cRNA for 18 hours (blue data points) transposed on a comparison of intensities obtained with unmodified cRNA hybridized for 18 hours versus unmodified cRNA hybridized for 42 hours (orange data points). (B) Comparison of intensities obtained with unmodified cRNA hybridized for 18 hours versus partially DAP-modified cRNA hybridized for 42 hours. (C) Plot showing the relative increases with respect to the 18 hour, unmodified control for increased hybridization time (18 hour ⇒ 42 hour), 1:1 DAP:A substitution, and a combination of increased hybridization time/DAP substitution. The total number of probes was divided into three equally sized bins according to their signal intensities, with the bins representing low, medium, and high expressers. Relative percent increases for each probe were derived from the equation: 100 * (modified sample signal [time, DAP, or combination of both] - control sample signal) / control sample signal, and are shown with standard deviation error bars.

Because of the differential effects of increased hybridization time and DAP substitution on low and high expressers, we wanted determine whether the increases seen of using both DAP modification and an extended hybridization time could be additive. We therefore hybridized unmodified (control) cRNA for 18 hours and a DAP-modified cRNA for 42 hours (Figure 3B). The median slide intensity increased by 110% +/- 17%, which is approximately double the increase seen with either the 1:1 DAP:A or the longer hybridization time by themselves. When all of the approximately 9,000 probes are divided into three equally sized bins representing low, medium, and high expressers according to their respective signal intensities, the effects of increased hybridization time and DAP become very evident (Figure 3C). Figure 3C shows that the relative increases in signal intensities due to increased hybridization time are biased towards the medium and high expressers, with signals increasing an average of 48% and 84%, respectively, whereas increases due to DAP substitution are more pronounced for low and medium expressers (57% and 81% increase, respectively). Together, the two modifications of increased hybridization time and DAP substitution can be used in concert to amplify signal intensities for the entire range of probe signals, and as figure 3C indicates, these boosts in signals are approximately additive for the 42 hour, DAP-containing samples.

We next determined the effect of modified NTPs on hybridization specificity (the ability to distinguish sequences up to a certain homology). We addressed this issue by designing probes which had one to four adjacent, centrally located mismatches and comparing hybridization signals generated from these mismatched probes to signals generated from their corresponding perfect matches. In the control situation (hybridization of unmodified cRNA), the hybridization intensity decreased as the number of mismatches increased, with two mismatches generally destabilizing the duplex sufficiently to reduce the hybridization signal to 0% of the parent signal. There was one exception (Figure 4C) where, even in the presence of four mismatches, the signal was not reduced below 60% of the parent signal. This signal was low in the perfect match and, therefore, this situation most likely reflects a lowering to the noise levels. In fact, the signal was lowered to below threshold levels in the presence of mismatches. When equimolar ratios of DAP and unmodified ATP were incorporated into the cRNA target (1:1 DAP:A), specificity was not significantly affected in two of the four probe sequences (Figure 4A and 4B). However, in one probe sequence (Figure 4C), specificity was enhanced. For this test probe, the unmodified sample produced unusually high signals for base mismatches. Nevertheless, other groups have seen similar improvements in mismatch discrimination with diaminopurine-containing oligo mers for different applications [13]. A fourth probe sequence demonstrated a smaller destabilizing effect of a single mismatch with the partially modified cRNA compared to the control (Figure 4D). In this sequence, the hybridization was reduced to the same extent with three mismatches using either the control or partially modified cRNA. Full substitution of adenosine by DAP showed varied effects on specificity in the four different probe sequences. In two of the probe sequences, the fully modified cRNA behaved similarly to the control cRNA (Figures 4A and 4B). In a third probe sequence, the specificity improved for the fully modified cRNA, as it did for the partially modified cRNA (Figure 4C). The improvement in specificity for this probe depended on the extent of DAP modification. In a fourth probe sequence, the fully modified cRNA showed a dramatic decrease in specificity with one- or two-base mismatches, although three mismatches reduced the hybridization intensity to ~10% of that of the perfect match. The loss in specificity for this probe sequence also depended on the extent of DAP modification, with one, two, or three mismatches required to reduce the hybridization intensity to ~ 10% for the control, partially modified, and fully modified, respectively, cRNA. A fifth test probe showed hybridization signal intensity below the negative control threshold in both the control and modified ATP samples, suggesting either an absence of the transcript from the target sample or abundance too low to quantify (data not shown). Repeat experiments with cRNA containing modified ATP supported the general specificity trend: unmodified ATP > 1:1 DAP:A > all DAP (data not shown). Attempts to increase specificity in the all DAP condition by raising the temperature during hybridization from 37°C to 42°C or 47°C were unsuccessful (data not shown). Others have been able to distinguish sequences of up to 90% homology [6] using the same platform (3-base mismatch/ 30-mer oligonucleotide signal near or below negative control threshold), and we support their findings here with the control and 1:1 DAP:A condition. The fully MeC-substituted cRNA targets were examined with respect to specificity in two test sequences (Figures 4B and 4D). In one test sequence (Figure B), the fully MeC-substituted cRNA targets behaved nearly identically to the partially DAP-substituted targets. In the other test sequence (Figure 4D), the MeC-substituted cRNA behaved nearly identically to the unmodified cRNA target and showed better specificity than that of the partially DAP-substituted cRNA target with respect to the effect of one mismatch. We conclude that the specificity of fully MeC-substituted cRNA and partially DAP-substituted cRNA targets are, for the most part, equivalent and essentially no loss in specificity (as determined by the effect of two mismatches) is observed with these modified targets.

Figure 4
figure 4

Mismatch discrimination of DAP- or MeC-substituted and control samples. Specificity was determined by introducing single or multiple adjacent, centrally located mismatches in several test probes. Signal intensities of mismatch (MM) probes are plotted as a function of the percent of the perfect match (PM) control for unmodified (diamonds), partially DAP-modified (triangles), fully DAP-substituted (boxes), and fully-MeC-substituted (x) cRNA targets for the transcripts corresponding to the following accession numbers: (A) (X79067), (B) (AF067139), (C) (Z83844), and (D) (NM_004323).

An additional metric of specificity on the Codelink bioarrays is the use of 54 negative control bacterial probes spotted in 4X redundancy, which were designed and empirically shown not to cross hybridize to human transcripts [6]. While the all DAP condition resulted in an increase in the hybridization intensity for three of the negative control probes, the 1:1 DAP:A condition produced lower background signals similar to those of the unmodified control (data not shown). Thus, increases in hybridization signal intensities in the 1:1 DAP:A condition are attributable to specific modified cRNA target/DNA probe interactions. We conclude that partial substitution of adenosine with DAP does not significantly compromise specificity and was, therefore, investigated further.

Effect of DAP incorporation on differential expression ratios

Ultimately, the goal of any global expression profiling system measuring relative transcript abundance is to accurately and reproducibly determine the changes in expression levels between different target samples. We therefore determined if the increases in signal intensity caused by DAP incorporation had an effect on differential ratio calls. Kidney and lymphoma cRNA samples containing either no modified NTPs or 1:1 DAP:A were hybridized in duplicate to Human Uniset I arrays; average kidney to lymphoma ratios were calculated and plotted (Figure 5) after removing signal outliers defined as a two-fold difference between replicates. Differential expression ratios demonstrate very good correlation between the unmodified control sample and the 1:1 DAP:A sample (r = 0.95) on the Human Uniset I microarray.

Figure 5
figure 5

Correlation of kidney to lymphoma differential expression ratios using partially DAP-modified and unmodified control samples. Duplicate hybridizations were performed for each sample, resulting in four possible calls for kidney to lymphoma ratios for each probe (i.e., K1:L1, K1:L2, K2:L1, K2:L2). Data plotted are the average kidney to liver ratio for the four calls on both axes.

In addition to investigating the correlation of the differential expression ratios generated from modified and unmodified targets, we determined whether DAP incorporation affected the variability of the ratio calls. One method to measure microarray reproducibility is by calculating the coefficients of variation (CVs) for each replicate probe across arrays. Likewise, CVs can also be calculated for multiple ratio calculations across different tissue samples. Figure 6 shows a plot of the CVs of the ratios as a function of the mean ratios for all of the data points, regardless of the intensity level of the probe. Consistent with earlier findings [6], the majority of the CVs in the of the kidney to lymphoma ratios are below 30%, with an average of 13.2%, and the variability increases as the ratio approaches unity for the unmodified targets (Figure 6A). Figure 6B shows that the 1:1 DAP:A condition produces ratios with similar overall variability (average CV of 12.4%) compared to the unmodified control targets. When the CVs in the ratios generated using either the control or partially modified cRNA are binned according to the magnitude of the CV, the partially modified cRNA generates more CVs in the less than 20% bins (Figure 6C). These data demonstrate that the low variability in ratio calls that we are able to routinely obtain [6] are maintained during DAP incorporation. However, a closer inspection of the ratios in Figures 6A and 6B reveals that a possible caveat of DAP use is a slight compression of ratios along the entire range of calls. This compression is also observed when the observed ratio is binned according to the magnitude of the fold change (Figure 6D). The partially modified cRNA generates more ratios in the one to 1.5 fold change bin and fewer ratio changes in the other bins. This compression is not significant but also manifests itself in the slope of the correlation line in Figure 5. The slope of this line was found to be greater than unity.

Figure 6
figure 6

Effect of partial DAP substitution on the variability and magnitude of differential expression ratios. (A and B) CVs of the ratios plotted as a function of the mean ratio for unmodified (A) and modified (B) samples. Signal outliers, defined as any pair of signals which were greater than two-fold different in intensity, were removed from all samples. (C) Plot showing the number of probes having a designated CV across a range of CVs for control and partially DAP-modified cRNA targets. (D) Plot showing the number of probes having a designated fold change (ratio) across a range of ratios for control and partially DAP-modified cRNA targets.

The differential expression ratios calculated when using DAP (Figures 5 and 6) suggest that incorporation of DAP has little effect on correlation and reproducibility of ratio calls between tissues. However, the ability to accurately discriminate between signals due to true hybridization events and what may be considered background noise is paramount to making accurate differential assessments. A lower limit of detection was previously defined using our platform by developing a negative control threshold in order to assign a confidence level to such signal or noise queries [6]. Briefly, this threshold was determined by taking the mean signal of bacterial negative control probes mentioned above (minus a 10% trim to account for weak cross hybridization of high expressers or for true hybridization to sequences not in the database) and adding three standard deviations (99.7% confidence). Using only probes that were at or above the negative control threshold in both tissues, we observed a significantly greater number of probes in the 1:1 DAP:A sample for which we could confidently assign a ratio (Figure 7). As Figure 7 shows, samples containing DAP generated over 400 more above-threshold ratios than the control samples containing only unmodified NTPs. Most of these ratios identified only in the 1:1 DAP:A sample were small in magnitude, with fold changes between 1.1 and 1.9.

Figure 7
figure 7

Number of above-threshold ratios generated by partially DAP-modified and unmodified target samples. A negative control threshold was imposed to define the lower limit of detection and was calculated by taking the 80% trimmed mean (top 10% and bottom 10% of signals removed from the population) of 216 negative control probes and adding three standard deviations to the mean (99.7% confidence). Embryonic kidney and Burkitt's lymphoma samples were performed in duplicate and data were screened for probes that displayed above-threshold signal intensities. Kidney to lymphoma ratio calls were calculated using only above-threshold probe signals with a maximum of four ratios for each probe. The data represent the average of the number of calls for any two kidney and lymphoma slides +/- standard deviation error bars.

Discussion

The effect of modified NTPs on hybridization intensities

Two important issues for identifying differentially expressed genes using DNA microarrays are 1) methods to increase signal intensity and 2) the ability to accurately separate true hybridization signals from background noise at the low end of the signal dynamic range. In this study, we utilized modified nucleoside triphosphates with the aim of enhancing hybridization signal intensity across a wide range of probe sequences. Modified ribo- and deoxyribonucleoside analogs have been used in a variety of applications [12–14], but their use has been limited in the field of microarrays [23]. We demonstrate that 2-aminoadenosine or 2,6-diaminopurine (DAP), an analog of adenosine, significantly increases signal intensity for a wide range of probe sequences, whereas C-5 methylated pyrimidine analogs of cytosine and uridine decreased signal intensities across the entire range of probe sequences on our platform (Figure 2).

Although incorporation of DAP did not uniformly increase the hybridization stability for all target-probe duplexes, even moderate incorporation of DAP (1:1 DAP:A) resulted in signal intensity increases (up to 30-fold) for the majority of probes on our platform. More importantly, a significant number of probes that went previously undetected (i.e., below a negative control threshold; [6]) in the control sample exhibited signal intensities above this threshold in DAP-modified samples. Utilization of such a modification may prove especially valuable for low abundant transcripts which may be difficult to quantify because of weak hybridization kinetics. As shown in Figures 2B and 2C, the largest increases in signal intensities occurred for low and medium expressers relative to the unmodified control sample. It is apparent that the effect of DAP depends not only on the sequence compositions of target and probe, but also on the absolute abundance of specific transcripts in the target sample.

It is of interest to note that the results we report here differ from those obtained by others [23]. Those studies reported increases in signal intensity for targets containing 5-methyl UTP while completely substituted targets containing DAP had the opposite or no effect. Although we cannot completely account for the disparity of the results, one key difference in methodology between the studies is that their group was not able to normalize concentrations for modified and unmodified ATP targets because of dramatic differences in cRNA yield. Therefore, mass input of cRNA target samples were much greater for unmodified samples compared to the DAP-containing samples. Because the sample preparation used in this protocol is highly robust and reproducible [24], we were able to use equivalent mass inputs of all modified and unmodified targets during hybridization, allowing us to present our results quantitatively rather than qualitatively. However, we agree that complete substitution of DAP for adenosine may cause specificity to decline in certain sequence contexts, which may limit its usefulness in microarray applications. However, we have found that reducing the proportion of DAP incorporated into target samples (e.g., incorporating a 1:1 ratio of DAP:adenosine) is one approach to avoid such decreases in specificity. Other factors that may contribute to the dissimilarity of the results include differences in probe sequence and size, microarray fabrication/chemistries, labeling/detection methods, and the vigorous mixing that is employed in our microarray experiments. We have previously shown the dramatic impact of mixing on hybridization signal intensities [6]. Presumably, this impact is due to the effect of three dimensional diffusion of the target molecules and to the effect on how many probes reach equilibrium in the hybridization. It is possible that the degree of mixing during the hybridization can affect the number of probes which show a difference in intensity with substitution of DAP for adenosine as well as the magnitude of this difference. Given these differences, it is not surprising that our results are not in line with those previously reported.

The results in our study are in good agreement with previous reports which include a wide variety of interaction types, including DNA:DNA, DNA:RNA, and DNA:PNA interactions. [11–13, 25]. These studies have shown that DAP substitution in oligonucleotide probes can increase thermodynamic affinities as reported directly by increases in Tm and indirectly through increases in hybridization intensities.

Biophysical considerations in using modified NTPs

Modified nucleotides appear to drive the equilibrium (particularly for low copy transcripts binding with their targets) towards duplex formation. These findings suggest that DAP use could be exploited in RNA samples that are limited, such as tumor biopsies. Based on our experiments with extended hybridization times (Figure 3), use of DAP may prove to be a useful alternative to increase array sensitivity for studies that are unable to generate enough target to hybridize or are unable to increase hybridization times because of throughput constraints.

Why do we see differential effects of DAP, MeC, and MeU on hybridization intensities? The fact that DAP was able to increase hybridization intensities while MeC and MeU decreased hybridization intensities (Figures 2D and 2E) is interesting in light of the fact that DAP is a purine incorporated into an RNA strand while MeC and MeU are pyrimidines incorporated into an RNA strand. Many studies have shown that a ribopurine rich strand bound to a deoxypyrimidine rich strand has a higher stability than the corresponding all deoxy strands which, in turn, have a higher stability than a deoxypurine rich strand bound to a ribopyrimidine rich strand [26, 27]. These differences also suggest that further studies examining DAP or MeC incorporation into first strand cDNA, followed by formation of a DNA-DNA duplex, may not necessarily show the same effects as incorporation of these modified nucleotides into cRNA. Finally, earlier studies have noted that methylation of cytosines (generating MeC) in ribopolynucleotides stabilizes duplexes to the same extent as substitution of thymine (MeU) for uracil [28]. Thus, although it is still surprising that MeC and MeU decrease hybridization intensities, it is not surprising that the magnitude of the effects generated by MeC and MeU residues are similar.

Could the differential effects of DAP versus MeC or MeU or the differential effects of DAP on different probe sequences also be related to structural consequences? Earlier studies have shown that DAP, in DNA fragments, can widen the minor groove, as detected by reactivity towards uranyl nitrate or susceptibility to Dnase cleavage [29], and can relieve compressions in the minor groove associated with A-tracts [30]. In contrast, NMR and molecular modeling studies of an oligonucleotide duplex containing MeC have demonstrated that, while this duplex still has a B-DNA conformation, there are differences in the structural parameters and thermal stability relative to the unmodified duplex [31]. An earlier study also found, in two closely related octanucleotide duplexes, that although the methylated duplexes retained their B-DNA conformation, different structural and thermal stability effects were seen [32]. Although these studies have been carried out with DNA-DNA and not DNA-RNA duplexes, it is possible that MeC substitution may have different structural consequences than DAP substitution.

We have shown that while complete substitution of DAP for ATP may cause decreased base mismatch discrimination in certain sequence contexts, equimolar ratios of adenosine and DAP in cRNA target samples did not have significant detrimental effects on specificity (Figure 4). Remarkably, in one sequence context, the DAP modification enhanced specificity against mismatches. Enhanced specificity against mismatches has also been observed by other groups when DAP is incorporated into PNA oligomers [13] and after C5-(1-propynyl)ation of pyrimidines in DNA-RNA duplexes [33].

The effect of modified NTPs on differential expression ratios

The use of DAP to increase hybridization signals is supported by the results that show equivalency of both ratio calls and ratio variability in 1:1 DAP:A and control samples (Figures 5 and 6). When all data points are considered in the differential expression ratio analyses, the two methods appear equivalent. However, when only those data points which are above a threshold value [6] are used, ensuring a higher confidence that these intensities represent true expression levels and not noise or weak cross hybridization, the DAP-modified cRNA enabled an ~ 10% increase in the number of ratios that could be generated (Figure 7). Such increases are critical in order to sample as many genes as possible in any given microarray experiment. For example, some investigators have analyzed only data for which the signal intensity is greater than approximately 0.4% of the total signal range in both channels [34]. Although this method minimizes the variance associated with the ratios, many genes and their expression changes are missed. Other platforms use a minimal intensity level to determine whether a gene is present or absent, prior to including this gene in expression analyses [35]. It is plausible that DAP modification, by increasing intensity levels, particularly for low expressors, could enable wider coverage in genome-wide expression analyses.

In summary, incorporation of modified nucleotides may not only lead to discovery and better quantitation of rare expressers in complex samples but also impact other aspects of the microarray experiment such as probe design prior to array fabrication and the statistical design of microarray experiments. The target preparation procedure (generation of amplified cRNA or of first strand cDNA) and the subsequent duplex formation may dictate the choice of which modified nucleotide to use, as we and others have found differential effects with various nucleotides. We believe further investigation in this area could be fruitful.

Methods

Target preparation

Five μgs of human embryonic kidney or Burkitt's lymphoma total RNA (Ambion, Inc., Austin, TX) were added to a reaction mix in a final volume of 12 μl, containing 0.5 pmol T7-(dT)24 oligonucleotide primer. The mixture was incubated for 10 minutes at 70°C and chilled on ice. With the mixture remaining on ice, 4 μl of 5X first-strand buffer, 2 μl 0.1 M DTT, 1 μl of 10 mM dNTP mix and 1 μl Superscript™ II RNaseH- reverse transcriptase (200 U/μl) was added to make a final volume of 20 μl, and the mixture incubated for one hour in a 42°C water bath. Second-strand cDNA was synthesized in a final volume of 150 μl, in a mixture containing 30 μl of 5X second-strand buffer, 3 μl of 10 mM dNTP mix, 4 μl of E. coli DNA polymerase I (10 U/μl) and 1 μl of RNase H (2 U/μl) for 2 hours at 16°C. The cDNA was purified using a Qiagen QIAquick purification kit, dried down, and resuspended in IVT reaction mix, containing 3.0 μl nuclease-free water, 4.0 μl 10X reaction buffer, 4.0 μl 75 mM ATP, 4.0 μl 75 mM GTP 3.0 μl 75 mM CTP, 3.0 μl 75 mM UTP, 7.5 μl 10 mM Biotin-11-CTP, 7.5 μl 10 mM Biotin 11-UTP and 4.0 μl enzyme mix (unmodified control condition). Commercially available 2-aminoadenosine-5'-triphosphate, 5-methylcytidine-5'-triphosphate, and 5-methyluridine-5'-triphosphate (TriLink Biotechnologies, Inc., San Diego, CA) were substituted for ATP, CTP, and UTP, respectively, in separate reactions containing either complete substitution, 1:1, or 1:3 ratio of modified: unmodified NTP, keeping molar input of nucleotide constant. The reaction mix was incubated for 14 hours at 37°C and cRNA target purified using an RNeasy® Kit (Qiagen). cRNA yield was quantitated by measuring the UV absorbance at 260 nm, and fragmented in 40 mM Tris-acetate (TrisOAc), pH 7.9, 100 mM KOAc, and 31.5 mM MgOAc, at 94°C, for 20 minutes. This typically resulted in fragmented target with a size range between 100–200 bases.

Array hybridization

Two μg of fragmented target cRNA was used for hybridization of each UniSet Human I Expression Bioarray (Motorola Life Sciences) containing 9589 probes (representing 9,203 unique accession numbers (genes), corresponding to approximately 8,935 unique clusters and 386 control probes, selected initially from GenBank Unigene build #125) or for hybridization to a microarray containing probes corresponding to 1100 human genes, each spotted 6 times per array. All probes on these microarrays are 30-mer oligonucleotides spotted by piezoelectric technologies and covalently attached to a polymeric matrix [6]. These microarrays were hybridized, washed, and processed using a direct detection method of the biotin-containing transcripts by a Streptavidin-Alexa647 conjugate as previously described [6]. Processed slides were scanned using an Axon GenePix Scanner with the laser set to 635 nm, a PMT voltage of 600, and a scan resolution of 10 microns.

Data analysis

Slides were scanned and images for each slide were quantitated using CodeLink Scanning and Analysis Software (Motorola Life Sciences). Signal intensities for each spot were calculated by summation of the pixel intensities for each spot, followed by local background subtraction (based on the median pixel intensity of the area surrounding each spot). Whole array data normalization, when used, was performed independently for each slide by dividing each spot's intensity (after background subtraction) by the median signal intensity of all test probes. All false positives, determined by visual inspection of the images, which were greater than 2-fold different between duplicate arrays were removed.

Digestion and chromatography of cRNA

Four units of P1 nuclease were used to digest 20–50 μg of cRNA to generate nucleotide monophosphates. The enzyme was incubated with the cRNA at 55°C for 6 hours, then for 6 hours at 37°C in the presence of 10 units of calf intestine alkaline phosphatase to generate the nucleosides. The digested products were purified using Microcon YM-3 columns followed by centrifugation at 8000 g for 30–60 minutes. The mix was then concentrated using a SpeedVac to 100 μl. This solution was analyzed on an HPLC column equilibrated with 0.03 M TEAA (Solvent A) at a flow rate of 1 ml/min. The following gradient was used: 0–1% Solvent B (95% AcCN, 5% Solvent A1) over 5 minutes, 1–15% Solvent B over the next 15 minutes, 15–45% Solvent B over the next 30 minutes, 45–100% Solvent B over the next 20 minutes, and hold at 100% B for 2 minutes. The concentration of the heterocycles was determined by the absorbance values at 260 nm (the wavelength where maximal absorption occurs for the heterocycles) and the biotin-containing nucleoside concentrations were determined by the absorbance values at 294 nm (the wavelength where maximal absorption occurs for biotinylated cytosine and biotinylated uridine). The biotin-11-UTP peak was measured at an absorbance of 294 nm with the extinction coefficient = 13000 M-1cm-1. The 2-amino ATP was measured at an absorbance of 260 nm with the extinction coefficient = 9894 M-1cm-1.

References

  1. Heller R, Schena M, Chai A, Shalon D, Bedilion T, Gilmore J, Woolley DE, Davis RW: Discovery and analysis of inflammatory disease-related genes using cDNA microarrays. Proc Natl Acad Sci USA. 1997, 94: 2150-2155. 10.1073/pnas.94.6.2150.

    Article  CAS  Google Scholar 

  2. Perou CM, Sorlie T, Eisen MB, van de Rijn M, Jeffrey SS, Rees CA, Pollack JR, Ross DT, Johnsen H, Akslen LA, Fluge O, Pergamenschikov A, Williams C, Zhu SX, Lonning PE, Borresen-Dale A-L, Brown PO, Botstein D: Molecular portraits of human breast tumors. Nature. 2000, 406: 747-752. 10.1038/35021093.

    Article  CAS  Google Scholar 

  3. Bittner M, Meltzer P, Chen Y, Jiang Y, Seftor E, Hendrix M, Radmacher M, Simon R, Yakhini Z, Ben-Dor A, Sampas N, Dougherty E, Wang E, Marincola F, Gooden C, Lueders J, Glatfelter A, Pollock P, Carpten J, Gillanders E, Leja D, Dietrich K, Beaudry C, Berens M, Alberts D, Sondak V, Hayward N, Trent J: Molecular classification of cutaneous malignant melanoma by gene expression profiling. Nature. 2000, 406: 536-540. 10.1038/35020115.

    Article  CAS  Google Scholar 

  4. Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA, Bloomfield CD, Lander ES: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science. 1999, 286: 531-537. 10.1126/science.286.5439.531.

    Article  CAS  Google Scholar 

  5. Hu GK, Madore SJ, Moldover B, Jatkoe T, Balaban D, Thomas J, Wang Y: Predicting splice variant from DNA chip expression data. Genome Res. 2001, 11: 1237-1245. 10.1101/gr.165501.

    Article  CAS  Google Scholar 

  6. Ramakrishnan R, Dorris D, Lublinsky A, Nguyen A, Domanus M, Prokhorova A, Gieser L, Touma E, Lockner R, Tata M, Zhu X, Patterson M, Shippy R, Sendera T, Mazumder A: An assessment of Motorola Codelinkâ„¢ microarray performance for gene expression profiling applications. Nucleic Acids Res. 2002, 30: e30-10.1093/nar/30.7.e30.

    Article  Google Scholar 

  7. Stears RL, Getts RC, Gullans SR: A novel, sensitive detection system for high-density microarrays using dendrimer technology. Physiol Genomics. 2000, 9: 93-99.

    Google Scholar 

  8. Kane MD, Jatkoe TA, Stumpf CR, Lu J, Thomas JD, Madore SJ: Assessment of the sensitivity and specificity of oligonucleotide (50 mer) microarrays. Nucleic Acids Res. 2000, 28: 4552-4557. 10.1093/nar/28.22.4552.

    Article  CAS  Google Scholar 

  9. Weiler J, Gausepohl H, Hauser N, Jensen ON, Hoheisel JD: Hybridisation based DNA screening on peptide nucleic acid (PNA) oligomer arrays. Nucleic Acids Res. 1997, 25: 2792-2799. 10.1093/nar/25.14.2792.

    Article  CAS  Google Scholar 

  10. Broude NE, Woodward K, Cavallo R, Cantor DR, Englert D: DNA microarrays with stem-loop DNA probes: preparation and applications. Nucleic Acids Res. 2001, 29: e92-

    CAS  Google Scholar 

  11. Cheong C, Tinoco I, Chollet A: Thermodynamic studies of base pairing involving 2,6-diaminopurine. Nucleic Acids Res. 1988, 16: 5115-5122.

    Article  CAS  Google Scholar 

  12. Matray T, Gamsey S, Pongracz K, Gryaznov S: A remarkable stabilization of complexes formed by 2,6-diaminopurine oligonucleotide N3' → P5' phophoramidates. Nucleosides Nucleotides Nucleic Acids. 2000, 19: 1553-1567.

    Article  CAS  Google Scholar 

  13. Haaima G, Hansen H, Christensen L, Dahl O, Nielsen P: Increased DNA binding and sequence discrimination of PNA oligomers containing 2,6-diaminopurine. Nucleic Acids Res. 1997, 25: 4639-4643. 10.1093/nar/25.22.4639.

    Article  CAS  Google Scholar 

  14. Prosnyak MI, Veselovskaya SI, Myasnikov VA, Efremova EJ, Potapov VK, Limborska SA, Sverdlov ED: Substitution of 2-aminoadenine and 5-methylcytosine for adenine and cytosine in hybridization probes increases the sensitivity of DNA fingerprinting. Genomics. 1994, 21: 490-494. 10.1006/geno.1994.1306.

    Article  CAS  Google Scholar 

  15. Hoheisel JD, Craig AG, Lehrach H: Effect of 5-bromo- and 5-methyldeoxycytosine on duplex stability and discrimination of the NotI octadeoxynucleotide. Quantitative measurements using thin-layer chromatography. J Biol Chem. 1990, 265: 16656-16660.

    CAS  Google Scholar 

  16. Khudiakov IIa, Kirnos MD, Aleksandrushkina NI, Vaniushin BF: [2,6-Diaminopurine–a new adenine substituent base in the DNA of cyanophage S-2]. Dokl Akad Nauk SSSR. 1977, 232: 965-968.

    Google Scholar 

  17. Rackwitz HR, Scheit KH: The stereochemical basis of template function. Eur J Biochem. 1977, 72: 191-200.

    Article  CAS  Google Scholar 

  18. Vaish NK, Fraley AW, Szostak JW, McLaughlin LW: Expanding the structural and functional diversity of RNA: analog uridine triphosphates as candidates for in vitro selection of nucleic acids. Nucleic Acids Res. 2000, 28: 3316-3322. 10.1093/nar/28.17.3316.

    Article  CAS  Google Scholar 

  19. Hacia JG, Novotny EA, Mayer RA, Woski SA, Ashlock MA, Collins FS: Design of modified oligodeoxynucleotide probes to detect telomere repeat sequences in FISH assays. Nucleic Acids Res. 1999, 27: 4034-4039. 10.1093/nar/27.20.4034.

    Article  CAS  Google Scholar 

  20. Stump MD, Cherry JL, Weiss RB: The use of modified primers to eliminate cycle sequencing artifacts. Nucleic Acids Res. 1999, 27: 4642-4648. 10.1093/nar/27.23.4642.

    Article  CAS  Google Scholar 

  21. Fidanza JA, McGall GH: High-density nucleoside probe arrays for enhanced hybridization. Nucleosides Nucleotides. 1999, 18: 1293-1295.

    Article  CAS  Google Scholar 

  22. Timofeev E, Mirzabekov A: Binding specificity and stability of duplexes formed by modified oligonucleotides with a 4096-hexanucleotide microarray. Nucleic Acids Res. 2001, 29: 2626-2634. 10.1093/nar/29.12.2626.

    Article  CAS  Google Scholar 

  23. Hacia JG, Woski SA, Fidanza J, Edgemon K, Hunt N, McGall G, Fodor SP, Collins FS: Enhanced high density oligonucleotide array-based sequence analysis using modified nucleoside triphosphates. Nucleic Acids Res. 1998, 26: 4975-4982. 10.1093/nar/26.21.4975.

    Article  CAS  Google Scholar 

  24. Dorris DR, Ramakrishnan R, Trakas D, Dudzik F, Belval R, Zhao C, Nguyen A, Domanus M, Mazumder A: A highly reproducible, linear, and automated sample preparation method for DNA microarrays. Genome Res. 2002, 12: 976-984. 10.1101/gr.227402.

    Article  CAS  Google Scholar 

  25. Lebedev Y, Akopyants N, Azhikina T, Shevchenko Y, Potapov V, Stecenko D, Berg D, Sverdlov E: Oligonucleotides containing 2-aminoadenine and 5-methylcytosine are more effective as primers for PCR amplification than their nonmodified counterparts. Genet Analy. 1996, 13: 15-21. 10.1016/1050-3862(96)00139-8.

    Article  CAS  Google Scholar 

  26. Lesnik EA, Freier SM: Relative thermodynamic stability of DNA, RNA, and DNA:RNA hybrid duplexes: relationship with base composition and structure. Biochemistry. 1995, 34: 10807-10815.

    Article  CAS  Google Scholar 

  27. Gyi JI, Conn GL, Lane AN, Brown T: Comparison of the thermodynamic stabilities and solution conformations of DNA/RNA hybrids containing purine-rich and pyrimidine-rich strands with DNA and RNA duplexes. Biochemistry. 1996, 35: 12538-12548. 10.1021/bi960948z.

    Article  CAS  Google Scholar 

  28. Collins M, Myers RM: Alterations in DNA helix stability due to base modifications can be evaluated using denaturing gradient gel electrophoresis. J Mol Biol. 1987, 198: 737-744.

    Article  CAS  Google Scholar 

  29. Bailly C, Mollegaard NE, Nielsen PE, Waring MJ: The influence of the 2-amino group of guanine on DNA conformation. Uranyl and Dnase I probing of inosine/diaminopurine substituted DNA. EMBO Journal. 1995, 14: 2121-2131.

    CAS  Google Scholar 

  30. Lavigne M, Buc H: Compression of the DNA minor groove is responsible for termination of DNA synthesis by HIV-1 reverse transcriptase. J Mol Biol. 1999, 285: 977-995. 10.1006/jmbi.1998.2367.

    Article  CAS  Google Scholar 

  31. Marcourt L, Cordier C, Couesnon T, Dodin G: Impact of C5-cytosine methylation on the solution structure of d(GAAAACGTTTTC)2. An NMR and molecular modeling investigation. Eur J Biochem. 1999, 265: 1032-1042. 10.1046/j.1432-1327.1999.00819.x.

    Article  CAS  Google Scholar 

  32. Lefebvre A, Mauffret O, el Antri S, Monnot M, Lescot E, Fermandjian S: Sequence-dependent effects of CpG cytosine methylation. A joint 1H-NMR and 31P-NMR study. Eur J Biochem. 1995, 229: 445-454.

    Article  CAS  Google Scholar 

  33. Barnes III TW, Turner DH: C5-(1-propynyl)-2'-deoxy-pyrimidines enhance mismatch penalties of DNA:RNA duplex formation. Biochemistry. 2001, 40: 12738-12745. 10.1021/bi011033+.

    Article  Google Scholar 

  34. Ross DT, Scherf U, Eisen MB, Perou CM, Rees C, Spellman P, Iyer V, Jeffrey SS, Van de Rijn M, Waltham M, Pergamenschikov A, Lee JCF, Lashkari D, Shalon D, Myers TG, Weinstein JN, Botstein D, Brown PO: Systematic variation in gene expression patterns in human cancer cell lines. Nature Genetics. 2000, 24: 227-235. 10.1038/73432.

    Article  CAS  Google Scholar 

  35. Hill AA, Hunter CP, Tsung BT, Tucker-Kellogg G, Brown EL: Genomic analysis of gene expression in C. elegans. Science. 2000, 290: 809-812. 10.1126/science.290.5492.809.

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Abhijit Mazumder.

Additional information

Authors' contributions

AN performed the target preparations, microarray hybridizations, and data analyses, and drafted the manuscript. CZ performed the digestion and chromatography of the cRNAs. DD participated in design of the study and data analyses. AM conceived the study, participated in its design and coordination, and aided in drafting the manuscript.

Authors’ original submitted files for images

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nguyen, A., Zhao, C., Dorris, D. et al. Quantitative assessment of the use of modified nucleoside triphosphates in expression profiling: differential effects on signal intensities and impacts on expression ratios. BMC Biotechnol 2, 14 (2002). https://0-doi-org.brum.beds.ac.uk/10.1186/1472-6750-2-14

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/1472-6750-2-14

Keywords