3. Results

The performance of the Leginon system was analyzed in terms of efficiency and accuracy as compared to the performance of a human operator.

3.1 Efficiency of Image Acquisition

Currently, the automated system can acquire on average one high magnification image every 90 seconds; which translates to approximately 1000 images in a 24 hour period. This compares with an average time of 110s per image for a human operator using the computer assisted interface as described above. The principle bottlenecks are image acquisition time, network bandwidth for transferring images, and processing performance (identification of features of interest in the low magnification images and assessment of the quality of high magnification images). With planned improvements in the network and the efficiency of the software we are confident that the automated acquisition time can be reduced to less than 45s an image.

3.2 Accuracy and Precision of Microscope Controls

For automated acquisition it is necessary to (i) accurately control the goniometer in order to locate the features of interest in the center of the field of view of the low magnification images; (ii) maintain the feature of interest in the center of the field of view when the magnification is increased by several orders of magnitude; and (iii) accurately focus the image at high magnification. These steps require precise characterization of the response of the microscope goniometer, image shift and beam tilt coils. The accuracy with which the goniometer can be moved to a precise location is approximately 500nm and this does not allow for centering the feature of interest in a single step. Ideally, we would like to move with an accuracy of a single pixel on the low magnification image which would require accuracy to 30nm. We currently work around this problem by implementing a multi-step method in which the current location is refined by cross correlation to the target location.

Maintaining the feature of interest in the center of the field of view as the magnification changes also posed difficulties due to hysteresis effects at the very low magnification settings (660x). This occasionally causes the center of the feature of interest to shift as much as several hundred nm when returning to the low magnification setting. Again, we have implemented a refinement step to compensate for these effects.

3.3 Assessing the quality of the images acquired

The overall system performance was assessed by calculating the percentage of acceptable quality images acquired by the system during a session and comparing this performance with that achieved by a microscopist. As described above we can automatically assess the quality of our images by counting diffraction spots in the power spectrum. In order to evaluate the reliability of this method we compared the machine quality assessment to the quality assessment as made by a human observer. The human observer simply examined each power spectrum and counted the number of layer lines which could be observed (see fig. 3b). For each image, the machine quality assessment (the total number of diffraction spots with SNR >3.5) was then plotted against the number of observed layer lines (fig 5).

Figure 5.

The results show that there is good correlation (r²=0.98; n=2194) between the machine and human quality assessment. If we use the criterion of three or more layer lines as a measure of an acceptable images then this corresponds to a machine quality assessment of greater than 65. Thus automated assessment of image quality provides a reliable measure of an acceptable image and could be used to evaluate the overall performance of the system.

In order to compare the automated system to a microscopist we used the computer assisted interface, described above, to establish a baseline for the performance of a human operator. In this experiment, the microscopist identified the location of the features of interest on the low magnification image. A total of 288 high magnification images were acquired in this way and 79% of these were acceptable as defined above. In comparison, using the same grid, the fully automated image acquisition system was used to acquire 380 images of which 51% were acceptable.

Analysis of these results indicated a correlation between average feature intensity and image quality [fig 6]. This feature intensity is related to the thickness of the catalase crystal and indicates that thinner crystals result in more acceptable images. The fully automated target selection criteria was therefore further refined by incorporating an assessment of crystal thickness into the model. By acquiring high magnification images of only those features that have an average intensity greater than a preset threshold the percentage of acceptable images can be significantly improved. For example, if the threshold is set to 6000, the percentage of acceptable images improves to 86% from a baseline of 51%.

Figure 6.

Page 4 of 7