3 Representative images of the 17 pollen types. Images from different angles are presented for selected pollen types. a: Asteraceae (AS), b: Coprosma (CO), c: Echium (EC), d, e: Geniostoma (GE), f: Griselinia (GR), g: Ixerba brexiodes (IX), h, I: Knightia excelsa (KN), j, k: Leptospermum/Kunzea type , l: Lotus (LO), m, n: Lycopodium (LY), O, p: Metrosideros (ME), q: Poaceae (PO), r: Quintinia (QU), s: Salix (SA). t: Taraxacum (TA), u: Trifolium (TR), v: Weinmannia (WE) 

Contexts in source publication

Context 1

... are the most similar amongst the species in the data set. Similar observations can be made from the confusion matrices for the SVM and RF models and so their confusion matrices are not displayed here. Table 12.4 shows the full confusion matrix for the LDA model on the COMB data. The two species with the lowest classification accuracy are PA with 69 % and IP with 72 %. The high number of IP observations accounts for most of the model’s overall classification error. Another feature of the confusion matrix is that all species other than AR are confused with the PA species. Despite this, the error is more balanced between species which would appear to be a desirable result. In practice, the classification results need not be the end point of an investigation. Typically a palynolgist would review and adjust the classification results. The LDA model presents a simple means for assisting the review stage. LDA works by transforming the data into an optimal space for discrimination. For each species, a mean value for each discriminant is then calculated. The model then measures the Mahalanobis distance between a sample and the set of mean discriminants for each species, with the lowest distance informing the choice of species for classification. This distance itself provides a measure of how far from the training data a particular sample is. Taking the ratio of lowest over the second lowest distance score gives a measure of how ‘borderline’ a classification decision is. So, for example, a ratio close to 0 would indicate a strong decision while a value close to 1 would indicate possible confusion. Table 12.5 shows the top 20 classification results for the LDA model on the COMB FF data ranked on decreasing values of this ratio. It shows that out of the 20 results only 4 are correctly classified. So in the context of reviewing the data, if a user were to sort their observations based on the ratio, they could easily and efficiently adjust decisions for the most borderline cases. The ratio is similar in spirit to the posterior probability which can be calculated for the LDA models and the RF models. For each classification, a posterior probability is assigned for each class. It is then possible to use this to rank the data in a similar fashion to the ranking, noting that a posterior probability close to 1 corresponds to a strong decision. This would allow the user to use the RF model to perform a similar type of assisted review. Another, more automated approach is to simply exclude a proportion of the classification results based on the ratio. For example, after ranking the results on decreasing values of the ratio, one can exclude the worst N percent of the results. For the LDA model on the COMB FF data, if we exclude 20 % of the results based on this strategy, the overall performance of the classifier increases to 0.94. This compared to the performance 0.84, when all data is used, is a significant increase. If we excluded 50 % of the data the performance increases to 0.99. However, when one examines the confusion matrix corresponding to only 50 % of the data, the relative proportions of the species are modified in a reasonably substantial way. For example, the IP, NP and AP species account for 18, 15 and 10 % of the species present in the COMB FF data, respectively. When we exclude the worst 20 % of the data, the relative percentages are 16, 16 and 11 which is not too dissimilar. However, when we exclude 50 % of the data, the relative percentages are 5, 23 and 13 which is very different to the known abundances. So if the goal of a palynologist is to study relative abundance of species in a sample, one would need to find an appropriate percentage for exclusion which would preserve relative abundance. In this section we work through a real world case study to demonstrate the use of the Classifynder and how various classification strategies employing neural networks and linear discriminant analysis can be incorporated into the workflow. The data source for this section is a suite of slides containing pollen extracted from 45 different New Zealand honeys. The pollen composition of these slides has already been determined through manual counting, with the results presented (as proportions) by Mildenhall and Tremain [23]. Across the 45 samples, Mildenhall and Tremain recognize 21 common pollen types, however, the individual classifications at the image level are unavailable. The slides also contain Lycopodium marker spores, which are added to determine pollen concentration per gram of honey [31]. A subset of slides were selected to supply images for a library. These slides were scanned on the Classifynder and the resulting images manually classified and sorted into folders to create a master set (Table 12.6). Insufficient numbers of images were obtained for 5 of the 21 common honey pollen types, preventing the generation of a library class for these types. The final library thus contained images of 17 taxa (Table 12.6 and Fig. 12.3). The image set for each taxon was divided into two, with one half to be used for training the classifiers, and the other half for testing. This training and testing set was used to train and test both the Classifynder neural network classifier and the LDA classifier. The software of the Classifynder trains the neural network by splitting the training set provided to it and first training itself on half of this test set. It then tests its performance on the other half of the training set. It then takes any misclassified images and adds them to its training set file and retests itself in an attempt to improve performance of the neural network. Once this automated training and testing was complete, the neural network was tested on the separate manually generated test set. Classification accuracy on the training set was 89 %. The LDA classifier was also applied to the training and test sets also achieving an accuracy of 89 %. Once trained, both classifiers were applied to image sets gathered from three different slides from the original 45 slide set, none of which had been used as a source of images for training and testing. The three slides contain a different combination and number of the 17 taxa in the training set. These three slides were scanned on the Classifynder. The Classifynder software sourced preliminary morphological limits from the neural network for use in identifying pollen grains on the slides at low magnification. These were then imaged using the high resolution camera. The resulting image sets were then classified by the two different classifiers (Classifynder NN and LDA). In assessing the performance of the classifiers, allowances had to be made for the fact that these ‘real world’ datasets contained numerous images that were of objects not present in the training set. These images fell into one of several categories described in Table 12.7. When assessing the accuracy of the classifications (i.e., proportion of images correctly classified), images assigned to these categories by the analyst were omitted from the calculations. Because both classifiers are forced, they have to classify every image as something within the training set. Therefore, because these types of image were not included in the training set, it is not reasonable (or in many cases possible) to expect correct classification. Although, in some cases images of clumps comprising grains of the same taxon were correctly classified as that taxon. Likewise, images of clumps of different taxa were sometimes classified as one of the taxa in the clump. But because at this stage there is no mechanism to adequately deal with clumps (i.e., recognizing multiple grains in the one image) we have chosen to omit these altogether. As discussed earlier, in contrast to the NN, the LDA provides an indication of the strength of the classification of each image and these scores potentially provide a basis to further improve the results of the classification. The majority of incorrectly classified images have very high (i.e., weak) classification strength scores. When examining these, many are images which belong to the clumps, unknown, other or junk categories. One possible option to improve the classification is to automatically ignore or disregard a portion of images with the highest (weakest) classification strength scores. We have assessed this approach with two subsets of the LDA classification of the initial complete image set, which take the top 80 % (LDA80) and top 50 % (LDA50) of the classified images, based on their classification strength scores, and then calculated the proportions for these in the same way as was done with the previous classifications. Results are presented in Table 12.8. After assessing classification accuracy, any incorrectly classified images were sorted into their correct categories manually by a human palynologist, and the proportions of the different taxa as detected by the Classifynder’s imaging systems calculated. These are presented in Table 12.9, and compared with the original human counts of the samples/slides, as presented in [23]. This data is a measure of how well the Classifynder can find and image the pollen using the basic morphological information extracted from the neural network, but it still has an element of manual classification through the correcting of misclassifications. These proportions are calculated by taking the sum of all images of single pollen grains (i.e., no clump images) of a taxon (whether correctly classified or not) and dividing it by the sum of all images of single pollen grains, excluding images of non-pollen, clumps, and unknown, and also Lycopodium spores. Images of clumps, unknowns, other pollen and junk are also excluded, for the reasons mentioned earlier Lycopodium is excluded because the number of Lycopodium grains in each slide was not originally presented by Mildenhall and Tremain [23]. Lycopodium spores are added to honey samples to aid in determination ...

Context 2

... intervals are calculated for the human counts (following the method of [21], and assuming a total count of 500 grains), the Classifynder counts fall within these limits. The degree of difference between the original human counts and the Classifynder counts is much greater for slide 2, with the Classifynder values lying outside the 95. The raw proportions produced by the classifiers (i.e., any misclassifications left uncorrected) are also compared with the two sets of ‘true proportions’ for each slide in Table 12.10. These proportions are calculated by taking the total number of images classified as a taxon, whether correct or not, is then divided by two different totals to give two sets of proportions. The first total includes all images captured, including clumps, non-pollen, other and unknown pollen, but excludes images of Lycopodium spores (for reasons discussed previously). The second total excludes images of clumps, non-pollen, other pollen, unknown pollen, and Lycopodium . The purpose of these two different totals is to investigate what impact the presence of junk, clumps and unrecognizable pollen has on the raw results and to determine whether there are any taxa which these types of images are more likely to be classified as. The root mean square error calculated is also presented to provide an additional indication of the success of each classification. The first point of interest from Table 12.8 is that the LDA performs slightly better than the NN on 2 out of the 3 slides. When the best 80 and 50 % sets are considered, the LDA performance is significantly better than that of the NN. However, this is not true for slide 2 where the performance of the LDA is only comparable to the NN when the best 50 % of the data is retained. It is unclear why this difference occurs. One possibility is that the number of taxa present in slide 2, based on human counts, is larger (10 for slide 2 compared with, 5 and 3 for slides 12 and 26) and has an impact on classification, although, our initial assessments of classifier performance (in Table 12.2 for example) suggest comparable performance for a relatively similar number of classes. Another notable issue is highlighted in Table 12.10. When proportions are compared with the manual classifications, the majority are outside the 95 % confidence limits (for the human counts). Further to this, there are many instances where images have been classified as taxa which are not actually present in the sample (i.e., ‘false positives’). In some cases these false positives comprise in excess of 20 % of the total (i.e., ‘LDA’ classification for slide 2 in Table 12.10). False positives are particularly problematic to the ideal goal of applying automated classification to routine/commercial honey pollen analysis. Obviously, it is far from satisfactory to have the proportions of the taxa present in the sample incorrect. But reporting the presence of species in a sample which aren’t actually there (or completely omitting types which are present) adds an additional layer of complication. Often, certain pollen types are specific to a particular geographic region, and presence/absence information (regardless of abundance) can be used to trace where a honey sample has come from. Therefore false positives/negatives can further influence the interpretation of the results, beyond just determining the nectar contributions. Even though reducing the images sets based on classification strength scores improves classification accuracy overall, it does not deal to the problem of false positives, with 3.7 % FP remaining in slide 26 (L5*). In all cases, ignoring the images of clumps, unknowns, other and junk has resulted in proportions closer to actual (i.e., lower RMSE), indicating that these are in part responsible for inaccurate results. However, when the proportions for the classifications with reduced images calculated against the sum excluding the clumps, unknowns, other and junk are examined, the proportions are still outside the 95 % confidence limits in most cases. This indicates that not all error can be explained by the presence of these types of images, and that some of the error is still resulting from confusion. Some of the most common confusions in both the NN and LDA classifications (excluding clumps, junk, etc.) are between the following: • ‘LE’, ‘EC’, ‘LO’ and ‘ME’ (e.g. slides 12 and 26) • ‘TR’, ‘GE’, ‘GR’ and ‘ME’ (e.g. slides 2 and 26) Confusions of images as EC, GE or GR are responsible for many of the false positive instances discussed earlier. Confusions between LE, EC, and LO seem logical, due to the similarities in size (Fig. 12.1). Likewise, confusions between LE and ME are likely related to overall morphological similarity, even though there is a reasonable size difference. Also, in the case of slide 12, ME was by far the dominant pollen type, so therefore it is likely to be best represented in misclassifications. TR and GR pollen types are also morphologically similar (Fig. 12.3), although less so with GE. Many of these more common confusables have relatively low numbers of images in their training sets (Table 12.6). Perhaps notably, EC, GE and GR, none of which are actually present in any of the samples all have less than 50 images in their training sets. Likewise, TR, which is a dominant pollen type in 2 out of the 3 slides has only 92 images in its training set. Therefore, confusions between TR, GR and GE may be explained by undertraining. LE, LO and ME have higher numbers in their training sets (greater than 120) so undertraining is likely not the problem for confusions between these three, but can explain confusions between them and ...