More Research

What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits

¹Harish Babu Manogaran, ¹M. Maruf, ⁶Arka Daw, ¹Kazi Sajeed Mehrab, ¹Caleb Charpentier, ¹Josef Uyeda, ²Wasila Dahdul, ³Matthew J Thompson, ³Elizabeth G Campolongo, ³Kaiya L Provost, ⁴Paula Mabee, ⁵Hilmar Lapp, ¹Anuj Karpatne

¹Virginia Tech, ²University of California, Irvine, ³The Ohio State University, ⁴Battelle, ⁵Duke University, ⁶Oak Ridge National Lab,

harishbabu@vt.edu, karpatne@vt.edu

Paper Code

HComP-Net

We introduce an hierarchical interpretable image classification framework called HComP-Net that can be applied to discover potential evolutionary traits from images by making use of the Phylogenetic tree also called as Tree-Of-Life. HComP-Net idenitfies evolutionary traits by means of prototypes that are representative of the visual trait that is unique to a species or a group of species that share a common ancestor in the phylogeny. HComPNet can generate hypothesis for potential evolutionary traits by learning semantically meaningful non-over-specific prototypes at each internal node of the hierarchy. In order to discover evolutionary traits through prototypes, our approach features the following,

Hierarchical Prototypes: We learn prototypes at each internal node of the phylogeny such that each prototype captures a evolutionary trait of the species that are leaf descendants of the node.
Avoiding Over-specificity: Over-specific prototypes represents traits that are specific to one or few of the leaf descendant species, rather than representing a common shared trait. In order to avoid over-specificity.

We introduce a novel over-specificity loss that enforces each prototype to represent a common trait of all leaf descendants.
We introduce masking module that helps to idenitfy and discard over-specific prototypes that are learned by the model.

Learning Discriminative Traits: We introduce discriminative loss to ensure the prototypes represent traits that are absent in the contrasting set of species that have evolved from a different ancestor.

In addition, we also adopt self-supervised learning of prototypes over the hierarchy to ensure the prototypes are semantically consistent, such that they always represent the same visual concept in different images.

Figure 2: Schematic illustration of HComP-Net model architecture.

Results

Classification Performance of HComP-Net

The comparison of the fine-grained accuracy calculated for HComP-Net and the baselines on Bird, Fish, and Butterfly datasets are given below

% Accuracy
Model	Hierarchy	Bird	Butterfly	Fish
ResNet-50	No	74.18	95.76	86.63
INTR	No	69.22	95.53	86.73
HPnet	Yes	36.18	94.69	77.51
HComP-Net	Yes	70.01	97.35	90.80

Semantic Quality of Prototypes

We calculate the semantic quality of each prototype by means of part purity on Bird dataset. Prototypes that represent the same part (eye, beak, tail, etc.) in Top-10 closest images have part purity close to 1. We report the mean and standard deviation of the part purity of all the prototypes learned. % masked denotes the percentage of prototypes that were identified as over-specific and removed using masking module.

Part purity of prototypes on Bird dataset.
Model	L_ovsp	Masking	Part purity	% masked
HPnet	-	-	0.14 ± 0.09	-
HComP-Net	-	-	0.68 ± 0.22	-
HComP-Net	-	✓	0.75 ± 0.17	21.42%
HComP-Net	✓	-	0.72 ± 0.19	-
HComP-Net	✓	✓	0.77 ± 0.16	16.53%

Prototype Visualizations

Figure 3: Visualizing the hierarchy of prototypes discovered by HComP-Net for birds and fishes. *Note that the textual descriptions of the hypothesized traits shown for every prototype are based on human interpretation.

Figure 4: We trace the prototypes learned for Western Grebe at three different levels in the phylogenetic tree (corresponding to different periods of time in evolution). Text in blue is the interpretation of common traits of descendants found by HComP-Net at every ancestor node of Western Grebe.

Figure 5: Visualizing the hierarchy of prototypes discovered by HComP-Net over three levels in the phylogeny of seven species from Butterfly dataset. For each prototype, we visualize one image from each of its leaf descendants.

References

Please cite our paper if you use our code, data, model or results.

      @article{manogaran2024you,
        title={What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits},
        author={Manogaran, Harish Babu and Maruf, M and Daw, Arka and Mehrab, Kazi Sajeed and Charpentier, Caleb Patrick and Uyeda, Josef C and Dahdul, Wasila and Thompson, Matthew J and Campolongo, Elizabeth G and Provost, Kaiya L and others},
        journal={arXiv preprint arXiv:2409.02335},
        year={2024}
      }

Acknowledgements

This research is supported by National Science Foundation (NSF) awards for the HDR Imageomics Institute (OAC-Award #2118240). We are thankful for the support of computational resources provided by the Advanced Research Computing (ARC) Center at Virginia Tech. This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains, and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript or allow others to do so for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (https://www.energy.gov/doe-public-access-plan).