More KABR Components

KABR: In-Situ Kenyan Animal Behavior Recognition

Data and Models KABR Tools Package KABR Tools Docs

KABR lightly overlayed on an image of zebras in a field

KABR: Kenyan Animal Behavior Recognition from Drone Videos

We present a collection of novel in-situ datasets, methods, and models for animal behavior recognition from drone videos. These video and mini-scene datasets were curated from drone videos of wildlife taken at the Mpala Research Centre in Kenya in January 2023. The datasets contain giraffe, plains zebra, and Grevy’s zebra behaviors. We focus on seven types of behaviors along with an additional category for occluded views. Altogether, this collection is a valuable resource for experts in both machine learning and animal behavior:

It provides challenging data for the development of new machine-learning algorithms for animal behavior recognition. It complements recently released, larger datasets that used videos scraped from online sources because it was gathered in-situ and therefore is more representative of how behavior recognition algorithms may be used in practice.
It demonstrates the effectiveness of a new animal behavior curation pipeline for videos collected in-situ using drones.
It provides a test set for evaluating the impact of a change in fieldwork protocols by research scientists studying animal behavior to the use of drones and recorded videos.

Furthermore, the raw video datasets support various computer vision tasks including:

Animal Detection: Detecting giraffes, plains zebras, and Grevy's zebras in natural settings.
Animal Tracking: Following individual animals across video sequences.
Behavior Analysis: Developing methods for temporal behavior recognition.
Environmental Context Analysis: Studying animal behavior in relation to environmental factors.
Video Preprocessing: Developing pipelines for extracting regions of interest from drone footage.
Multi-object Tracking: Tracking multiple animals simultaneously in wide-field drone videos.

KABR Datasets

The overall KABR project contains multiple datasets, whose raw data was collected at the Mpala Research Centre in Laikipia, Kenya. Specifically, these datasets are comprised of a combination of scan and focal obersvations, drone videos, telemetry data for the drone flights, and mini-scenes derived from the drone videos.

KABR Mini-Scenes

The KABR mini-scene dataset was the first published, and is described in greater detail on the KABR Mini-Scene Website. This dataset is comprised of cropped drone videos of giraffes, plains zebras, and Grevy's zebras at the Mpala Research Centre with behavior labels. Raw drone videos for the KABR mini-scene dataset are coming soon to the KABR Mini-Scene Raw Video Dataset.

KABR Methodology

The KABR Methodology Dataset contains time-budget data, focal observations, scan samples, and object-detection annotations for African ungulates, recorded both from the ground and from drones. This dataset complements the original KABR Mini-Scene Dataset, providing ground-based sampling to correspond with a subset of the published mini-scenes. Specifically designed to compare Kenyan animal behavior observation methods, this dataset supports object detection, behavior classification and time-budget analyses from per-individual CSV logs, and methodological comparisons between focal, scan, and drone-based sampling.

The KABR Worked Examples Dataset is comprised of manually annotated bounding box detections, mini-scenes, behavior annotations, and associated telemetry for three drone video sessions that were used for kaber-tools case studies. These examples were derived from the KABR Raw Videos, described below.

KABR Raw Videos

There are two raw video datasets:

The KABR Mini-Scene Raw Videos dataset contains the raw drone videos from which the KABR mini-scenes were derived.
The KABR Raw Videos dataset contains the raw drone videos that were not used for the original KABR mini-scene dataset.

Both raw video datasets contain original drone footage from the Mpala Research Centre in Kenya, taken during the same three-week period in January 2023; they were just processed for different tasks.

KABR Telemetry

The KABR Telemetry Dataset contains the flight telemetry data associated with the KABR mini-scene dataset. This telemetry dataset contains information about the status drone during the missions, including location and altitude, along with the bounding box dimensions of the wildlife in the frame and behavior annotation information. It is intended to be used to provide guidance on executing wildlife behavior collection missions with drones, which can be conducted by drone pilots manually, or integrated into an autonomous navigation framework.

The KABR Datapalooza 2023 Subset consists of synchronized drone telemetry, camera metadata, behavior annotations, and drone status data collected during wildlife monitoring operation for initial mini-scene release. The data was captured using drones equipped with cameras to observe animal behavior in natural habitats.

KABR Methods

Our approach to curating animal behaviors from drone videos is a method that we refer to as mini-scenes extraction. We use object detection and tracking methods to simulate centering the camera’s field of view on each individual animal and zooming in on the animal and its immediate surroundings. This compensates for drone and animal movements and provides a focused context for categorizing individual animal behavior. To learn more about the mini-scenes extraction process, please see the KABR Mini-Scene Website and the KABR methods paper (coming soon!).

Since the initial KABR mini-scene publication, we have developed the kabr-tools package, which can be used to implement the full mini-scenes extraction pipeline, and used to create similar datasets with other animal videos. It is not exclusively for generating mini-scenes, as the intermediate stages of the pipeline can be used for other tasks, such as object detection and tracking. Please see the kabr-tools documentation for more information about the package and how to use it.

The pipeline, as used for the KABR mini-scene dataset and KABR Worked Examples dataset, is visually represented below; and a step-by-step description of the process is provide in the kabr-tools GitHub repository.

A visual representation of the pipeline used to create the mini-scenes for the original KABR dataset

KABR Model

The X3D KABR Kinetics Model is a behavior recognition model for in situ drone videos of zebras and giraffes, built using X3D model initialized on Kinetics weights. It was trained on the KABR mini-scene dataset, which is comprised of 10 hours of aerial video footage of reticulated giraffes (Giraffa reticulata), Plains zebras (Equus quagga), and Grevy’s zebras (Equus grevyi) captured using a DJI Mavic 2S drone. It includes both spatiotemporal (i.e., mini-scenes) and behavior annotations provided by an expert behavioral ecologist.

This model is available for use with the kabr-tools package, though custom models are easily inserted into this pipeline as well.

KABR Collection Papers

KABR: In-Situ Dataset for Kenyan Animal Behavior Recognition from Drone Videos
This is the original KABR paper, describing the mini-scene dataset, process, and benchmarking behavior recognition models.
A Framework for Autonomic Computing for In Situ Imageomics
In this vision paper, we make the case for a research agenda for in situ imageomics that depends on significant advances in autonomic and self-aware computing. Precisely, we seek autonomous data collection that manages camera angles, aircraft positioning, conflicting actions for multiple traits of interest, energy availability, and cost factors.
Integrating Biological Data into Autonomous Remote Sensing Systems for In Situ Imageomics
We analyzed the flight telemetry data from portions of UAV videos (from the KABR Mini-Scene Raw Videos dataset) that were usable for inferring wildlife behavior, and demonstrate how these insights can be integrated into an autonomous remote sensing system to track wildlife in real time.
Deep dive into KABR: a dataset for understanding ungulate behavior from in-situ drone videos
This paper extends the dataset statistics, model benchmarks, and performance analysis for the KABR mini-scene dataset (beyond that done in the original KABR paper, listed above).

Please cite the appropriate paper(s) and associated artifact(s) if you use them in your research.

Acknowledgements

This work was supported by the Imageomics Institute, which is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under Award #2118240 (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Additional support was also provided by the AI Institute for Intelligent Cyberinfrastructure with Computational Learning in the Environment (ICICLE), which is funded by the US National Science Foundation under Award #2112606. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

We thank the field experts at Mpala Research Centre for their support with data collection and logistics.