Multimodal ensemble of UAV-borne hyperspectral, thermal, and RGB imagery to identify combined nitrogen and water deficiencies in field-grown sesame

Hyperspectral reflectance as well as thermal infrared emittance unmanned aerial vehicle (UAV)-borne imagery are widely used for determining plant status. However, they have certain limitations to distinguish crops subjected to combined environmental stresses such as nitrogen and water deficiencies. Studies on combined stresses would require a multimodal analysis integrating remotely sensed information from a multitude of sensors. This research identified field-grown sesame plants’ combined nitrogen and water status when subjected to these treatment combinations by exploiting the potential of multimodal remotely sensed dataset. Sesame (Sesamum indicum L.; indeterminate crop) was grown under three nitrogen regimes: low, medium, and high, combined with two irrigation treatments: well-watered and water limited. With the removal of high nitrogen treated sesame plots due to adverse effects on crop development, the effects of combined treatments were analyzed using remotely acquired dataset- UAV-borne sesame canopy hyperspectral at 400 – 1020 nm, red–green–blue, thermal infrared imagery, and contact full range hyperspectral reflectance (400 – 2350 nm) of youngest fully developed leaves in the growing season. Selected leaf traits- leaf nitrogen content, chlorophyll a and b, leaf mass per area, leaf water content, and leaf area index were measured on ground and estimated from UAV-borne hyperspectral dataset using genetic algorithm inspired partial least squares regression models (R² ranging from 0.5 to 0.9). These estimated trait maps were used to classify the sesame plots for combined treatments with a 40 – 55 % accuracy, indicating its limitation. The reduced separability among the combined treatments was resolved by implementing a multimodal convolutional neural network classification approach integrating UAV-borne hyperspectral, RGB, and normalized thermal infrared imagery that enhanced the accuracy to 65 – 90 %. The ability to remotely distinguish between combined nitrogen and irrigation treatments was demonstrated for field-grown sesame based on the availability of ground truth data, combined treatments, and the developed ensembled multimodal timeline modeling approach.