Hong Lab

The Hong lab is a radiation oncology lab with a focus on machine learning and individualized clinical care. I am leading computer vision efforts within the Hong Lab. My current research focuses on applying deep learning frameworks to H&E stained, post-radical prostatectomy whole slide images and tabular clinical data. We hope to classify patterns of recurrence and Gleason grade four subtypes to improve prognostication and treatment planning for prostate cancer.

Example Pipeline

Tabular Clinical DataWSIPatch EncodingSlide EncodingPatient-level FusionClassification Output

Example pathology whole slide image and clinical data processing workflow. Feature encoder pairings may include Gigapath or Conch + Titan.
Honors Thesis: DANTURESDaylight Associated Noise for Training UAV-based Multispectral Detectors, Robust to Environmental Shifts

Abstract: Multispectral object detection has applications in defense, agriculture, and industry. This thesis explores improvements to UAV-based object detection by embedding environmental variance into multispectral models. Combining forward-looking infrared (FLIR) and RGB cameras, I address detection accuracy challenges caused by infrared variations and RGB visibility issues across changing times of day.

I implemented novel adaptations of the YOLOv7 framework integrating RGB, infrared, and time-of-day information through custom fusion layers, gated multimodal units (GMU), and cross-attention mechanisms. The work introduces several extensible PyTorch layers (DualLayer, FusionLayer, SymmetricCrossAttention) that enable time-conditioned modality fusion for improved detection across environmental conditions.

Architecture Overview

RGBIRTime-of-DayEmbeddingFusion Layer(DualLayer +FusionLayer)Gated / GMU /Cross-AttentionYOLOv7Backbone(FeatureExtraction)DetectionHeadBounding Boxes+ ClassesOutputDetections
GMU approach batch 1GMU Approach Batch 1
GMU approach batch 2GMU Approach Batch 2
Learned fusion batch 1Learned Fusion Batch 1
Learned fusion batch 2Learned Fusion Batch 2
Highlighted model outputs. We see that GMU and Learned Fusion approaches predict with high confidence but struggle to distinguish between car and truck features, likely due to dataset issues. We believe that with refinement, this would quickly outperform the baseline.
Feature activation heatmap 1Feature activation heatmap 2Feature activation heatmap 3Feature activation heatmap 4
Heatmap visualization of feature activations from my mid-fusion approach
Pre-sunrise/post-sunset fusionPre-sunrise / Post-sunset
Post-sunrise or pre-sunset fusionPost-sunrise or Pre-sunset
Noon fusionNoon
Early fusion results at different time-of-day labels

Top Model Performance (mAP@0.5)

MethodPrecisionRecallmAP@0.5mAP@0.5:0.95
IR Only (Full)0.9310.9250.9660.637
Early Fusion (Perceptron)0.8990.8890.9330.580
GMU (Baseline)0.8810.9120.9430.586
Early Fusion (Manual)0.8590.9100.9230.574
Middle Fusion (Learned)0.9100.8490.9260.572
GMU (Learned)0.8490.9060.9290.564
RGB Only (Full)0.8760.8290.8880.485
Cross-Attention (Learned)0.7930.8610.8740.518

Key Contributions: Created a time-labeled RGB-IR dataset of 12,000+ image pairs; achieved 93.3% mAP@0.5 with perceptron-based early fusion; demonstrated adaptive gating can potentially learn more meaningful feature representations across varying lighting conditions while maintaining real-time performance.

GitHub RepositoryFull Thesis (PDF)