HybridHunter

LoRA placement experimentation framework for hybrid language models.

🎯 Purpose

Replicating and extending the LoRA placement hybrid study (2604.22127v1) for Qwen3.6-27B-MTP on otter_den's RTX 4090.

🔍 Research Questions

1. Does attention-only LoRA remain optimal in Qwen3.6-27B-MTP?
2. How does MTP integration affect component-type placement?
3. What efficiency ratios can we achieve?

🏗️ Project Structure

hybridhunter/
├── notebooks/          # Jupyter notebooks for experiments
├── src/                # Python modules
├── configs/            # Training configurations
├── results/            # Experiment outputs
│   ├── discovery/      # Module discovery manifests
│   ├── checkpoints/    # Trained LoRA adapters
│   └── eval_details/   # Per-instance evaluation
├── tables/             # Aggregate results
├── figures/            # Publication figures
└── docs/               # Documentation

🚀 Quick Start

# Install dependencies
pip install -r requirements.txt

# Run discovery
jupyter notebook notebooks/01_model_discovery.ipynb

# Run training
jupyter notebook notebooks/02_training.ipynb

📚 References

• Paper 2604.22127v1 — "Where Should LoRA Go?"
• Original replication

🧠 Fox Wisdom

"The best trap is one the prey doesn't know exists."
— Reynard the Overseer

Let the attention pathway reveal itself through its precision, not its weight.