Dragonfly, a CNCF Graduated P2P file distribution system, now natively supports hf:// and modelscope:// URL protocols for downloading AI models directly from Hugging Face and ModelScope hubs. Instead of each GPU node independently downloading large models (e.g., a 130 GB DeepSeek-V3 across 200 nodes = 26 TB of origin traffic), Dragonfly's seed peer fetches the model once and distributes pieces across the P2P mesh, reducing origin traffic by ~99.5%. The new backends are implemented in Rust using a pluggable Backend trait, support authentication tokens, revision pinning, recursive repository downloads, and single-file downloads. Key use cases include multi-node GPU cluster deployments, CI/CD ML pipelines, air-gapped environments, and dataset distribution for training jobs. The feature is available via dfget CLI and integrates natively with Kubernetes via Helm.
Table of contents
The problem: AI model distribution is broken at scaleWhat Is Dragonfly?Introducing native model hub protocols in DragonflyThe hf:// Protocol — Hugging Face hubThe modelscope:// Protocol — ModelScope hubUnder the hood: Technical deep diveReal-world impact: Where this mattersComparison: Why not just use platform CLIs?Getting startedWhat’s nextContributingConclusionSort: