Magnetite

Self-hosted BitTorrent DHT crawler, classifier and torrent search engine. A Rust + React rebuild of bitmagnet that is a drop-in replacement for it (same Torznab API, import format, classifier rules and ports), with:

• Login, roles and API keys (admin / user / viewer), audit log
• Instant indexing: new torrents are searchable seconds after the crawler finds them
• Torrent sites as extra sources: YTS, EZTV and The Pirate Bay are polled for new uploads (or backfilled), and their torrents are downloaded from the DHT with full file lists and the site's IMDb/quality/episode info as classifier hints
• SOCKS5 proxy support: send all DHT, metadata and tracker traffic (and its DNS) through a proxy, with no fallback to the host network
• Fast search with facets and exact counts (within a time budget; estimates only when a count is slow)
• Torrent health: seeders/leechers from public trackers (UDP and HTTP(S)) and from the DHT itself (BEP 33 scrapes), re-checked daily (busy torrents every few hours), with history
• Trending: most active torrents and the fastest risers over 24 hours or 7 days, per content type
• Classifier editor in the browser with a dry run on real torrents before you save
• Import from bitmagnet: copy an existing bitmagnet database in one job
• About 4 KB per torrent on disk (bitmagnet: ~8 KB) and under 4 GB RAM for the whole stack

No TMDB: classification is local and never waits on an external API.

Quick start

cp .env.example .env    # change the database passwords
docker compose up -d --build

1. Open http://localhost:3333 and create the admin account.
2. Choose how to connect: the host network or a SOCKS5 proxy. Nothing is crawled until you do; you can change it later in Admin → Network.
3. Within a minute the crawler joins the DHT; watch Stats.
4. Settings → API keys → New key for Prowlarr/Sonarr/Radarr and scripts.

On the host network the crawler uses UDP 3334. Forwarding that port on your router (or VPN) lets other DHT nodes reach you, which helps discovery.

With a SOCKS5 proxy, the proxy must support UDP ASSOCIATE (the DHT is UDP). Most VPN providers' SOCKS5 proxies and Dante do; ssh -D and Tor do not. The crawler (DHT, metadata downloads) and the worker's tracker health checks all use it, and host names are resolved with DNS over TCP through the proxy. If the proxy is unreachable or drops the connection, nothing is sent: the crawler retries (Status shows why) and never falls back to the host network. Changing the setting restarts the crawler (Docker brings it back).

Torrent sites

Admin → Sources turns on polling of YTS (movies), EZTV (TV) and The Pirate Bay (recent uploads); all are off by default. The worker fetches each site's newest uploads every few minutes over the chosen network and sends hashes the catalog lacks to the crawler, which downloads the metadata from the DHT (torrents nobody seeds are retried three times, an hour apart, then dropped). Backfill history pages through a site's whole catalog (YTS, EZTV) as a job. Sites move domains often; change the API address there when one stops answering. cargo test -p magnetite-scrapers --test live -- --ignored checks the sites still parse.

Download pipeline

Every hash the crawler finds on the DHT is downloaded (its metadata, from peers) unless the catalog already has it. When the download queue is full, hashes wait in a backlog on disk (NATS) instead of being dropped: hot ones first (seen on the DHT twice, or announced with a peer), then the rest. A download that finds no peers, or whose peers all fail, is retried after 1 hour, 6 hours and 24 hours, with a deeper DHT lookup, then given up. Peers that other peers mention (PEX) are tried too. Stats → Download pipeline shows where hashes went and why peer connections failed.

Sonarr / Radarr / Prowlarr

Add a Generic Torznab indexer:

• URL: http://<magnetite-host>:3333/torznab
• API key: a key from Settings → API keys

Same endpoint, parameters, categories and XML as bitmagnet, so an existing bitmagnet indexer only needs the API key added (or set MAGNETITE_TORZNAB_REQUIRE_API_KEY=false on the api service to keep it keyless on a trusted network). Extra profiles (/torznab/<id>/api) can be configured with MAGNETITE_TORZNAB_PROFILES.

Coming from bitmagnet

• Torrents: Admin → Jobs → Import from bitmagnet with your bitmagnet database URL (postgres://user:pass@host:5432/bitmagnet). Torrents are re-classified on the way in.
• Classifier rules: your classifier.yml works as-is: put it in config/classifier.yml (mounted at /etc/magnetite/classifier.yml) or paste it into Admin → Classifier.
• Imports: POST /import accepts bitmagnet's newline-delimited JSON (send the key as X-Api-Key).

Using it

Page	What
Search	Full-text search ("phrases", -exclude, or), filters, sort by seeders or activity, file-name search
Trending	Most active (seeders + leechers now) and Rising (gain vs. yesterday / last week)
Torrent	Files, health history, magnet
Stats	Overview (catalog growth and saturation, disk, crawler, services), Catalog (quality by release year, seeders, survival, sizes, sources, records; recounted every 15 minutes), DHT network (node table, churn, world map of nodes and metadata peers, clients), Download pipeline, Usage & database (searches, Torznab, latency, per API key; tables and indexes)
Admin	Jobs (reclassify, bitmagnet import, site backfill), classifier editor + dry run, crawler limits (presets, rates, sizes), network (host / SOCKS5), sources (torrent sites), users, audit log

API docs (OpenAPI): http://localhost:3333/api/docs

Configuration

Everything is configured with environment variables (MAGNETITE_*) in compose.yaml / .env.

api

Variable	Default
MAGNETITE_TORZNAB_REQUIRE_API_KEY	true	Torznab needs apikey
MAGNETITE_TORZNAB_PROFILES	–	e.g. [{id=anime,tags=[anime]}]
MAGNETITE_COOKIE_SECURE	false	Set true behind HTTPS
MAGNETITE_SESSION_DAYS	30
MAGNETITE_MAGNET_TRACKERS	4 public trackers	Added to magnet links

worker

Variable	Default
MAGNETITE_CLASSIFIER_WORKFLOW	default	Workflow to run (bitmagnet classifier.workflow)
MAGNETITE_CLASSIFIER_FILES	/etc/magnetite/classifier.yml, ./classifier.yml	bitmagnet-style overrides
MAGNETITE_HEALTH_TRACKERS	4 public UDP trackers	udp://, http:// or https:// announce URLs; [] disables health checks
MAGNETITE_HEALTH_RECHECK_HOURS	24
MAGNETITE_HEALTH_HISTORY_DAYS	90
MAGNETITE_RETENTION_DEAD_DAYS	off	Delete torrents without seeders for N days
MAGNETITE_RETENTION_UNKNOWN_DAYS	off	Delete torrents still unclassified after N days
MAGNETITE_MAX_FILES	10000	File-list entries stored per torrent
MAGNETITE_GEOIP_URL	DB-IP Lite	IP → country table for Stats → DHT network ({month} = 2026-10), fetched monthly over the chosen network; empty turns the map off

crawler

Variable	Default
MAGNETITE_DHT_PORT	3334	UDP port
MAGNETITE_BOOTSTRAP_NODES	bitmagnet's list	[host:port, ...]

Rates, concurrency and sizes (DHT queries/s, metadata and site downloads at once, node table, queues, …) are not env vars: set them in Admin → Crawler, with Gentle / Default / Aggressive presets. Rates and concurrency apply within seconds; queue, node-table and filter sizes restart the crawler.

Architecture

browser ─┐                                  ┌─ app-db (users, keys, settings, jobs)
*arr ────┼─ web (Caddy :3333) ─ api ────────┤
         │                                  └─ data-db (torrents, health)
         │                     NATS JetStream ▲
DHT ─ UDP 3334 ─ crawler ──torrents.raw──▶ worker (classify, store, health, jobs)
                       ▲                      │
                       └──scraped.torrents────┘ (hashes from torrent sites)

Torrent traffic (DHT, metadata, trackers, sites) uses the host network or a SOCKS5 proxy.

Container	RAM limit	Role
web	128 MB	Serves the UI, proxies the API
api	256 MB	REST + OpenAPI, Torznab, import, SSE
worker	384 MB	Ingest + classifier, tracker health, site polling, jobs, retention
crawler	384 MB	DHT node, metadata downloads
app-db / data-db	256 MB / 1.6 GB	PostgreSQL 18
nats	96 MB	Ingest queue and live events

Crate layout: crates/{bencode, krpc, dht, peer-wire, metainfo, classifier, torznab, tracker-scrape, net, scrapers, storage, common}, services in services/{api, worker, crawler}, UI in web/.

Health and metrics

Every service serves /healthz (used by the Docker health checks) and /metrics (counters in Prometheus text format, handy for debugging): the api on :3333 inside its container, the worker on :3336, the crawler on 127.0.0.1:3335 on the host. The Stats page in the UI shows the important numbers without any extra tools.

Development

Needs the latest stable Rust (rustup update stable), Node 26 (.node-version; fnm or nvm pick it up) and pnpm 12 (web/package.json → packageManager).

docker compose up -d app-db data-db nats
cargo run --bin api            # :3333
cargo run --bin worker
cargo run --bin crawler
cd web && pnpm install && pnpm dev   # :5173, proxies to :3333

After changing the API: cd web && pnpm gen:api (regenerates the OpenAPI spec and the typed client).

Checks (also in CI, .github/workflows/ci.yml, run by the Forgejo runner): cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test, the OpenAPI spec and API client being up to date, pnpm lint, pnpm build. CI does not build the Docker images; run docker compose build yourself after changing a Dockerfile. The classifier is tested against bitmagnet's own output on 1,250 real torrents (crates/classifier/tests/golden.rs).

Tests that need PostgreSQL or NATS are #[ignore]d (CI skips them). Run them against throwaway servers, never the stack's own (they create databases and purge streams):

docker run -d --rm --name magnetite-test-pg -p 127.0.0.1:55433:5432 \
    -e POSTGRES_USER=magnetite -e POSTGRES_PASSWORD=magnetite postgres:18-alpine
docker run -d --rm --name magnetite-test-nats -p 127.0.0.1:54223:4222 nats:2.12-alpine -js
export MAGNETITE_TEST_PG_URL=postgres://magnetite:magnetite@127.0.0.1:55433/magnetite
export MAGNETITE_TEST_NATS_URL=nats://127.0.0.1:54223
cargo test -- --ignored --skip perf_                          # storage, crawler → NATS, NATS → worker → search
cargo test --release -- --ignored --nocapture --test-threads=1 perf_   # benchmarks

The benchmarks (perf_*) print throughput for KRPC messages, the bloom filters, the DHT over loopback UDP, metadata downloads and parsing, the classifier, bulk writes and search latency on a 200 000-torrent catalog (MAGNETITE_PERF_TORRENTS changes the size), and the whole ingest path.

Security notes

• Change the database passwords in .env before exposing anything.
• The SOCKS5 proxy password is stored in app-db; keep the databases off the network (they listen on 127.0.0.1 only).
• Serve over HTTPS (e.g. add a domain to web/Caddyfile) and set MAGNETITE_COOKIE_SECURE=true if Magnetite is reachable from the internet.

License

MIT. Includes material from bitmagnet (MIT), see THIRD_PARTY_NOTICES.