Parameter Golf Calculator #

Why this calculator exists #

This calculator helps you decide whether a model idea is worth running before you spend money/time on GPU training.

In the Parameter Golf challenge, your model must be:

1. Good enough (low validation bits-per-byte score, val_bpb), and
2. Small enough (artifact under 16,000,000 bytes), and
3. Fast enough (train/eval runtime limits).

This tool gives a fast pre-check for those constraints.

---

What is the goal? #

The goal is to find input values that produce:

• val_bpb_pred as low as possible,
• artifact_ok = true,
• train_runtime_ok = true,
• eval_runtime_ok = true,
• and therefore eligible = true.

If eligible = false, your candidate likely fails challenge constraints and should be adjusted before expensive runs.

---

Inputs #

Model shape inputs #

• V: Vocabulary size (how many token IDs your tokenizer can output).
• d: Model width (hidden size).
• L: Number of layers in your model.
• m: MLP (multilayer perceptron) multiplier (how wide feed-forward blocks are).
• H: Number of attention query heads.
• K: Number of key/value heads.

Performance/packing assumptions #

• tokens_per_byte: How many tokens are used to represent one byte of text for your tokenizer family.
  • Lower often means longer tokens and can help val_bpb.
  • Example ballpark: 0.27 (SP8192-ish behavior).
• val_loss_estimate: Your expected validation loss (nats per token).
• weight_bits: Average bits used to store each parameter after quantization (e.g. 6 for int6-like planning).
• compression_ratio: Expected additional compression effect after packing.
• train_seconds: Estimated training runtime.
• eval_seconds: Estimated evaluation runtime.

---

How it works #

1) Parameter count estimate #

The calculator computes model parameter count N_params from (V, d, L, m, H, K) using block-level formulas.

Core pieces:

• d_kv = d * K / H
• N_embedding = V * d
• N_per_layer_attn = d*d + d*d_kv + d*d_kv + d*d + H
• N_per_layer_mlp = 2 * d * (m*d)
• N_params = N_embedding + L*(N_per_layer_attn + N_per_layer_mlp + N_per_layer_ctrl) + N_skip + N_extra

2) Artifact size estimate #

• raw_bytes = N_params * (weight_bits / 8)
• with_overhead = raw_bytes * (1 + scale_overhead)
• quantized_model_bytes_est = with_overhead / compression_ratio
• total_artifact_bytes_est = quantized_model_bytes_est + code_bytes

3) Score prediction #

The predicted challenge score is:

• val_bpb_pred = (val_loss_estimate / ln(2)) * tokens_per_byte

where ln(2) ≈ 0.6931.

4) Eligibility checks #

The tool checks if your candidate is under artifact/runtime limits (with safety margin).

---

Output walkthrough #

Given this example of outputs:

• N_params = 35,944,536
• quantized_model_bytes_est = 17,354,471
• total_artifact_bytes_est = 17,374,471
• val_bpb_pred = 1.080797...
• artifact_ok = false
• train_runtime_ok = true
• eval_runtime_ok = true
• eligible = false

What this means:

1. Model quality estimate looks strong (val_bpb_pred ≈ 1.08 is competitive-like),
2. But size estimate is too large, so it fails artifact constraint,
3. Runtime assumptions pass,
4. Final result is eligible = false because one required constraint failed.

In plain language: “Promising quality, but still too big to submit.”

---

What is a “positive” output? #

A positive output is:

• eligible = true

and practically:

• low val_bpb_pred,
• all three booleans true (artifact_ok, train_runtime_ok, eval_runtime_ok).

---

What to do next when output is satisfying #

If output is satisfying (eligible = true + good predicted score):

1. Move to a real training run on RunPod (as OpenAI recommends for challenge workflows).
2. Start with a short sanity run.
3. Run a full train + eval under challenge-like constraints.
4. Compare actual val_bpb vs predicted val_bpb_pred.
5. Update assumptions (tokens_per_byte, val_loss_estimate, compression_ratio) and iterate.

If output is not satisfying:

• Reduce size pressure (d, L, m, V) or improve packing (weight_bits, compression_ratio),
• Improve expected loss without exploding size,
• Re-run calculator before spending GPU budget.

---

Important reminder #

This is a planner, not a final grader.

It helps prioritize candidates and avoid expensive dead ends. Final truth always comes from actual train/eval logs.