THINKING MACHINES LAB

TINKER

You write the learning experiment. Tinker makes the large model actually move.

PYTHON OUTSIDEACCELERATORS INSIDEJUDGMENT THROUGHOUT
00THE COMPRESSION

Tinker is an API boundary around distributed post-training.

It is lower-level than “upload a dataset and receive a model,” and higher-level than assembling a GPU cluster. Your Python code owns the data, loss, rewards, rollout environment, and experimental logic. Thinking Machines owns placement, parallelism, scheduling, failure recovery, and moving giant tensors through hardware.

That boundary is the product. It keeps the part where research judgment lives and removes the part where NCCL develops opinions about your evening.

01WHERE THE SYSTEM CUTS

YOUR PROCESS

The experiment

  • Examples and renderers
  • Loss functions
  • Reward and grading logic
  • Rollout environments
  • Evaluation and interpretation
SAMPLEFORWARD_BACKWARDAPIOPTIM_STEPSAVE_WEIGHTS

TINKER SERVICE

The machinery

  • Model sharding
  • GPU allocation
  • Distributed execution
  • Checkpoint storage
  • Failure recovery

Switching from an 8B dense model to a much larger mixture-of-experts model can be one string change because the hardware layout stays behind the boundary.

02THE FIVE VERBS
01

sample

Generate candidate tokens from a base model or your current adapter.

02

compute_logprobs

Ask how probable tokens were under a particular policy.

03

forward_backward

Evaluate a loss on data and accumulate gradients.

04

optim_step

Use those gradients to update the trainable adapter.

05

save_weights

Freeze a named checkpoint or turn it into a sampling client.

Every method is a composition of these verbs.

03SUPERVISED FINE-TUNING

Show it the behavior you want.

SFT is next-token prediction over examples you chose. The model sees a prompt and completion; cross-entropy pushes probability toward the completion tokens. The intellectual work is mostly upstream: what counts as a good example, what context is included, and which tokens are allowed to teach.

EXAMPLETOKENS + MASKCROSS-ENTROPYGRADIENTUPDATE
THE LOOPPYTHON
service = tinker.ServiceClient()
trainer = service.create_lora_training_client(
    base_model="Qwen/Qwen3-8B",
    rank=32,
)

for batch in dataset:
    future = await trainer.forward_backward_async(
        data=batch,
        loss_fn="cross_entropy",
    )
    metrics = await future.result_async()

    step = await trainer.optim_step_async(
        types.AdamParams(learning_rate=1e-4)
    )
    await step.result_async()
THE DATUMTOKENS
tokens   = [prompt tokens..., answer tokens...]
targets  = tokens[1:]
weights  = [0, 0, 0, ..., 1, 1, 1, ...]

datum = types.Datum(
    model_input=types.ModelInput.from_ints(tokens[:-1]),
    loss_fn_inputs={
        "target_tokens": targets,
        "weights": weights,
    },
)
04LORA, WITHOUT THE FOG

W′ = W + γBA

ORIGINAL WEIGHTS + SMALL LEARNED UPDATE

Do not rewrite the whole model. Learn a compact change.

A model layer contains a large weight matrix W. LoRA keeps it fixed and learns two skinny matrices, B and A. Their product is low-rank: it can express a structured change using far fewer trainable parameters.

Rank controls the dimensionality of that change, not the model’s intelligence. Too little rank can become a capacity bottleneck. More rank costs storage and training memory. Thinking Machines’ experiments found a broad low-regret regime for ordinary post-training, especially when LoRA is applied across all weight matrices rather than attention alone.

CHEAPERTrain and store a small adapter.
SWAPPABLEMany adapters can share one base.
ENOUGHUsually, when the dataset fits its capacity.
05REINFORCEMENT LEARNING

Let the model act, then teach from consequences.

RL closes a loop. The current policy samples several attempts. An environment or grader scores them. Advantages express which attempts were better than their local baseline. Training increases the probability of better trajectories and decreases the probability of worse ones.

1POLICYcurrent adapter
2ROLLOUTSmultiple attempts
3REWARDenvironment judgment
4ADVANTAGErelative signal
5UPDATEnew policy
THE LOOPSCHEMATIC
while training:
    policy = trainer.save_weights_and_get_sampling_client()
    result = await policy.sample_async(
        prompt=problem,
        num_samples=8,
        sampling_params=params,
    )

    rollouts = result.sequences
    rewards = [environment.score(x) for x in rollouts]
    baseline = sum(rewards) / len(rewards)
    advantages = [reward - baseline for reward in rewards]

    await trainer.forward_backward_async(
        data=make_rl_data(rollouts, advantages),
        loss_fn="importance_sampling",
    )
    await trainer.optim_step_async(optimizer)
06WHY AGENTS FIT

An agent already emits the raw material for post-training.

STATETHOUGHTTOOLOBSERVATIONANSWER
A

Trajectory as data

A whole attempt can become an example, a comparison, or a rollout.

B

Human correction as signal

Edits, approvals, reversals, and “that is the wrong frame” can become labels.

C

Environment as grader

Tests, receipts, task state, and user judgment can supply rewards.

The scarce thing is not text. It is trustworthy judgment attached to text.

07EVALUATION IS THE EXPERIMENT

Training loss tells you that learning happened. It does not tell you what was learned.

A run without held-out evaluation is an expensive anecdote. Decide what improvement means before the first optimizer step, then preserve examples the model never trains on.

01

Loss

Did optimization move in the expected direction?

02

Task metric

Did held-out accuracy, reward, or preference rate improve?

03

Behavior

Did the model acquire the intended habit rather than a shortcut?

04

Regression

What unrelated ability or style got worse?

08A FIRST EXPERIMENT WORTH RUNNING

TRAJECTORY JUDGMENT / SFT FIRST

Can a small model learn to recognize the better agent run?

This is narrow enough to finish and close enough to real agent work to be diagnostic. It tests the entire Tinker surface without requiring a synthetic math environment or pretending that reward design is solved.

  1. 01
    Collect pairs

    Two attempts at the same task, plus a human choice and one-sentence reason.

  2. 02
    Render carefully

    Include the task, compact traces, outputs, and receipts. Remove irrelevant noise.

  3. 03
    Train a judge

    Start with SFT: predict A or B, then explain the decisive evidence.

  4. 04
    Hold out hard cases

    Especially cases where polish conflicts with correctness or claimed success lacks a receipt.

  5. 05
    Interrogate errors

    The useful output is not one score. It is a taxonomy of judgment the model failed to acquire.

HYPOTHESIS

Expert judgment will transfer where it is visible in concrete labels and reasons. It will fail where the “expertise” only exists as tacit context withheld from the training example.

09THE LEARNING MAP
NOW

Build one SFT run

Tokenization, masks, batches, loss curves, checkpoints, held-out samples.

NEXT

Compare objectives

SFT versus preference training on the same underlying judgments.

THEN

Close the RL loop

Generate fresh trajectories, grade them, and train on-policy.

FINALLY

Change the question

Use observed failures to decide what data and reward should exist next.

10WORDS THAT STOP BEING MYSTERIOUS
Base model
The frozen pretrained or instruction-tuned network an adapter modifies.
Renderer
Code that turns structured examples or messages into the exact tokens a model sees.
Logprob
The log of a model’s assigned probability to a token. Conveniently additive across a sequence.
Gradient
The local direction in parameter space that changes the loss.
Checkpoint
A named saved state: adapter weights, and sometimes optimizer state for resuming.
On-policy
Training from trajectories generated by the current or very recent policy.
Advantage
How much better an action or trajectory was than an expected baseline.
Distillation
Training one model to reproduce information carried by another model’s outputs or probabilities.
Appearance