DENSE ORDINAL REPLICA LOSS

ARC's ten colours are not ten unrelated labels — they sit on a circle. This loss teaches the network that, by leaking a little probability mass onto nearby colours instead of using a one-hot cliff.

Ported from Opinion@nca (CircularReplicaLoss) and Opinion 3773f4e (DenseOrdinalReplicaLoss), checked against references generated from those sources on every build.

Recorded effect

1.36%→64.66%

Grid accuracy, swapping one-hot targets for the circular replica targets below. The figure is recorded in the loss module itself.

Circular, then dense ordinal

CircularReplicaLoss

Opinion@nca · earlier

Treats the ten colours as a ring. Target mass leaks onto neighbouring colours, weighted by inverse circular distance.

d(a,b)=\min\bigl(|a-b|,\; N-|a-b|\bigr)

Smooth between colours. But it asserts that colour 4 is nearer colour 3 than colour 8 — and in ARC, colour indices are arbitrary labels.

DenseOrdinalReplicaLoss

Opinion 3773f4e · 2025-10-26

Keeps the replicas, drops the ring. Distance is linear within a colour's own replica block, and a flat Hamming penalty everywhere else.

d(r, c^{\star}) = \begin{cases} |r - m^{\star}| & \lfloor r/R \rfloor = c^{\star} \\[0.5em] R & \text{otherwise} \end{cases}

Smoothness lives where it is real — replica index is a genuine continuum — and every wrong colour is equally wrong.

Distance to every one of the 100 classes

Target colour 0. Each cell is one class; darker means further from the target.

circular — mass spreads across colour blocks

dense ordinal — a V inside one block, flat elsewhere

The dense version has structure in exactly one block — a V centred on replica 5 of colour 0 — and is deliberately featureless across the other 9 colours. That flatness is the claim: nothing is known about how colours relate, so nothing is encoded.

Target colour

Spillovers = 0.030

the default used in training

The colour circle

Line thickness is the mass colour 0 donates. Note that 9 and 1 are equally close — the palette wraps.

d(a,b)=\min\bigl(|a-b|,\; N-|a-b|\bigr)

Target mass per colour

0d0

97.000%

1d1

0.687%

2d2

0.344%

3d3

0.229%

4d4

0.172%

5d5

0.137%

6d4

0.172%

7d3

0.229%

8d2

0.344%

9d1

0.687%

Bars are on a shared square-root scale so the spillover stays visible next to the 97.0% on the target itself.

Why replicas

p(r \mid c^{\star}) = \begin{cases} \dfrac{1-s}{R} & r \in [c^{\star}R,\; (c^{\star}+1)R) \\[1.1em] \dfrac{s\, \tilde{w}\bigl(d(c,c^{\star})\bigr)}{R} & r \in [cR,\; (c+1)R),\; c \neq c^{\star} \end{cases}

with inverse-distance weights $w(d)=1/d$ normalised so that $\textstyle\sum_{c \neq c^{\star}} \tilde{w} = 1$ .

Each colour owns 10 output classes rather than one, so the head predicts over 100 classes. Mass is split evenly inside a colour's block and spread across blocks by circular distance — the spillover is between colours, never between replicas of the same colour.

100 classes, ten per colour. Opacity is the probability assigned to each class.

What the network feels

A prediction that is wrong but adjacent should be penalised less than one that is wrong and far away. One-hot cannot express that. Below, the answer is colour 0 and the model confidently predicts each colour in turn.

Cost of confidently predicting each wrong colour. The right answer is excluded: it costs 2.48 against roughly 8.3 for a miss, and at that scale the differences between misses would be invisible.

1d1

2d2

3d3

4d4

5d5

6d4

7d3

8d2

9d1

one-hotreplica each column scaled to its own range — spread: one-hot 0.0000, replica 0.0330

Under one-hot every wrong colour costs exactly the same — the grey bars are flat, so the gradient carries no information about which colours are close. Under the replica target the cost rises with circular distance, giving a V centred on the answer. At the default s = 0.03 the effect is deliberately small; raise the spillover slider above and the V deepens.

The loss

\mathcal{L} \;=\; \underbrace{\mathrm{KL}\bigl(p^{\star} \,\|\, p_{\theta}\bigr)}_{\text{soft cross-entropy}} \;-\; \lambda \underbrace{H(p_{\theta})}_{\text{entropy bonus}}

with $\lambda = 0.01$ , $s = 0.03$ , $R = 10$ and $N = 10$ colours, giving $N \cdot R = 100$ output classes.