Supervised

In a restless world, pursue what does not change

Implement the paper. Keep the code. Landmark work, staged and judged — a path that lasts as long as the text.

See the wall first

Invite-only. We send a serial when a seat opens.

Without Supervised

The paper is clear. The path is not.

Landmark work arrives as PDF — then blogs, half-dead repos, and threads. The hard bit still goes ungraded.

The maskAttention(Q,K,V) = softmax(QKᵀ/√dₖ + M) V
Tutorials wave at M. Repos bury it. Nothing grades your fixture.
The shapesAssertionError: expected (B,H,T,D), got (B,T,H,D)
You pass unit tests you wrote yourself. Production still blows up.
The stack40 files · no order · no boss stage
Encoder, PE, decoder, training loop — which week? Which file first?

arXiv · 1706.03762.pdfMedium · ‘Attention from scratch’github.com/…/transformerSO · RuntimeError shapes#ml-help · 3 unread

Attention Is All You Need

We propose a new simple network architecture, the Transformer…

§3.2.1 Scaled Dot-Product Attention — see equation (1).

python train.py --config big

RuntimeError: mat1 and mat2 shapes cannot be multiplied (512x64 and 8x64)

# which of the 11 attention impls is canonical?

_

02:14 — rewrite PE again
03:02 — blog disagrees with paper
04:40 — still no test for M

Supervised stages that same paper — implement it, judge it, keep the Thesauros.

With Supervised · Peira → Judge

From the text, a path.

Official unlocks in recipe order; Community is a flat open set. Open a stage to see its problems, then Open a brick for the live Judge — apply stays on the footer when you want a serial.

Peira · Attention Is All You Needlive UI

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin

Official — staged track. Open a stage to see its problems, then Open a problem for the live Judge. Stages / Paper focuses the PDF like product.

Paper subsystems

Why this order: Open the ideas that unlock the first stage.

Gnosis

0 of 10

Scaled Dot-Product

Stage 1

Multi-Head Split

Stage 2

Attention Mask

Stage 3

Positional Encoding

Stage 4

Encoder Layer

Stage 5

Decoder Layer

Stage 6

Cross Attention

Stage 7

FFN

Stage 8

Full Encoder

Stage 9

Full Decoder

Stage 10

Boss · Mini Transformer

Boss

Abstract

The Transformer is a model architecture based solely on attention mechanisms.

Gnosis You'll Need

Optional reference. You can start Stage 1 without reviewing these.

Stages

Recipe order — clear each stage to unlock the next. Open Stage 1 to see its problems.

2

Multi-Head Split

Locked220 Drachma4 problems

Locked

3

Attention Mask

Locked220 Drachma4 problems

Locked

4

Positional Encoding

Locked220 Drachma4 problems

Locked

5

Encoder Layer

Locked220 Drachma4 problems

Locked

6

Decoder Layer

Locked220 Drachma4 problems

Locked

7

Cross Attention

Locked220 Drachma4 problems

Locked

8

FFN

Locked220 Drachma4 problems

Locked

9

Full Encoder

Locked220 Drachma4 problems

Locked

10

Full Decoder

Locked220 Drachma4 problems

Locked

11

Boss: Mini Transformer

Boss

Locked320 Drachma4 problems

Locked

1706.03762.pdf

Gnosis

Ideas that open the stage

Explain · Example · Quiz · Mark learned — concepts that sit beside the stage work.

Gnosis · Scaled Dot-Product Attentionlive UI

Gnosis

Scaled Dot-Product Attention

In progressQuiz 0/1~18 min

INTERMEDIATE~18 min

What it is

Given queries, keys, and values, compute $\mathrm{softmax}(QK^\top/\sqrt{d_k}+M)V$ . Dividing by $\sqrt{d_k}$ keeps logits from growing with dimension.

In plain language

Score how much each position should attend, shrink by $\sqrt{d_k}$ , add mask $M$ , then mix values.

Why it shows up in papers

The Transformer and GPT-2 stacks are built from this block — Stage 1 implements the mask that makes it causal.

Prerequisites

Leads to

Truth

The literature, in one room

Search arXiv for the paper room — PDF, notes, Agora repost. Link Google Scholar to claim authorship when you’re on the byline.

arXivGoogle Scholar

Truth · 1706.03762live UI

Truth

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin · 2017 · arXiv:1706.03762

0 CommentsarXiv PDF

Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms.

Verified authorship

Optional — link Google Scholar to claim this paper if you’re an author. Name must match an arXiv author; citation tiers come from Semantic Scholar.

No verified authors yet.

Comments

No comments yet.

1706.03762.pdf

Open PDF arXiv abstract

Drachma · Arete · Polis

Earn a season. Keep the marks.

Peira clears, Agora bounties, and Continuity feed one seasonal Drachma tally on Polis. Cross a threshold and Arete moves — Scholaris through Archon; Hierophant is the top ten. The quarter ends, Drachma resets; the marks you earned stay.

Polis · seasonlive UI

Δρ

Drachma

One season’s worth of work

Clears, bounties, and Continuity add to a seasonal Drachma tally of how you showed up. Cross a hall threshold and Arete moves. When the quarter ends, Drachma resets; the marks stay.

Arete (season marks) · 2026-Q3

38 days left · then Drachma (season wallet · 6 ὀβ = 1 Δρ) resets

Cross seasonal Drachma (season wallet · 6 ὀβ = 1 Δρ) thresholds to earn Arete (season marks). Hierophant (Arete · season top 10 · Elenchos arbiter) is ranked — top 10 by season total at close. Balances reset each quarter; marks you earned stay.

Seasonal Drachma (season wallet · 6 ὀβ = 1 Δρ)

14,200 Drachma (season wallet · 6 ὀβ = 1 Δρ)

Next: Scholarch
4,800 Drachma (season wallet · 6 ὀβ = 1 Δρ) to go

ScholarisScholarAcademicianScholarchArchonHierophant

Your Arete (season marks)

Scholaris
2026-Q3 · 4,000 Drachma (season wallet · 6 ὀβ = 1 Δρ) · 2026-07-18
Scholaris
Scholar
2026-Q3 · 8,500 Drachma (season wallet · 6 ὀβ = 1 Δρ) · 2026-08-02
Scholar
Academician
2026-Q3 · 12,500 Drachma (season wallet · 6 ὀβ = 1 Δρ) · 2026-08-20
Academician

Climb the halls

ScholarisPnyx
4,000 Drachma
ScholarBouleuterion
8,500 Drachma
AcademicianAreopagus
12,500 Drachma
ScholarchLyceum
19,000 Drachma
ArchonBasileus
25,000 Drachma
HierophantEleusinion
Top 10 at close

Academician is the season mean. Archon is dedicated. Hierophant is ranked — the top ten by season total.

Agora

Where the campus talks papers

Truth reposts, stage Questions with a fixed title and a Drachma bounty, and open campus talk — help while you implement.

Agora · feedlive UI

Channels

#attention
#papers
#cv
#campus
#questions

Thread

Truth · arXiv:1512.03385

Repost from Truth — worth a Peira CV track?

From Truth

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

CVPR 2016 · CNN

arXiv:1512.03385 PDF

Pulled ResNet from Truth. Residual blocks are still the cleanest “CNN → modern stack” on-ramp — notes on §3.2 shortcut connections attached.

@atticus-fanWe should Atticus this — Stage 1 identity shortcut, Stage 2 bottleneck.
@cv-scholarReposted so #papers sees it before Official catalog picks it up.
@hetairia-leadPinning in the CV Hetairia room.
@mathētēsTruth → Agora is how I find weekend papers.
@archonGood catch — CNN tracks were thin this season.

Atticus

Author the track

Link Official Peira, draft problems, compile preview. Save draft or submit for review opens early-bird apply on this page.

Atticus · Propose a Peira tracklive UI

Atticus · author ceremony

Propose a Peira track

Same composer as after login — paper metadata plus multiple problems. Save or submit here to request early access.

1 · Source Peira

Link an Official Peira paper you have cleared at least one problem on. Community tracks publish under that paper’s Community catalog (flat open problems, no stages).

OfficialAttention Is All You Need

Paper titleLinked to an Official Peira paper. Unlink to change.

Authors

VenueYear

Topic categoryTrack weight

Abstract

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

2 · Problems (2)

Tracks need at least 2 problems to submit (max 24). You can remove down to one while drafting. Each clear pays the Drachma reward you set below.

Problem title

Community problems are all open — no stages or unlock order.

Clear reward (Drachma)Community cap 30 Drachma per problem.

Allowed languages

Python

C++

C

Java

At least one language required. Starter below stays Python for now.

Statement (Markdown)

Sample inputSample output

Judge inputJudge output

Starter code (Python)

3 · Bounty

Challenge bounty (Drachma)Δρ50Paid from your balance if a beta challenge wins (min 50 Δρ).

Preview

Peira track

Attention Is All You Need

Vaswani et al. · NeurIPS 2017

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

Problem 1 / 2

Scaled Dot-Product Attention

Why this brick

Scaled attention is the core operator of the Transformer — clear this before stacking heads and layers.

Task

Implement scaled dot-product attention for one head, including an optional additive mask M.

Ground truth

Correctness follows Attention Is All You Need §3.2.1. Fair edge cases left open by the paper can be raised in beta.

Input

Q, K, V matrices and optional mask

Output

Attention(Q, K, V) with softmax(QKᵀ/√dₖ + M) V

Pitfalls

Keep √d_k scaling; tiny fixtures; print floats to the stated precision.

Sample input

d_k=64
Q K V shapes (T,d)

Sample output

attn (T,d)

Thesauros

The tree you cleared stays yours

Implement the paper. Keep the code — ZIP, vscode.dev, Polis shelf.

Polis · Thesauroslive UI

Thesauros — keep what you cleared

attention-is-all-you-need

From Peira: Attention Is All You Need

Cleared Peira paper — full stage tree

@scholar / attention-is-all-you-need · Cleared 11/13/2026 · 5 files

← Polis shelf

Files

README.md

# Attention Is All You Need

Cleared on Peira.

Peira catalog

What’s on the shelf

Official shelf — recipe strip, Gnosis, Stages / Paper PDF. Stage 1 opens into its brick list; Open a problem for the live Judge.

Peira · Officiallive UI

28 Official tracks — open a paper → stage → bricks, then Open a problem for the Judge (Paper tab has the PDF).

01

Nitish Srivastava et al. · JMLR · 2014 · 11 stages · 2030 Δρ

Neural Nets

02

Tomas Mikolov et al. · ICLR Workshop · 2013 · 12 stages · 2160 Δρ

NLP

03

Volodymyr Mnih et al. · NIPS Deep Learning Workshop · 2013 · 12 stages · 2160 Δρ

RL

04

Diederik P. Kingma et al. · ICLR · 2014 · 11 stages · 2030 Δρ

Generative

05

Jeffrey Pennington et al. · EMNLP · 2014 · 9 stages · 1770 Δρ

NLP

06

Ian J. Goodfellow et al. · NeurIPS · 2014 · 11 stages · 2030 Δρ

Generative

07

Dzmitry Bahdanau et al. · ICLR · 2015 · 11 stages · 2030 Δρ

NLP

08

Karen Simonyan et al. · ICLR · 2015 · 10 stages · 1900 Δρ

CV

09

Diederik P. Kingma et al. · ICLR · 2015 · 9 stages · 1770 Δρ

Optimization

10

Sergey Ioffe et al. · ICML · 2015 · 10 stages · 1900 Δρ

Neural Nets

11

Olaf Ronneberger et al. · MICCAI · 2015 · 11 stages · 2030 Δρ

CV

12

Joseph Redmon et al. · CVPR · 2016 · 12 stages · 2160 Δρ

CV

13

Oriol Vinyals et al. · NeurIPS · 2015 · 11 stages · 2030 Δρ

NLP

14

Kaiming He et al. · CVPR · 2016 · 11 stages · 2030 Δρ

CV

15

Jimmy Lei Ba et al. · arXiv · 2016 · 9 stages · 1770 Δρ

Neural Nets

16

John Schulman et al. · arXiv · 2017 · 11 stages · 2030 Δρ

RL

17

Jacob Devlin et al. · NAACL · 2019 · 16 stages · 2680 Δρ

NLP

18

Yinhan Liu et al. · arXiv · 2019 · 10 stages · 1900 Δρ

NLP

19

Colin Raffel et al. · JMLR · 2020 · 11 stages · 2030 Δρ

NLP

20

Jonathan Ho et al. · NeurIPS · 2020 · 10 stages · 1900 Δρ

Generative

21

Alexey Dosovitskiy et al. · ICLR · 2021 · 11 stages · 2030 Δρ

CV

22

Alec Radford et al. · ICML · 2021 · 12 stages · 2160 Δρ

Multimodal

23

Edward J. Hu et al. · ICLR · 2022 · 10 stages · 1900 Δρ

Transformers

24

Tri Dao et al. · NeurIPS · 2022 · 11 stages · 2030 Δρ

Transformers

25

Hugo Touvron et al. · arXiv · 2023 · 13 stages · 2290 Δρ

Transformers

26

Mingxing Tan et al. · ICML · 2019 · 10 stages · 1900 Δρ

CV

27

Alec Radford et al. · OpenAI · 2019 · 11 stages · 2030 Δρ

NLP

28

Ashish Vaswani et al. · NeurIPS · 2017 · 11 stages · 2030 Δρ

TRANSFORMERS

Topics

PathfinderBeginner packs · curated paths4 packs · ~6–15h · Hetairia study circlesOpen curriculum

⌘K · Ctrl+K

One keystroke. The whole campus.

Press ⌘K anywhere — after you touch this band it stays here; until then ⌘K is a page overlay. The loop tries agora, Attention Is All You Need, and gnosis so Peira leads then Truth.

Command palette · livelive UI

Shortcut

⌘K

Live loop: Jump · agora · Attention Is All You Need · gnosis. Touch the frame to take over.

Don’t touch here — ⌘K anywhere else opens over the whole page.

Watch the loop — or take over

Touch this frame to type yourself. Try or . ⌘K outside this band opens over the page until you claim it.

Also on campus

Peira
Thesauros
Agora
Gnosis
Truth
Atticus
Hetairia
Arete
Polis
Neo

Early bird

From supervised to Unsupervised.

Email, a light profile, and your role. If we approve, you get a serial and the redeem link.

Email

Name (optional)

X handle (optional)

Your role

Helps us understand who’s waiting for a seat.

UndergraduateMaster’s studentPhD studentPostdocProfessorResearcherPrefer not to say

Supervised