The Benchmark.

IV · Sample task: Input validation

What the verifier
actually looks like.

Every task ships with four files. The verifier suite is the judge. It runs the agent’s output and asserts on the contract. Below are three tests excerpted from tasks/01-input-validation/verifier/validation.test.js.

// three of the 29 tests in validation.test.js

test('rejects null input', () => {
  expect(() => processUserInput(null)).toThrow(ValidationError);
  try { processUserInput(null); } catch (e) {
    expect(e.code).toBe('invalid_input');
    expect(e.field).toBe('data');
  }
});

test('rejects malformed email (no @)', () => {
  try { processUserInput({ email: 'notanemail', age: 25, name: 'A' }); }
  catch (e) {
    expect(e.code).toBe('invalid_email');
    expect(e.field).toBe('email');
  }
});

test('rejects too many tags', () => {
  const tags = Array(11).fill('x');
  try { processUserInput({ email: 'a@b.com', age: 25, name: 'A', tags }); }
  catch (e) { expect(e.code).toBe('invalid_tags'); }
});

The agent doesn’t see this file. It sees the prompt and the starter code. Its output is then run against this suite. Pass means green tests. There is no LLM-as-judge. There is no judgment call.