AWE eval

Agent Web Evaluation

A computer-use eval harness for AI agents. Drive one of our sub-sites with a browser, finish, and we grade the result deterministically against a per-eval seed.

Human mode Agent mode

Pick an eval to start

Click Start on any eval to spin up a fresh session and drop into the sub-site.

0 pass 0 fail 0 active 24 untouched 24 total

Todo /todo

Summarize by priority

todo.summarize-by-priority

Read the todo list. Add one new todo that groups the existing todos by priority in the format "High: item1, item2; Med: item1; Low: item1", preserving the listed order within each priority group.
Complete today

todo.complete-today

Mark every todo due today as complete.
Edit and complete

todo.edit-and-complete

Create a todo titled "draft", rename it to "Updated", then complete it.
Delete highest priority

todo.delete-highest-priority

Delete the todo with the highest priority.
Dismiss popup and complete

todo.dismiss-popup-and-complete

Mark the "Submit timesheet" todo as complete.

Calendar /cal

First free slot

cal.first-free-slot

Find the first open 30-minute slot tomorrow after 09:00 UTC. Create an event titled "Design review" in that slot.
Delete off-week event

cal.delete-offweek-event

Go to the week containing the event titled "Quarterly Planning" and delete that event.
Replace attendee

cal.replace-attendee

Open "Hiring Panel" and replace bob@example.com with carol@example.com in its guest list.
Overnight event

cal.overnight-event

Create an event titled "Maintenance window" from tomorrow at 23:30 UTC to the following day at 00:15 UTC.
Three focus blocks

cal.three-blocks

Add three 1-hour events titled "Focus" tomorrow, back-to-back, starting at 09:00 UTC.
Cancel Old Sync

cal.cancel-old-sync

Delete every event titled "Old Sync".
Mirror today to tomorrow

cal.mirror-today-to-tomorrow

Read every event on today's calendar. Create matching events on tomorrow with the same titles, the same times of day (UTC), and the same durations.

Social /soc

Scroll like target

soc.scroll-like-target

Find the post that says "release candidate ready in row twelve" and like it.
Unlike and unfollow

soc.unlike-and-unfollow

Unfollow @bob and unlike the post "Build notes are posted".
Profile stat post

soc.profile-stat-post

Visit @dave's profile. Post a status update that says how many posts @dave has, in the format "@dave has N posts".
Post hello world

soc.post-hello-world

Post a status update with the text "Hello world".
Follow and like

soc.follow-and-like

Follow @alice and like her most recent post.
Reply thanks

soc.reply-thanks

Reply to @bob's most recent post with "Thanks!".
Delete mistake

soc.delete-mistake

Delete your own post that contains the word "mistake".
DM Carol

soc.dm-carol

Send a direct message to @carol with the text "Meeting at 3?".
List authors

soc.list-authors

Look at every top-level post in the feed. Post a status update whose text is the handles of all distinct authors (other than yourself), in alphabetical order, separated by single spaces — for example: "@alice @bob".

Directory /data

Promote engineer

data.promote-engineer

Marcus Chen just got promoted. Find him in the employee directory and click the Promote button on his row.
Promote Marcus's manager

data.promote-managers-chain

Marcus Chen's manager is up for promotion. Find Marcus's manager in the employee directory and click Promote on their row.
Terminate contractor

data.terminate-contractor

Wesley Xu's contract just ended. Terminate his record in the employee directory.