1. Use Jev to decide the next operation#

Jev does not generate text; instead, it handles decisions like noul, choice, and score. Each step in a browser operation typically involves judging “Is it done?”, “Does it need login?”, or “Which one to click next?”, making it more suitable for this scenario than general LLMs.

A single request can share the same state and ask multiple questions simultaneously:

1
import os
2
import requests
3

4
def ask(state, questions):
5
    response = requests.post(
6
        "<https://openrouter.ai/api/alpha/decisions>",
7
        headers={
8
            "Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"
9
        },
10
        json={
11
            "model": "typesafe/jev-1.13",
12
            "state": state,
13
            "questions": questions,
14
        },
15
        timeout=30,
16
    )
17
    response.raise_for_status()
18
    return response.json()["answers"]

When there are too many candidate elements, do not throw all of them to Jev at once. You can group them into batches of 40 elements, keep the top few from each group, and then proceed to the next round of selection.

2. Use Playwright to capture and operate pages#

Playwright is responsible for scanning pages, iframes, videos, and clickable elements, then organizing them into text that Jev can interpret:

1
<button> 下一步 @ bottom-center
2
<button> 設定 @ top-right [highlighted]
3
<div> 閱讀更多 @ middle-center

The source of elements should not only scan button; in practice, it also needs to include:

• a, [role=button]
• cursor: pointer
• classes containing btn, next, arrow
• interactive elements scanned by elementFromPoint()
• elements within multiple layers of iframes Like tools such as Articulate Storyline, buttons on the screen might just be images, while the actual named elements are transparent elements underneath. In such cases, you cannot directly call the element’s click(); instead, you need to get the coordinates and click with the mouse.

3. Establish a fixed decision loop#

Each step only does a few things:

1. Capture the current page and candidate elements.
2. Determine if login is required.
3. Process currently playing videos.
4. Handle single-choice, multiple-choice, or text answers from the configuration file.
5. Ask Jev if it’s complete and which element to click next.
6. Wait for the page to stabilize after clicking.
7. Detect repetitive operations to avoid infinite loops.

1
flowchart TD
2
    A[擷取頁面] --> B{需要登入？}
3
    B -->|是| C[等待人工登入]
4
    B -->|否| D[處理影片與題目]
5
    C --> D
6
    D --> E[Jev 選擇下一步]
7
    E --> F{敏感操作？}
8
    F -->|是| G[人工確認]
9
    F -->|否| H[執行操作]
10
    G --> H
11
    H --> I{完成？}
12
    I -->|否| A
13
    I -->|是| J[結束]

4. Retain human handoff mechanism#

Full automation can easily get stuck on unknown interfaces, so the design focus is not “no human needed at all,” but rather letting humans handle only the parts Jev doesn’t know.

In practice, we retain:

• Waiting for options: Wait when there are no reasonable operations, instead of clicking randomly.
• Low-confidence query: Let humans choose when confidence is below the threshold.
• Safety confirmation: Confirm operations like submit, exit, or delete.
• Human prompt: Add user prompts to the next round of state.
• Remember buttons: After a human clicks once, remember the text for subsequent prioritized selection. For example, “Information Point 1, 2, 3” can be treated uniformly as “Information Point” after ignoring numbers and punctuation, avoiding re-querying for every similar button.