feat(chrome): let the IDE agent use the user's Chrome and beta.42
The AITURK Chrome extension now registers the user's Chrome as the desktop backend's browser controller, the way the ChatGPT extension lets its agent work in the user's browser. The agent's existing browser tools (navigate, snapshot, click, type, scroll, back, press, get_images, vision) run in its own background tabs grouped as "AITURK", or a tab the user hands over. Backend: a local desktop lane in the browser-control broker. One controller registered from the loopback desktop transport is offered to every desktop session as a lane template; each session still gets its own exact scope on first use. The lane has its own switch (browser.extension_control.local_desktop, on by default); with no extension registered, or after it disconnects, the normal browser backend runs unchanged. browser_vision turns the extension's capture into the usual native/auxiliary vision result. Bridge: forwards controller commands only to a connection that registered, refuses unknown or oversized commands, relays results. Extension: service-worker controller with its own native connection, trusted chrome.debugger input (press/release only: a hover move waits 5 s on background tabs), refusal of password and payment-card fields and of browser, extension, file and Web Store pages, panel switch, Stop, tab hand-over, recent actions, Alt+Shift+A and right-click "send to AITURK". Cancelling Chrome's debugging bar stops the agent. Existing panel flows and privacy rules are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5c76c24c9c
commit
86a6621936
32 files changed
+2755
-195
No files matched your search
@@ -16727,6 +16727,9 @@ def _ws_auth_reason(ws: "WebSocket") -> tuple[Optional[str], str]:
|
||||
if not token:
|
||||
return "no_credential", "none"
|
||||
if hmac.compare_digest(token.encode(), _SESSION_TOKEN.encode()):
|
||||
# Only reachable on the loopback / --insecure path: the desktop IDE and
|
||||
# the bridges it spawns. Marks the local desktop browser-controller lane.
|
||||
ws._hermes_local_desktop = True
|
||||
return None, "token"
|
||||
return "token_mismatch", "token"
|
||||
|
||||
@@ -17864,6 +17867,7 @@ async def gateway_ws(ws: WebSocket) -> None:
|
||||
ws,
|
||||
auth_identity=getattr(ws, "_hermes_auth_identity", None),
|
||||
subprotocol=getattr(ws, "_hermes_ws_subprotocol", None),
|
||||
local_desktop=getattr(ws, "_hermes_local_desktop", False) is True,
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user