From d87f388efd36c9772499f434a43e5d15f64a3ef2 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 22:01:52 +0800 Subject: [PATCH 01/26] Give the GUI a searchable navigation panel, token-based dark and light themes, and tabs built on first open Forty-five of the 48 tabs could only be reached through View > Tabs > category, the window built every one of them before showing three, and its look was a third-party sheet that could not be adjusted. The panel lists every feature and searches it (Ctrl+K), the theme comes from one set of tokens, and a tab's module is imported when the tab is opened. The qt_material guard test goes with the import it guarded, so the main-window tests it kept skipped in CI now run there. --- CHANGELOG.md | 17 + Progress.md | 18 +- README.md | 2 +- README/README_zh-CN.md | 2 +- README/README_zh-TW.md | 2 +- architecture.md | 7 +- architecture_explore.md | 23 +- .../doc/new_features/v223_features_doc.rst | 35 +- .../Zh/doc/new_features/v223_features_doc.rst | 28 +- docs/updates/2026-10.md | 14 + docs/updates/README.md | 3 +- je_auto_control/gui/__init__.py | 2 + .../gui/language_wrapper/english.py | 8 + .../gui/language_wrapper/japanese.py | 8 + .../language_wrapper/simplified_chinese.py | 8 + .../language_wrapper/traditional_chinese.py | 8 + je_auto_control/gui/main_widget.py | 395 ++++++++---------- je_auto_control/gui/main_window.py | 114 ++++- je_auto_control/gui/navigation.py | 200 +++++++++ je_auto_control/gui/tab_registry.py | 123 ++++++ je_auto_control/gui/theme.py | 178 ++++++++ .../headless/test_gui_lazy_registry.py | 154 +++++++ .../headless/test_gui_slots_menu_audit.py | 2 - .../headless/test_gui_tab_lifecycle_audit.py | 5 +- .../headless/test_gui_workspace_navigation.py | 226 ++++++++++ .../headless/test_qt_material_guard.py | 64 --- .../headless/test_r3_gui_main_window.py | 10 +- 27 files changed, 1309 insertions(+), 347 deletions(-) create mode 100644 je_auto_control/gui/navigation.py create mode 100644 je_auto_control/gui/tab_registry.py create mode 100644 je_auto_control/gui/theme.py create mode 100644 test/unit_test/headless/test_gui_lazy_registry.py create mode 100644 test/unit_test/headless/test_gui_workspace_navigation.py delete mode 100644 test/unit_test/headless/test_qt_material_guard.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 062ccdf3e..f78dd9825 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,12 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's ### Added +- GUI: a navigation panel on the left of the main window lists every tab by + category, with a search box (`Ctrl+K`; `Ctrl+B` hides the panel). + `AutoControlGUIWidget.activate_tab(key)` opens a tab or brings it to the + front, and `current_tab_key()` names the tab on screen. +- GUI: **View → Theme** switches between a dark and a light theme, both built + from the tokens in `gui/theme.py` (`AutoControlGUIUI.set_theme(name)`). - AI-agent documentation and a dedicated `AI.md` explain computer-use positioning, MCP aliases, safe tool selection, and OpenAI integration. - `AC_run_agent` now uses a focused computer-use allow-list by default instead of exposing the full `AC_*` command catalogue to the model. - `write_secret(secret)` / `AC_write_secret` (`secret`): type a password or @@ -155,6 +161,17 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's ### Changed +- GUI tabs are built the first time they are opened. `AutoControlGUIWidget` + registers all 48 tabs from `gui/tab_registry.py` but constructs only the + three it opens on and its own forms; `list_registered_tabs()` builds + nothing, and reading `entry.widget` on a `_tab_entries` row builds that tab. + An embedder that relied on every tab existing after construction (a tab's + timer or listener started at start-up) has to open the tab first. +- The main window no longer imports `qt_material`; it styles itself from + `gui/theme.py`. The `[gui]` extra still lists `qt-material` for now. +- GUI text size "Auto" is 10 / 11 / 13 pt by screen height (was 12 / 14 / 16), + and the default window is 1280×800 (was 1000×760). The font family is the + platform's UI font instead of Lato. - `je_auto_control_dev`, the dev-channel package, declares what `je_auto_control` declares: the same pinned dependencies and platform markers (`defusedxml`, `cryptography` and the `opencv-python` bound are new diff --git a/Progress.md b/Progress.md index 276069c77..7741dff80 100644 --- a/Progress.md +++ b/Progress.md @@ -8,7 +8,21 @@ `utils/{config_sync,remote_desktop,mcp_server,self_healing,codegen,executor}/` 與型別／文件驗證。 核准設計：[跨平台自動化與 GUI 改版](docs/superpowers/specs/2026-10-02-platform-gui-modernization-design.md)。 實作計畫：[分階段交付計畫](docs/superpowers/plans/2026-10-02-modernization-index.md)，待審閱。 -現有 `[Answer]` 決策沿用；產品實作尚未開始。 +現有 `[Answer]` 決策沿用。 + +`WIP` — 計畫 F（GUI）：F1 的延遲分頁註冊與 F2 的導覽／搜尋／主題已交付（U-20261008-02），其餘各子計畫尚未開始。F 還缺： + +- **窄視窗的內容是被擠壓而不是可捲動**：`gui/main_widget.py` 給 `QTabWidget` 明確的最小尺寸讓視窗能縮到 640×420， + 但分頁內容沒有包進 `QScrollArea`；包進去會改變 `tabs.indexOf(entry.widget)` 這個 PyBreeze 與測試都在用的關係，要一起設計。 +- **主題與面板狀態不會記住**：`AutoControlGUIUI.set_theme`、導覽面板的顯示與寬度、字級都只活在當次執行； + 要用 `QSettings` 存，並讓測試不寫到使用者的設定。 +- **`qt-material` 還在 `[gui]` extra**：`gui/main_window.py` 已不匯入它。移除要同時改 `pyproject.toml`、`dev.toml`、 + `requirements.txt`、`uv.lock` 與 mypy 的 override，並先確認 PyBreeze 沒有靠這個 extra 取得它。 +- **分頁的關閉鈕是 Fusion 內建圖示**：計畫規定主題不新增點陣圖相依，換圖示要用 Qt 內建向量或既有資產。 +- **Remote Desktop 與 Script Builder 仍在啟動時建立**（預設開啟），啟動時間 2.6–2.7 秒裡大半是它們與門面匯入。 +- **F1 的 `TabRegistry.open/close` 介面與 `close` 釋放訂閱**：現在關閉分頁只是從分頁列移除，widget 留著。 +- **F3**（共用 worker、取消、關閉時不碰已銷毀物件、`webrtc_panel.py` 拆分）與 **F4**（啟動／記憶體基準、mixed-DPI、 + 功能對等測試）尚未開始。 **只記未完成的事。** 完成的工作記在 [docs/updates/](docs/updates/README.md)（每月一個批次檔， 索引與查詢指令在它的 README），相容性變更寫進 [CHANGELOG.md](CHANGELOG.md)；完成的項目 @@ -507,7 +521,7 @@ be at 2x if on a Retina screen」，`scale_down=True` 只在帶 `bbox` 時生效 ## `test_usb_acl_prompt.py` 讓 Python 3.10 的 headless 測試間歇 segfault -`TODO` — `test/unit_test/headless/test_usb_acl_prompt.py::test_bridge_remember_persists_acl_rule` 在 `coverage run -m pytest` 下讓行程 SIGSEGV（exit 139），整個 `pytest-headless` job 因此失敗：2026-09-26 連續三次 AutoControl Code Quality（ubuntu-22.04／3.10），2026-09-30 一次（macos-14／3.10）；同一次其他版本都過，之後的 run 又過，所以是間歇的。原因還沒查：先在 3.10 開 `faulthandler` 重跑這一支，看崩在哪個原生呼叫。 +`TODO` — `test/unit_test/headless/test_usb_acl_prompt.py::test_bridge_remember_persists_acl_rule` 在 `coverage run -m pytest` 下讓行程 SIGSEGV（exit 139），整個 `pytest-headless` job 因此失敗：2026-09-26 連續三次 AutoControl Code Quality（ubuntu-22.04／3.10），2026-09-30 一次（macos-14／3.10），2026-10-08 一次（ubuntu-22.04／3.10，PR #501，重跑該 job 後通過）；同一次其他版本都過，之後的 run 又過，所以是間歇的。原因還沒查：先在 3.10 開 `faulthandler` 重跑這一支，看崩在哪個原生呼叫。 --- diff --git a/README.md b/README.md index ec9c28d20..ba7d30e84 100644 --- a/README.md +++ b/README.md @@ -120,7 +120,7 @@ Optional extras, installed only when you need them: | Extra | Enables | |---|---| -| `gui` | PySide6 desktop application (48 tabs) | +| `gui` | PySide6 desktop application (48 tabs): a searchable navigation panel (`Ctrl+K`) lists every feature by category, tabs are built the first time they are opened, and **View → Theme** switches dark / light | | `webrtc` | WebRTC remote desktop, USB passthrough (`aiortc`, `av`) | | `signaling` | Standalone signaling / rendezvous server (`fastapi`, `uvicorn`) | | `discovery` | mDNS / Zeroconf LAN host discovery | diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index a8eb3e953..429fdd30b 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -115,7 +115,7 @@ pip install je_auto_control[gui] # 加上 PySide6 桌面应用 | Extra | 启用的功能 | |---|---| -| `gui` | PySide6 桌面应用（48 个标签页） | +| `gui` | PySide6 桌面应用（48 个标签页）：左侧可搜索的导航面板（`Ctrl+K`）按分类列出全部功能，标签页在第一次打开时才创建，**View → Theme** 切换深色／浅色 | | `webrtc` | WebRTC 远程桌面、USB 直通（`aiortc`、`av`） | | `signaling` | 独立的信令／rendezvous 服务器（`fastapi`、`uvicorn`） | | `discovery` | mDNS / Zeroconf 局域网主机发现 | diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index 88674e276..b4e4520fc 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -114,7 +114,7 @@ pip install je_auto_control[gui] # 加上 PySide6 桌面應用程式 | Extra | 啟用的功能 | |---|---| -| `gui` | PySide6 桌面應用程式（48 個分頁） | +| `gui` | PySide6 桌面應用程式（48 個分頁）：左側可搜尋的導覽面板（`Ctrl+K`）依分類列出全部功能，分頁在第一次開啟時才建立，**View → Theme** 切換深色／淺色 | | `webrtc` | WebRTC 遠端桌面、USB 直通（`aiortc`、`av`） | | `signaling` | 獨立的訊令／rendezvous 伺服器（`fastapi`、`uvicorn`） | | `discovery` | mDNS / Zeroconf 區網主機探索 | diff --git a/architecture.md b/architecture.md index c4ef2ffe6..627fbc98b 100644 --- a/architecture.md +++ b/architecture.md @@ -29,7 +29,7 @@ entry points → execution core (`utils/executor/`) → headless capabilities (` | `je_auto_control/wrapper/` | Platform-neutral API (`auto_control_mouse/keyboard/screen/image/record/window.py`); `platform_wrapper.py` picks the backend; `backend_contract.py` types the seam; `window_backends/`. | | `je_auto_control/{windows,osx,linux_with_x11,linux_wayland}/` | Desktop OS backends; only the running OS's backend is imported. | | `je_auto_control/{android,ios}/` | Mobile device control (adb / uiautomator2, WebDriverAgent). | -| `je_auto_control/gui/` | Optional PySide6 GUI (`[gui]` extra): `main_window.py`, tab registry `main_widget.py`, `script_builder/`, `remote_desktop/`, `language_wrapper/`. | +| `je_auto_control/gui/` | Optional PySide6 GUI (`[gui]` extra): `main_window.py` (menus, navigation dock, theme), `main_widget.py` (the tabbed workspace), the tab table `tab_registry.py` (tabs are built on first open), `navigation.py`, `theme.py`, `script_builder/`, `remote_desktop/`, `language_wrapper/`. | | `autocontrol-lsp/` | Separate distribution: language server for `AC_*` action JSON, plus a `vscode/` client. | | `test/` | `unit_test/headless/` (CI gate), `unit_test/flow_control/`, `integrated_test/`, `gui_test/`, `manual_test/`, `verify/`. | | `docs/` | Sphinx docs, `API_LIFECYCLE.md`, `CAPABILITY_MATRIX.md`. | @@ -108,8 +108,9 @@ wrapper/auto_control_record.record → OS listener (e.g. windows/record/win32_in `_handlers_scheduling.py`, `_handlers_remote.py`, `_handlers_locators.py`, `_handlers_operations.py`, `_handlers_qa.py`, `_handlers_executor_bridge.py` (a three-line delegation to an executor function), or `_handlers.py` for data, text and the WebRunner bridge. -6. GUI: thin widget in `gui/`, registered in `gui/main_widget.py` (`_add_tab`) with commands exposed through - `menu_actions()`; strings in every `gui/language_wrapper/*.py` catalogue. +6. GUI: thin widget in `gui/`, registered by one `TabSpec` row in `gui/tab_registry.py` (module and class name, so + it is imported only when opened) with commands exposed through `menu_actions()`; strings in every + `gui/language_wrapper/*.py` catalogue. 7. Headless test in `test/unit_test/headless/`. 8. Update `architecture_explore.md` (and `README.md` + `README/` translations if a quoted count changes), then run `python test/unit_test/headless/test_doc_line_counts.py --fix`. Regenerate the typed stub with diff --git a/architecture_explore.md b/architecture_explore.md index de1263480..e466e94c7 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -19,8 +19,8 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | -| Python 模組總數（含周邊子專案） | 1,065 | -| 程式碼總行數 | 157,741 | +| Python 模組總數（含周邊子專案） | 1,068 | +| 程式碼總行數 | 158,285 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 778 | | 套件門面 `__all__` 公開名稱數 | 1,244 | @@ -880,9 +880,12 @@ GUI 是**選用 extra**（`pip install je_auto_control[gui]`，PySide6 + qt-mate | 模組 | 行數 | 職責 | | --- | ---: | --- | -| `gui/__init__.py` | 23 | `start_autocontrol_gui()`：**唯一**會延遲匯入 PySide6 的地方，維持頂層套件 Qt-free。 | -| `main_window.py` | 301 | `QMainWindow`：選單列（File／Actions／View／…）、可關閉分頁、即時語言切換、字級預設、qt-material 主題。分頁分為 core／editing／detection／automation／system 五類。 | -| `main_widget.py` | 437 | 擁有 `QTabWidget`，註冊 48 個分頁，並暴露 show/hide/list API 給選單列。核心分頁在註冊時直接宣告 `(label_key, handler)` 動作對；分頁本體都在下列 mixin。 | +| `gui/__init__.py` | 25 | `start_autocontrol_gui()`：**唯一**會延遲匯入 PySide6 的地方，維持頂層套件 Qt-free。 | +| `main_window.py` | 379 | `QMainWindow`：選單列（File／Actions／View／…）、左側導覽面板 dock（`Ctrl+K` 搜尋、`Ctrl+B` 收合）、即時語言切換、字級預設、深色／淺色主題（`theme.py` 的 token，不再用 qt-material）。分頁分為 core／editing／detection／automation／system 五類。 | +| `main_widget.py` | 368 | 工作區：擁有 `QTabWidget`，依 `tab_registry.TAB_SPECS` 註冊 48 個分頁，並暴露 show/hide/activate/list API 給選單列與導覽面板。只有預設開啟的三個分頁與自己的 mixin 表單在啟動時建立，其餘第一次開啟才匯入模組、建立 widget。核心分頁在 `_own_tab_builders` 宣告 `(label_key, handler)` 動作對。 | +| `tab_registry.py` | 123 | 分頁表：每個分頁一筆 `TabSpec`（鍵、標題鍵、分類、模組與類別名），`TabEntry` 在第一次存取 `widget` 時才呼叫 factory。不匯入 Qt。 | +| `navigation.py` | 200 | `NavigationPanel`：搜尋框 + 依分類的功能樹，列出每個已註冊分頁（開啟中的以粗體標示），只回報被選的鍵，開啟分頁仍由視窗負責。 | +| `theme.py` | 178 | 設計 token（`ThemeTokens`：顏色、圓角、間距、字族）、深色與淺色兩組值、由 token 產生的樣式表與對應的 `QPalette`；不載入圖檔或字型檔。 | | `_auto_click_tab.py` | 291 | 自動點擊分頁的 mixin 建構器。 | | `_screenshot_tab.py` | 137 | 截圖／取像素分頁 mixin。 | | `_image_detect_tab.py` | 115 | 影像偵測分頁 mixin。 | @@ -894,11 +897,11 @@ GUI 是**選用 extra**（`pip install je_auto_control[gui]`，PySide6 + qt-mate | `_screen_geometry.py` | 52 | Qt 邏輯座標與截圖用的原生像素互轉：`native_region()`、`screen_at_native()`、`logical_point()`（每個螢幕的左上角在兩者相同，螢幕內依 device pixel ratio 縮放）。區域選取與主機端標註覆蓋層都用它。 | | `_daemon_thread.py` | 79 | `DaemonThread`：`QThread` 的替代品，保留遠端桌面 worker 用到的介面（`start`／`run`／`isRunning`／`wait`／`requestInterruption`／`started`／`finished`），但 `run()` 跑在 daemon `threading.Thread` 上，刪除物件或程式結束都不會銷毀執行中的執行緒。 | | `_worker_thread.py` | 216 | `start_worker()`：在 daemon `threading.Thread` 上執行 `QObject` worker 的 `run()`（沒有 `QThread` 可被銷毀），並經由分頁擁有的中繼物件回報結果（回呼一律在 GUI 執行緒；worker 沒處理的例外也送到 `on_fail`）；worker 留在模組登錄表直到 GUI 執行緒看到它結束，回傳 `WorkerHandle`（`isRunning()`）；程式結束時先呼叫 worker 的 `request_stop()`，最多等 10 秒，仍在跑的隨行程結束。 | -| `language_wrapper/` | 5,031 | 四語系字典（英／日／簡中／繁中）+ `multi_language_wrapper` 執行期切換器與監聽註冊表。 | +| `language_wrapper/` | 5,063 | 四語系字典（英／日／簡中／繁中）+ `multi_language_wrapper` 執行期切換器與監聽註冊表。 | | `selector/` | 216 | 拖曳選取螢幕區域的半透明全螢幕覆蓋層與樣板裁切工具（互動式，但都有對應的程式化 API）。 | > **分頁指令一律走 Actions 選單**：分頁本身只放輸入、表格與結果檢視，指令由視窗層選單暴露。 -> 核心分頁在 `main_widget.py` 註冊時宣告動作；功能分頁實作 `menu_actions()`（目前 40 個檔案有此 hook）。 +> 核心分頁在 `main_widget.py` 的 `_own_tab_builders` 宣告動作；功能分頁實作 `menu_actions()`（目前 40 個檔案有此 hook）。 > `test/unit_test/headless/test_actions_menu_gui.py` 會守住這個契約——沒有動作宣告的新分頁會讓 CI 失敗。 #### 48 個分頁 @@ -1005,7 +1008,7 @@ GUI 是**選用 extra**（`pip install je_auto_control[gui]`，PySide6 + qt-mate | **新平台後端** | 新增 `je_auto_control//` 實作 backend 介面，並在 `wrapper/platform_wrapper.py` 加一個分支 | 所有 wrapper 模組與上層 | | **新 `AC_*` 指令** | 在 `utils/` 寫無頭實作 → 加進 `Executor.event_dict` → 加進 `gui/script_builder/command_schema.py` | executor 分派邏輯本身 | | **執行期外掛指令** | `add_command_to_executor({"AC_x": fn})`，或用 `utils/plugin_loader`（掃描目錄）／`utils/plugin_sdk`（entry points） | 核心程式碼 | -| **新 GUI 分頁** | 在 `gui/` 新增 widget（只做 UI 翻譯）→ 在 `main_widget.py` `_add_tab` 註冊 → 提供 `menu_actions()` | 主視窗選單建構邏輯 | +| **新 GUI 分頁** | 在 `gui/` 新增 widget（只做 UI 翻譯）→ 在 `gui/tab_registry.py` 的 `TAB_SPECS` 加一筆 `TabSpec` → 提供 `menu_actions()` | 主視窗選單建構邏輯 | | **新 OCR／VLM／LLM／a11y 後端** | 在對應 `backends/` 實作 base 協定 | 呼叫端 | | **新報表格式** | 仿 `generate_report/` 既有三者的骨架新增產生器 | 執行紀錄收集 | | **新 MCP 工具** | 在 `mcp_server/tools/_factories.py` 加工廠、`_handlers.py` 加 adapter（QA 主題加在 `_handlers_qa.py`） | 傳輸層 | @@ -1076,7 +1079,7 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | 層／子系統 | 檔案數 | 行數 | | --- | ---: | ---: | -| `gui/` | 95 | 27,853 | +| `gui/` | 98 | 28,397 | | `utils/mcp_server/` | 35 | 18,898 | | `utils/remote_desktop/` | 56 | 13,014 | | `utils/executor/` | 8 | 9,606 | @@ -1097,5 +1100,5 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 852 | | 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 682 | 57,156 | -| **總計** | **1,059** | **157,676** | +| **總計** | **1,062** | **158,220** | diff --git a/docs/source/Eng/doc/new_features/v223_features_doc.rst b/docs/source/Eng/doc/new_features/v223_features_doc.rst index ef5264f69..8b64c4fc4 100644 --- a/docs/source/Eng/doc/new_features/v223_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v223_features_doc.rst @@ -25,6 +25,36 @@ replace stay in place: per-page browse buttons inside stacked trigger forms, the visibility-toggled data-source browse button, and stateful auto-refresh checkboxes. +The navigation panel +-------------------- + +Every registered tab is listed on the left of the window, grouped by the same +five categories, whether it is open or not; open tabs are shown in bold. +Click a feature to open it (or bring it to the front). The search box filters +the list as you type — by title, by key (``usb_devices``) or by category — and +**Return** opens the first match. ``Ctrl+K`` (**View → Search Features...**) +puts the cursor in the search box from anywhere, and ``Ctrl+B`` +(**View → Navigation Panel**) hides or shows the panel. + +A tab is built the first time it is opened: the window starts with the three +default tabs and the forms the main widget owns, and imports the module of +any other tab only when you open it. + +The navigation panel +-------------------- + +Every registered tab is listed on the left of the window, grouped by the same +five categories, whether it is open or not; open tabs are shown in bold. +Click a feature to open it (or bring it to the front). The search box filters +the list as you type — by title, by key (``usb_devices``) or by category — and +**Return** opens the first match. ``Ctrl+K`` (**View → Search Features...**) +puts the cursor in the search box from anywhere, and ``Ctrl+B`` +(**View → Navigation Panel**) hides or shows the panel. + +A tab is built the first time it is opened: the window starts with the three +default tabs and the forms the main widget owns, and imports the module of +any other tab only when you open it. + The View menu ------------- @@ -33,8 +63,11 @@ The View menu default layout opens with just Record, Script Builder, and Remote Desktop; everything else is one menu click away. Tabs are closable — closing one is the same as unchecking it in the View menu. +* **View → Theme** switches between the dark and the light theme. Both come + from one set of design tokens in ``gui/theme.py``; the window no longer + uses ``qt-material``. * **View → Text Size** offers auto (screen-height based) and preset font - sizes applied live. + sizes applied live, on top of the active theme. The contract test ----------------- diff --git a/docs/source/Zh/doc/new_features/v223_features_doc.rst b/docs/source/Zh/doc/new_features/v223_features_doc.rst index 4b872c04b..6f0e01764 100644 --- a/docs/source/Zh/doc/new_features/v223_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v223_features_doc.rst @@ -22,6 +22,30 @@ Actions 選單 逐頁瀏覽按鈕、隨可見性切換的資料來源瀏覽按鈕,以及有狀態的自動更新 核取方塊。 +導覽面板 +-------- + +視窗左側依同樣五個分類列出每一個已註冊的分頁,不論是否已開啟;已開啟的以粗體 +顯示。點一下就開啟該功能(已開啟則切到最前面)。搜尋框會隨輸入過濾清單——比對 +標題、鍵名(``usb_devices``)或分類——按 **Return** 開啟第一個符合的項目。 +``Ctrl+K``(**View → Search Features...**)從任何地方把游標移到搜尋框, +``Ctrl+B``(**View → Navigation Panel**)隱藏或顯示面板。 + +分頁在第一次開啟時才建立:視窗啟動時只有三個預設分頁與主元件自己的表單,其他 +分頁的模組要到開啟時才匯入。 + +導覽面板 +-------- + +視窗左側依同樣五個分類列出每一個已註冊的分頁,不論是否已開啟;已開啟的以粗體 +顯示。點一下就開啟該功能(已開啟則切到最前面)。搜尋框會隨輸入過濾清單——比對 +標題、鍵名(``usb_devices``)或分類——按 **Return** 開啟第一個符合的項目。 +``Ctrl+K``(**View → Search Features...**)從任何地方把游標移到搜尋框, +``Ctrl+B``(**View → Navigation Panel**)隱藏或顯示面板。 + +分頁在第一次開啟時才建立:視窗啟動時只有三個預設分頁與主元件自己的表單,其他 +分頁的模組要到開啟時才匯入。 + View 選單 --------- @@ -29,7 +53,9 @@ View 選單 顯示或隱藏任一已註冊分頁。預設版面只開啟錄製、Script Builder 與遠端 桌面;其餘分頁一個選單點擊即可叫出。分頁可關閉——關閉等同於在 View 選單取消勾選。 -* **View → Text Size** 提供自動(依螢幕高度)與預設字級,即時套用。 +* **View → Theme** 在深色與淺色主題之間切換。兩者都出自 ``gui/theme.py`` 的同一組 + 設計 token;視窗不再使用 ``qt-material``。 +* **View → Text Size** 提供自動(依螢幕高度)與預設字級,即時套用在目前的主題上。 契約測試 -------- diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index fa95e543c..1ff1082b7 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -204,3 +204,17 @@ Jeffrey_RPA kept its own `KEY_ALIASES` (40 friendly names) and `_EXTRA_KEY_CODES - **Tests**: `test/unit_test/headless/test_cross_project_contracts.py` +3 (real child processes), `test/unit_test/headless/test_report_path.py`. - **Docs**: `README.md`, `README/README_zh-TW.md`, `README/README_zh-CN.md`, `architecture.md`. - **Files**: `je_auto_control/__main__.py`, `je_auto_control/utils/generate_report/` (`report_path.py` new; three writers), the tests and docs above. + +## U-20261008-02 · 2026-10-08 · GUI: navigation panel with search, dark and light themes from tokens, tabs built on first open · #feature #gui #F1 #F2 + +- **What**: the first two tasks of plan F (`docs/superpowers/plans/2026-10-02-f-gui.md`), without the parts that wait on plans A–E. + - **Navigation** (`gui/navigation.py`, docked in `main_window.py`): every registered tab is listed on the left under its category, open or not, open ones in bold. The search box filters by title, key or category, every word, any case; Return opens the first match; no match shows a message instead of an empty list. `Ctrl+K` focuses the search and reveals a hidden panel, `Ctrl+B` toggles the panel. Both shortcuts belong to the window, not to a menu, so rebuilding the menu bar on a language switch does not register them again. Before, a tab that was not one of the three default ones could only be reached through View → Tabs → category. + - **Theme** (`gui/theme.py`): `ThemeTokens` (12 colours, radius, spacing, font family), `DARK` and `LIGHT`, `build_stylesheet(tokens)` and a matching `QPalette`, so what Qt draws itself follows the theme. View → Theme switches; the text-size rule is still appended to the theme's sheet. `main_window.py` no longer imports `qt_material`. The launcher sets the Fusion style. + - **Lazy tabs** (`gui/tab_registry.py`): 48 `TabSpec` rows (key, title key, category, module, class). `TabEntry.widget` calls its factory on first access. `main_widget.py` builds the three default tabs and its own six mixin forms (Record is both, so eight widgets) and nothing else; `list_registered_tabs()`, `retranslate()`, `sync_engine_tabs()` and `hide_tab()` build nothing. `_add_tab` still takes a ready widget. `main_widget.py` went from 437 to 368 lines and no longer imports the tab modules at the top. + - The tab widget has an explicit minimum size, so the window shrinks to 640×420; before, the Script Builder's minimum width kept it at 1064 px with the panel docked. +- **Result / numbers** (Windows 11, Python 3.14, offscreen, `AutoControlGUIWidget()` from a cold interpreter, four runs each): widgets built at start 48 → 8; `je_auto_control` modules imported 892 → 844; time to a built workspace 2.90–3.08 s → 2.63–2.74 s. Most of what is left is the facade import and the Remote Desktop and Script Builder tabs, which open by default. +- **Tests**: `test_gui_lazy_registry.py` (50: the table without Qt, each lazy class name read from its source, and a child-process probe of what is built at start, after listing, and on open / hide / reopen) and `test_gui_workspace_navigation.py` (17: the style sheet, WCAG contrast of both themes, the panel's search, and a child-process probe of the real window). `test_qt_material_guard.py` is removed with the import it guarded, and the three `importorskip("qt_material")` lines with it, so `test_r3_gui_main_window.py`, the tabs-menu leak test and the window-listener test now run in CI. The window-listener probe opens the Presence tab before counting listeners, because the tab is no longer built at start. +- **Checked by eye**: both themes and the 640×420 window rendered with `WA_DontShowOnScreen` on the Windows platform plugin (real fonts, nothing shown on the desktop). +- **Docs**: the three READMEs (extras table), `docs/source/{Eng,Zh}/doc/new_features/v223_features_doc.rst`, `architecture.md` §2 and §5, `architecture_explore.md` (GUI file table, extension points), `CHANGELOG.md`. +- **Files**: `je_auto_control/gui/{tab_registry,navigation,theme}.py` (new), `je_auto_control/gui/{main_widget,main_window}.py`, the four `gui/language_wrapper/` catalogues (8 keys each), the tests above. +- **Open items**: in `Progress.md` › 跨平台與 GUI 全面改版 — page content is squeezed, not scrolled, in a narrow window; the theme and panel state are not remembered between runs; `qt-material` is still in the `[gui]` extra; tab close buttons are Fusion's; F3 (workers, cancel, close) and F4 (benchmarks, parity) have not started. diff --git a/docs/updates/README.md b/docs/updates/README.md index da7d0ceaa..73e4ad061 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-02 | 2026-10-08 | GUI: navigation panel with search, dark and light themes from tokens, tabs built on first open | #feature #gui #F1 #F2 | [2026-10](2026-10.md) | | U-20261008-01 | 2026-10-08 | The legacy CLI flags exit 1 when an action failed; reports go below TestPioneer's artifact directory | #feature #cli #report #testpioneer | [2026-10](2026-10.md) | | U-20261006-01 | 2026-10-06 | Make AutoControl discoverable and safer for AI agents | #feature #ai #mcp #docs | [2026-10](2026-10.md) | | U-20261001-10 | 2026-10-01 | The publish jobs install hash-locked build tooling and build with the locked setuptools | #release #ci #security #X-13 | [2026-10](2026-10.md) | @@ -348,7 +349,7 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 12 | +| [2026-10.md](2026-10.md) | 2026-10 | 13 | | [2026-09-e.md](2026-09-e.md) | 2026-09 | 7 | | [2026-09-d.md](2026-09-d.md) | 2026-09 | 55 | | [2026-09-c.md](2026-09-c.md) | 2026-09 | 38 | diff --git a/je_auto_control/gui/__init__.py b/je_auto_control/gui/__init__.py index 26e60a0b6..931f3e884 100644 --- a/je_auto_control/gui/__init__.py +++ b/je_auto_control/gui/__init__.py @@ -16,8 +16,10 @@ def start_autocontrol_gui() -> None: from PySide6.QtWidgets import QApplication from je_auto_control.gui.main_window import AutoControlGUIUI + from je_auto_control.gui.theme import prepare_application app = QApplication(sys.argv) + prepare_application(app) window = AutoControlGUIUI() window.show() sys.exit(app.exec()) diff --git a/je_auto_control/gui/language_wrapper/english.py b/je_auto_control/gui/language_wrapper/english.py index 72c2fcff4..a5ccc74ca 100644 --- a/je_auto_control/gui/language_wrapper/english.py +++ b/je_auto_control/gui/language_wrapper/english.py @@ -1318,6 +1318,14 @@ "menu_view_text_large": "Large (14pt)", "menu_view_text_xlarge": "Extra Large (16pt)", "menu_view_text_xxlarge": "Huge (20pt)", + "menu_view_search": "Search Features...", + "menu_view_sidebar": "Navigation Panel", + "menu_view_theme": "Theme", + "menu_view_theme_dark": "Dark", + "menu_view_theme_light": "Light", + "nav_search_placeholder": "Search features (Ctrl+K)", + "nav_no_results": "No feature matches this search.", + "nav_open_hint": "Open in the workspace", "menu_tools": "Tools", "menu_tools_start_hotkeys": "Start hotkey daemon", "menu_tools_start_scheduler": "Start scheduler", diff --git a/je_auto_control/gui/language_wrapper/japanese.py b/je_auto_control/gui/language_wrapper/japanese.py index b55ba6cc3..5e7faea6b 100644 --- a/je_auto_control/gui/language_wrapper/japanese.py +++ b/je_auto_control/gui/language_wrapper/japanese.py @@ -1205,6 +1205,14 @@ "menu_view_text_large": "大 (14pt)", "menu_view_text_xlarge": "特大 (16pt)", "menu_view_text_xxlarge": "超大 (20pt)", + "menu_view_search": "機能を検索...", + "menu_view_sidebar": "ナビゲーションパネル", + "menu_view_theme": "テーマ", + "menu_view_theme_dark": "ダーク", + "menu_view_theme_light": "ライト", + "nav_search_placeholder": "機能を検索 (Ctrl+K)", + "nav_no_results": "一致する機能がありません。", + "nav_open_hint": "ワークスペースで開いています", "menu_tools": "ツール", "menu_tools_start_hotkeys": "ホットキーデーモン開始", "menu_tools_start_scheduler": "スケジューラー開始", diff --git a/je_auto_control/gui/language_wrapper/simplified_chinese.py b/je_auto_control/gui/language_wrapper/simplified_chinese.py index 7f08c1c07..424def77e 100644 --- a/je_auto_control/gui/language_wrapper/simplified_chinese.py +++ b/je_auto_control/gui/language_wrapper/simplified_chinese.py @@ -1184,6 +1184,14 @@ "menu_view_text_large": "大 (14pt)", "menu_view_text_xlarge": "特大 (16pt)", "menu_view_text_xxlarge": "超大 (20pt)", + "menu_view_search": "搜索功能...", + "menu_view_sidebar": "导航面板", + "menu_view_theme": "主题", + "menu_view_theme_dark": "深色", + "menu_view_theme_light": "浅色", + "nav_search_placeholder": "搜索功能 (Ctrl+K)", + "nav_no_results": "没有匹配的功能。", + "nav_open_hint": "已在工作区打开", "menu_tools": "工具", "menu_tools_start_hotkeys": "启动热键守护进程", "menu_tools_start_scheduler": "启动调度器", diff --git a/je_auto_control/gui/language_wrapper/traditional_chinese.py b/je_auto_control/gui/language_wrapper/traditional_chinese.py index 9f9a645ed..db29e507d 100644 --- a/je_auto_control/gui/language_wrapper/traditional_chinese.py +++ b/je_auto_control/gui/language_wrapper/traditional_chinese.py @@ -1185,6 +1185,14 @@ "menu_view_text_large": "大 (14pt)", "menu_view_text_xlarge": "特大 (16pt)", "menu_view_text_xxlarge": "超大 (20pt)", + "menu_view_search": "搜尋功能...", + "menu_view_sidebar": "導覽面板", + "menu_view_theme": "佈景主題", + "menu_view_theme_dark": "深色", + "menu_view_theme_light": "淺色", + "nav_search_placeholder": "搜尋功能 (Ctrl+K)", + "nav_no_results": "沒有符合的功能。", + "nav_open_hint": "已在工作區開啟", "menu_tools": "工具", "menu_tools_start_hotkeys": "啟動熱鍵服務", "menu_tools_start_scheduler": "啟動排程器", diff --git a/je_auto_control/gui/main_widget.py b/je_auto_control/gui/main_widget.py index b543cf55c..4ee4e4ac0 100644 --- a/je_auto_control/gui/main_widget.py +++ b/je_auto_control/gui/main_widget.py @@ -1,6 +1,6 @@ +"""Workspace widget: owns the tab bar and the registry of every tab it can open.""" import json -from dataclasses import dataclass -from typing import Optional +from typing import Any, Dict, List, Optional, Union from PySide6.QtCore import QTimer, Signal, QObject from PySide6.QtGui import QKeyEvent, Qt @@ -12,80 +12,25 @@ from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._image_detect_tab import ImageDetectTabMixin from je_auto_control.gui._record_tab import RecordTabMixin +from je_auto_control.gui._report_tab import ReportTabMixin from je_auto_control.gui._screenshot_tab import ScreenshotTabMixin from je_auto_control.gui._script_tab import ScriptTabMixin -from je_auto_control.gui.accessibility_tab import AccessibilityTab -from je_auto_control.gui.assertions_tab import AssertionsTab -from je_auto_control.gui.data_source_tab import DataSourceTab -from je_auto_control.gui.flakiness_tab import FlakinessTab -from je_auto_control.gui.test_suite_tab import TestSuiteTab -from je_auto_control.gui.a11y_audit_tab import A11yAuditTab -from je_auto_control.gui.device_matrix_tab import DeviceMatrixTab -from je_auto_control.gui.media_checks_tab import MediaChecksTab -from je_auto_control.gui.computer_use_tab import ComputerUseTab -from je_auto_control.gui.chatops_tab import ChatOpsTab -from je_auto_control.gui.dag_tab import DagTab -from je_auto_control.gui.trace_replay_tab import TraceReplayTab -from je_auto_control.gui._report_tab import ReportTabMixin -from je_auto_control.gui.hotkeys_tab import HotkeysTab from je_auto_control.gui.language_wrapper.multi_language_wrapper import language_wrapper -from je_auto_control.gui.live_hud_tab import LiveHUDTab -from je_auto_control.gui.llm_planner_tab import LLMPlannerTab -from je_auto_control.gui.ocr_tab import OCRReaderTab -from je_auto_control.gui.plugins_tab import PluginsTab -from je_auto_control.gui.presence_tab import PresenceTab -from je_auto_control.gui.profiler_tab import ProfilerTab -from je_auto_control.gui.secrets_tab import SecretsTab -from je_auto_control.gui.admin_console_tab import AdminConsoleTab -from je_auto_control.gui.audit_log_tab import AuditLogTab -from je_auto_control.gui.diagnostics_tab import DiagnosticsTab -from je_auto_control.gui.inspector_tab import InspectorTab -from je_auto_control.gui.recording_editor_tab import RecordingEditorTab -from je_auto_control.gui.usb_browser_tab import UsbBrowserTab -from je_auto_control.gui.usb_devices_tab import UsbDevicesTab -from je_auto_control.gui.usb_passthrough_panel import UsbPassthroughPanel -# Remote desktop relies on the optional `webrtc` extra (aiortc + PyAV). -# Importing it eagerly would break embedders (e.g. PyBreeze) that install -# je_auto_control without the extra; fall back to a placeholder tab that -# tells the user how to enable it. -try: - from je_auto_control.gui.remote_desktop_tab import RemoteDesktopTab - _REMOTE_DESKTOP_IMPORT_ERROR: Optional[ImportError] = None -except ImportError as _remote_desktop_error: - RemoteDesktopTab = None # type: ignore[assignment,misc] # reason: name is a class or None - _REMOTE_DESKTOP_IMPORT_ERROR = _remote_desktop_error -from je_auto_control.gui.rest_api_tab import RestApiTab -from je_auto_control.gui.run_history_tab import RunHistoryTab -from je_auto_control.gui.scheduler_tab import SchedulerTab -from je_auto_control.gui.flow_editor import FlowEditorTab -from je_auto_control.gui.script_builder import ScriptBuilderTab -from je_auto_control.gui.self_healing_tab import SelfHealingTab -from je_auto_control.gui.triggers_tab import TriggersTab -from je_auto_control.gui.webhooks_tab import WebhooksTab -from je_auto_control.gui.email_triggers_tab import EmailTriggersTab -from je_auto_control.gui.variables_tab import VariablesTab -from je_auto_control.gui.vlm_tab import VLMTab -from je_auto_control.gui.webrunner_tab import WebRunnerTab -from je_auto_control.gui.window_tab import WindowManagerTab +from je_auto_control.gui.tab_registry import ( + TAB_SPECS, MenuActions, TabEntry, WidgetFactory, lazy_factory, +) from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.json.json_file import read_action_json +# Kept under its old name: tests and embedders read ``_tab_entries`` rows as this. +_TabEntry = TabEntry + class _WorkerSignals(QObject): finished = Signal(str) error = Signal(str) -@dataclass -class _TabEntry: - key: str - title_key: str - widget: QWidget - category: str = "core" - default_visible: bool = False - actions: tuple = () - - # ============================================================================= # Main Widget # ============================================================================= @@ -103,154 +48,29 @@ def __init__(self, parent=None): super().__init__(parent) self._tr_init() layout = QVBoxLayout() + layout.setContentsMargins(0, 0, 0, 0) - self._tab_entries: list = [] + self._tab_entries: List[TabEntry] = [] self.tabs = QTabWidget() self.tabs.setTabsClosable(True) + self.tabs.setDocumentMode(True) + self.tabs.setUsesScrollButtons(True) + # Without this the widest page's minimum size is the window's: the + # Script Builder alone kept the window from going under ~860 px. + self.tabs.setMinimumSize(360, 240) self.tabs.tabCloseRequested.connect(self._on_tab_close_requested) - # Default UI keeps only the last three of the previously-visible - # tabs (record / script_builder / remote_desktop) so the launcher - # opens on a focused capture+script+remote workflow. The earlier - # core tabs (auto_click / screenshot / image_detect) are still - # registered and reachable from the View menu's "show tab" list. - self._add_tab("auto_click", "tab_auto_click", self._build_auto_click_tab(), - category="core", actions=( - ("start", self._start_auto_click), - ("stop", self._stop_auto_click), - ("get_position", self._get_mouse_pos), - ("hotkey_send", self._send_hotkey), - ("write_send", self._send_write), - ("scroll_send", self._send_scroll), - )) - self._add_tab("screenshot", "tab_screenshot", self._build_screenshot_tab(), - category="core", actions=( - ("take_screenshot", self._take_screenshot), - ("browse", self._browse_ss_path), - ("pick_region", self._pick_ss_region), - ("get_screen_size", self._get_screen_size), - ("get_pixel_label", self._get_pixel_color), - )) - self._add_tab("image_detect", "tab_image_detect", self._build_image_detect_tab(), - category="core", actions=( - ("browse", self._browse_img), - ("crop_template", self._crop_template), - ("locate_image", self._locate_image), - ("locate_all", self._locate_all), - ("locate_click", self._locate_click), - )) - self._add_tab("record", "tab_record", self._build_record_tab(), - category="core", default_visible=True, actions=( - ("start_record", self._start_record), - ("stop_record", self._stop_record), - ("playback", self._playback_record), - ("save_record", self._save_record), - ("load_record", self._load_record), - )) - self._add_tab("script_builder", "tab_script_builder", ScriptBuilderTab(), - category="core", default_visible=True) - self._add_tab("flow_editor", "tab_flow_editor", FlowEditorTab(), - category="editing") - self._add_tab("script", "tab_script", self._build_script_tab(), - category="editing", actions=( - ("load_script", self._browse_script), - ("execute_script", self._execute_script), - ("menu_choose_script_dir", self._browse_script_dir), - ("execute_dir", self._execute_dir), - ("execute_editor_script", self._execute_manual_script), - )) - self._add_tab("recording_editor", "tab_recording_editor", RecordingEditorTab(), - category="editing") - self._add_tab("variables", "tab_variables", VariablesTab(), - category="editing") - self._add_tab("secrets", "tab_secrets", SecretsTab(), - category="editing") - self._add_tab("vlm", "tab_vlm", VLMTab(), - category="detection") - self._add_tab("self_healing", "tab_self_healing", SelfHealingTab(), - category="detection") - self._add_tab("ocr_reader", "tab_ocr_reader", OCRReaderTab(), - category="detection") - self._add_tab("accessibility", "tab_accessibility", AccessibilityTab(), - category="detection") - self._add_tab("live_hud", "tab_live_hud", LiveHUDTab(), - category="detection") - self._add_tab("llm_planner", "tab_llm_planner", LLMPlannerTab(), - category="detection") - self._add_tab("computer_use", "tab_computer_use", ComputerUseTab(), - category="detection") - self._add_tab("scheduler", "tab_scheduler", SchedulerTab(), - category="automation") - self._add_tab("hotkeys", "tab_hotkeys", HotkeysTab(), - category="automation") - self._add_tab("triggers", "tab_triggers", TriggersTab(), - category="automation") - self._add_tab("webhooks", "tab_webhooks", WebhooksTab(), - category="automation") - self._add_tab("email_triggers", "tab_email_triggers", - EmailTriggersTab(), category="automation") - self._add_tab("test_suite", "tab_test_suite", TestSuiteTab(), - category="core") - self._add_tab("assertions", "tab_assertions", AssertionsTab(), - category="core") - self._add_tab("data_source", "tab_data_source", DataSourceTab(), - category="core") - self._add_tab("flakiness", "tab_flakiness", FlakinessTab(), - category="system") - self._add_tab("a11y_audit", "tab_a11y_audit", A11yAuditTab(), - category="core") - self._add_tab("device_matrix", "tab_device_matrix", DeviceMatrixTab(), - category="core") - self._add_tab("media_checks", "tab_media_checks", MediaChecksTab(), - category="core") - self._add_tab("run_history", "tab_run_history", RunHistoryTab(), - category="automation") - self._add_tab("profiler", "tab_profiler", ProfilerTab(), - category="automation") - self._add_tab("window_manager", "tab_window_manager", WindowManagerTab(), - category="system") - self._add_tab("plugins", "tab_plugins", PluginsTab(), - category="system") - self._add_tab("webrunner", "tab_webrunner", WebRunnerTab(), - category="automation") - self._add_tab("dag_runner", "tab_dag_runner", DagTab(), - category="automation") - self._add_tab("chatops", "tab_chatops", ChatOpsTab(), - category="automation") - self._add_tab("trace_replay", "tab_trace_replay", TraceReplayTab(), - category="automation") - self._add_tab( - "remote_desktop", "tab_remote_desktop", - self._build_remote_desktop_tab(), - category="system", default_visible=True, - ) - self._add_tab("presence", "tab_presence", PresenceTab(), - category="system") - self._add_tab("rest_api", "tab_rest_api", RestApiTab(), - category="system") - self._add_tab("admin_console", "tab_admin_console", AdminConsoleTab(), - category="system") - self._add_tab("audit_log", "tab_audit_log", AuditLogTab(), - category="system") - self._add_tab("inspector", "tab_inspector", InspectorTab(), - category="system") - self._add_tab("usb_devices", "tab_usb_devices", UsbDevicesTab(), - category="system") - self._add_tab("usb_browser", "tab_usb_browser", UsbBrowserTab(), - category="system") - self._add_tab("usb_share", "tab_usb_share", UsbPassthroughPanel(), - category="system") - self._add_tab("diagnostics", "tab_diagnostics", DiagnosticsTab(), - category="system") - self._add_tab("report", "tab_report", self._build_report_tab(), - category="system", actions=( - ("enable_test_record", self._enable_test_record), - ("disable_test_record", self._disable_test_record), - ("generate_html_report", self._gen_html), - ("generate_json_report", self._gen_json), - ("generate_xml_report", self._gen_xml), - )) + # The window opens on record / script_builder / remote_desktop. Every + # other tab is registered but not built: its module is imported and + # its widget constructed the first time it is opened, from the + # navigation panel or the View menu. + own_tabs = self._own_tab_builders() + for spec in TAB_SPECS: + builder, actions = own_tabs.get(spec.key, (None, ())) + self._add_tab(spec.key, spec.title_key, + builder or lazy_factory(spec.module, spec.class_name), + category=spec.category, default_visible=spec.default_visible, actions=actions) layout.addWidget(self.tabs) self.setLayout(layout) @@ -262,11 +82,74 @@ def __init__(self, parent=None): self.repeat_max = 0 self._record_data = [] + def _own_tab_builders(self) -> Dict[str, Any]: + """Tabs this widget builds itself: ``key -> (builder, menu actions)``. + + The mixin tabs are built now, not on first open: their builders create + attributes other slots read (``script_path_input``, the record status + label), and they are plain forms with nothing running behind them. + """ + return { + "auto_click": (self._build_auto_click_tab(), ( + ("start", self._start_auto_click), + ("stop", self._stop_auto_click), + ("get_position", self._get_mouse_pos), + ("hotkey_send", self._send_hotkey), + ("write_send", self._send_write), + ("scroll_send", self._send_scroll), + )), + "screenshot": (self._build_screenshot_tab(), ( + ("take_screenshot", self._take_screenshot), + ("browse", self._browse_ss_path), + ("pick_region", self._pick_ss_region), + ("get_screen_size", self._get_screen_size), + ("get_pixel_label", self._get_pixel_color), + )), + "image_detect": (self._build_image_detect_tab(), ( + ("browse", self._browse_img), + ("crop_template", self._crop_template), + ("locate_image", self._locate_image), + ("locate_all", self._locate_all), + ("locate_click", self._locate_click), + )), + "record": (self._build_record_tab(), ( + ("start_record", self._start_record), + ("stop_record", self._stop_record), + ("playback", self._playback_record), + ("save_record", self._save_record), + ("load_record", self._load_record), + )), + "script": (self._build_script_tab(), ( + ("load_script", self._browse_script), + ("execute_script", self._execute_script), + ("menu_choose_script_dir", self._browse_script_dir), + ("execute_dir", self._execute_dir), + ("execute_editor_script", self._execute_manual_script), + )), + "remote_desktop": (self._build_remote_desktop_tab, ()), + "report": (self._build_report_tab(), ( + ("enable_test_record", self._enable_test_record), + ("disable_test_record", self._disable_test_record), + ("generate_html_report", self._gen_html), + ("generate_json_report", self._gen_json), + ("generate_xml_report", self._gen_xml), + )), + } + @staticmethod def _build_remote_desktop_tab() -> QWidget: """Return the real remote-desktop tab, or a placeholder if the - ``webrtc`` extra is not installed.""" - if RemoteDesktopTab is not None: + ``webrtc`` extra is not installed. + + Remote desktop relies on the optional ``webrtc`` extra (aiortc + PyAV); + embedders such as PyBreeze install je_auto_control without it, so a + failed import becomes a tab that says how to enable it. + """ + try: + from je_auto_control.gui.remote_desktop_tab import RemoteDesktopTab + except ImportError as error: + import_error: ImportError = error + else: return RemoteDesktopTab() placeholder = QWidget() layout = QVBoxLayout(placeholder) @@ -274,7 +157,7 @@ def _build_remote_desktop_tab() -> QWidget: "Remote Desktop is unavailable: the optional 'webrtc' extra " "(aiortc + PyAV) is not installed.\n\n" "Install with:\n pip install je_auto_control[webrtc]\n\n" - f"Underlying error: {_REMOTE_DESKTOP_IMPORT_ERROR!r}", + f"Underlying error: {import_error!r}", ) message.setWordWrap(True) message.setTextInteractionFlags(Qt.TextSelectableByMouse) @@ -285,24 +168,43 @@ def _build_remote_desktop_tab() -> QWidget: # --- tab registry API ---------------------------------------------------- def _add_tab( - self, key: str, title_key: str, widget: QWidget, + self, key: str, title_key: str, widget: Union[QWidget, WidgetFactory], category: str = "core", default_visible: bool = False, - actions: tuple = (), + actions: MenuActions = (), ) -> None: - self._tab_entries.append(_TabEntry( - key=key, title_key=title_key, widget=widget, - category=category, default_visible=default_visible, - actions=actions, - )) - if default_visible: - self.tabs.addTab(widget, language_wrapper.translate(title_key, title_key)) + """Register a tab from a widget, or from a factory called on first open.""" + if isinstance(widget, QWidget): + built = widget + entry = TabEntry(key=key, title_key=title_key, factory=lambda: built, + category=category, default_visible=default_visible, actions=actions) else: - # Owned from the start: an unparented hidden tab outlived this - # widget, and one a registry held a listener of (Presence) kept - # its timer running after every window that built it was gone. + entry = TabEntry(key=key, title_key=title_key, factory=widget, + category=category, default_visible=default_visible, actions=actions, + on_build=lambda page: AutoControlGUIWidget._adopt_hidden_tab(self, page)) + self._tab_entries.append(entry) + if default_visible: + self.tabs.addTab(entry.widget, language_wrapper.translate(title_key, title_key)) + elif isinstance(widget, QWidget): + AutoControlGUIWidget._adopt_hidden_tab(self, entry.widget) + + def _adopt_hidden_tab(self, widget: QWidget) -> None: + # Owned from the start: an unparented hidden tab outlived this + # widget, and one a registry held a listener of (Presence) kept + # its timer running after every window that built it was gone. + if self.tabs.indexOf(widget) == -1: widget.setParent(self) widget.hide() + @staticmethod + def _is_built(entry: Any) -> bool: + return bool(getattr(entry, "built", True)) + + def _built_entries(self) -> List[Any]: + return [entry for entry in self._tab_entries if self._is_built(entry)] + + def _is_open(self, entry: Any) -> bool: + return self._is_built(entry) and self.tabs.indexOf(entry.widget) != -1 + def _on_current_tab_changed(self, _index: int) -> None: self.current_tab_changed.emit() @@ -317,7 +219,7 @@ def current_tab_menu_actions(self) -> list: widget = self.tabs.currentWidget() if widget is None: return [] - for entry in self._tab_entries: + for entry in self._built_entries(): if entry.widget is widget: if entry.actions: return list(entry.actions) @@ -334,42 +236,71 @@ def _find_entry(self, key: str): return None def sync_engine_tabs(self) -> None: - """Let every tab that mirrors an engine re-read its state (after Tools > Start).""" + """Let every built tab that mirrors an engine re-read its state (after Tools > Start). + + A tab not built yet reads the engine when it is. + """ for entry in self._tab_entries: + if not AutoControlGUIWidget._is_built(entry): + continue sync = getattr(entry.widget, "sync_with_engine", None) if callable(sync): sync() def list_registered_tabs(self) -> list: - """Return metadata for the View → Tabs menu.""" + """Return metadata for the View → Tabs menu and the navigation panel. + + Reading it builds nothing: a tab never opened is listed as not visible. + """ return [ { "key": entry.key, "title": language_wrapper.translate(entry.title_key, entry.title_key), - "visible": self.tabs.indexOf(entry.widget) != -1, + "visible": self._is_open(entry), "category": entry.category, } for entry in self._tab_entries ] + def current_tab_key(self) -> Optional[str]: + """Key of the tab on screen, or ``None`` when every tab is closed.""" + widget = self.tabs.currentWidget() + for entry in self._built_entries(): + if entry.widget is widget: + return str(entry.key) + return None + def show_tab(self, key: str) -> None: + """Open the tab ``key`` (building it if this is its first time) and select it.""" entry = self._find_entry(key) - if entry is None or self.tabs.indexOf(entry.widget) != -1: + if entry is None or self._is_open(entry): return target_index = 0 for candidate in self._tab_entries: if candidate.key == key: break - if self.tabs.indexOf(candidate.widget) != -1: + if self._is_open(candidate): target_index += 1 title = language_wrapper.translate(entry.title_key, entry.title_key) self.tabs.insertTab(target_index, entry.widget, title) self.tabs.setCurrentWidget(entry.widget) self.tabs_changed.emit() - def hide_tab(self, key: str) -> None: + def activate_tab(self, key: str) -> bool: + """Bring the tab ``key`` to the front, opening it first if needed.""" entry = self._find_entry(key) if entry is None: + return False + if self._is_open(entry): + self.tabs.setCurrentWidget(entry.widget) + else: + self.show_tab(key) + return True + + def hide_tab(self, key: str) -> None: + """Close the tab ``key``; its widget is kept for the next time it is opened.""" + entry = self._find_entry(key) + if entry is None or not self._is_built(entry): return index = self.tabs.indexOf(entry.widget) if index != -1: @@ -378,7 +309,7 @@ def hide_tab(self, key: str) -> None: def _on_tab_close_requested(self, index: int) -> None: widget = self.tabs.widget(index) - for entry in self._tab_entries: + for entry in self._built_entries(): if entry.widget is widget: self.hide_tab(entry.key) return @@ -388,7 +319,7 @@ def _translate(self, key: str) -> str: def retranslate(self) -> None: """Relabel tab titles and propagate into every child tab.""" - for entry in self._tab_entries: + for entry in self._built_entries(): index = self.tabs.indexOf(entry.widget) if index != -1: self.tabs.setTabText( @@ -404,7 +335,7 @@ def retranslate(self) -> None: if hasattr(self, "_record_retranslate"): self._record_retranslate() # Child class tabs get their own retranslate if they implement one. - for entry in self._tab_entries: + for entry in self._built_entries(): callback = getattr(entry.widget, "retranslate", None) if callable(callback) and entry.widget is not self: try: @@ -415,7 +346,7 @@ def retranslate(self) -> None: def open_script_file(self, path: str) -> None: """Load a JSON script into the Script Executor tab and focus it.""" entry = self._find_entry("script") - if entry is not None and self.tabs.indexOf(entry.widget) == -1: + if entry is not None and not self._is_open(entry): self.show_tab("script") self.script_path_input.setText(path) try: diff --git a/je_auto_control/gui/main_window.py b/je_auto_control/gui/main_window.py index 8ff7f29c8..a8ef06859 100644 --- a/je_auto_control/gui/main_window.py +++ b/je_auto_control/gui/main_window.py @@ -1,17 +1,20 @@ -"""Top-level window with menu bar, closable tabs, and live language switching.""" +"""Top-level window: menu bar, feature navigation, tabbed workspace, themes, live language switching.""" import sys from PySide6.QtCore import Qt -from PySide6.QtGui import QAction, QActionGroup +from PySide6.QtGui import QAction, QActionGroup, QKeySequence from PySide6.QtWidgets import ( - QApplication, QFileDialog, QMainWindow, QMenu, QMessageBox, + QApplication, QDockWidget, QFileDialog, QMainWindow, QMenu, QMessageBox, QWidget, ) -from qt_material import QtStyleTools from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( language_wrapper, ) from je_auto_control.gui.main_widget import AutoControlGUIWidget +from je_auto_control.gui.navigation import NavigationPanel +from je_auto_control.gui.theme import ( + DEFAULT_THEME, THEMES, apply_theme, font_rule, prepare_application, theme_named, +) def _t(key: str, default: str = "") -> str: @@ -36,8 +39,14 @@ def _t(key: str, default: str = "") -> str: ) -class AutoControlGUIUI(QMainWindow, QtStyleTools): - """Main window: menu bar + AutoControlGUIWidget (which owns the tabs).""" +_THEME_LABELS = { + "dark": ("menu_view_theme_dark", "Dark"), + "light": ("menu_view_theme_light", "Light"), +} + + +class AutoControlGUIUI(QMainWindow): + """Main window: menu bar, navigation panel and AutoControlGUIWidget (which owns the tabs).""" def __init__(self) -> None: super().__init__() @@ -47,24 +56,25 @@ def __init__(self) -> None: windll.shell32.SetCurrentProcessExplicitAppUserModelID(self.app_id) self._user_font_pt: int = 0 # 0 means auto-detect from screen - self.apply_stylesheet(self, "dark_amber.xml") - # qt_material writes the theme into this window's stylesheet; capture it - # so _apply_font_pt can append the font rule instead of replacing (and - # thereby wiping) the theme. - self._theme_stylesheet: str = self.styleSheet() + self._theme_name: str = DEFAULT_THEME + # The theme's style sheet is kept so _apply_font_pt can append the + # font rule instead of replacing (and thereby wiping) the theme. + self._theme_stylesheet: str = apply_theme(self, theme_named(self._theme_name)) self._apply_font_pt(self._user_font_pt) self.setWindowTitle(_t("application_name", "AutoControlGUI")) - self.resize(1000, 760) + self.resize(1280, 800) self.auto_control_gui_widget = AutoControlGUIWidget(parent=self) self.setCentralWidget(self.auto_control_gui_widget) + self._build_navigation() self._view_menu: QMenu = None self._actions_menu: QMenu = None self._tab_actions: list = [] self._build_menu_bar() self.auto_control_gui_widget.tabs_changed.connect(self._rebuild_tabs_menu) + self.auto_control_gui_widget.tabs_changed.connect(self._refresh_navigation) self.auto_control_gui_widget.tabs_changed.connect(self._rebuild_actions_menu) self.auto_control_gui_widget.current_tab_changed.connect( self._rebuild_actions_menu, @@ -75,6 +85,41 @@ def __init__(self) -> None: listener = self._on_language_changed self.destroyed.connect(lambda *_args: language_wrapper.remove_listener(listener)) + # --- navigation ---------------------------------------------------------- + + def _build_navigation(self) -> None: + """Dock the searchable feature list on the left of the workspace.""" + self.navigation = NavigationPanel(self) + self.navigation.feature_activated.connect(self.auto_control_gui_widget.activate_tab) + self._navigation_dock = QDockWidget(self) + self._navigation_dock.setObjectName("NavigationDock") + self._navigation_dock.setFeatures(QDockWidget.DockWidgetFeature.NoDockWidgetFeatures) + self._navigation_dock.setTitleBarWidget(QWidget(self._navigation_dock)) + self._navigation_dock.setWidget(self.navigation) + self.addDockWidget(Qt.DockWidgetArea.LeftDockWidgetArea, self._navigation_dock) + self.resizeDocks([self._navigation_dock], [260], Qt.Orientation.Horizontal) + # Owned by the window, not by a menu: the menu bar is rebuilt on every + # language switch, and a shortcut on a per-menu action would be + # registered once more each time. + self._search_action = QAction(self) + self._search_action.setShortcut(QKeySequence("Ctrl+K")) + self._search_action.triggered.connect(self._focus_feature_search) + self.addAction(self._search_action) + self._sidebar_action = self._navigation_dock.toggleViewAction() + self._sidebar_action.setShortcut(QKeySequence("Ctrl+B")) + self.addAction(self._sidebar_action) + self._refresh_navigation() + + def _refresh_navigation(self) -> None: + self.navigation.set_entries( + self.auto_control_gui_widget.list_registered_tabs(), _TAB_CATEGORIES, + ) + + def _focus_feature_search(self) -> None: + """Show the navigation panel and put the cursor in its search box (Ctrl+K).""" + self._navigation_dock.setVisible(True) + self.navigation.focus_search() + # --- menu construction --------------------------------------------------- def _build_menu_bar(self) -> None: @@ -122,14 +167,44 @@ def _build_file_menu(self) -> QMenu: def _build_view_menu(self) -> QMenu: menu = QMenu(_t("menu_view", "View"), self) + self._search_action.setText(_t("menu_view_search", "Search Features...")) + menu.addAction(self._search_action) + self._sidebar_action.setText(_t("menu_view_sidebar", "Navigation Panel")) + menu.addAction(self._sidebar_action) tabs_menu = menu.addMenu(_t("menu_view_tabs", "Tabs")) self._view_menu = tabs_menu self._rebuild_tabs_menu() menu.addSeparator() + theme_menu = menu.addMenu(_t("menu_view_theme", "Theme")) + self._build_theme_menu(theme_menu) text_menu = menu.addMenu(_t("menu_view_text_size", "Text Size")) self._build_text_size_menu(text_menu) return menu + def _build_theme_menu(self, menu: QMenu) -> None: + group = QActionGroup(menu) + group.setExclusive(True) + for name in THEMES: + label_key, default_label = _THEME_LABELS.get(name, (name, name.title())) + action = QAction(_t(label_key, default_label), menu, checkable=True) + action.setData(name) + action.setChecked(name == self._theme_name) + action.triggered.connect(self._on_theme_selected) + group.addAction(action) + menu.addAction(action) + + def _on_theme_selected(self) -> None: + action = self.sender() + if isinstance(action, QAction) and action.data(): + self.set_theme(str(action.data())) + + def set_theme(self, name: str) -> None: + """Switch to the theme called ``name`` (``dark`` or ``light``), keeping the text size.""" + tokens = theme_named(name) + self._theme_name = tokens.name + self._theme_stylesheet = apply_theme(self, tokens) + self._apply_font_pt(self._user_font_pt) + def _rebuild_tabs_menu(self) -> None: if self._view_menu is None: return @@ -177,21 +252,21 @@ def _detect_auto_font_pt(self) -> int: return 12 height = screen.geometry().height() if height >= 2000: - return 16 + return 13 if height >= 1300: - return 14 - return 12 + return 11 + return 10 def _apply_font_pt(self, pt: int) -> None: """Apply the font size on top of the active theme stylesheet. The theme lives in this window's stylesheet, so the font rule is appended rather than assigned — assigning would replace (and wipe) the - qt_material theme on startup and on every text-size change. + theme on startup and on every text-size change. The font family is + the theme's; only the size is set here. """ effective = pt if pt > 0 else self._detect_auto_font_pt() - font_rule = f"* {{ font-size: {effective}pt; font-family: 'Lato'; }}" - self.setStyleSheet(f"{self._theme_stylesheet}\n{font_rule}") + self.setStyleSheet(f"{self._theme_stylesheet}\n{font_rule(effective)}") def _on_text_size_selected(self) -> None: action = self.sender() @@ -265,6 +340,8 @@ def _on_language_selected(self) -> None: def _on_language_changed(self, _language: str) -> None: self.setWindowTitle(_t("application_name", "AutoControlGUI")) self.auto_control_gui_widget.retranslate() + self.navigation.retranslate() + self._refresh_navigation() self._build_menu_bar() def _on_about(self) -> None: @@ -296,6 +373,7 @@ def _start_triggers(self) -> None: if "__main__" == __name__: app = QApplication(sys.argv) + prepare_application(app) window = AutoControlGUIUI() window.show() sys.exit(app.exec()) diff --git a/je_auto_control/gui/navigation.py b/je_auto_control/gui/navigation.py new file mode 100644 index 000000000..4cc143aef --- /dev/null +++ b/je_auto_control/gui/navigation.py @@ -0,0 +1,200 @@ +"""Searchable feature list for the left side of the main window. + +Every registered tab appears here under its category, open or not, so a +feature is one click or a few typed letters away instead of three menus deep. +The panel only reports which key was chosen; opening the tab stays with the +window that owns the tab registry. +""" +from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple + +from PySide6.QtCore import Qt, Signal +from PySide6.QtGui import QFont, QKeyEvent +from PySide6.QtWidgets import ( + QLabel, QLineEdit, QTreeWidget, QTreeWidgetItem, QVBoxLayout, QWidget, +) + +from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( + language_wrapper, +) + +Category = Tuple[str, str, str] +_KEY_ROLE = Qt.ItemDataRole.UserRole + + +def _t(key: str, default: str) -> str: + return language_wrapper.translate(key, default) + + +def matches(query: str, entry: Dict[str, Any], category_label: str = "") -> bool: + """Whether every word of ``query`` occurs in the entry's title, key or category.""" + haystack = " ".join((str(entry.get("title", "")), str(entry.get("key", "")).replace("_", " "), + str(entry.get("key", "")), category_label)).casefold() + return all(word in haystack for word in query.casefold().split()) + + +class _SearchField(QLineEdit): + """Line edit that hands Down / Return to the list below it.""" + + move_down = Signal() + + def keyPressEvent(self, event: QKeyEvent) -> None: # noqa: N802 # reason: Qt override + if event.key() == Qt.Key.Key_Down: + self.move_down.emit() + return + super().keyPressEvent(event) + + +class NavigationPanel(QWidget): + """Search box over a category tree of every registered tab.""" + + feature_activated = Signal(str) + + def __init__(self, parent: Optional[QWidget] = None) -> None: + super().__init__(parent) + self.setObjectName("NavigationPanel") + self.setAttribute(Qt.WidgetAttribute.WA_StyledBackground, True) + self._entries: List[Dict[str, Any]] = [] + self._categories: Sequence[Category] = () + + self.search = _SearchField(self) + self.search.setClearButtonEnabled(True) + self.search.textChanged.connect(self.apply_filter) + self.search.returnPressed.connect(self.activate_first_match) + self.search.move_down.connect(self._focus_first_match) + + self.tree = QTreeWidget(self) + self.tree.setHeaderHidden(True) + self.tree.setRootIsDecorated(False) + self.tree.setIndentation(12) + self.tree.setUniformRowHeights(True) + self.tree.itemClicked.connect(self._on_item_chosen) + self.tree.itemActivated.connect(self._on_item_chosen) + + self.empty = QLabel(self) + self.empty.setObjectName("NavigationEmpty") + self.empty.setWordWrap(True) + self.empty.setAlignment(Qt.AlignmentFlag.AlignTop | Qt.AlignmentFlag.AlignHCenter) + self.empty.hide() + + layout = QVBoxLayout(self) + layout.setContentsMargins(8, 8, 8, 8) + layout.setSpacing(8) + layout.addWidget(self.search) + layout.addWidget(self.tree, 1) + layout.addWidget(self.empty, 1) + self.setMinimumWidth(200) + self.retranslate() + + # --- content ------------------------------------------------------------- + + def set_entries(self, entries: Iterable[Dict[str, Any]], categories: Sequence[Category]) -> None: + """Show ``entries`` (``key`` / ``title`` / ``category`` / ``visible``) grouped by ``categories``. + + A category is ``(key, title_key, default_title)``; entries of a + category not listed come last, under the category key itself. + """ + self._entries = [dict(entry) for entry in entries] + self._categories = tuple(categories) + self._rebuild() + + def _category_labels(self) -> List[Tuple[str, str]]: + labels = [(key, _t(title_key, default)) for key, title_key, default in self._categories] + known = {key for key, _label in labels} + for entry in self._entries: + category = str(entry.get("category", "")) + if category not in known: + known.add(category) + labels.append((category, category.title())) + return labels + + def _rebuild(self) -> None: + self.tree.clear() + heading = QFont(self.font()) + heading.setBold(True) + opened = QFont(self.font()) + opened.setBold(True) + for category, label in self._category_labels(): + members = [entry for entry in self._entries if entry.get("category") == category] + if not members: + continue + group = QTreeWidgetItem(self.tree, [label]) + group.setFont(0, heading) + group.setFlags(Qt.ItemFlag.ItemIsEnabled) + for entry in members: + item = QTreeWidgetItem(group, [str(entry.get("title", entry.get("key", "")))]) + item.setData(0, _KEY_ROLE, entry.get("key")) + if entry.get("visible"): + item.setFont(0, opened) + item.setToolTip(0, _t("nav_open_hint", "Open in the workspace")) + group.setExpanded(True) + self.apply_filter(self.search.text()) + + # --- search -------------------------------------------------------------- + + def apply_filter(self, text: str = "") -> int: + """Hide what ``text`` does not match; return how many features remain.""" + shown = 0 + for index in range(self.tree.topLevelItemCount()): + group = self.tree.topLevelItem(index) + in_group = 0 + for row in range(group.childCount()): + item = group.child(row) + entry = {"title": item.text(0), "key": item.data(0, _KEY_ROLE)} + hit = matches(text, entry, group.text(0)) + item.setHidden(not hit) + in_group += int(hit) + group.setHidden(in_group == 0) + shown += in_group + self.tree.setVisible(shown > 0) + self.empty.setVisible(shown == 0) + return shown + + def visible_keys(self) -> List[str]: + """Keys of the features the current search leaves on screen, top to bottom.""" + keys: List[str] = [] + for index in range(self.tree.topLevelItemCount()): + group = self.tree.topLevelItem(index) + for row in range(group.childCount()): + item = group.child(row) + if not item.isHidden(): + keys.append(str(item.data(0, _KEY_ROLE))) + return keys + + def _first_match(self) -> Optional[QTreeWidgetItem]: + for index in range(self.tree.topLevelItemCount()): + group = self.tree.topLevelItem(index) + for row in range(group.childCount()): + if not group.child(row).isHidden(): + return group.child(row) + return None + + def activate_first_match(self) -> None: + """Open the first feature the search shows (Return in the search box).""" + item = self._first_match() + if item is not None: + self._on_item_chosen(item) + + def _focus_first_match(self) -> None: + item = self._first_match() + if item is not None: + self.tree.setCurrentItem(item) + self.tree.setFocus(Qt.FocusReason.ShortcutFocusReason) + + def focus_search(self) -> None: + """Put the cursor in the search box with its text selected.""" + self.search.setFocus(Qt.FocusReason.ShortcutFocusReason) + self.search.selectAll() + + def _on_item_chosen(self, item: QTreeWidgetItem, _column: int = 0) -> None: + key = item.data(0, _KEY_ROLE) + if key: + self.feature_activated.emit(str(key)) + + # --- language ------------------------------------------------------------ + + def retranslate(self) -> None: + """Re-read every label from the language wrapper.""" + self.search.setPlaceholderText(_t("nav_search_placeholder", "Search features (Ctrl+K)")) + self.empty.setText(_t("nav_no_results", "No feature matches this search.")) + if self._entries: + self._rebuild() diff --git a/je_auto_control/gui/tab_registry.py b/je_auto_control/gui/tab_registry.py new file mode 100644 index 000000000..424b18266 --- /dev/null +++ b/je_auto_control/gui/tab_registry.py @@ -0,0 +1,123 @@ +"""Tab registry: what the workspace can open, and when each tab is built. + +Every feature tab is described by a :class:`TabSpec` — key, title key, +category and where its widget class lives — and nothing more is loaded until +the tab is opened. :class:`TabEntry` holds one registered tab and builds the +widget on first access to ``widget``, so the window no longer imports and +constructs some fifty panels (with the timers and helper threads several of +them start) before it can show the three it opens on. +""" +from dataclasses import dataclass, field +from importlib import import_module +from typing import Any, Callable, Optional, Tuple + +MenuActions = Tuple[Tuple[str, Callable[[], Any]], ...] +WidgetFactory = Callable[[], Any] + + +@dataclass(frozen=True) +class TabSpec: + """A tab the workspace can open; ``module`` is imported on first use.""" + + key: str + title_key: str + category: str + module: str = "" + class_name: str = "" + default_visible: bool = False + + +def lazy_factory(module: str, class_name: str) -> WidgetFactory: + """Return a factory that imports ``module`` and builds ``class_name``.""" + def build() -> Any: + return getattr(import_module(module), class_name)() + return build + + +@dataclass +class TabEntry: + """One registered tab; the widget is built the first time it is needed.""" + + key: str + title_key: str + factory: WidgetFactory + category: str = "core" + default_visible: bool = False + actions: MenuActions = () + on_build: Optional[Callable[[Any], None]] = None + _widget: Any = field(default=None, repr=False) + + @property + def built(self) -> bool: + """Whether the widget exists yet.""" + return self._widget is not None + + @property + def widget(self) -> Any: + """The tab's widget, building it on first access.""" + if self._widget is None: + self._widget = self.factory() + if self.on_build is not None: + self.on_build(self._widget) + return self._widget + + +_GUI = "je_auto_control.gui" + +# Registration order: it is the order of the View > Tabs menu, of the +# navigation panel inside each category, and of the tab bar. A spec without a +# module is one the main widget builds itself (its mixin tabs, and Remote +# Desktop, which needs an optional extra and falls back to a placeholder). +TAB_SPECS: Tuple[TabSpec, ...] = ( + TabSpec("auto_click", "tab_auto_click", "core"), + TabSpec("screenshot", "tab_screenshot", "core"), + TabSpec("image_detect", "tab_image_detect", "core"), + TabSpec("record", "tab_record", "core", default_visible=True), + TabSpec("script_builder", "tab_script_builder", "core", + f"{_GUI}.script_builder", "ScriptBuilderTab", default_visible=True), + TabSpec("flow_editor", "tab_flow_editor", "editing", f"{_GUI}.flow_editor", "FlowEditorTab"), + TabSpec("script", "tab_script", "editing"), + TabSpec("recording_editor", "tab_recording_editor", "editing", + f"{_GUI}.recording_editor_tab", "RecordingEditorTab"), + TabSpec("variables", "tab_variables", "editing", f"{_GUI}.variables_tab", "VariablesTab"), + TabSpec("secrets", "tab_secrets", "editing", f"{_GUI}.secrets_tab", "SecretsTab"), + TabSpec("vlm", "tab_vlm", "detection", f"{_GUI}.vlm_tab", "VLMTab"), + TabSpec("self_healing", "tab_self_healing", "detection", f"{_GUI}.self_healing_tab", "SelfHealingTab"), + TabSpec("ocr_reader", "tab_ocr_reader", "detection", f"{_GUI}.ocr_tab", "OCRReaderTab"), + TabSpec("accessibility", "tab_accessibility", "detection", f"{_GUI}.accessibility_tab", "AccessibilityTab"), + TabSpec("live_hud", "tab_live_hud", "detection", f"{_GUI}.live_hud_tab", "LiveHUDTab"), + TabSpec("llm_planner", "tab_llm_planner", "detection", f"{_GUI}.llm_planner_tab", "LLMPlannerTab"), + TabSpec("computer_use", "tab_computer_use", "detection", f"{_GUI}.computer_use_tab", "ComputerUseTab"), + TabSpec("scheduler", "tab_scheduler", "automation", f"{_GUI}.scheduler_tab", "SchedulerTab"), + TabSpec("hotkeys", "tab_hotkeys", "automation", f"{_GUI}.hotkeys_tab", "HotkeysTab"), + TabSpec("triggers", "tab_triggers", "automation", f"{_GUI}.triggers_tab", "TriggersTab"), + TabSpec("webhooks", "tab_webhooks", "automation", f"{_GUI}.webhooks_tab", "WebhooksTab"), + TabSpec("email_triggers", "tab_email_triggers", "automation", + f"{_GUI}.email_triggers_tab", "EmailTriggersTab"), + TabSpec("test_suite", "tab_test_suite", "core", f"{_GUI}.test_suite_tab", "TestSuiteTab"), + TabSpec("assertions", "tab_assertions", "core", f"{_GUI}.assertions_tab", "AssertionsTab"), + TabSpec("data_source", "tab_data_source", "core", f"{_GUI}.data_source_tab", "DataSourceTab"), + TabSpec("flakiness", "tab_flakiness", "system", f"{_GUI}.flakiness_tab", "FlakinessTab"), + TabSpec("a11y_audit", "tab_a11y_audit", "core", f"{_GUI}.a11y_audit_tab", "A11yAuditTab"), + TabSpec("device_matrix", "tab_device_matrix", "core", f"{_GUI}.device_matrix_tab", "DeviceMatrixTab"), + TabSpec("media_checks", "tab_media_checks", "core", f"{_GUI}.media_checks_tab", "MediaChecksTab"), + TabSpec("run_history", "tab_run_history", "automation", f"{_GUI}.run_history_tab", "RunHistoryTab"), + TabSpec("profiler", "tab_profiler", "automation", f"{_GUI}.profiler_tab", "ProfilerTab"), + TabSpec("window_manager", "tab_window_manager", "system", f"{_GUI}.window_tab", "WindowManagerTab"), + TabSpec("plugins", "tab_plugins", "system", f"{_GUI}.plugins_tab", "PluginsTab"), + TabSpec("webrunner", "tab_webrunner", "automation", f"{_GUI}.webrunner_tab", "WebRunnerTab"), + TabSpec("dag_runner", "tab_dag_runner", "automation", f"{_GUI}.dag_tab", "DagTab"), + TabSpec("chatops", "tab_chatops", "automation", f"{_GUI}.chatops_tab", "ChatOpsTab"), + TabSpec("trace_replay", "tab_trace_replay", "automation", f"{_GUI}.trace_replay_tab", "TraceReplayTab"), + TabSpec("remote_desktop", "tab_remote_desktop", "system", default_visible=True), + TabSpec("presence", "tab_presence", "system", f"{_GUI}.presence_tab", "PresenceTab"), + TabSpec("rest_api", "tab_rest_api", "system", f"{_GUI}.rest_api_tab", "RestApiTab"), + TabSpec("admin_console", "tab_admin_console", "system", f"{_GUI}.admin_console_tab", "AdminConsoleTab"), + TabSpec("audit_log", "tab_audit_log", "system", f"{_GUI}.audit_log_tab", "AuditLogTab"), + TabSpec("inspector", "tab_inspector", "system", f"{_GUI}.inspector_tab", "InspectorTab"), + TabSpec("usb_devices", "tab_usb_devices", "system", f"{_GUI}.usb_devices_tab", "UsbDevicesTab"), + TabSpec("usb_browser", "tab_usb_browser", "system", f"{_GUI}.usb_browser_tab", "UsbBrowserTab"), + TabSpec("usb_share", "tab_usb_share", "system", f"{_GUI}.usb_passthrough_panel", "UsbPassthroughPanel"), + TabSpec("diagnostics", "tab_diagnostics", "system", f"{_GUI}.diagnostics_tab", "DiagnosticsTab"), + TabSpec("report", "tab_report", "system"), +) diff --git a/je_auto_control/gui/theme.py b/je_auto_control/gui/theme.py new file mode 100644 index 000000000..9a6ad7124 --- /dev/null +++ b/je_auto_control/gui/theme.py @@ -0,0 +1,178 @@ +"""Design tokens and the stylesheet built from them. + +One :class:`ThemeTokens` value holds every colour, radius and spacing the +window uses; :func:`build_stylesheet` turns it into Qt style sheet text and +:func:`apply_theme` puts it on a window together with a matching palette, so +the parts Qt draws itself (arrows, check marks, scroll handles) agree with it. +Nothing here loads an image or a font file. +""" +from dataclasses import dataclass +from typing import Any, Dict, Optional + +FONT_FAMILY = ('"Segoe UI Variable Text", "Segoe UI", "SF Pro Text", "Helvetica Neue", ' + '"Noto Sans", "Microsoft JhengHei UI", "PingFang TC", "Noto Sans CJK TC", sans-serif') + + +@dataclass(frozen=True) +class ThemeTokens: + """Colours and metrics for one theme.""" + + name: str + window: str + surface: str + surface_raised: str + border: str + text: str + text_muted: str + accent: str + accent_hover: str + accent_text: str + selection: str + hover: str + danger: str + radius: int = 6 + spacing: int = 8 + font_family: str = FONT_FAMILY + + +DARK = ThemeTokens( + name="dark", window="#16181d", surface="#1d2026", surface_raised="#262a32", border="#343944", + text="#e7e9ee", text_muted="#9aa1ae", accent="#5b9dff", accent_hover="#7ab0ff", accent_text="#0c1220", + selection="#2c4470", hover="#2b303a", danger="#ff6b6b", +) +LIGHT = ThemeTokens( + name="light", window="#f4f5f8", surface="#ffffff", surface_raised="#eceef3", border="#d3d7df", + text="#1c2029", text_muted="#5f6775", accent="#2563eb", accent_hover="#1d4fd0", accent_text="#ffffff", + selection="#d4e2ff", hover="#e6e9f0", danger="#c62828", +) +THEMES: Dict[str, ThemeTokens] = {DARK.name: DARK, LIGHT.name: LIGHT} +DEFAULT_THEME = DARK.name + + +def theme_named(name: str) -> ThemeTokens: + """Return the theme called ``name``, or the default for an unknown name.""" + return THEMES.get(name, THEMES[DEFAULT_THEME]) + + +def font_rule(point_size: int) -> str: + """Style sheet rule setting the text size everywhere.""" + return f"* {{ font-size: {int(point_size)}pt; }}" + + +_TEMPLATE = """ +QWidget {{ background-color: {window}; color: {text}; font-family: {font_family}; }} +QMainWindow, QDialog, QDockWidget {{ background-color: {window}; }} +QLabel, QCheckBox, QRadioButton {{ background: transparent; }} +QToolTip {{ background-color: {surface_raised}; color: {text}; border: 1px solid {border}; + padding: {half}px {spacing}px; border-radius: {radius}px; }} + +QMenuBar {{ background-color: {window}; border-bottom: 1px solid {border}; padding: 2px {half}px; }} +QMenuBar::item {{ background: transparent; padding: {half}px {spacing}px; border-radius: {radius}px; }} +QMenuBar::item:selected, QMenuBar::item:pressed {{ background-color: {hover}; }} +QMenu {{ background-color: {surface}; border: 1px solid {border}; border-radius: {radius}px; padding: {half}px; }} +QMenu::item {{ padding: {half}px {wide}px {half}px {wide}px; border-radius: {small}px; }} +QMenu::item:selected {{ background-color: {selection}; }} +QMenu::item:disabled {{ color: {text_muted}; }} +QMenu::separator {{ height: 1px; background: {border}; margin: {half}px {spacing}px; }} + +QTabWidget::pane {{ border: 1px solid {border}; border-radius: {radius}px; background-color: {surface}; top: -1px; }} +QTabBar {{ background: transparent; qproperty-drawBase: 0; }} +QTabBar::tab {{ background: transparent; color: {text_muted}; padding: {spacing}px {wide}px; + border: 1px solid transparent; border-top-left-radius: {radius}px; border-top-right-radius: {radius}px; + margin-right: 2px; }} +QTabBar::tab:hover {{ color: {text}; background-color: {hover}; }} +QTabBar::tab:selected {{ color: {text}; background-color: {surface}; border-color: {border}; + border-bottom: 2px solid {accent}; }} + +QPushButton, QToolButton {{ background-color: {surface_raised}; border: 1px solid {border}; + border-radius: {radius}px; padding: {half}px {wide}px; min-height: 20px; }} +QPushButton:hover, QToolButton:hover {{ background-color: {hover}; border-color: {accent}; }} +QPushButton:pressed, QToolButton:pressed {{ background-color: {selection}; }} +QPushButton:default {{ background-color: {accent}; color: {accent_text}; border-color: {accent}; }} +QPushButton:default:hover {{ background-color: {accent_hover}; }} +QPushButton:disabled, QToolButton:disabled {{ color: {text_muted}; background-color: {window}; }} +QTabBar QToolButton {{ padding: 0; min-height: 0; border-radius: {small}px; }} + +QLineEdit, QTextEdit, QPlainTextEdit, QSpinBox, QDoubleSpinBox, QComboBox, QDateTimeEdit, QTimeEdit {{ + background-color: {surface}; border: 1px solid {border}; border-radius: {radius}px; + padding: {half}px {spacing}px; selection-background-color: {selection}; selection-color: {text}; }} +QLineEdit:focus, QTextEdit:focus, QPlainTextEdit:focus, QSpinBox:focus, QDoubleSpinBox:focus, +QComboBox:focus {{ border-color: {accent}; }} +QLineEdit:disabled, QTextEdit:disabled, QPlainTextEdit:disabled, QComboBox:disabled {{ color: {text_muted}; }} +QComboBox QAbstractItemView {{ background-color: {surface}; border: 1px solid {border}; + selection-background-color: {selection}; selection-color: {text}; outline: 0; }} + +QTreeView, QListView, QTableView {{ background-color: {surface}; alternate-background-color: {window}; + border: 1px solid {border}; border-radius: {radius}px; gridline-color: {border}; outline: 0; }} +QTreeView::item, QListView::item {{ padding: {small}px {half}px; border-radius: {small}px; }} +QTreeView::item:hover, QListView::item:hover, QTableView::item:hover {{ background-color: {hover}; }} +QTreeView::item:selected, QListView::item:selected, QTableView::item:selected {{ + background-color: {selection}; color: {text}; }} +QHeaderView::section {{ background-color: {surface_raised}; color: {text_muted}; border: 0; + border-bottom: 1px solid {border}; padding: {half}px {spacing}px; }} +QTableCornerButton::section {{ background-color: {surface_raised}; border: 0; }} + +QGroupBox {{ border: 1px solid {border}; border-radius: {radius}px; margin-top: {wide}px; + padding: {spacing}px; background-color: transparent; }} +QGroupBox::title {{ subcontrol-origin: margin; left: {spacing}px; padding: 0 {half}px; color: {text_muted}; }} + +QScrollBar:vertical {{ background: transparent; width: 10px; margin: 0; }} +QScrollBar:horizontal {{ background: transparent; height: 10px; margin: 0; }} +QScrollBar::handle {{ background-color: {border}; border-radius: 4px; min-height: 24px; min-width: 24px; }} +QScrollBar::handle:hover {{ background-color: {text_muted}; }} +QScrollBar::add-line, QScrollBar::sub-line {{ width: 0; height: 0; }} +QScrollBar::add-page, QScrollBar::sub-page {{ background: transparent; }} + +QSplitter::handle {{ background-color: {border}; }} +QSplitter::handle:horizontal {{ width: 1px; }} +QSplitter::handle:vertical {{ height: 1px; }} +QProgressBar {{ background-color: {surface_raised}; border: 0; border-radius: {small}px; text-align: center; }} +QProgressBar::chunk {{ background-color: {accent}; border-radius: {small}px; }} +QStatusBar {{ background-color: {window}; border-top: 1px solid {border}; color: {text_muted}; }} + +#NavigationPanel {{ background-color: {surface}; border-right: 1px solid {border}; }} +#NavigationPanel QTreeWidget {{ background-color: {surface}; border: 0; border-radius: 0; }} +#NavigationPanel QLineEdit {{ background-color: {window}; }} +#NavigationEmpty {{ color: {text_muted}; padding: {wide}px; }} +""" + + +def build_stylesheet(tokens: ThemeTokens) -> str: + """Return the Qt style sheet for ``tokens``.""" + values = dict(vars(tokens)) + values.update(half=tokens.spacing // 2, wide=tokens.spacing * 2, small=max(tokens.radius - 2, 2)) + return _TEMPLATE.format(**values).strip() + + +def _palette(tokens: ThemeTokens) -> Any: + from PySide6.QtGui import QColor, QPalette + role = QPalette.ColorRole + palette = QPalette() + for target, colour in ( + (role.Window, tokens.window), (role.WindowText, tokens.text), (role.Base, tokens.surface), + (role.AlternateBase, tokens.window), (role.Text, tokens.text), (role.Button, tokens.surface_raised), + (role.ButtonText, tokens.text), (role.ToolTipBase, tokens.surface_raised), + (role.ToolTipText, tokens.text), (role.Highlight, tokens.accent), + (role.HighlightedText, tokens.accent_text), (role.PlaceholderText, tokens.text_muted), + (role.Link, tokens.accent), (role.BrightText, tokens.danger)): + palette.setColor(target, QColor(colour)) + for target in (role.WindowText, role.Text, role.ButtonText): + palette.setColor(QPalette.ColorGroup.Disabled, target, QColor(tokens.text_muted)) + return palette + + +def apply_theme(window: Any, tokens: ThemeTokens, point_size: Optional[int] = None) -> str: + """Style ``window`` with ``tokens``; return the style sheet without the font rule. + + ``point_size`` appends :func:`font_rule`, so the size is set on top of the + theme instead of replacing it. + """ + sheet = build_stylesheet(tokens) + window.setPalette(_palette(tokens)) + window.setStyleSheet(sheet if point_size is None else f"{sheet}\n{font_rule(point_size)}") + return sheet + + +def prepare_application(app: Any) -> None: + """Give ``app`` the Fusion style, which follows the palette on every platform.""" + app.setStyle("Fusion") diff --git a/test/unit_test/headless/test_gui_lazy_registry.py b/test/unit_test/headless/test_gui_lazy_registry.py new file mode 100644 index 000000000..f711236c0 --- /dev/null +++ b/test/unit_test/headless/test_gui_lazy_registry.py @@ -0,0 +1,154 @@ +"""The tab registry: every tab is listed, and none is built before it is opened. + +The table and ``TabEntry`` are checked without Qt. What the real widget +builds at start-up is measured in a child process: in this one, other test +modules have long since imported most tab modules. +""" +import importlib.util +import json +import os +import pathlib +import subprocess # nosec B404 # reason: runs this file's own probe with a fixed argv +import sys + +import pytest + +from je_auto_control.gui.tab_registry import TAB_SPECS, TabEntry, lazy_factory + +ROOT = pathlib.Path(__file__).resolve().parents[3] +# Built by the main widget itself: its mixin tabs, and Remote Desktop's placeholder fallback. +OWN_TABS = {"auto_click", "screenshot", "image_detect", "record", "script", "remote_desktop", "report"} +OPEN_AT_START = ["record", "script_builder", "remote_desktop"] + + +def test_keys_are_unique_and_the_start_tabs_are_the_three_documented(): + keys = [spec.key for spec in TAB_SPECS] + assert len(keys) == len(set(keys)) + assert [spec.key for spec in TAB_SPECS if spec.default_visible] == OPEN_AT_START + assert {spec.key for spec in TAB_SPECS if not spec.module} == OWN_TABS + + +def test_importing_the_registry_loads_no_qt(): + probe = ("import sys; import je_auto_control.gui.tab_registry; " + "print(any(name.startswith('PySide6') for name in sys.modules))") + done = subprocess.run([sys.executable, "-c", probe], capture_output=True, text=True, # nosec B603 # nosemgrep # reason: fixed argv, no shell + timeout=120, cwd=str(ROOT), check=False) + assert done.returncode == 0, done.stderr[-2000:] + assert done.stdout.strip() == "False" + + +@pytest.mark.parametrize("spec", [spec for spec in TAB_SPECS if spec.module], ids=lambda spec: spec.key) +def test_every_lazy_tab_names_a_class_that_exists(spec): + """Read, not imported: a typo here would otherwise surface when a user opens the tab.""" + found = importlib.util.find_spec(spec.module) + assert found is not None and found.origin, spec.module + source = pathlib.Path(found.origin).read_text(encoding="utf-8") + assert (f"class {spec.class_name}(" in source or f"import {spec.class_name}" in source + or f"{spec.class_name}," in source or f'"{spec.class_name}"' in source), (spec.module, spec.class_name) + + +def test_an_entry_builds_its_widget_once_and_only_when_asked(): + built, adopted = [], [] + + def factory(): + built.append(object()) + return built[-1] + + entry = TabEntry("k", "tab_k", factory, on_build=adopted.append) + assert not entry.built and built == [] + first = entry.widget + assert entry.built and entry.widget is first + assert built == [first] and adopted == [first] + + +def test_a_lazy_factory_imports_at_call_time(): + factory = lazy_factory("collections", "OrderedDict") + assert type(factory()).__name__ == "OrderedDict" + with pytest.raises(ModuleNotFoundError): + lazy_factory("je_auto_control.gui.no_such_tab", "Missing")() + + +_PROBE = r""" +import json, os, sys +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +from PySide6.QtWidgets import QApplication +from je_auto_control.gui.main_widget import AutoControlGUIWidget +from je_auto_control.gui.tab_registry import TAB_SPECS + +app = QApplication.instance() or QApplication([]) +widget = AutoControlGUIWidget() +entries = {entry.key: entry for entry in widget._tab_entries} +report = { + "registered": [entry.key for entry in widget._tab_entries], + "specs": [spec.key for spec in TAB_SPECS], + "built_at_start": [key for key, entry in entries.items() if entry.built], + "open_at_start": [row["key"] for row in widget.list_registered_tabs() if row["visible"]], + "variables_imported_at_start": "je_auto_control.gui.variables_tab" in sys.modules, + "presence_imported_at_start": "je_auto_control.gui.presence_tab" in sys.modules, +} +widget.list_registered_tabs() +widget.retranslate() +widget.sync_engine_tabs() +widget.hide_tab("variables") +report["built_after_listing"] = [key for key, entry in entries.items() if entry.built] + +changes = [] +widget.tabs_changed.connect(lambda: changes.append(1)) +widget.show_tab("variables") +first = entries["variables"].widget +report["variables_imported_after_open"] = "je_auto_control.gui.variables_tab" in sys.modules +report["current_after_open"] = widget.current_tab_key() +widget.hide_tab("variables") +report["hidden_is_owned"] = widget.isAncestorOf(first) and widget.tabs.indexOf(first) == -1 +report["activated"] = widget.activate_tab("variables") +report["same_widget_on_reopen"] = entries["variables"].widget is first +report["tab_order"] = [row["key"] for row in widget.list_registered_tabs() if row["visible"]] +widget.activate_tab("record") +report["current_after_activate"] = widget.current_tab_key() +report["changes"] = len(changes) +report["unknown"] = widget.activate_tab("no_such_tab") +sys.stdout.write(json.dumps(report)) +sys.stdout.flush() +os._exit(0) +""" + + +@pytest.fixture(scope="module") +def report(): + pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) + env = dict(os.environ, PYTHONPATH=str(ROOT)) + env.setdefault("QT_QPA_PLATFORM", "offscreen") + done = subprocess.run([sys.executable, "-c", _PROBE], capture_output=True, text=True, # nosec B603 # nosemgrep # reason: fixed argv, no shell + timeout=180, env=env, cwd=str(ROOT), check=False) + assert done.returncode == 0, done.stderr[-2000:] + return json.loads(done.stdout) + + +def test_every_spec_is_registered_in_order(report): + assert report["registered"] == report["specs"] + + +def test_only_the_start_tabs_and_the_widgets_own_forms_are_built(report): + assert set(report["built_at_start"]) == OWN_TABS | set(OPEN_AT_START) + assert report["open_at_start"] == OPEN_AT_START + assert not report["variables_imported_at_start"] + assert not report["presence_imported_at_start"] + + +def test_listing_translating_and_hiding_build_nothing(report): + assert report["built_after_listing"] == report["built_at_start"] + + +def test_opening_builds_the_tab_and_reopening_reuses_it(report): + assert report["variables_imported_after_open"] + assert report["current_after_open"] == "variables" + assert report["hidden_is_owned"] + assert report["activated"] and report["same_widget_on_reopen"] + # Registration order, not opening order: Variables sits between Script Builder and Remote Desktop. + assert report["tab_order"] == ["record", "script_builder", "variables", "remote_desktop"] + + +def test_activating_an_open_tab_only_selects_it(report): + assert report["current_after_activate"] == "record" + assert report["changes"] == 3 # opened, hidden, opened again + assert report["unknown"] is False diff --git a/test/unit_test/headless/test_gui_slots_menu_audit.py b/test/unit_test/headless/test_gui_slots_menu_audit.py index e006301e8..cff7b4567 100644 --- a/test/unit_test/headless/test_gui_slots_menu_audit.py +++ b/test/unit_test/headless/test_gui_slots_menu_audit.py @@ -184,8 +184,6 @@ def sample_pixel(_x, _y): def test_rebuilding_the_tabs_menu_does_not_leak(): - # main_window imports qt_material (its theme), which the headless CI job does not install. - pytest.importorskip("qt_material", exc_type=ImportError) app = _app() from PySide6.QtCore import QCoreApplication, QEvent from PySide6.QtWidgets import QMenu, QWidget diff --git a/test/unit_test/headless/test_gui_tab_lifecycle_audit.py b/test/unit_test/headless/test_gui_tab_lifecycle_audit.py index 584c3f985..1bdd195c9 100644 --- a/test/unit_test/headless/test_gui_tab_lifecycle_audit.py +++ b/test/unit_test/headless/test_gui_tab_lifecycle_audit.py @@ -281,6 +281,8 @@ def test_each_usb_prompt_dialog_is_deleted(qapp, monkeypatch): before = (len(language_wrapper._listeners), len(default_presence_registry()._listeners)) from je_auto_control.gui.main_window import AutoControlGUIUI window = AutoControlGUIUI() + # Tabs are built on first open; Presence is the one holding a registry listener. + window.auto_control_gui_widget.show_tab("presence") during = (len(language_wrapper._listeners), len(default_presence_registry()._listeners)) window.deleteLater() del window @@ -292,9 +294,6 @@ def test_each_usb_prompt_dialog_is_deleted(qapp, monkeypatch): def test_destroying_the_window_removes_its_listeners(): - # The main window styles itself with qt_material, which the headless CI - # jobs do not install; the probe imports the real window. - pytest.importorskip("qt_material", exc_type=ImportError) env = dict(os.environ, PYTHONPATH=str(_REPO_ROOT), QT_QPA_PLATFORM="offscreen") argv = [sys.executable, "-c", _WINDOW_PROBE] done = subprocess.run(argv, capture_output=True, text=True, timeout=180, env=env, cwd=str(_REPO_ROOT), check=False) # nosec B603 # nosemgrep # reason: this test's own probe, fixed argv diff --git a/test/unit_test/headless/test_gui_workspace_navigation.py b/test/unit_test/headless/test_gui_workspace_navigation.py new file mode 100644 index 000000000..a54e806bf --- /dev/null +++ b/test/unit_test/headless/test_gui_workspace_navigation.py @@ -0,0 +1,226 @@ +"""Theme tokens, the navigation panel's search, and the window that hosts both.""" +import json +import os +import pathlib +import re +import subprocess # nosec B404 # reason: runs this file's own probe with a fixed argv +import sys + +import pytest + +from je_auto_control.gui.tab_registry import TAB_SPECS +from je_auto_control.gui.theme import ( + DARK, DEFAULT_THEME, LIGHT, THEMES, build_stylesheet, font_rule, theme_named, +) + +CATEGORIES = (("core", "menu_view_cat_core", "Core"), ("system", "menu_view_cat_system", "System")) +ENTRIES = [ + {"key": "record", "title": "Record / Playback", "category": "core", "visible": True}, + {"key": "usb_devices", "title": "USB Devices", "category": "system", "visible": False}, + {"key": "usb_browser", "title": "USB Browser", "category": "system", "visible": False}, + {"key": "live_hud", "title": "Live HUD", "category": "detection", "visible": False}, +] + + +# --- theme: no Qt needed ----------------------------------------------------- + +@pytest.mark.parametrize("tokens", [DARK, LIGHT], ids=lambda tokens: tokens.name) +def test_the_stylesheet_is_complete(tokens): + sheet = build_stylesheet(tokens) + assert not re.search(r"\{[a-z_]+\}", sheet), "an unfilled placeholder is left" + assert sheet.count("{") == sheet.count("}") + for colour in (tokens.window, tokens.surface, tokens.text, tokens.accent, tokens.border, tokens.selection): + assert colour in sheet + assert "#NavigationPanel" in sheet and "QTabBar::tab:selected" in sheet + + +def test_the_two_themes_differ_and_an_unknown_name_falls_back(): + assert set(THEMES) == {"dark", "light"} + assert build_stylesheet(DARK) != build_stylesheet(LIGHT) + assert theme_named("light") is LIGHT + assert theme_named("no-such-theme") is THEMES[DEFAULT_THEME] + + +def test_the_font_rule_sets_only_the_size(): + assert font_rule(14) == "* { font-size: 14pt; }" + + +def test_text_contrast_is_readable_in_both_themes(): + def luminance(colour): + channels = [int(colour[i:i + 2], 16) / 255 for i in (1, 3, 5)] + linear = [c / 12.92 if c <= 0.03928 else ((c + 0.055) / 1.055) ** 2.4 for c in channels] + return 0.2126 * linear[0] + 0.7152 * linear[1] + 0.0722 * linear[2] + + def contrast(first, second): + high, low = sorted((luminance(first), luminance(second)), reverse=True) + return (high + 0.05) / (low + 0.05) + + for tokens in (DARK, LIGHT): + assert contrast(tokens.text, tokens.window) >= 7, tokens.name + assert contrast(tokens.text, tokens.surface) >= 7, tokens.name + assert contrast(tokens.text_muted, tokens.surface) >= 4.5, tokens.name + assert contrast(tokens.accent_text, tokens.accent) >= 4.5, tokens.name + + +# --- navigation panel -------------------------------------------------------- + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") + + +@pytest.fixture(scope="module") +def qapp(): + widgets = pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) + return widgets.QApplication.instance() or widgets.QApplication([]) + + +@pytest.fixture +def panel(qapp): + from je_auto_control.gui.navigation import NavigationPanel + navigation = NavigationPanel() + navigation.set_entries(ENTRIES, CATEGORIES) + yield navigation + navigation.deleteLater() + + +def test_every_entry_is_listed_under_its_category(panel): + assert panel.visible_keys() == ["record", "usb_devices", "usb_browser", "live_hud"] + groups = [panel.tree.topLevelItem(i).text(0) for i in range(panel.tree.topLevelItemCount())] + assert groups == ["Core", "System", "Detection"] # an unlisted category comes last + + +def test_search_matches_title_key_and_category(panel): + assert panel.apply_filter("usb") == 2 + assert panel.visible_keys() == ["usb_devices", "usb_browser"] + assert panel.apply_filter("USB brow") == 1 # every word, any case + assert panel.apply_filter("live_hud") == 1 # the key as written in scripts + assert panel.apply_filter("system") == 2 # the category name + assert panel.apply_filter("") == 4 + + +def test_no_match_shows_the_empty_state_instead_of_a_blank_list(panel): + assert panel.apply_filter("zzz") == 0 + assert panel.tree.isHidden() and not panel.empty.isHidden() + panel.apply_filter("") + assert not panel.tree.isHidden() and panel.empty.isHidden() + + +def test_return_in_the_search_box_opens_the_first_match(panel): + chosen = [] + panel.feature_activated.connect(chosen.append) + panel.search.setText("usb") + panel.search.returnPressed.emit() + assert chosen == ["usb_devices"] + panel.search.setText("zzz") + panel.search.returnPressed.emit() + assert chosen == ["usb_devices"] # nothing to open + + +def test_clicking_a_category_heading_opens_nothing(panel): + chosen = [] + panel.feature_activated.connect(chosen.append) + panel.tree.itemClicked.emit(panel.tree.topLevelItem(0), 0) + panel.tree.itemClicked.emit(panel.tree.topLevelItem(0).child(0), 0) + assert chosen == ["record"] + + +def test_a_search_survives_the_list_being_refreshed(panel): + panel.search.setText("usb") + panel.set_entries(ENTRIES, CATEGORIES) # what a tab opening triggers + assert panel.visible_keys() == ["usb_devices", "usb_browser"] + + +# --- the window -------------------------------------------------------------- +# In a child process: the window builds Remote Desktop and the Script Builder, +# whose native helper threads must not be torn down inside this interpreter. + +_PROBE = r""" +import json, os, sys +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +from PySide6.QtWidgets import QApplication +from je_auto_control.gui.main_window import AutoControlGUIUI +from je_auto_control.gui.tab_registry import TAB_SPECS +from je_auto_control.gui.theme import DARK, LIGHT, font_rule + +app = QApplication.instance() or QApplication([]) +window = AutoControlGUIUI() +window.show() +app.processEvents() +workspace, navigation = window.auto_control_gui_widget, window.navigation +report = {"reachable": sorted(navigation.visible_keys()), "registered": sorted(s.key for s in TAB_SPECS)} + +navigation.search.setText("variables") +navigation.activate_first_match() +report["opened"] = workspace.current_tab_key() +report["listed_open"] = "variables" in {r["key"] for r in workspace.list_registered_tabs() if r["visible"]} +navigation.search.setText("") +workspace.hide_tab("variables") + +window._navigation_dock.setVisible(False) +window._search_action.trigger() +report["panel_shown_by_shortcut"] = not window._navigation_dock.isHidden() +report["shortcuts"] = [window._search_action.shortcut().toString(), window._sidebar_action.shortcut().toString()] + +window._user_font_pt = 14 +window.set_theme("light") +report["light"] = LIGHT.window in window.styleSheet() and font_rule(14) in window.styleSheet() +window.set_theme("dark") +report["dark"] = DARK.window in window.styleSheet() and font_rule(14) in window.styleSheet() +window._user_font_pt = 0 +window._apply_font_pt(0) + +before = len(window.actions()) +window._on_language_changed("English") +window._on_language_changed("English") +report["actions_before"], report["actions_after"] = before, len(window.actions()) +report["placeholder"] = navigation.search.placeholderText() +view = next(a.menu() for a in window.menuBar().actions() if a.menu() is not None + and window._search_action in a.menu().actions()) +report["view_menu"] = [a.text() for a in view.actions() if a.text()] + +window.resize(640, 420) +app.processEvents() +report["small"] = [window.width(), window.height(), workspace.width(), navigation.isVisible()] +sys.stdout.write(json.dumps(report)) +sys.stdout.flush() +os._exit(0) +""" + + +@pytest.fixture(scope="module") +def window(): + pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) + root = pathlib.Path(__file__).resolve().parents[3] + env = dict(os.environ, PYTHONPATH=str(root), QT_QPA_PLATFORM="offscreen") + done = subprocess.run([sys.executable, "-c", _PROBE], capture_output=True, text=True, # nosec B603 # nosemgrep # reason: fixed argv, no shell + timeout=180, env=env, cwd=str(root), check=False) + assert done.returncode == 0, done.stderr[-2000:] + return json.loads(done.stdout) + + +def test_search_reaches_every_registered_feature(window): + assert window["reachable"] == window["registered"] == sorted(spec.key for spec in TAB_SPECS) + + +def test_choosing_a_feature_opens_it_and_marks_it_open(window): + assert window["opened"] == "variables" and window["listed_open"] + + +def test_the_search_shortcut_reveals_a_hidden_panel(window): + assert window["panel_shown_by_shortcut"] + assert window["shortcuts"] == ["Ctrl+K", "Ctrl+B"] + + +def test_switching_theme_keeps_the_text_size(window): + assert window["light"] and window["dark"] + + +def test_rebuilding_the_menus_for_a_language_switch_adds_no_shortcut(window): + assert window["actions_after"] == window["actions_before"] + assert window["placeholder"] + assert window["view_menu"] == ["Search Features...", "Navigation Panel", "Tabs", "Theme", "Text Size"] + + +def test_the_window_shrinks_to_a_small_screen_with_both_panes_usable(window): + width, height, workspace_width, panel_visible = window["small"] + assert (width, height) == (640, 420), "the window refused to shrink" + assert panel_visible and workspace_width >= 300 diff --git a/test/unit_test/headless/test_qt_material_guard.py b/test/unit_test/headless/test_qt_material_guard.py deleted file mode 100644 index cbef871cf..000000000 --- a/test/unit_test/headless/test_qt_material_guard.py +++ /dev/null @@ -1,64 +0,0 @@ -"""Every headless test that imports the main window skips without qt_material. - -``gui/main_window.py`` imports ``qt_material`` for its theme. Developers have -it installed, the ``pytest-headless`` CI job does not, so a test that imports -the window without ``pytest.importorskip("qt_material")`` passes locally and -fails every CI square. This reads the test sources, so it fails on the -developer's machine too. -""" -import ast -from pathlib import Path - -_HEADLESS = Path(__file__).resolve().parent -_WINDOW = "je_auto_control.gui.main_window" - - -def _imports_window(node: ast.AST) -> bool: - if isinstance(node, ast.ImportFrom): - return node.module == _WINDOW or (node.module == "je_auto_control.gui" - and any(alias.name == "main_window" for alias in node.names)) - if isinstance(node, ast.Import): - return any(alias.name == _WINDOW for alias in node.names) - return isinstance(node, ast.Constant) and isinstance(node.value, str) and _WINDOW in node.value - - -def _skips_without_theme(node: ast.AST) -> bool: - return (isinstance(node, ast.Call) and getattr(node.func, "attr", None) == "importorskip" - and bool(node.args) and isinstance(node.args[0], ast.Constant) - and node.args[0].value == "qt_material") - - -def _unguarded(tree: ast.Module) -> list: - """Names of the functions (or ````) that import the window unguarded.""" - if any(_skips_without_theme(node) for statement in tree.body for node in ast.walk(statement) - if not isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))): - return [] - scopes = [node for node in ast.walk(tree) if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))] - missing = [scope.name for scope in scopes - if any(_imports_window(node) for node in ast.walk(scope)) - and not any(_skips_without_theme(node) for node in ast.walk(scope))] - module_level = [node for statement in tree.body - if not isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)) - for node in ast.walk(statement)] - if any(isinstance(node, (ast.Import, ast.ImportFrom)) and _imports_window(node) for node in module_level): - missing.append("") - return missing - - -def test_every_main_window_import_skips_without_qt_material(): - offenders = {} - for path in sorted(_HEADLESS.glob("test_*.py")): - if path.name == Path(__file__).name: - continue # its own fixtures are strings, not imports - missing = _unguarded(ast.parse(path.read_text(encoding="utf-8"))) - if missing: - offenders[path.name] = missing - assert not offenders, offenders - - -def test_the_check_sees_an_unguarded_import(): - source = "def test_x():\n from je_auto_control.gui.main_window import AutoControlGUIUI\n" - assert _unguarded(ast.parse(source)) == ["test_x"] - guarded = ("import pytest\ndef test_x():\n pytest.importorskip('qt_material', exc_type=ImportError)\n" - " from je_auto_control.gui.main_window import AutoControlGUIUI\n") - assert _unguarded(ast.parse(guarded)) == [] diff --git a/test/unit_test/headless/test_r3_gui_main_window.py b/test/unit_test/headless/test_r3_gui_main_window.py index 69202b3cb..22a37bf5f 100644 --- a/test/unit_test/headless/test_r3_gui_main_window.py +++ b/test/unit_test/headless/test_r3_gui_main_window.py @@ -1,9 +1,9 @@ """Round-3 GUI audit regression: applying the font must not wipe the theme. ``_apply_font_pt`` used to call ``setStyleSheet("font-size: ...")`` which -*replaces* the widget stylesheet, discarding the qt_material theme that -``apply_stylesheet`` had just installed (finding 5). The font rule must now be -merged on top of the captured theme stylesheet instead. +*replaces* the widget stylesheet, discarding the theme that had just been +installed (finding 5). The font rule must now be merged on top of the captured +theme stylesheet instead. """ import os @@ -11,10 +11,6 @@ os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) -# main_window imports qt_material (the theme); the headless CI job installs -# PySide6 but not the GUI theme extra, so skip cleanly there rather than erroring -# out collection for the whole suite. -pytest.importorskip("qt_material") from PySide6.QtWidgets import QApplication, QMainWindow # noqa: E402 From 8ab69ab2b11969fd0f4efc7affabea6622c7a78c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 8 Oct 2026 14:04:56 +0000 Subject: [PATCH 02/26] Update mypy requirement from >=2.3.0 to >=2.4.0 Updates the requirements on [mypy](https://github.com/python/mypy) to permit the latest version. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](https://github.com/python/mypy/compare/v2.3.0...v2.4.0) --- updated-dependencies: - dependency-name: mypy dependency-version: 2.4.0 dependency-type: direct:development ... Signed-off-by: dependabot[bot] --- dev_requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev_requirements.txt b/dev_requirements.txt index 541244f75..e2eded39f 100644 --- a/dev_requirements.txt +++ b/dev_requirements.txt @@ -34,4 +34,4 @@ coverage==7.15.4 # The complexity limit in CLAUDE.md is measured by this, both in the # pre-commit list and by test/unit_test/headless/test_complexity_budget.py. radon==6.0.1 -mypy>=2.3.0 +mypy>=2.4.0 From 2a8d2ce3bae762d4d4ac05194cf6d5a846f97178 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 23:40:21 +0800 Subject: [PATCH 03/26] Load the pytest plugin from a top-level module so pytest start-up no longer imports the facade Every pytest run in an environment with this package installed imported about 1,350 modules through the pytest11 entry point, because the plugin was a submodule of the package. je_auto_control_pytest imports only pytest; the old import path re-exports the same objects. --- .github/workflows/quality.yml | 15 ++- CLAUDE.md | 20 ++- architecture.md | 2 +- dev.toml | 7 +- je_auto_control/utils/pytest_plugin/plugin.py | 119 ++--------------- je_auto_control_pytest.py | 122 ++++++++++++++++++ pyproject.toml | 9 +- .../headless/test_coverage_measurement.py | 40 +++--- .../headless/test_pytest_entrypoint_light.py | 61 +++++++++ test/unit_test/headless/test_pytest_plugin.py | 2 +- 10 files changed, 249 insertions(+), 148 deletions(-) create mode 100644 je_auto_control_pytest.py create mode 100644 test/unit_test/headless/test_pytest_entrypoint_light.py diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml index 1be3d695f..c75423f23 100644 --- a/.github/workflows/quality.yml +++ b/.github/workflows/quality.yml @@ -40,7 +40,7 @@ jobs: run: "pip install --only-binary :all: ruff==0.16.0" - name: Run ruff - run: ruff check je_auto_control/ + run: ruff check je_auto_control/ je_auto_control_pytest.py security: runs-on: ubuntu-latest @@ -57,7 +57,7 @@ jobs: run: "pip install --only-binary :all: bandit==1.9.4" - name: Run bandit (recursive, skip tests + i18n dicts) - run: bandit -r je_auto_control/ -c pyproject.toml + run: bandit -r je_auto_control/ je_auto_control_pytest.py -c pyproject.toml pytest-headless: # The suite ran on Windows alone for its whole life, so every @@ -150,10 +150,13 @@ jobs: # would hide exactly the breakage this job exists to find. # # `coverage run -m pytest`, NOT `pytest --cov`, and that is load-bearing: - # this package registers a `pytest11` entry point, so pytest imports - # `je_auto_control.utils.pytest_plugin.plugin` while loading plugins — - # which executes `je_auto_control/__init__.py`, the facade, and with it - # several hundred modules. pytest-cov only starts measuring after that, + # this package registers a `pytest11` entry point, and while it pointed + # at `je_auto_control.utils.pytest_plugin.plugin` pytest imported that + # submodule while loading plugins — which executes + # `je_auto_control/__init__.py`, the facade, and with it several hundred + # modules. (The entry point is the top-level `je_auto_control_pytest` + # now; `coverage run` stays because it does not depend on what a plugin + # imports.) pytest-cov only starts measuring after that, # so every one of those modules' import-time lines was recorded as never # executed. Measured on this tree: 11,962 statements, 24 points, and the # worst-hit files were the biggest ones (`action_executor` +786, diff --git a/CLAUDE.md b/CLAUDE.md index 458a3cb6a..d7a4010c3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,13 +33,19 @@ python -m build # build ``` **Coverage is measured with `coverage run -m pytest`, never `pytest --cov`.** -This package registers a `pytest11` entry point, so pytest imports -`je_auto_control.utils.pytest_plugin.plugin` — and with it the whole facade — -while loading plugins, before pytest-cov starts. Several hundred modules then -have their import-time lines recorded as never executed: measured, that is -11,962 statements and ~24 percentage points (52.22% vs 72.05% on the same -suite). `test/unit_test/headless/test_coverage_measurement.py` holds CI to the -correct spelling. +This package registers a `pytest11` entry point. While it pointed at +`je_auto_control.utils.pytest_plugin.plugin`, pytest imported that submodule — +and with it the whole facade — while loading plugins, before pytest-cov starts. +Several hundred modules then had their import-time lines recorded as never +executed: measured, that was 11,962 statements and ~24 percentage points +(52.22% vs 72.05% on the same suite). The entry point is now the top-level +module `je_auto_control_pytest`, which imports only pytest +(`test_pytest_entrypoint_light.py` keeps it so), but `coverage run` remains the +rule: it does not depend on what a plugin imports or on which build is +installed, and `pytest --cov` has not been re-measured since. +`test/unit_test/headless/test_coverage_measurement.py` holds CI to the +correct spelling. After changing the entry point, reinstall (`pip install -e .`) +for it to take effect. **Measure it with the `[webrtc]` extra installed**, which is why it is in the line above. Eleven modules under `utils/remote_desktop` raise `ImportError` at diff --git a/architecture.md b/architecture.md index 627fbc98b..d5e112d30 100644 --- a/architecture.md +++ b/architecture.md @@ -49,7 +49,7 @@ entry points → execution core (`utils/executor/`) → headless capabilities (` | MCP server | `je_auto_control_mcp` → `utils/mcp_server/__main__.py:main` | stdio; `start_mcp_stdio_server()`; HTTP transport via the `AC_start_mcp_http_server` command. | | REST API | `je_auto_control start-rest`, `python -m je_auto_control.utils.rest_api`, `start_rest_api_server()` | Default `127.0.0.1:9939`, bearer token + rate limit. | | TCP server | `je_auto_control start-server`, `start_autocontrol_socket_server()` | `utils/socket_server/auto_control_socket_server.py`, default `127.0.0.1:9938`, JSON action lists. | -| pytest plugin | `pytest11` entry point `je_auto_control.utils.pytest_plugin.plugin` | Loaded automatically once the package is installed (see coverage rule in §7). | +| pytest plugin | `pytest11` entry point `je_auto_control_pytest` (top-level module, imports only pytest; `je_auto_control.utils.pytest_plugin` re-exports it) | Loaded automatically once the package is installed, without importing the package (see coverage rule in §7). | | LSP | `autocontrol-lsp` → `autocontrol_lsp.server.server:run`; `python -m autocontrol_lsp.server` | Command list is read from the live executor. | | GUI | `start_autocontrol_gui()` in `gui/__init__.py`; `exe/start_autocontrol_gui.py` | Needs `pip install je_auto_control[gui]`; PySide6 is imported only under `gui/`. | | Action lint | `python -m je_auto_control.utils.action_lint` | Used by `.github/workflows/action-json-lint.yml`. | diff --git a/dev.toml b/dev.toml index 6220026a2..ac1a0c941 100644 --- a/dev.toml +++ b/dev.toml @@ -48,7 +48,9 @@ je_auto_control = "je_auto_control.cli:main" je_auto_control_mcp = "je_auto_control.utils.mcp_server.__main__:main" [project.entry-points.pytest11] -je_auto_control = "je_auto_control.utils.pytest_plugin.plugin" +# A top-level module that imports only pytest: as a submodule of the package +# it made every pytest start-up on the machine import the whole facade. +je_auto_control = "je_auto_control_pytest" [project.urls] Homepage = "https://github.com/Integration-Automation/AutoControlGUI" @@ -59,6 +61,9 @@ Code = "https://github.com/Integration-Automation/AutoControlGUI" file = "README.md" content-type = "text/markdown" +[tool.setuptools] +py-modules = ["je_auto_control_pytest"] + [tool.setuptools.packages] find = { namespaces = false } diff --git a/je_auto_control/utils/pytest_plugin/plugin.py b/je_auto_control/utils/pytest_plugin/plugin.py index ad0c4686d..1408084df 100644 --- a/je_auto_control/utils/pytest_plugin/plugin.py +++ b/je_auto_control/utils/pytest_plugin/plugin.py @@ -1,113 +1,16 @@ -"""pytest plugin for AutoControl. +"""pytest plugin for AutoControl: the names of ``je_auto_control_pytest``. -Exposes: - -* ``autocontrol`` fixture — the package module itself, so a test can - call ``autocontrol.screen_size()`` without re-importing it; -* ``autocontrol_screenshot_dir`` fixture — a ``pathlib.Path`` to - ``tmp_path/'autocontrol_screenshots'``; created on first access; -* ``@pytest.mark.autocontrol`` marker — when a test marked with it - fails, the plugin captures a screenshot to the screenshot dir and - attaches the path to the failure report. - -Registered via the ``pytest11`` entry point so installing -``je_auto_control`` makes the plugin available automatically. +The plugin itself lives in the top-level module ``je_auto_control_pytest`` so +that the ``pytest11`` entry point can load it without importing this package. +This module keeps the old import path, and +``pytest_plugins = ["je_auto_control.utils.pytest_plugin"]``, working: every +name here is the same object as there. """ -from __future__ import annotations - -import os -from pathlib import Path -from typing import Optional - -import pytest - - -_MARKER_NAME = "autocontrol" -_DEFAULT_SUBDIR = "autocontrol_screenshots" - - -def pytest_configure(config: "pytest.Config") -> None: - """Register the ``autocontrol`` marker so ``--strict-markers`` is happy.""" - config.addinivalue_line( - "markers", - f"{_MARKER_NAME}: AutoControl GUI test; capture a screenshot on failure.", - ) - - -@pytest.fixture -def autocontrol(): - """Return the ``je_auto_control`` module without forcing a global import.""" - import je_auto_control - return je_auto_control - - -@pytest.fixture -def autocontrol_screenshot_dir(tmp_path) -> Path: - """Per-test directory under ``tmp_path`` for screenshots / artefacts.""" - target = tmp_path / _DEFAULT_SUBDIR - target.mkdir(parents=True, exist_ok=True) - return target - - -@pytest.fixture -def autocontrol_executor(): - """Yield the executor singleton (callable via ``executor.event_dict``).""" - from je_auto_control.utils.executor.action_executor import executor - return executor - - -def _capture_failure_screenshot(item: "pytest.Item", - directory: Path) -> Optional[Path]: - """Best-effort screenshot capture on failure; returns the path or None.""" - name = item.nodeid.replace("/", "_").replace("::", "__") - target = directory / f"{name}.png" - from je_auto_control.utils.exception.exceptions import AutoControlException - try: - from je_auto_control.wrapper.auto_control_screen import screenshot - screenshot(file_path=str(target)) - # AutoControlScreenException (no display) and ImportError (no cv2) turned - # a failed test into an INTERNALERROR that ended the whole session. - except (OSError, RuntimeError, ValueError, ImportError, AutoControlException) as exc: - item.add_report_section( - "call", "autocontrol-screenshot", - f"failed to capture screenshot: {exc!r}", - ) - return None - return target - - -@pytest.hookimpl(hookwrapper=True, tryfirst=True) -def pytest_runtest_makereport(item, call): # noqa: D401 - """Attach a screenshot path to the failure report for ``autocontrol`` tests.""" - outcome = yield - report = outcome.get_result() - # Failures only: a skipped or xfailed test is not passed either, and was - # screenshotted as if it had failed. - if report.when != "call" or not report.failed or hasattr(report, "wasxfail"): - return - if item.get_closest_marker(_MARKER_NAME) is None: - return - directory = _resolve_dir(item) - captured = _capture_failure_screenshot(item, directory) - if captured is not None: - report.sections.append( - ("autocontrol-screenshot", f"screenshot: {captured}"), - ) - - -def _resolve_dir(item: "pytest.Item") -> Path: - """Pick the per-test ``autocontrol_screenshot_dir`` if it exists.""" - funcargs = getattr(item, "funcargs", {}) or {} - directory = funcargs.get("autocontrol_screenshot_dir") - if isinstance(directory, Path): - return directory - fallback = Path(os.environ.get( - "JE_AUTOCONTROL_PYTEST_ARTIFACTS", - str(Path.cwd() / _DEFAULT_SUBDIR), - )) - fallback.mkdir(parents=True, exist_ok=True) - return fallback - +from je_auto_control_pytest import ( # noqa: F401 # reason: the private helpers stay importable from the old path + _capture_failure_screenshot, _resolve_dir, + autocontrol, autocontrol_executor, autocontrol_screenshot_dir, + pytest_configure, pytest_runtest_makereport, +) __all__ = [ "autocontrol", "autocontrol_executor", "autocontrol_screenshot_dir", diff --git a/je_auto_control_pytest.py b/je_auto_control_pytest.py new file mode 100644 index 000000000..cefdffc29 --- /dev/null +++ b/je_auto_control_pytest.py @@ -0,0 +1,122 @@ +"""pytest plugin for AutoControl. + +Exposes: + +* ``autocontrol`` fixture — the package module itself, so a test can + call ``autocontrol.screen_size()`` without re-importing it; +* ``autocontrol_screenshot_dir`` fixture — a ``pathlib.Path`` to + ``tmp_path/'autocontrol_screenshots'``; created on first access; +* ``@pytest.mark.autocontrol`` marker — when a test marked with it + fails, the plugin captures a screenshot to the screenshot dir and + attaches the path to the failure report. + +Registered via the ``pytest11`` entry point so installing +``je_auto_control`` makes the plugin available automatically. + +This is a top-level module on purpose. pytest imports every ``pytest11`` +entry point at start-up, in every environment the package is installed in; +as a submodule of ``je_auto_control`` the plugin dragged the whole facade +(about 1,350 modules) into each of those runs. Here it imports ``pytest`` +and nothing else, and the package is imported inside the fixtures that need +it. ``je_auto_control.utils.pytest_plugin.plugin`` re-exports these names. +""" +from __future__ import annotations + +import os +from pathlib import Path +from typing import Optional + +import pytest + + +_MARKER_NAME = "autocontrol" +_DEFAULT_SUBDIR = "autocontrol_screenshots" + + +def pytest_configure(config: "pytest.Config") -> None: + """Register the ``autocontrol`` marker so ``--strict-markers`` is happy.""" + config.addinivalue_line( + "markers", + f"{_MARKER_NAME}: AutoControl GUI test; capture a screenshot on failure.", + ) + + +@pytest.fixture +def autocontrol(): + """Return the ``je_auto_control`` module without forcing a global import.""" + import je_auto_control + return je_auto_control + + +@pytest.fixture +def autocontrol_screenshot_dir(tmp_path) -> Path: + """Per-test directory under ``tmp_path`` for screenshots / artefacts.""" + target = tmp_path / _DEFAULT_SUBDIR + target.mkdir(parents=True, exist_ok=True) + return target + + +@pytest.fixture +def autocontrol_executor(): + """Yield the executor singleton (callable via ``executor.event_dict``).""" + from je_auto_control.utils.executor.action_executor import executor + return executor + + +def _capture_failure_screenshot(item: "pytest.Item", + directory: Path) -> Optional[Path]: + """Best-effort screenshot capture on failure; returns the path or None.""" + name = item.nodeid.replace("/", "_").replace("::", "__") + target = directory / f"{name}.png" + from je_auto_control.utils.exception.exceptions import AutoControlException + try: + from je_auto_control.wrapper.auto_control_screen import screenshot + screenshot(file_path=str(target)) + # AutoControlScreenException (no display) and ImportError (no cv2) turned + # a failed test into an INTERNALERROR that ended the whole session. + except (OSError, RuntimeError, ValueError, ImportError, AutoControlException) as exc: + item.add_report_section( + "call", "autocontrol-screenshot", + f"failed to capture screenshot: {exc!r}", + ) + return None + return target + + +@pytest.hookimpl(hookwrapper=True, tryfirst=True) +def pytest_runtest_makereport(item, call): # noqa: D401 + """Attach a screenshot path to the failure report for ``autocontrol`` tests.""" + outcome = yield + report = outcome.get_result() + # Failures only: a skipped or xfailed test is not passed either, and was + # screenshotted as if it had failed. + if report.when != "call" or not report.failed or hasattr(report, "wasxfail"): + return + if item.get_closest_marker(_MARKER_NAME) is None: + return + directory = _resolve_dir(item) + captured = _capture_failure_screenshot(item, directory) + if captured is not None: + report.sections.append( + ("autocontrol-screenshot", f"screenshot: {captured}"), + ) + + +def _resolve_dir(item: "pytest.Item") -> Path: + """Pick the per-test ``autocontrol_screenshot_dir`` if it exists.""" + funcargs = getattr(item, "funcargs", {}) or {} + directory = funcargs.get("autocontrol_screenshot_dir") + if isinstance(directory, Path): + return directory + fallback = Path(os.environ.get( + "JE_AUTOCONTROL_PYTEST_ARTIFACTS", + str(Path.cwd() / _DEFAULT_SUBDIR), + )) + fallback.mkdir(parents=True, exist_ok=True) + return fallback + + +__all__ = [ + "autocontrol", "autocontrol_executor", "autocontrol_screenshot_dir", + "pytest_configure", "pytest_runtest_makereport", +] diff --git a/pyproject.toml b/pyproject.toml index eb5aa6d1b..ef440abdc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -71,7 +71,9 @@ je_auto_control = "je_auto_control.cli:main" je_auto_control_mcp = "je_auto_control.utils.mcp_server.__main__:main" [project.entry-points.pytest11] -je_auto_control = "je_auto_control.utils.pytest_plugin.plugin" +# A top-level module that imports only pytest: as a submodule of the package +# it made every pytest start-up on the machine import the whole facade. +je_auto_control = "je_auto_control_pytest" [project.urls] Homepage = "https://github.com/Integration-Automation/AutoControlGUI" @@ -82,6 +84,9 @@ Code = "https://github.com/Integration-Automation/AutoControlGUI" file = "README.md" content-type = "text/markdown" +[tool.setuptools] +py-modules = ["je_auto_control_pytest"] + [tool.setuptools.packages] find = { namespaces = false } @@ -148,7 +153,7 @@ python_files = ["test_*.py"] [tool.coverage.run] branch = true -source = ["je_auto_control"] +source = ["je_auto_control", "je_auto_control_pytest"] omit = ["*/gui/*", "*/language_wrapper/*"] [tool.coverage.report] diff --git a/test/unit_test/headless/test_coverage_measurement.py b/test/unit_test/headless/test_coverage_measurement.py index 2d1496743..48fbc3473 100644 --- a/test/unit_test/headless/test_coverage_measurement.py +++ b/test/unit_test/headless/test_coverage_measurement.py @@ -1,12 +1,12 @@ -"""Why this package cannot measure its own coverage with ``pytest --cov``. +"""Why this package measures its coverage with ``coverage run -m pytest``. -``je_auto_control`` registers a ``pytest11`` entry point, so pytest imports -``je_auto_control.utils.pytest_plugin.plugin`` while it is loading plugins — -and importing that submodule executes ``je_auto_control/__init__.py``, the -facade, which imports several hundred modules. ``pytest-cov`` starts measuring -after plugin loading, so every one of those modules had its import-time lines -(``def`` lines, class bodies, constants, the dispatch tables) recorded as never -executed. +``je_auto_control`` registers a ``pytest11`` entry point. Until 2026-10 it +pointed at ``je_auto_control.utils.pytest_plugin.plugin``, so pytest imported +that submodule while loading plugins — which executes +``je_auto_control/__init__.py``, the facade, and with it several hundred +modules. ``pytest-cov`` starts measuring after plugin loading, so every one of +those modules had its import-time lines (``def`` lines, class bodies, +constants, the dispatch tables) recorded as never executed. That is not a small correction. Measured 2026-08-23 on one machine, same suite and same ``[tool.coverage.run]`` config, the *only* difference being when @@ -17,27 +17,23 @@ So ``quality.yml`` runs ``coverage run -m pytest``, which starts before pytest loads anything. These tests pin that, because the difference between the two -spellings is invisible in a green build: reverting to ``pytest --cov`` gives -back 24 points and every job still passes. +spellings is invisible in a green build: reverting to ``pytest --cov`` gave +back 24 points and every job still passed. + +The entry point is now the top-level module ``je_auto_control_pytest``, which +imports only pytest (``test_pytest_entrypoint_light.py``), so the plugin no +longer pulls the facade in. ``coverage run`` stays all the same: it does not +depend on what any plugin imports or on which build of the package is +installed — an environment that still has an older install keeps the old +entry point until it is reinstalled — and nothing has re-measured +``pytest --cov`` against it since the move. """ -import sys from pathlib import Path _REPO_ROOT = Path(__file__).resolve().parents[3] _QUALITY_YML = _REPO_ROOT / ".github" / "workflows" / "quality.yml" -def test_the_facade_is_imported_before_any_test_runs(): - """The premise: pytest has already imported the package by test time. - - This is what makes the measurement order matter. It holds no matter which - test file runs first and with no conftest of ours involved, because the - entry point pulls the facade in during plugin loading. - """ - assert "je_auto_control" in sys.modules - assert "je_auto_control.utils.pytest_plugin.plugin" in sys.modules - - def test_ci_starts_coverage_before_pytest_loads_plugins(): """``coverage run -m pytest``, not ``pytest --cov``. See the module docstring.""" workflow = _QUALITY_YML.read_text(encoding="utf-8") diff --git a/test/unit_test/headless/test_pytest_entrypoint_light.py b/test/unit_test/headless/test_pytest_entrypoint_light.py new file mode 100644 index 000000000..2d56723f7 --- /dev/null +++ b/test/unit_test/headless/test_pytest_entrypoint_light.py @@ -0,0 +1,61 @@ +"""The ``pytest11`` entry point loads without importing the package. + +pytest imports every ``pytest11`` entry point at start-up, in every +environment the package is installed in. The plugin used to be a submodule of +``je_auto_control``, so each of those runs imported the facade first. It is now +the top-level module ``je_auto_control_pytest``; these tests keep it that way. + +The checks run in child interpreters: this process imported the package long +ago, through the tests around this one. +""" +import pathlib +import subprocess # nosec B404 # reason: runs fixed probes with this interpreter +import sys + +try: + import tomllib +except ModuleNotFoundError: # Python 3.10 + import tomli as tomllib # type: ignore[no-redef] # reason: same API under the older name + +ROOT = pathlib.Path(__file__).resolve().parents[3] +PLUGIN_NAMES = ("autocontrol", "autocontrol_executor", "autocontrol_screenshot_dir", + "pytest_configure", "pytest_runtest_makereport") + + +def _probe(code: str) -> str: + done = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, # nosec B603 # nosemgrep # reason: fixed argv, no shell + timeout=120, cwd=str(ROOT), check=False) + assert done.returncode == 0, done.stderr[-2000:] + return done.stdout.strip() + + +def test_entrypoint_import_is_light(): + out = _probe("import sys, je_auto_control_pytest; " + "print(sorted(name for name in sys.modules if name.split('.')[0] == 'je_auto_control'))") + assert out == "[]" + + +def test_the_fixture_imports_the_package_only_when_used(): + out = _probe("import sys, je_auto_control_pytest as plugin; " + "before = 'je_auto_control' in sys.modules; " + "module = plugin.autocontrol.__wrapped__(); " + "print(before, module.__name__, 'je_auto_control' in sys.modules)") + assert out == "False je_auto_control True" + + +def test_legacy_plugin_exports_same_hooks(): + import je_auto_control_pytest as standalone + from je_auto_control.utils import pytest_plugin as package + from je_auto_control.utils.pytest_plugin import plugin as legacy + for name in PLUGIN_NAMES: + assert getattr(legacy, name) is getattr(standalone, name), name + assert getattr(package, name) is getattr(standalone, name), name + + +def test_both_metadata_files_point_the_entry_point_at_the_top_level_module(): + for name in ("pyproject.toml", "dev.toml"): + data = tomllib.loads((ROOT / name).read_text(encoding="utf-8")) + assert data["project"]["entry-points"]["pytest11"] == {"je_auto_control": "je_auto_control_pytest"}, name + # Without this the module is not in the wheel and the entry point cannot load. + assert data["tool"]["setuptools"]["py-modules"] == ["je_auto_control_pytest"], name + assert (ROOT / "je_auto_control_pytest.py").is_file() diff --git a/test/unit_test/headless/test_pytest_plugin.py b/test/unit_test/headless/test_pytest_plugin.py index 3c7658c84..3df964e63 100644 --- a/test/unit_test/headless/test_pytest_plugin.py +++ b/test/unit_test/headless/test_pytest_plugin.py @@ -154,7 +154,7 @@ def test_pyproject_registers_pytest11_entry_point(): "pyproject.toml", ).read_text(encoding="utf-8") assert re.search(r"\[project\.entry-points\.pytest11\]", raw) - assert "je_auto_control.utils.pytest_plugin.plugin" in raw + assert 'je_auto_control = "je_auto_control_pytest"' in raw # === plugin via pytester ================================================== From 6817b3838258fbf8375dc6245cb7f7c7e8eadcb9 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 23:42:24 +0800 Subject: [PATCH 04/26] Stop rewriting sent agent turns; restart the history from a summary instead Replacing older screenshots in turns the Messages API had already seen missed the prompt cache on every step and, on models whose thinking blocks are bound to the conversation before them, got the request refused with HTTP 400. Both Anthropic backends now leave sent turns alone and, past the screenshot limit, open a new history of one summary message (goal, executed actions and outcomes) plus the current screenshot. The OpenAI backend keeps pruning in place. --- .../Eng/doc/new_features/v2_features_doc.rst | 9 + .../Zh/doc/new_features/v2_features_doc.rst | 5 + .../utils/agent/backends/anthropic.py | 42 ++-- .../agent/backends/anthropic_computer_use.py | 61 +++-- je_auto_control/utils/agent/backends/base.py | 162 +++++++++++- .../test_agent_append_only_history.py | 231 ++++++++++++++++++ .../headless/test_llm_agent_audit.py | 6 +- 7 files changed, 466 insertions(+), 50 deletions(-) create mode 100644 test/unit_test/headless/test_agent_append_only_history.py diff --git a/docs/source/Eng/doc/new_features/v2_features_doc.rst b/docs/source/Eng/doc/new_features/v2_features_doc.rst index ba7f0eb99..edeec5565 100644 --- a/docs/source/Eng/doc/new_features/v2_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v2_features_doc.rst @@ -239,6 +239,15 @@ budgets so a runaway loop can't drain the API; setting ``stop_event=`` (a **Computer Use** tab, whose Actions menu has **Stop**. Closing the window asks a running job to stop and waits up to 10 seconds for it. +The Anthropic backends (this one and ``AnthropicAgentBackend`` behind +``AC_run_agent``) never edit a turn they have already sent. When the +conversation would hold more than three screenshots (or more than 20 MB of +them), the next request starts a new history instead: one message with the +goal, the list of actions executed so far with their outcomes, and the current +screenshot. Earlier turns and their thinking blocks are not replayed, so the +model continues from that summary alone. The OpenAI backend still replaces +older screenshots with a text note in place. + WebRunner executor + MCP integration ------------------------------------ diff --git a/docs/source/Zh/doc/new_features/v2_features_doc.rst b/docs/source/Zh/doc/new_features/v2_features_doc.rst index d5b457d7f..df7709db9 100644 --- a/docs/source/Zh/doc/new_features/v2_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v2_features_doc.rst @@ -221,6 +221,11 @@ beta 工具也一樣:它宣告縮放後的大小為螢幕大小,再把模型給 ``final_message`` 為 ``"stopped"``。Executor：``AC_computer_use``。 GUI：**Computer Use** 分頁，Actions 選單有 **停止**。關閉視窗時會請執行中的工作停止，最多等 10 秒。 +Anthropic 的兩個 backend（這一個，以及 ``AC_run_agent`` 背後的 ``AnthropicAgentBackend``）不會改寫已送出的回合。 +對話裡的截圖將超過三張（或合計超過 20 MB）時，下一個請求改開一段新的對話：只有一則訊息，內容是目標、 +目前為止執行過的動作與各自的結果，再加上當下的截圖。較早的回合與其中的 thinking 區塊不會重播， +模型只憑這份摘要接續。OpenAI backend 仍是就地把較舊的截圖換成文字。 + WebRunner 接入 executor + MCP ----------------------------- diff --git a/je_auto_control/utils/agent/backends/anthropic.py b/je_auto_control/utils/agent/backends/anthropic.py index 8b2ae2a3f..4b6bee72e 100644 --- a/je_auto_control/utils/agent/backends/anthropic.py +++ b/je_auto_control/utils/agent/backends/anthropic.py @@ -9,8 +9,8 @@ ) from je_auto_control.utils.agent.backends.base import ( REQUEST_TIMEOUT_S, AgentBackendError, build_default_system_prompt, - encode_screenshot_b64, offered_tool_names, prune_old_screenshots, - require_offered, + compact_history, image_block, needs_compaction, offered_tool_names, + require_offered, summarise_steps, ) @@ -72,15 +72,24 @@ def decide_next_action(self, goal: str, # A new run: the last run's conversation ended on an unanswered # tool_use, which the API rejects. self._conversation = [] - # Track the previous turn's tool_result, if any. - self._ingest_history(history) # Always attach the latest screenshot so the model has fresh # state — text-only context drifts quickly during a long run. if screenshot: screenshot, self._scale = fit_screenshot(screenshot, self._tier) user_content = _build_user_content(screenshot) - self._conversation.append({"role": "user", "content": user_content}) - prune_old_screenshots(self._conversation) + if needs_compaction(self._conversation, user_content): + # Sent turns are never edited (that broke the prompt cache and, + # with thinking blocks bound to the conversation, the request): + # past the screenshot limit a new history starts instead, whose + # summary holds the last tool's outcome in place of a tool_result. + self._conversation = compact_history( + self._conversation, + f"{summarise_steps(goal, history)}\n\n{_NEXT_TOOL_PROMPT}", + image_block(screenshot)) + else: + # Track the previous turn's tool_result, if any. + self._ingest_history(history) + self._conversation.append({"role": "user", "content": user_content}) client = self._resolve_client() try: response = client.messages.create( @@ -177,23 +186,16 @@ def _ingest_history(self, history: Sequence[AgentStep]) -> None: def _build_user_content(screenshot: Optional[bytes]) -> List[Dict[str, Any]]: blocks: List[Dict[str, Any]] = [] - encoded = encode_screenshot_b64(screenshot) - if encoded: - blocks.append({ - "type": "image", - "source": { - "type": "base64", - "media_type": "image/png", - "data": encoded, - }, - }) - blocks.append({ - "type": "text", - "text": "Latest screenshot above. Pick the next AC_* tool to call.", - }) + image = image_block(screenshot) + if image is not None: + blocks.append(image) + blocks.append({"type": "text", "text": _NEXT_TOOL_PROMPT}) return blocks +_NEXT_TOOL_PROMPT = "Latest screenshot above. Pick the next AC_* tool to call." + + def _attr(block: Any, name: str) -> Any: if isinstance(block, dict): return block.get(name) diff --git a/je_auto_control/utils/agent/backends/anthropic_computer_use.py b/je_auto_control/utils/agent/backends/anthropic_computer_use.py index 7a24da754..1fc5b844d 100644 --- a/je_auto_control/utils/agent/backends/anthropic_computer_use.py +++ b/je_auto_control/utils/agent/backends/anthropic_computer_use.py @@ -36,7 +36,8 @@ ) from je_auto_control.utils.agent.backends.base import ( REQUEST_TIMEOUT_S, AgentBackendError, build_default_system_prompt, - encode_screenshot_b64, prune_old_screenshots, + compact_history, encode_screenshot_b64, image_block, needs_compaction, + summarise_steps, ) @@ -204,14 +205,30 @@ def decide_next_action(self, return self._decide_with_toolset(self._batch, goal, screenshot, history) if screenshot and self._declared is not None: screenshot = resize_png(screenshot, self._declared) - self._ingest_history(history, screenshot) + self._extend(self._pending_result(history, screenshot), goal, screenshot, history) + return self._handle_response(self._create(goal, beta=True)) + + def _extend(self, results: List[Dict[str, Any]], goal: str, + screenshot: Optional[bytes], history: Sequence[AgentStep]) -> None: + """Append the turn's ``tool_result`` blocks, or start a new history. + + Sent turns are never edited: replacing their screenshots broke the + prompt cache every step and, where thinking blocks are bound to the + conversation before them, the request itself. Past the screenshot + limit the history restarts from a summary and the current full + screenshot (``screenshot``, already fitted) — the results are in the + summary, since their ``tool_use`` blocks are not replayed. + """ + if results and needs_compaction(self._conversation, results): + self._conversation = compact_history( + self._conversation, summarise_steps(goal, history), image_block(screenshot)) + elif results: + self._conversation.append({"role": "user", "content": results}) if not self._conversation: self._conversation.append({ "role": "user", "content": _initial_user_content(goal, screenshot), }) - prune_old_screenshots(self._conversation) - return self._handle_response(self._create(goal, beta=True)) def _new_run(self) -> None: """Forget the previous run: its conversation ended on an unanswered tool_use.""" @@ -257,14 +274,10 @@ def _decide_with_toolset(self, batch: ToolsetBatch, goal: str, if batch.has_next(): return batch.next_decision() results = batch.drain_results() - if results: - self._conversation.append({"role": "user", "content": results}) - if not self._conversation: - self._conversation.append({ - "role": "user", - "content": _initial_user_content(goal, self._fit(screenshot)), - }) - prune_old_screenshots(self._conversation) + if not self._conversation or needs_compaction(self._conversation, results): + # Only a history that opens with this frame takes its scale. + screenshot = self._fit(screenshot) + self._extend(results, goal, screenshot, history) return self._handle_toolset_response(self._create(goal, beta=False), batch) def _fit(self, screenshot: Optional[bytes]) -> Optional[bytes]: @@ -344,22 +357,20 @@ def _handle_response(self, response: Any) -> Dict[str, Any]: return _clamp_decision(decision, *self._display) return _final_answer(response, content) - def _ingest_history(self, history: Sequence[AgentStep], - screenshot: Optional[bytes]) -> None: + def _pending_result(self, history: Sequence[AgentStep], + screenshot: Optional[bytes]) -> List[Dict[str, Any]]: + """The ``tool_result`` answering the last turn's call, if one is pending.""" if not history or self._pending_tool_use_id is None: - return + return [] last = history[-1] content = _tool_result_content(last, screenshot, self._scale) - self._conversation.append({ - "role": "user", - "content": [{ - "type": "tool_result", - "tool_use_id": self._pending_tool_use_id, - "content": content, - "is_error": bool(last.error), - }], - }) - self._pending_tool_use_id = None + tool_use_id, self._pending_tool_use_id = self._pending_tool_use_id, None + return [{ + "type": "tool_result", + "tool_use_id": tool_use_id, + "content": content, + "is_error": bool(last.error), + }] def _resolve_client(self) -> Any: if self._client is not None: diff --git a/je_auto_control/utils/agent/backends/base.py b/je_auto_control/utils/agent/backends/base.py index 4f1362e70..583c1109a 100644 --- a/je_auto_control/utils/agent/backends/base.py +++ b/je_auto_control/utils/agent/backends/base.py @@ -2,7 +2,8 @@ from __future__ import annotations import base64 -from typing import Any, Dict, FrozenSet, Iterable, List, Mapping, Optional +import json +from typing import Any, Dict, FrozenSet, Iterable, List, Mapping, Optional, Sequence, Tuple from je_auto_control.utils.exception.exceptions import AutoControlException @@ -74,6 +75,8 @@ def encode_screenshot_b64(screenshot: Optional[bytes]) -> Optional[str]: # capture and the whole conversation is resent, so a default 25-step run # carried 25 full-screen PNGs — past the Messages API's 32 MB request limit # for ordinary desktop captures — and paid for every earlier frame each step. +# The OpenAI backend drops the older ones in place (prune_old_screenshots); +# the Anthropic backends start a new history instead (compact_history). SCREENSHOTS_KEPT = 3 _IMAGE_BLOCK_TYPES = frozenset({"image", "image_url"}) @@ -110,8 +113,159 @@ def _prune_blocks(content: Any, keep: int, seen: int) -> int: return seen +# --- append-only histories (Anthropic) ------------------------------- +# +# prune_old_screenshots edits turns the API has already seen. On the Anthropic +# backends that is not allowed: the prompt cache is a prefix match, so every +# step missed it, and a thinking block is valid only in front of the exact +# conversation that produced it, so the replayed turns were refused with +# HTTP 400 ("bound to a different conversation"). Those backends instead +# leave sent turns alone and, once the screenshots pass the limit, open a new +# history: one summary message plus the current screenshot, nothing replayed. + +#: Base64 characters of screenshots a request may carry. The Messages API +#: refuses a request over 32 MB; this leaves the rest of it a wide margin. +MAX_IMAGE_CHARS = 20_000_000 + +#: The most recent actions a summary lists, and the characters kept of each +#: action's arguments and outcome. +_SUMMARY_STEPS = 60 +_SUMMARY_FIELD_CHARS = 240 + +_SUMMARY_HEAD = ( + "This session continues a task that is already under way. The earlier " + "conversation is not included; this message is the record of it." +) +_SUMMARY_TAIL = ( + "The attached screenshot shows the whole screen as it is now, after all " + "of the actions above. Continue toward the goal from this state: do not " + "repeat an action that already succeeded, and when the goal is met, stop " + "and say so." +) + + +def _image_stats(content: Any) -> Tuple[int, int]: + """``(images, base64 characters)`` in one content list, nested ones included.""" + if not isinstance(content, list): + return 0, 0 + count = chars = 0 + for block in content: + if not isinstance(block, dict): + continue + if block.get("type") in _IMAGE_BLOCK_TYPES: + source = block.get("source") + data = source.get("data") if isinstance(source, dict) else None + count += 1 + chars += len(data) if isinstance(data, str) else 0 + continue + nested = _image_stats(block.get("content")) + count += nested[0] + chars += nested[1] + return count, chars + + +def _history_image_stats(messages: Sequence[Any]) -> Tuple[int, int]: + count = chars = 0 + for message in messages: + if isinstance(message, dict): + found = _image_stats(message.get("content")) + count += found[0] + chars += found[1] + return count, chars + + +def count_screenshots(messages: Sequence[Any]) -> int: + """Image blocks in ``messages``, those inside a ``tool_result`` included.""" + return _history_image_stats(messages)[0] + + +def needs_compaction(messages: Sequence[Any], incoming: Any, *, + keep: int = SCREENSHOTS_KEPT, + max_image_chars: int = MAX_IMAGE_CHARS) -> bool: + """Whether appending the content list ``incoming`` passes the screenshot limits. + + Only a turn that brings a screenshot can pass them, so a history is + never restarted for a text-only turn. + """ + new_count, new_chars = _image_stats(incoming) + if not new_count: + return False + count, chars = _history_image_stats(messages) + return count + new_count > keep or chars + new_chars > max_image_chars + + +def _clip(text: str) -> str: + if len(text) <= _SUMMARY_FIELD_CHARS: + return text + return text[:_SUMMARY_FIELD_CHARS] + "..." + + +def _step_line(number: int, step: Any) -> str: + """One executed action: its tool, arguments and outcome, each bounded.""" + try: + arguments = json.dumps(step.arguments or {}, ensure_ascii=False, default=repr) + except (TypeError, ValueError): + arguments = repr(step.arguments) + if step.error: + outcome = f"error: {step.error}" + elif step.result is None: + outcome = "ok" + else: + outcome = str(step.result) + return f"{number}. {step.tool} {_clip(arguments)} -> {_clip(outcome)}" + + +def summarise_steps(goal: str, history: Sequence[Any]) -> str: + """The summary a restarted history opens with: the goal and the actions run so far. + + ``history`` is the loop's ``AgentStep`` list. The newest actions are + listed in full; a very long run says how many older ones are left out. + """ + steps = [step for step in history if getattr(step, "tool", None)] + lines = [_SUMMARY_HEAD, "", f"Goal: {goal.strip()}", ""] + if not steps: + lines.append("No action has been executed yet.") + else: + skipped = max(0, len(steps) - _SUMMARY_STEPS) + lines.append("Actions executed so far, oldest first (tool, arguments -> outcome):") + if skipped: + lines.append(f"({skipped} earlier actions are not listed.)") + lines.extend(_step_line(skipped + offset + 1, step) + for offset, step in enumerate(steps[skipped:])) + lines.extend(["", _SUMMARY_TAIL]) + return "\n".join(lines) + + +def image_block(screenshot: Optional[bytes]) -> Optional[Dict[str, Any]]: + """A PNG as an Anthropic base64 image block, or ``None`` without one.""" + encoded = encode_screenshot_b64(screenshot) + if not encoded: + return None + return {"type": "image", + "source": {"type": "base64", "media_type": "image/png", "data": encoded}} + + +def compact_history(messages: Sequence[Any], summary: str, + latest_screenshot: Optional[Dict[str, Any]]) -> List[Dict[str, Any]]: + """A new Anthropic history: one user message with the screenshot and ``summary``. + + Nothing of ``messages`` is carried over and it is not modified: a turn + replayed behind a summary would bring thinking blocks made in front of a + different conversation, and a ``tool_result`` whose ``tool_use`` is gone. + Call it only between tool rounds, with every executed call's outcome in + ``summary``. ``latest_screenshot`` is an image block (:func:`image_block`). + """ + del messages # the old history is deliberately not read + content: List[Dict[str, Any]] = [] + if latest_screenshot is not None: + content.append(latest_screenshot) + content.append({"type": "text", "text": summary}) + return [{"role": "user", "content": content}] + + __all__ = [ - "AgentBackendError", "REQUEST_TIMEOUT_S", "SCREENSHOTS_KEPT", - "build_default_system_prompt", "encode_screenshot_b64", - "prune_old_screenshots", + "AgentBackendError", "MAX_IMAGE_CHARS", "REQUEST_TIMEOUT_S", "SCREENSHOTS_KEPT", + "build_default_system_prompt", "compact_history", "count_screenshots", + "encode_screenshot_b64", "image_block", "needs_compaction", + "prune_old_screenshots", "summarise_steps", ] diff --git a/test/unit_test/headless/test_agent_append_only_history.py b/test/unit_test/headless/test_agent_append_only_history.py new file mode 100644 index 000000000..280f3bba2 --- /dev/null +++ b/test/unit_test/headless/test_agent_append_only_history.py @@ -0,0 +1,231 @@ +"""The Anthropic agent backends never rewrite a turn they have already sent. + +Every step used to replace the older screenshots of the replayed conversation +with a text note. That edits messages the API has already seen: the prompt +cache misses on every step, and on models whose thinking blocks are bound to +the conversation before them the request is refused with HTTP 400. The +backends now leave sent turns alone and, once the screenshots pass the limit, +open a new history made of one summary message and the current screenshot. + +Fake clients only: nothing here reaches the network. +""" +from __future__ import annotations + +import copy +import io +import json +from types import SimpleNamespace +from typing import Any, Dict, List + +import pytest + +from je_auto_control.utils.agent.agent_loop import AgentStep +from je_auto_control.utils.agent.backends import base +from je_auto_control.utils.agent.backends._computer_toolset import TOOLSET_TYPE +from je_auto_control.utils.agent.backends.anthropic import AnthropicAgentBackend +from je_auto_control.utils.agent.backends.anthropic_computer_use import ( + ComputerUseAgentBackend, +) + +GOAL = "open the settings page" + + +def _png() -> bytes: + from PIL import Image + buffer = io.BytesIO() + Image.new("RGB", (64, 48), (10, 20, 30)).save(buffer, format="PNG") + return buffer.getvalue() + + +def _plain(value: Any) -> Any: + """The JSON-shaped form of a message tree holding SDK-like objects.""" + if isinstance(value, SimpleNamespace): + return {key: _plain(item) for key, item in vars(value).items()} + if isinstance(value, dict): + return {key: _plain(item) for key, item in value.items()} + if isinstance(value, (list, tuple)): + return [_plain(item) for item in value] + return value + + +class _Client: + """Stands in for the SDK client; snapshots each request as it was sent.""" + + def __init__(self, reply) -> None: + self._reply = reply + self.sent: List[List[Dict[str, Any]]] = [] + self.live: List[List[Dict[str, Any]]] = [] + self.requests: List[Dict[str, Any]] = [] + self.messages = self + self.beta = SimpleNamespace(messages=self) + + def create(self, **kwargs): + self.requests.append(kwargs) + self.live.append(kwargs["messages"]) + self.sent.append(copy.deepcopy(_plain(kwargs["messages"]))) + return self._reply(len(self.sent)) + + +def _thinking() -> SimpleNamespace: + return SimpleNamespace(type="thinking", thinking="", signature="sig") + + +def _reply(name: str, tool_input_for): + def reply(number: int) -> SimpleNamespace: + return SimpleNamespace(stop_reason="tool_use", content=[ + _thinking(), + SimpleNamespace(type="tool_use", id=f"tu{number}", name=name, + input=tool_input_for(number)), + ]) + return reply + + +def _drive(backend, steps: int, tool: str = "AC_screenshot") -> List[AgentStep]: + history: List[AgentStep] = [] + for index in range(steps): + decision = backend.decide_next_action(GOAL, _png(), history) + history.append(AgentStep(index=index, tool=decision["tool"], + arguments=decision["input"], result=f"result-{index}")) + return history + + +def _ac_backend(client) -> AnthropicAgentBackend: + tools = [{"name": "AC_click_mouse", "input_schema": {"type": "object"}}] + return AnthropicAgentBackend(tools=tools, client=client) + + +def _ac_client() -> _Client: + return _Client(_reply("AC_click_mouse", lambda number: {"x": number, "y": 2})) + + +def _beta_backend(client) -> ComputerUseAgentBackend: + return ComputerUseAgentBackend(display_width_px=64, display_height_px=48, + client=client, model="claude-opus-5") + + +def _toolset_backend(client) -> ComputerUseAgentBackend: + return ComputerUseAgentBackend(display_width_px=64, display_height_px=48, + client=client, model="claude-opus-5", + tool_type=TOOLSET_TYPE) + + +def _cu_client(toolset: bool) -> _Client: + if toolset: + return _Client(_reply("screenshot", lambda number: {})) + return _Client(_reply("computer", lambda number: {"action": "screenshot"})) + + +_PATHS = [ + pytest.param(_ac_backend, _ac_client, id="ac-tools"), + pytest.param(_beta_backend, lambda: _cu_client(False), id="computer-beta"), + pytest.param(_toolset_backend, lambda: _cu_client(True), id="computer-toolset"), +] + + +def _images(messages) -> int: + return base.count_screenshots(messages) + + +def _blocks(messages, block_type: str) -> List[Dict[str, Any]]: + return [block for message in messages for block in message["content"] + if isinstance(block, dict) and block.get("type") == block_type] + + +@pytest.mark.parametrize("make_backend, make_client", _PATHS) +def test_sent_turns_are_never_mutated(make_backend, make_client): + client = make_client() + _drive(make_backend(client), base.SCREENSHOTS_KEPT * 3 + 2) + # What each request held when it left is what its message list holds now. + for sent, live in zip(client.sent, client.live): + assert _plain(live)[:len(sent)] == sent + # A request either extends the previous one byte for byte or starts over. + restarts = 0 + for previous, current in zip(client.sent, client.sent[1:]): + if current[:len(previous)] == previous: + continue + restarts += 1 + assert len(current) == 1 + assert restarts >= 1 + assert not any("omitted" in json.dumps(sent) for sent in client.sent) + + +@pytest.mark.parametrize("make_backend, make_client", _PATHS) +def test_compaction_opens_new_history(make_backend, make_client): + client = make_client() + _drive(make_backend(client), base.SCREENSHOTS_KEPT * 3 + 2) + assert all(_images(sent) <= base.SCREENSHOTS_KEPT for sent in client.sent) + restarted = [(number, sent) for number, sent in enumerate(client.sent) + if number and len(sent) == 1] + assert restarted + for number, sent in restarted: + first = sent[0] + assert first["role"] == "user" + # No orphan tool_result, and no thinking block from the old conversation. + assert [block["type"] for block in first["content"]] == ["image", "text"] + text = first["content"][1]["text"] + assert GOAL in text + # Every action executed before this request is in the summary. + for index in range(number): + assert f"result-{index}" in text + for sent in client.sent: + answered = {block["tool_use_id"] for block in _blocks(sent, "tool_result")} + asked = {block["id"] for block in _blocks(sent, "tool_use")} + assert answered <= asked + assert sent[0]["role"] == "user" + + +def test_compact_history_leaves_the_old_history_alone(): + old = [{"role": "user", "content": [{"type": "text", "text": "go"}]}, + {"role": "assistant", "content": [_thinking()]}] + before = copy.deepcopy(_plain(old)) + image = {"type": "image", "source": {"type": "base64", "data": "abc"}} + new = base.compact_history(old, "the summary", image) + assert _plain(old) == before + assert new == [{"role": "user", "content": [image, {"type": "text", "text": "the summary"}]}] + assert base.compact_history(old, "s", None) == [ + {"role": "user", "content": [{"type": "text", "text": "s"}]}] + + +def test_summary_names_the_goal_and_bounds_each_action(): + steps = [ + AgentStep(index=0, tool="AC_click_mouse", arguments={"x": 1, "y": 2}, result="x" * 5000), + AgentStep(index=1, tool="AC_write", arguments={"write_string": "hi"}, + error="ValueError: boom"), + AgentStep(index=2, tool=None, arguments=None, stop_reason="done"), + ] + summary = base.summarise_steps(GOAL, steps) + assert GOAL in summary + assert "AC_click_mouse" in summary and '"x": 1' in summary + assert "error: ValueError: boom" in summary + assert len(summary) < 2000 + many = [AgentStep(index=i, tool="AC_press_key", arguments={"key": "a"}) for i in range(500)] + assert len(base.summarise_steps(GOAL, many)) < 40_000 + assert "earlier actions" in base.summarise_steps(GOAL, many) + + +def test_a_history_over_the_byte_budget_is_compacted_too(): + image = {"type": "image", "source": {"type": "base64", "data": "a" * 600}} + messages = [{"role": "user", "content": [image]}] + assert not base.needs_compaction(messages, [image]) + assert base.needs_compaction(messages, [image], max_image_chars=1000) + nested = [{"type": "tool_result", "tool_use_id": "t", "content": [image]}] + assert base.needs_compaction(messages, nested, keep=1) + assert not base.needs_compaction(messages, [{"type": "text", "text": "hi"}], keep=1) + + +def test_default_agent_tools_are_unchanged(): + from je_auto_control.utils.executor import action_executor + assert action_executor._DEFAULT_AGENT_TOOLSET == [ + "AC_screenshot", "AC_screen_size", "AC_set_mouse_position", + "AC_get_mouse_position", "AC_click_mouse", "AC_mouse_scroll", "AC_drag", + "AC_write", "AC_type_keyboard", "AC_hotkey", "AC_press_key", + "AC_locate_image_center", "AC_click_text", "AC_wait_text", "AC_wait_image", + "AC_a11y_find", "AC_a11y_click", "AC_list_windows", "AC_focus_window", + "AC_assert_text", + ] + + +def test_the_openai_backend_still_prunes_in_place(): + from je_auto_control.utils.agent.backends import openai + import inspect + assert "prune_old_screenshots(self._messages)" in inspect.getsource(openai) diff --git a/test/unit_test/headless/test_llm_agent_audit.py b/test/unit_test/headless/test_llm_agent_audit.py index 3c3026bdb..2031f4ca6 100644 --- a/test/unit_test/headless/test_llm_agent_audit.py +++ b/test/unit_test/headless/test_llm_agent_audit.py @@ -80,7 +80,11 @@ def test_agent_requests_carry_a_timeout_and_bounded_screenshots(backend_cls, too backend.decide_next_action("goal", b"png", history) history = [AgentStep(index=index, tool=None, arguments={})] # an empty one starts a new run assert all(call["timeout"] == base.REQUEST_TIMEOUT_S for call in client.calls) - assert client.calls[-1]["image_count"] == base.SCREENSHOTS_KEPT + # OpenAI drops the older frames in place; Anthropic starts a new history + # past the limit (sent turns are not edited), so its count falls back to 1. + counts = [call["image_count"] for call in client.calls] + assert max(counts) == base.SCREENSHOTS_KEPT + assert all(1 <= count <= base.SCREENSHOTS_KEPT for count in counts) def test_an_empty_only_list_exports_no_tools(): From e62e9d998ab18648c317ed408b807ca287dce398 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 23:52:54 +0800 Subject: [PATCH 05/26] Keep the action signing key apart from the right to execute AC_sign_action_file signed with the per-user HMAC key that verification also read, so anyone who could run an action could sign a file and then run it. Version-2 signatures are Ed25519: the signing machine holds the private key, execution endpoints hold only the public key, and an endpoint configured that way refuses to sign or mint a key pair instead of falling back to HMAC. HMAC sidecars stay the default when no key pair is configured and are accepted beside one only in an explicit migration mode. A missing cryptography wheel is now CryptographyUnavailableError, inside the framework family. --- .../Eng/doc/new_features/v4_features_doc.rst | 29 ++ .../Zh/doc/new_features/v4_features_doc.rst | 20 ++ je_auto_control/__init__.py | 8 +- .../gui/script_builder/command_schema.py | 14 +- .../utils/action_signing/__init__.py | 9 +- .../utils/action_signing/_key_file.py | 16 ++ .../utils/action_signing/asymmetric.py | 188 ++++++++++++ .../utils/action_signing/cipher.py | 6 +- .../utils/action_signing/config.py | 62 ++++ .../utils/action_signing/signer.py | 107 +++++-- je_auto_control/utils/exception/exceptions.py | 10 + .../utils/executor/action_executor.py | 22 +- je_auto_control/utils/secrets/secret_store.py | 6 +- .../headless/test_signing_execution_roles.py | 268 ++++++++++++++++++ 14 files changed, 729 insertions(+), 36 deletions(-) create mode 100644 je_auto_control/utils/action_signing/asymmetric.py create mode 100644 je_auto_control/utils/action_signing/config.py create mode 100644 test/unit_test/headless/test_signing_execution_roles.py diff --git a/docs/source/Eng/doc/new_features/v4_features_doc.rst b/docs/source/Eng/doc/new_features/v4_features_doc.rst index 378ba0571..933973b2c 100644 --- a/docs/source/Eng/doc/new_features/v4_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v4_features_doc.rst @@ -134,6 +134,35 @@ File security & safety ``read_executable_action_json`` for the same check. The per-user key file must hold at least 32 bytes. ``AC_sign_action_file`` / ``AC_verify_action_file``. +* **Signing key kept apart from execution (version-2 signatures)** — the HMAC + key is one shared secret, so whoever can run actions on an endpoint can also + sign a file for it. ``create_signing_keypair(private_path, public_path)`` + (``AC_create_signing_keypair``) makes an Ed25519 pair: the private key stays + on the signing machine, execution endpoints get the public key, which + verifies and cannot sign. ``sign_action_file(path, private_key_path=...)`` + writes a JSON envelope (``version`` 2, ``algorithm`` ``ed25519``) into the + same ``.sig`` sidecar; ``verify_action_file(path, public_key_path=...)`` + checks it. Three environment variables configure a process, and + ``action_signing_config()`` reports what it sees: + + * ``JE_AUTOCONTROL_ACTION_SIGNING_PRIVATE_KEY`` — path of the private key; + signing machine only. ``sign_action_file(path)`` then signs with it. + * ``JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY`` — path of the public key, on + every execution endpoint. With this and no private key the endpoint is + verify-only: ``AC_sign_action_file`` and ``AC_create_signing_keypair`` + raise instead of falling back to the per-user HMAC key, and an HMAC + sidecar no longer verifies. + * ``JE_AUTOCONTROL_ACCEPT_LEGACY_ACTION_SIGNATURES`` (``1`` / ``true`` / + ``yes`` / ``on``) — migration mode: HMAC sidecars are accepted alongside + version 2, with a warning in the log. + + With none of them set nothing changes: signing is the per-user HMAC as + before. To migrate an endpoint that enforces signatures: create the pair on + the signing machine; set the public-key variable and the migration variable + on the endpoint; sign every file again with the private key; then unset the + migration variable. Key-pair signing needs ``cryptography`` -- where it is + absent (Windows arm64) these calls raise ``CryptographyUnavailableError`` + with the install hint, and HMAC signing still works. * **Action-file encryption** — ``encrypt_action_file`` / ``decrypt_action_file`` keep a script's contents secret at rest with Fernet (AES-128-CBC + HMAC), keyed by a per-user 0600 key or by a diff --git a/docs/source/Zh/doc/new_features/v4_features_doc.rst b/docs/source/Zh/doc/new_features/v4_features_doc.rst index 2a010e192..cb864c18d 100644 --- a/docs/source/Zh/doc/new_features/v4_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v4_features_doc.rst @@ -114,6 +114,26 @@ Builder 項目。視覺與視窗功能的 geometry / IO 操作皆可注入,因 webhook、MCP 執行工具與 GUI -- 都會拒絕未簽章或被改過的檔案;自行載入時 用 ``read_executable_action_json`` 可得到同樣的檢查。每位使用者的金鑰檔 至少要有 32 位元組。``AC_sign_action_file`` / ``AC_verify_action_file``。 +* **簽章金鑰與執行權限分開(第 2 版簽章)** — HMAC 金鑰是一把共用密鑰,所以能在 + 某個端點執行動作的人,也能替它簽檔。``create_signing_keypair(private_path, public_path)`` + (``AC_create_signing_keypair``)產生一組 Ed25519 金鑰:私鑰留在簽章機,執行端只拿 + 公鑰,公鑰能驗章、不能簽章。``sign_action_file(path, private_key_path=...)`` 把 JSON + envelope(``version`` 2、``algorithm`` ``ed25519``)寫進同一個 ``.sig`` sidecar; + ``verify_action_file(path, public_key_path=...)`` 負責驗證。行程由三個環境變數設定, + ``action_signing_config()`` 會回報目前讀到的內容: + + * ``JE_AUTOCONTROL_ACTION_SIGNING_PRIVATE_KEY`` — 私鑰路徑;只設在簽章機。設了之後 + ``sign_action_file(path)`` 就用它簽。 + * ``JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY`` — 公鑰路徑,設在每個執行端。只有它而 + 沒有私鑰時,該端點只能驗章:``AC_sign_action_file`` 與 ``AC_create_signing_keypair`` + 會拋出例外,不會退回每位使用者的 HMAC 金鑰,HMAC sidecar 也不再通過驗證。 + * ``JE_AUTOCONTROL_ACCEPT_LEGACY_ACTION_SIGNATURES``(``1`` / ``true`` / ``yes`` / + ``on``)— 遷移模式:除第 2 版外也接受 HMAC sidecar,並在記錄中留下警告。 + + 三個都沒設時行為完全不變:簽章仍是每位使用者的 HMAC。強制驗章的端點遷移步驟:在 + 簽章機產生金鑰組;在端點設定公鑰變數與遷移變數;用私鑰把每個檔案重簽一次;最後取消 + 遷移變數。金鑰組簽章需要 ``cryptography`` —— 沒有該套件的平台(Windows arm64)上, + 這些呼叫會拋出帶安裝提示的 ``CryptographyUnavailableError``,HMAC 簽章仍可使用。 * **動作檔加密** — ``encrypt_action_file`` / ``decrypt_action_file`` 以 Fernet(AES-128-CBC + HMAC)讓腳本內容在靜態時保密,金鑰來自每位使用者的 0600 金鑰,或經 scrypt 與每個檔案各自的隨機鹽值衍生自通行碼。``AC_encrypt_action_file`` / diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 0117874ed..55598681e 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -1129,11 +1129,13 @@ SecretManager, SecretStoreError, SecretStoreLocked, default_secret_manager, default_secret_store_path, ) -# Action-file security (HMAC-SHA256 sign/verify + Fernet encrypt, headless) +# Action-file security (Ed25519 / HMAC-SHA256 sign/verify + Fernet encrypt, headless) from je_auto_control.utils.action_signing import ( - VerifyResult, decrypt_action_file, encrypt_action_file, + SigningConfig, VerifyResult, action_signing_config, create_signing_keypair, + decrypt_action_file, encrypt_action_file, require_signed_actions, sign_action_file, verify_action_file, ) +from je_auto_control.utils.exception.exceptions import CryptographyUnavailableError # Observability (Prometheus metrics + OpenTelemetry traces, headless) from je_auto_control.utils.observability import ( Counter as MetricCounter, @@ -1783,6 +1785,8 @@ def start_autocontrol_gui(*args, **kwargs): "default_secret_manager", "default_secret_store_path", # Action-file security (sign + encrypt) "VerifyResult", "sign_action_file", "verify_action_file", + "SigningConfig", "action_signing_config", "create_signing_keypair", + "CryptographyUnavailableError", "require_signed_actions", "encrypt_action_file", "decrypt_action_file", # Observability (Prometheus + OpenTelemetry) "MetricCounter", "MetricGauge", "MetricHistogram", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index a7d2150ec..a9638555d 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -5057,13 +5057,24 @@ def _add_work_queue_specs(specs: List[CommandSpec]) -> None: fields=(FieldSpec("path", FieldType.FILE_PATH),), description="Delete a file to the OS recycle bin (recoverable).", )) + specs.append(CommandSpec( + "AC_create_signing_keypair", "Security", "Create Signing Key Pair", + fields=( + FieldSpec("private_path", FieldType.STRING), + FieldSpec("public_path", FieldType.STRING), + ), + description="Create an Ed25519 key pair: keep the private key on the " + "signing machine, give execution endpoints the public key.", + )) specs.append(CommandSpec( "AC_sign_action_file", "Security", "Sign Action File", fields=( FieldSpec("path", FieldType.FILE_PATH), FieldSpec("key", FieldType.STRING, optional=True), + FieldSpec("private_key_path", FieldType.FILE_PATH, optional=True), ), - description="Write an HMAC-SHA256 signature sidecar for an action file.", + description="Write a signature sidecar for an action file " + "(Ed25519 with a private key, else HMAC-SHA256).", )) specs.append(CommandSpec( "AC_verify_action_file", "Security", "Verify Action File", @@ -5072,6 +5083,7 @@ def _add_work_queue_specs(specs: List[CommandSpec]) -> None: FieldSpec("key", FieldType.STRING, optional=True), FieldSpec("raise_on_fail", FieldType.BOOL, optional=True, default=False), + FieldSpec("public_key_path", FieldType.FILE_PATH, optional=True), ), description="Verify an action file against its signature sidecar.", )) diff --git a/je_auto_control/utils/action_signing/__init__.py b/je_auto_control/utils/action_signing/__init__.py index 3ca0f05c3..198e91814 100644 --- a/je_auto_control/utils/action_signing/__init__.py +++ b/je_auto_control/utils/action_signing/__init__.py @@ -1,14 +1,21 @@ -"""Action-file security: HMAC-SHA256 signing + Fernet encryption.""" +"""Action-file security: Ed25519 / HMAC-SHA256 signing + Fernet encryption.""" +from je_auto_control.utils.action_signing.asymmetric import create_signing_keypair from je_auto_control.utils.action_signing.cipher import ( decrypt_action_file, encrypt_action_file, ) +from je_auto_control.utils.action_signing.config import ( + SigningConfig, action_signing_config, +) from je_auto_control.utils.action_signing.signer import ( VerifyResult, read_signed_action_bytes, require_signed_actions, sign_action_file, signing_required, verify_action_file, ) __all__ = [ + "SigningConfig", "VerifyResult", + "action_signing_config", + "create_signing_keypair", "decrypt_action_file", "encrypt_action_file", "read_signed_action_bytes", diff --git a/je_auto_control/utils/action_signing/_key_file.py b/je_auto_control/utils/action_signing/_key_file.py index 23b390c46..9e21da603 100644 --- a/je_auto_control/utils/action_signing/_key_file.py +++ b/je_auto_control/utils/action_signing/_key_file.py @@ -55,3 +55,19 @@ def load_or_create_key_file(path: Path, generate: Callable[[], bytes], "must then be signed or encrypted again)", ) return key + + +def write_new_file(path: Path, data: bytes, mode: int) -> None: + """Write ``data`` to a file that must not exist yet, created with ``mode``. + + ``O_EXCL`` makes the refusal atomic, so a key already at ``path`` -- the + public key an endpoint trusts, say -- can never be replaced through here. + Raises :class:`AutoControlException` when ``path`` exists. + """ + path.parent.mkdir(parents=True, exist_ok=True) + try: + descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode) + except FileExistsError as error: + raise AutoControlException(f"key file {str(path)!r} already exists") from error + with os.fdopen(descriptor, "wb") as key_file: + key_file.write(data) diff --git a/je_auto_control/utils/action_signing/asymmetric.py b/je_auto_control/utils/action_signing/asymmetric.py new file mode 100644 index 000000000..a6b327eb8 --- /dev/null +++ b/je_auto_control/utils/action_signing/asymmetric.py @@ -0,0 +1,188 @@ +"""Version-2 action-file signatures: Ed25519, so verifying cannot sign. + +The HMAC sidecar is one shared secret -- whoever can check a signature can +write one. Here the signing machine keeps a private key and every execution +endpoint gets the public key, which verifies and nothing else. + +A version-2 sidecar is a small JSON envelope:: + + {"version": 2, "algorithm": "ed25519", "key_id": "...", "signature": "..."} + +``signature`` is base64 of the Ed25519 signature over a fixed context line +followed by the file's exact bytes; ``key_id`` names the public key (the first +16 hex digits of its SHA-256) so a file signed by another pair says so instead +of reading as tampered. Keys are unencrypted PEM: PKCS#8 for the private half +(created 0600), SubjectPublicKeyInfo for the public half. + +``cryptography`` is imported lazily -- it has no Windows arm64 wheel, and HMAC +signing must keep working without it. This module is GUI-free and imports no Qt. +""" +import base64 +import binascii +import hashlib +import json +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Dict, Optional, Union + +from je_auto_control.utils.action_signing._key_file import write_new_file +from je_auto_control.utils.action_signing.config import action_signing_config +from je_auto_control.utils.exception.exceptions import ( + AutoControlException, CryptographyUnavailableError, +) +from je_auto_control.utils.logging.logging_instance import autocontrol_logger + +SIGNATURE_VERSION = 2 +_ALGORITHM = "ed25519" +_CONTEXT = b"je_auto_control action file signature v2\n" +_KEY_ID_LENGTH = 16 + +PathLike = Union[str, Path] + + +def _ed25519() -> SimpleNamespace: + """Return the ``cryptography`` names used here, or explain their absence.""" + try: + from cryptography.exceptions import InvalidSignature + from cryptography.hazmat.primitives import serialization + from cryptography.hazmat.primitives.asymmetric import ed25519 + except ImportError as error: + raise CryptographyUnavailableError( + "Ed25519 action-file signatures require cryptography " + "(pip install cryptography). It has no Windows arm64 wheel, so " + "key-pair signing is unavailable there; HMAC signing still works." + ) from error + return SimpleNamespace( + invalid_signature=InvalidSignature, serialization=serialization, + private_key=ed25519.Ed25519PrivateKey, public_key=ed25519.Ed25519PublicKey, + ) + + +def create_signing_keypair(private_path: PathLike, public_path: PathLike) -> None: + """Create an Ed25519 key pair as two PEM files; neither may exist yet. + + Keep ``private_path`` on the machine that signs and copy ``public_path`` + to every execution endpoint. Raises :class:`AutoControlException` when a + file is already there, or when this process is configured to verify only + -- a key pair minted on an execution endpoint separates nothing. + """ + if action_signing_config().verify_only: + raise AutoControlException( + "this endpoint verifies only (a public key and no private key are " + "configured); create the key pair on the signing machine", + ) + private, public = Path(private_path), Path(public_path) + for target in (private, public): + if target.exists(): + raise AutoControlException(f"key file {str(target)!r} already exists") + crypto = _ed25519() + key = crypto.private_key.generate() + encoding = crypto.serialization.Encoding.PEM + write_new_file(private, key.private_bytes( + encoding, crypto.serialization.PrivateFormat.PKCS8, + crypto.serialization.NoEncryption()), 0o600) + try: + write_new_file(public, key.public_key().public_bytes( + encoding, crypto.serialization.PublicFormat.SubjectPublicKeyInfo), 0o644) + except (OSError, AutoControlException): + private.unlink(missing_ok=True) # never leave half a pair behind + raise + autocontrol_logger.info("created action signing key pair, public half at %s", public) + + +def _read_key_file(path: Path, kind: str) -> bytes: + try: + return path.read_bytes() + except OSError as error: + raise AutoControlException( + f"cannot read the {kind} signing key {str(path)!r}: {error}") from error + + +def _load_private_key(path: PathLike) -> Any: + crypto = _ed25519() + target = Path(path) + try: + key = crypto.serialization.load_pem_private_key( + _read_key_file(target, "private"), password=None) + except (ValueError, TypeError) as error: + raise AutoControlException( + f"{str(target)!r} is not an unencrypted PEM private key") from error + if not isinstance(key, crypto.private_key): + raise AutoControlException(f"{str(target)!r} is not an Ed25519 private key") + return key + + +def _load_public_key(path: PathLike) -> Any: + crypto = _ed25519() + target = Path(path) + try: + key = crypto.serialization.load_pem_public_key(_read_key_file(target, "public")) + except ValueError as error: + raise AutoControlException(f"{str(target)!r} is not a PEM public key") from error + if not isinstance(key, crypto.public_key): + raise AutoControlException(f"{str(target)!r} is not an Ed25519 public key") + return key + + +def _key_id(public_key: Any) -> str: + crypto = _ed25519() + raw = public_key.public_bytes(crypto.serialization.Encoding.Raw, + crypto.serialization.PublicFormat.Raw) + return hashlib.sha256(raw).hexdigest()[:_KEY_ID_LENGTH] + + +def sign_envelope(data: bytes, private_key_path: PathLike) -> str: + """Return the version-2 sidecar text for ``data``, signed by the private key.""" + key = _load_private_key(private_key_path) + signature = key.sign(_CONTEXT + data) + return json.dumps({ + "version": SIGNATURE_VERSION, + "algorithm": _ALGORITHM, + "key_id": _key_id(key.public_key()), + "signature": base64.b64encode(signature).decode("ascii"), + }) + + +def is_envelope(sidecar_text: str) -> bool: + """Whether a sidecar is a JSON envelope rather than a bare HMAC hex digest.""" + return sidecar_text.lstrip().startswith("{") + + +def _parse_envelope(sidecar_text: str) -> Optional[Dict[str, Any]]: + """Return the envelope when it is a well-formed version-2 one, else ``None``.""" + try: + envelope = json.loads(sidecar_text) + except ValueError: + return None + if not isinstance(envelope, dict) or not isinstance(envelope.get("signature"), str): + return None + if envelope.get("version") != SIGNATURE_VERSION or envelope.get("algorithm") != _ALGORITHM: + return None + return envelope + + +def verification_key(public_key_path: Optional[PathLike], + private_key_path: Optional[PathLike]) -> Any: + """Load the public key, deriving it from the private one on a signing machine.""" + if public_key_path is not None: + return _load_public_key(public_key_path) + if private_key_path is None: + raise AutoControlException("no signing key is configured to verify with") + return _load_private_key(private_key_path).public_key() + + +def envelope_failure(sidecar_text: str, data: bytes, public_key: Any) -> Optional[str]: + """Return why ``sidecar_text`` does not sign ``data``, or ``None`` when it does.""" + envelope = _parse_envelope(sidecar_text) + if envelope is None: + return "malformed or unsupported signature envelope" + expected_id = _key_id(public_key) + signed_id = envelope.get("key_id") + if signed_id is not None and signed_id != expected_id: + return f"signed with a different key ({signed_id!r}, this endpoint trusts {expected_id!r})" + try: + signature = base64.b64decode(envelope["signature"], validate=True) + public_key.verify(signature, _CONTEXT + data) + except (binascii.Error, ValueError, _ed25519().invalid_signature): + return "signature mismatch (tampered or wrong key)" + return None diff --git a/je_auto_control/utils/action_signing/cipher.py b/je_auto_control/utils/action_signing/cipher.py index 6b6b15c35..f1e38988e 100644 --- a/je_auto_control/utils/action_signing/cipher.py +++ b/je_auto_control/utils/action_signing/cipher.py @@ -17,7 +17,9 @@ from typing import Optional, Union from je_auto_control.utils.action_signing._key_file import load_or_create_key_file -from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.exception.exceptions import ( + AutoControlException, CryptographyUnavailableError, +) from je_auto_control.utils.logging.logging_instance import autocontrol_logger @@ -47,7 +49,7 @@ def _fernet_types() -> tuple: try: from cryptography.fernet import Fernet, InvalidToken except ImportError as error: - raise RuntimeError( + raise CryptographyUnavailableError( "Action-file encryption requires cryptography (pip install cryptography). " "It has no Windows arm64 wheel, so encryption is unavailable there." ) from error diff --git a/je_auto_control/utils/action_signing/config.py b/je_auto_control/utils/action_signing/config.py new file mode 100644 index 000000000..97b15307d --- /dev/null +++ b/je_auto_control/utils/action_signing/config.py @@ -0,0 +1,62 @@ +"""Which signing keys this process was configured with, read at call time. + +Three environment variables separate the machine that signs from the machines +that execute: + +``JE_AUTOCONTROL_ACTION_SIGNING_PRIVATE_KEY`` + Path of the Ed25519 private key. Set it on the signing machine only. +``JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY`` + Path of the matching public key. Set it on every execution endpoint; an + endpoint that has this and no private key verifies and cannot sign. +``JE_AUTOCONTROL_ACCEPT_LEGACY_ACTION_SIGNATURES`` + Migration mode: also accept HMAC sidecars written before version 2. + Anyone who can execute actions can produce one, so switch it off again + once every file is signed with the key pair. + +With none of them set, signing is the per-user HMAC it has always been. +This module is GUI-free and imports no Qt. +""" +import os +from dataclasses import dataclass +from pathlib import Path +from typing import Optional + +PRIVATE_KEY_ENV = "JE_AUTOCONTROL_ACTION_SIGNING_PRIVATE_KEY" +PUBLIC_KEY_ENV = "JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY" +ACCEPT_LEGACY_ENV = "JE_AUTOCONTROL_ACCEPT_LEGACY_ACTION_SIGNATURES" + +_SWITCHED_ON = frozenset({"1", "true", "yes", "on"}) + + +@dataclass(frozen=True) +class SigningConfig: + """The signing keys one process holds, and whether it accepts HMAC sidecars.""" + + private_key_path: Optional[Path] + public_key_path: Optional[Path] + accept_legacy: bool + + @property + def asymmetric(self) -> bool: + """Whether version-2 (Ed25519) signatures are configured at all.""" + return self.private_key_path is not None or self.public_key_path is not None + + @property + def verify_only(self) -> bool: + """Whether this process holds the public key and no private key.""" + return self.public_key_path is not None and self.private_key_path is None + + +def _path_from(name: str) -> Optional[Path]: + value = os.environ.get(name, "").strip() + return Path(value) if value else None + + +def action_signing_config() -> SigningConfig: + """Return the signing configuration the environment describes right now.""" + legacy = os.environ.get(ACCEPT_LEGACY_ENV, "").strip().lower() in _SWITCHED_ON + return SigningConfig( + private_key_path=_path_from(PRIVATE_KEY_ENV), + public_key_path=_path_from(PUBLIC_KEY_ENV), + accept_legacy=legacy, + ) diff --git a/je_auto_control/utils/action_signing/signer.py b/je_auto_control/utils/action_signing/signer.py index ba1da8e5c..eba364d6f 100644 --- a/je_auto_control/utils/action_signing/signer.py +++ b/je_auto_control/utils/action_signing/signer.py @@ -1,13 +1,20 @@ -"""Sign and verify JSON action files with HMAC-SHA256. - -A signed action file gets a ``.sig`` sidecar holding the hex HMAC -of the file's exact bytes, keyed by a signing key. Verifying recomputes -the HMAC and compares it in constant time, so a tampered script (or one -signed with a different key) is rejected before it runs — closing the -"never trust action data from disk / the network" gap for replayed flows. - -The key is either supplied explicitly or read from the per-user file at -``~/.je_auto_control/action_signing_key`` (created on first use, 0600). +"""Sign and verify JSON action files: Ed25519 key pair, or HMAC-SHA256. + +A signed action file gets a ``.sig`` sidecar over the file's exact +bytes, so a tampered script (or one signed with a different key) is rejected +before it runs — closing the "never trust action data from disk / the +network" gap for replayed flows. + +Two sidecar formats exist. **Version 2** is an Ed25519 envelope +(:mod:`.asymmetric`): the signing machine holds the private key, execution +endpoints hold the public key, and an endpoint that only verifies refuses to +sign -- which keeps the right to sign apart from the right to execute. The +keys come from explicit arguments or from the environment variables in +:mod:`.config`. The **legacy** format is the hex HMAC-SHA256 keyed by an +explicit key or the per-user file at ``~/.je_auto_control/action_signing_key`` +(created on first use, 0600); it is what signing does when no key pair is +configured, and once one is configured it is accepted only in the explicit +migration mode (``JE_AUTOCONTROL_ACCEPT_LEGACY_ACTION_SIGNATURES``). With ``JE_AUTOCONTROL_REQUIRE_SIGNED_ACTIONS`` set, every path that runs an action file loads it through :func:`read_signed_action_bytes`, which reads @@ -22,7 +29,11 @@ from pathlib import Path from typing import Any, Dict, Optional, Union +from je_auto_control.utils.action_signing import asymmetric from je_auto_control.utils.action_signing._key_file import load_or_create_key_file +from je_auto_control.utils.action_signing.config import ( + ACCEPT_LEGACY_ENV, PUBLIC_KEY_ENV, action_signing_config, +) from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.logging.logging_instance import autocontrol_logger @@ -37,6 +48,7 @@ def _default_key_path() -> Path: _KEY_LENGTH = 32 KeyType = Optional[Union[bytes, str]] +KeyPath = Optional[Union[str, Path]] @dataclass(frozen=True) @@ -77,13 +89,35 @@ def _digest(data: bytes, key: bytes) -> str: return hmac.new(key, data, hashlib.sha256).hexdigest() -def sign_action_file(path: Union[str, Path], key: KeyType = None) -> str: - """Write an HMAC-SHA256 signature sidecar for the file at ``path``. +def _signature_text(data: bytes, key: KeyType, private_key_path: KeyPath) -> str: + """Sign ``data`` with the key pair when one applies, else with HMAC.""" + config = action_signing_config() + private = private_key_path + if private is None and key is None: + private = config.private_key_path + if private is not None: + return asymmetric.sign_envelope(data, private) + if config.verify_only: + raise AutoControlException( + "this endpoint verifies only (a public key and no private key are " + "configured); sign the file on the signing machine", + ) + return _digest(data, _load_or_create_key(key)) + + +def sign_action_file(path: Union[str, Path], key: KeyType = None, + *, private_key_path: KeyPath = None) -> str: + """Write a signature sidecar for the file at ``path``. - Returns the sidecar path (``.sig``). + With ``private_key_path`` -- or, when no ``key`` is given, the private key + named by ``JE_AUTOCONTROL_ACTION_SIGNING_PRIVATE_KEY`` -- the sidecar is a + version-2 Ed25519 envelope. Otherwise it is the HMAC-SHA256 of the file + under ``key`` or the per-user key. An endpoint configured with a public + key and no private key refuses instead of falling back to HMAC, raising + :class:`AutoControlException`. Returns the sidecar path (``.sig``). """ target = Path(path) - signature = _digest(target.read_bytes(), _load_or_create_key(key)) + signature = _signature_text(target.read_bytes(), key, private_key_path) sig_path = _sig_path(target) sig_path.write_text(signature, encoding="utf-8") autocontrol_logger.info("signed action file %s", target) @@ -91,21 +125,49 @@ def sign_action_file(path: Union[str, Path], key: KeyType = None) -> str: def verify_action_file(path: Union[str, Path], key: KeyType = None, - *, raise_on_fail: bool = False) -> VerifyResult: + *, raise_on_fail: bool = False, + public_key_path: KeyPath = None) -> VerifyResult: """Verify the action file at ``path`` against its ``.sig`` sidecar. - Returns a :class:`VerifyResult`. With ``raise_on_fail`` set, an - unverified file raises :class:`AutoControlException` instead. + A version-2 sidecar is checked with ``public_key_path`` or the configured + public key; no private key is read. An HMAC sidecar is checked with + ``key`` or the per-user key, and refused when a key pair is configured + unless migration mode is on. Returns a :class:`VerifyResult`. With + ``raise_on_fail`` set, an unverified file raises + :class:`AutoControlException` instead. """ try: data = Path(path).read_bytes() except OSError as error: return _fail(path, f"read error: {error}", raise_on_fail) - return _verify_bytes(path, data, key, raise_on_fail) + return _verify_bytes(path, data, key, raise_on_fail, public_key_path) + + +def _signature_failure(sidecar: str, data: bytes, key: KeyType, + public_key_path: KeyPath) -> Optional[str]: + """Return why ``sidecar`` does not sign ``data``, or ``None`` when it does.""" + config = action_signing_config() + public = public_key_path if public_key_path is not None else config.public_key_path + key_pair = public is not None or config.private_key_path is not None + if asymmetric.is_envelope(sidecar): + if not key_pair: + return f"Ed25519 signature but no public key is configured (set {PUBLIC_KEY_ENV})" + trusted = asymmetric.verification_key(public, config.private_key_path) + return asymmetric.envelope_failure(sidecar, data, trusted) + if key_pair and not config.accept_legacy: + return ("legacy HMAC signature refused: sign the file again with the key pair, " + f"or set {ACCEPT_LEGACY_ENV}=1 while migrating") + if key_pair: + autocontrol_logger.warning( + "accepting a legacy HMAC action signature because %s is set", ACCEPT_LEGACY_ENV) + actual = _digest(data, _load_or_create_key(key)) + if not hmac.compare_digest(sidecar.encode("utf-8"), actual.encode("utf-8")): + return "signature mismatch (tampered or wrong key)" + return None def _verify_bytes(path: Union[str, Path], data: bytes, key: KeyType, - raise_on_fail: bool) -> VerifyResult: + raise_on_fail: bool, public_key_path: KeyPath = None) -> VerifyResult: """Check ``data`` -- the content of ``path`` -- against its sidecar.""" sig_path = _sig_path(Path(path)) if not sig_path.exists(): @@ -114,10 +176,9 @@ def _verify_bytes(path: Union[str, Path], data: bytes, key: KeyType, expected = sig_path.read_text(encoding="utf-8").strip() except (OSError, UnicodeDecodeError) as error: return _fail(path, f"read error: {error}", raise_on_fail) - actual = _digest(data, _load_or_create_key(key)) - if not hmac.compare_digest(expected.encode("utf-8"), actual.encode("utf-8")): - return _fail(path, "signature mismatch (tampered or wrong key)", - raise_on_fail) + reason = _signature_failure(expected, data, key, public_key_path) + if reason is not None: + return _fail(path, reason, raise_on_fail) return VerifyResult(str(path), True, "signature valid") diff --git a/je_auto_control/utils/exception/exceptions.py b/je_auto_control/utils/exception/exceptions.py index b74d8e22a..86e6ef541 100644 --- a/je_auto_control/utils/exception/exceptions.py +++ b/je_auto_control/utils/exception/exceptions.py @@ -129,3 +129,13 @@ class AutoControlUnsupportedOperationException( slipped straight past all of them — aborting a whole script where a single action should have been reported as failed. """ + + +# Optional dependency +class CryptographyUnavailableError(AutoControlException, RuntimeError): + """A feature needs ``cryptography`` and the package is not installed. + + ``cryptography`` publishes no Windows arm64 wheel, so it is absent there by + design and the message carries the install hint. Inherits ``RuntimeError`` + because that is what these accessors raised before the type existed. + """ diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index b9a7bf614..2c1d7993a 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -2211,18 +2211,29 @@ def _human_type(text: str, base_delay: float = 0.05, jitter: float = 0.04, return {"chars": len(str(text)), "total_delay_s": round(sum(delays), 3)} -def _sign_action_file(path: str, key: Optional[str] = None) -> Dict[str, Any]: - """Executor adapter: write an HMAC-SHA256 signature sidecar for a file.""" +def _sign_action_file(path: str, key: Optional[str] = None, + private_key_path: Optional[str] = None) -> Dict[str, Any]: + """Executor adapter: write an Ed25519 or HMAC signature sidecar for a file.""" from je_auto_control.utils.action_signing import sign_action_file - return {"signature_path": sign_action_file(path, key)} + return {"signature_path": sign_action_file( + path, key, private_key_path=private_key_path)} + + +def _create_signing_keypair(private_path: str, public_path: str) -> Dict[str, Any]: + """Executor adapter: create an Ed25519 action-signing key pair.""" + from je_auto_control.utils.action_signing import create_signing_keypair + create_signing_keypair(private_path, public_path) + return {"private_path": str(private_path), "public_path": str(public_path)} def _verify_action_file(path: str, key: Optional[str] = None, - raise_on_fail: bool = False) -> Dict[str, Any]: + raise_on_fail: bool = False, + public_key_path: Optional[str] = None) -> Dict[str, Any]: """Executor adapter: verify an action file against its signature sidecar.""" from je_auto_control.utils.action_signing import verify_action_file return verify_action_file( path, key, raise_on_fail=_as_bool(raise_on_fail), + public_key_path=public_key_path, ).to_dict() @@ -7793,7 +7804,8 @@ def __init__(self): "AC_assert_any": _assert_any, "AC_assert_eventually": _assert_eventually, - # Action-file integrity (HMAC-SHA256 sign / verify) + # Action-file integrity (Ed25519 / HMAC-SHA256 sign / verify) + "AC_create_signing_keypair": _create_signing_keypair, "AC_sign_action_file": _sign_action_file, "AC_verify_action_file": _verify_action_file, "AC_encrypt_action_file": _encrypt_action_file, diff --git a/je_auto_control/utils/secrets/secret_store.py b/je_auto_control/utils/secrets/secret_store.py index 3f95bccbb..ccba7b4c3 100644 --- a/je_auto_control/utils/secrets/secret_store.py +++ b/je_auto_control/utils/secrets/secret_store.py @@ -30,7 +30,9 @@ from pathlib import Path from typing import Any, Dict, Iterator, List, Optional, Tuple -from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.exception.exceptions import ( + AutoControlException, CryptographyUnavailableError, +) from je_auto_control.utils.json_store.json_store import _file_lock, atomic_write_text @@ -50,7 +52,7 @@ def _fernet_types() -> tuple: try: from cryptography.fernet import Fernet, InvalidToken except ImportError as error: - raise RuntimeError( + raise CryptographyUnavailableError( "The secret vault requires cryptography (pip install cryptography). " "It has no Windows arm64 wheel, so the vault is unavailable there." ) from error diff --git a/test/unit_test/headless/test_signing_execution_roles.py b/test/unit_test/headless/test_signing_execution_roles.py new file mode 100644 index 000000000..db49bf380 --- /dev/null +++ b/test/unit_test/headless/test_signing_execution_roles.py @@ -0,0 +1,268 @@ +"""The signing key is kept apart from the right to execute. No Qt. + +``AC_sign_action_file`` signed with the per-user HMAC key, and verification +read that same key -- so whoever could run an action through the socket, REST +or MCP could sign a file first and then run it, and +``JE_AUTOCONTROL_REQUIRE_SIGNED_ACTIONS`` only stopped someone who could write +files but not execute. Version-2 signatures are Ed25519: the execution endpoint +holds the public key alone, which verifies and cannot sign. +""" +import json +import os +import stat +import sys +from pathlib import Path + +import pytest + +pytest.importorskip("cryptography") + +from je_auto_control.utils.action_signing import ( # noqa: E402 # reason: after importorskip + SigningConfig, action_signing_config, create_signing_keypair, + sign_action_file, verify_action_file, +) +from je_auto_control.utils.action_signing import signer # noqa: E402 # reason: after importorskip +from je_auto_control.utils.exception.exceptions import ( # noqa: E402 # reason: after importorskip + AutoControlException, CryptographyUnavailableError, +) +from je_auto_control.utils.json import json_file # noqa: E402 # reason: after importorskip + +PRIVATE_ENV = "JE_AUTOCONTROL_ACTION_SIGNING_PRIVATE_KEY" +PUBLIC_ENV = "JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY" +LEGACY_ENV = "JE_AUTOCONTROL_ACCEPT_LEGACY_ACTION_SIGNATURES" +REQUIRE_ENV = "JE_AUTOCONTROL_REQUIRE_SIGNED_ACTIONS" +_HMAC_KEY = b"h" * 32 + + +@pytest.fixture(autouse=True) +def _clean_environment(monkeypatch, tmp_path): + """No signing variable leaks in, and the per-user key lives in tmp_path.""" + for name in (PRIVATE_ENV, PUBLIC_ENV, LEGACY_ENV, REQUIRE_ENV): + monkeypatch.delenv(name, raising=False) + monkeypatch.setattr(signer, "_default_key_path", + lambda: tmp_path / "home" / "action_signing_key") + + +@pytest.fixture +def keys(tmp_path): + """A key pair as the signing machine would create it.""" + private, public = tmp_path / "signer" / "private.pem", tmp_path / "public.pem" + create_signing_keypair(private, public) + return private, public + + +@pytest.fixture +def execution_endpoint(monkeypatch, keys): + """An endpoint that enforces signatures and holds the public key only.""" + monkeypatch.setenv(PUBLIC_ENV, str(keys[1])) + monkeypatch.setenv(REQUIRE_ENV, "1") + return keys + + +def _script(tmp_path, name="flow.json"): + path = tmp_path / name + path.write_text(json.dumps([["AC_noop"]]), encoding="utf-8") + return path + + +# --- the key pair ------------------------------------------------------------- + +def test_a_key_pair_is_two_pem_files_and_the_private_one_is_private(keys): + private, public = keys + assert b"PRIVATE KEY" in private.read_bytes() + assert b"PUBLIC KEY" in public.read_bytes() + assert b"PRIVATE" not in public.read_bytes() + if sys.platform != "win32": + assert stat.S_IMODE(os.stat(private).st_mode) == 0o600 + + +@pytest.mark.parametrize("existing", ["private", "public"]) +def test_an_existing_key_is_never_overwritten(tmp_path, existing): + private, public = tmp_path / "private.pem", tmp_path / "public.pem" + (private if existing == "private" else public).write_bytes(b"keep me") + with pytest.raises(AutoControlException, match="already exists"): + create_signing_keypair(private, public) + survivor = private if existing == "private" else public + assert survivor.read_bytes() == b"keep me" + assert not (public if existing == "private" else private).exists() + + +# --- version-2 signatures ------------------------------------------------------- + +def test_a_version_2_signature_verifies_with_the_public_key_alone(tmp_path, keys): + private, public = keys + path = _script(tmp_path) + sidecar = sign_action_file(path, private_key_path=private) + envelope = json.loads(Path(sidecar).read_text(encoding="utf-8")) + assert envelope["version"] == 2 + assert envelope["algorithm"] == "ed25519" + private.unlink() # the verifier never has it + assert verify_action_file(path, public_key_path=public).verified is True + + +def test_a_tampered_file_fails_version_2_verification(tmp_path, keys): + private, public = keys + path = _script(tmp_path) + sign_action_file(path, private_key_path=private) + path.write_text('[["AC_evil"]]', encoding="utf-8") + result = verify_action_file(path, public_key_path=public) + assert result.verified is False + assert "mismatch" in result.reason + + +def test_a_signature_from_another_key_pair_is_refused(tmp_path, keys): + other_private, other_public = tmp_path / "o.pem", tmp_path / "o.pub" + create_signing_keypair(other_private, other_public) + path = _script(tmp_path) + sign_action_file(path, private_key_path=other_private) + result = verify_action_file(path, public_key_path=keys[1]) + assert result.verified is False + assert "different key" in result.reason + + +@pytest.mark.parametrize("sidecar", [ + "{not json", '{"version": 3, "algorithm": "ed25519", "signature": "AAAA"}', + '{"version": 2, "algorithm": "none", "signature": "AAAA"}', + '{"version": 2, "algorithm": "ed25519", "signature": "***"}', + '{"version": 2, "algorithm": "ed25519"}', +]) +def test_a_malformed_envelope_is_unverified_not_a_crash(tmp_path, keys, sidecar): + path = _script(tmp_path) + Path(str(path) + ".sig").write_text(sidecar, encoding="utf-8") + assert verify_action_file(path, public_key_path=keys[1]).verified is False + + +def test_a_version_2_signature_without_a_public_key_is_unverified(tmp_path, keys): + path = _script(tmp_path) + sign_action_file(path, private_key_path=keys[0]) + result = verify_action_file(path) + assert result.verified is False + assert PUBLIC_ENV in result.reason + + +def test_the_signing_machine_signs_from_its_environment(tmp_path, keys, monkeypatch): + monkeypatch.setenv(PRIVATE_ENV, str(keys[0])) + path = _script(tmp_path) + sidecar = sign_action_file(path) + assert json.loads(Path(sidecar).read_text(encoding="utf-8"))["version"] == 2 + assert verify_action_file(path).verified is True # public half derived + + +# --- the execution endpoint ----------------------------------------------------- + +def test_verifier_has_no_private_key(execution_endpoint): + verifier = action_signing_config() + assert isinstance(verifier, SigningConfig) + assert verifier.private_key_path is None + assert verifier.public_key_path == execution_endpoint[1] + assert verifier.verify_only is True + + +def test_execution_endpoint_cannot_sign(tmp_path, execution_endpoint): + from je_auto_control.utils.executor.action_executor import executor + path = _script(tmp_path) + for arguments in ({"path": str(path)}, {"path": str(path), "key": "anything"}): + with pytest.raises(AutoControlException, match="verifies only"): + executor.event_dict["AC_sign_action_file"](**arguments) + assert not Path(str(path) + ".sig").exists() + assert not (tmp_path / "home" / "action_signing_key").exists(), \ + "the refusal must not fall back to creating a personal key" + + +def test_execution_endpoint_cannot_mint_a_key_pair(tmp_path, execution_endpoint): + from je_auto_control.utils.executor.action_executor import executor + with pytest.raises(AutoControlException, match="verifies only"): + executor.event_dict["AC_create_signing_keypair"]( + private_path=str(tmp_path / "p.pem"), public_path=str(tmp_path / "p.pub")) + assert not (tmp_path / "p.pem").exists() + + +def test_execution_endpoint_runs_a_file_signed_elsewhere(tmp_path, execution_endpoint): + path = _script(tmp_path) + sign_action_file(path, private_key_path=execution_endpoint[0]) + assert json_file.read_executable_action_json(str(path)) == [["AC_noop"]] + + +def test_legacy_signature_requires_explicit_migration(tmp_path, execution_endpoint, monkeypatch): + path = _script(tmp_path) + monkeypatch.delenv(PUBLIC_ENV) + sign_action_file(path) # the per-user HMAC key, as before version 2 + monkeypatch.setenv(PUBLIC_ENV, str(execution_endpoint[1])) + with pytest.raises(AutoControlException, match="legacy HMAC"): + json_file.read_executable_action_json(str(path)) + monkeypatch.setenv(LEGACY_ENV, "1") + assert json_file.read_executable_action_json(str(path)) == [["AC_noop"]] + + +@pytest.mark.parametrize("value", ["", "0", "false", "no", "off"]) +def test_migration_mode_is_off_unless_switched_on(monkeypatch, value): + monkeypatch.setenv(LEGACY_ENV, value) + assert action_signing_config().accept_legacy is False + + +def test_an_explicit_hmac_key_does_not_bypass_version_2(tmp_path, execution_endpoint, monkeypatch): + path = _script(tmp_path) + monkeypatch.delenv(PUBLIC_ENV) + sign_action_file(path, _HMAC_KEY) + monkeypatch.setenv(PUBLIC_ENV, str(execution_endpoint[1])) + assert verify_action_file(path, _HMAC_KEY).verified is False + + +# --- deployments that never configured a key pair ------------------------------- + +def test_without_a_key_pair_signing_is_the_hmac_sidecar_it_always_was(tmp_path, monkeypatch): + path = _script(tmp_path) + sidecar = Path(sign_action_file(path)) + assert len(sidecar.read_text(encoding="utf-8")) == 64 # bare hex HMAC-SHA256 + monkeypatch.setenv(REQUIRE_ENV, "1") + assert json_file.read_executable_action_json(str(path)) == [["AC_noop"]] + + +# --- surfaces ------------------------------------------------------------------- + +def test_the_executor_commands_cover_the_whole_flow(tmp_path): + from je_auto_control.utils.executor.action_executor import executor + private, public = tmp_path / "k.pem", tmp_path / "k.pub" + path = _script(tmp_path) + created = executor.event_dict["AC_create_signing_keypair"]( + private_path=str(private), public_path=str(public)) + assert created == {"private_path": str(private), "public_path": str(public)} + executor.event_dict["AC_sign_action_file"](path=str(path), private_key_path=str(private)) + verified = executor.event_dict["AC_verify_action_file"]( + path=str(path), public_key_path=str(public)) + assert verified["verified"] is True + + +def test_the_facade_and_the_script_builder_expose_the_key_pair_command(): + import je_auto_control + from je_auto_control.gui.script_builder.command_schema import _build_specs + for name in ("create_signing_keypair", "action_signing_config", "SigningConfig", + "CryptographyUnavailableError"): + assert name in je_auto_control.__all__ and hasattr(je_auto_control, name) + specs = {spec.command: spec for spec in _build_specs()} + assert [field.name for field in specs["AC_create_signing_keypair"].fields] == [ + "private_path", "public_path"] + assert "private_key_path" in [f.name for f in specs["AC_sign_action_file"].fields] + assert "public_key_path" in [f.name for f in specs["AC_verify_action_file"].fields] + + +# --- a platform without the cryptography wheel ---------------------------------- + +def test_a_missing_cryptography_wheel_is_a_typed_error_with_an_install_hint( + tmp_path, monkeypatch): + monkeypatch.setitem(sys.modules, "cryptography.exceptions", None) + with pytest.raises(CryptographyUnavailableError) as caught: + create_signing_keypair(tmp_path / "a.pem", tmp_path / "a.pub") + assert isinstance(caught.value, AutoControlException) + assert isinstance(caught.value, RuntimeError) + assert "pip install cryptography" in str(caught.value) + assert "Windows arm64" in str(caught.value) + assert not (tmp_path / "a.pem").exists() + + +def test_hmac_signing_still_works_without_cryptography(tmp_path, monkeypatch): + for name in [m for m in sys.modules if m.split(".")[0] == "cryptography"]: + monkeypatch.setitem(sys.modules, name, None) + path = _script(tmp_path) + sign_action_file(path, _HMAC_KEY) + assert verify_action_file(path, _HMAC_KEY).verified is True From 6f32e8211dfd571702ff968d8514625df7990aa5 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 23:52:55 +0800 Subject: [PATCH 06/26] Raise the cryptography floor to 50.0.0 The 48.0.1 floor still admitted GHSA-g6cj-pr64-35w5 (PKCS#7 EnvelopedData decryption oracle, >=44.0.0,<50.0.0). The maintainer accepted the cost: 49.0.0 dropped the macOS universal2 wheel, so Intel Macs now build cryptography from source. The Windows arm64 marker is unchanged. --- .github/workflows/platform-smoke.yml | 7 +- dev.toml | 2 +- docs/CAPABILITY_MATRIX.md | 8 +- pyproject.toml | 10 ++- .../test_crypto_fifty_install_contract.py | 83 +++++++++++++++++++ .../test_dependency_security_floors.py | 2 +- uv.lock | 4 +- 7 files changed, 103 insertions(+), 13 deletions(-) create mode 100644 test/unit_test/headless/test_crypto_fifty_install_contract.py diff --git a/.github/workflows/platform-smoke.yml b/.github/workflows/platform-smoke.yml index c5c89bfb0..4814f59c6 100644 --- a/.github/workflows/platform-smoke.yml +++ b/.github/workflows/platform-smoke.yml @@ -27,8 +27,9 @@ jobs: # back to building from source and CMake could not # configure for ARM64. Twelve minutes, then failure. # cryptography — wheels stop at 46.0.3; 46.0.4 onwards ship none. - # Our floor is >=48.0.1 and that is a security floor - # (GHSA-537c-gmf6-5ccf), so it cannot be lowered. + # Our floor is >=50.0.0 and that is a security floor + # (GHSA-537c-gmf6-5ccf, GHSA-g6cj-pr64-35w5), so it + # cannot be lowered. # # Nothing in the package imports either one at import time, so what # this square proves is real: the install succeeds and the stable API @@ -38,7 +39,7 @@ jobs: # # pip install --dry-run --only-binary=:all: --platform win_arm64 \ # --python-version 3.12 --target /tmp/probe \ - # 'opencv-python>=4.8,<6' 'cryptography>=48.0.1' + # 'opencv-python>=4.8,<6' 'cryptography>=50.0.0' # # When both resolve, drop the markers from pyproject.toml. os: [windows-2022, ubuntu-22.04, macos-14, ubuntu-22.04-arm, windows-11-arm] diff --git a/dev.toml b/dev.toml index 6220026a2..912df3352 100644 --- a/dev.toml +++ b/dev.toml @@ -28,7 +28,7 @@ dependencies = [ "python-Xlib==0.33;platform_system=='Linux' or platform_system=='FreeBSD' or platform_system=='OpenBSD' or platform_system=='NetBSD'", "mss==10.2.0", "defusedxml==0.7.1", - "cryptography>=48.0.1; sys_platform != 'win32' or platform_machine != 'ARM64'" + "cryptography>=50.0.0; sys_platform != 'win32' or platform_machine != 'ARM64'" ] classifiers = [ "Programming Language :: Python :: 3.10", diff --git a/docs/CAPABILITY_MATRIX.md b/docs/CAPABILITY_MATRIX.md index 59358ebe5..d19029a09 100644 --- a/docs/CAPABILITY_MATRIX.md +++ b/docs/CAPABILITY_MATRIX.md @@ -130,8 +130,12 @@ measurement rather than assumption, and **two** dependencies are why: **opencv-python publishes no `win_arm64` wheel** in any version, so pip falls back to building from source and CMake cannot configure for ARM64; and **cryptography stopped publishing one after 46.0.3**, while this project's -floor is `>=48.0.1` — a security floor (GHSA-537c-gmf6-5ccf) that cannot be -lowered to reach a wheel. Neither is a CI problem to work around: the package +floor is `>=50.0.0` — a security floor (GHSA-537c-gmf6-5ccf, then +GHSA-g6cj-pr64-35w5) that cannot be lowered to reach a wheel. The same floor +costs Intel Macs their prebuilt wheel: cryptography 49.0.0 stopped publishing +`macosx_10_9_universal2` and ships `macosx_11_0_arm64` only, so `pip install` +on an x86_64 Mac compiles cryptography from source and needs a Rust toolchain +first. Neither is a CI problem to work around: the package genuinely cannot be installed on Windows arm64 today. `Progress.md` records both, alongside a `pip --dry-run --platform win_arm64` command that re-checks them in seconds without an arm64 machine. diff --git a/pyproject.toml b/pyproject.toml index eb5aa6d1b..eca5a2ba2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -23,8 +23,10 @@ dependencies = [ # that machine. PEP 508 markers have no boolean "not", hence "A or B". # # opencv-python no win_arm64 wheel in any version. - # cryptography wheels stop at 46.0.3; our floor >=48.0.1 is a security - # floor (GHSA-537c-gmf6-5ccf) and cannot be lowered. + # cryptography wheels stop at 46.0.3; our floor >=50.0.0 is a security + # floor (GHSA-537c-gmf6-5ccf, GHSA-g6cj-pr64-35w5) and + # cannot be lowered. 49.0.0 also dropped the Intel-mac + # universal2 wheel, so x86_64 macOS builds from source. # je_open_cv pure Python, but it depends on opencv-python, so leaving # it unmarked drags OpenCV back in through the side door. # @@ -34,7 +36,7 @@ dependencies = [ # # pip install --dry-run --only-binary=:all: --platform win_arm64 \ # --python-version 3.12 --target /tmp/probe \ - # 'opencv-python>=4.8,<6' 'cryptography>=48.0.1' + # 'opencv-python>=4.8,<6' 'cryptography>=50.0.0' # # When both resolve, drop the marker from all three lines. Progress.md # records what Windows arm64 gives up in the meantime. @@ -51,7 +53,7 @@ dependencies = [ "python-Xlib==0.33;platform_system=='Linux' or platform_system=='FreeBSD' or platform_system=='OpenBSD' or platform_system=='NetBSD'", "mss==10.2.0", "defusedxml==0.7.1", - "cryptography>=48.0.1; sys_platform != 'win32' or platform_machine != 'ARM64'" + "cryptography>=50.0.0; sys_platform != 'win32' or platform_machine != 'ARM64'" ] classifiers = [ "Programming Language :: Python :: 3.10", diff --git a/test/unit_test/headless/test_crypto_fifty_install_contract.py b/test/unit_test/headless/test_crypto_fifty_install_contract.py new file mode 100644 index 000000000..10f9445e5 --- /dev/null +++ b/test/unit_test/headless/test_crypto_fifty_install_contract.py @@ -0,0 +1,83 @@ +"""cryptography's floor is 50, and a missing wheel is a typed error. No Qt. + +``cryptography>=48.0.1`` still admitted GHSA-g6cj-pr64-35w5 (high; the PKCS#7 +EnvelopedData decryption oracle, ``>=44.0.0, <50.0.0``). The floor is declared +twice -- ``pyproject.toml`` for the stable package, ``dev.toml`` for the dev +one -- and resolved once in ``uv.lock``; all three have to agree. + +Raising it costs Intel Macs their prebuilt wheel (49.0.0 dropped +``macosx_10_9_universal2``), and Windows arm64 has had none since 46.0.3. What +those platforms get instead is an error that says so: every door onto +``cryptography`` raises :class:`CryptographyUnavailableError`, which is in the +framework family and carries the install command. +""" +import pathlib +import re +import sys +from typing import Callable, Tuple + +import pytest + +from je_auto_control.utils.exception.exceptions import ( + AutoControlException, CryptographyUnavailableError, +) + +ROOT = pathlib.Path(__file__).resolve().parents[3] +FLOOR = (50, 0, 0) +ARM64_MARKER = "sys_platform != 'win32' or platform_machine != 'ARM64'" + + +def _version(text: str) -> Tuple[int, ...]: + return tuple(int(part) for part in text.split(".")) + + +@pytest.mark.parametrize("manifest", ["pyproject.toml", "dev.toml"]) +def test_crypto_floor_is_fifty(manifest: str) -> None: + """Both manifests declare the floor, and keep the Windows arm64 marker.""" + text = (ROOT / manifest).read_text(encoding="utf-8") + declared = re.findall(r'^\s*"cryptography>=([\d.]+);\s*([^"]+)"', text, re.M) + assert len(declared) == 1, f"{manifest} must require cryptography exactly once" + minimum, marker = declared[0] + assert _version(minimum) >= FLOOR + assert marker.strip() == ARM64_MARKER + + +def test_the_lock_resolves_at_or_above_the_floor() -> None: + """uv.lock is what CI installs; a stale lock would keep an old version.""" + text = (ROOT / "uv.lock").read_text(encoding="utf-8") + locked = re.search(r'^name = "cryptography"\nversion = "([\d.]+)"', text, re.M) + assert locked is not None, "uv.lock no longer pins cryptography" + assert _version(locked.group(1)) >= FLOOR + assert "specifier = \">=50.0.0\"" in text, "uv.lock was not regenerated for the new floor" + + +def _vault_door() -> Callable[[], object]: + from je_auto_control.utils.secrets import secret_store + return secret_store._fernet_types + + +def _cipher_door() -> Callable[[], object]: + from je_auto_control.utils.action_signing import cipher + return cipher._fernet_types + + +def _key_pair_door() -> Callable[[], object]: + from je_auto_control.utils.action_signing import asymmetric + return asymmetric._ed25519 + + +@pytest.mark.parametrize("door,blocked", [ + (_vault_door, "cryptography.fernet"), + (_cipher_door, "cryptography.fernet"), + (_key_pair_door, "cryptography.exceptions"), +]) +def test_missing_crypto_feature_is_typed(monkeypatch, door, blocked: str) -> None: + """A containment boundary catching AutoControlException sees this failure.""" + accessor = door() + monkeypatch.setitem(sys.modules, blocked, None) + with pytest.raises(CryptographyUnavailableError) as caught: + accessor() + assert isinstance(caught.value, AutoControlException) + assert isinstance(caught.value, RuntimeError), "existing callers catch RuntimeError" + assert "pip install cryptography" in str(caught.value) + assert isinstance(caught.value.__cause__, ImportError) diff --git a/test/unit_test/headless/test_dependency_security_floors.py b/test/unit_test/headless/test_dependency_security_floors.py index 9fea27cfd..6ac37bc49 100644 --- a/test/unit_test/headless/test_dependency_security_floors.py +++ b/test/unit_test/headless/test_dependency_security_floors.py @@ -14,7 +14,7 @@ #: requirement name -> (minimum version, why). _FLOORS = { - "cryptography": ("48.0.1", "GHSA-537c-gmf6-5ccf"), + "cryptography": ("50.0.0", "GHSA-537c-gmf6-5ccf, GHSA-g6cj-pr64-35w5 (PKCS#7 decryption oracle)"), "starlette": ("1.0.1", "CVE-2026-48710, Host-header path confusion (signaling server)"), "zeroconf": ("0.149.16", "CVE-2026-47180 and siblings, mDNS memory / CPU exhaustion (discovery)"), } diff --git a/uv.lock b/uv.lock index 706be3524..9b8ba4cef 100644 --- a/uv.lock +++ b/uv.lock @@ -451,7 +451,7 @@ wheels = [ [[package]] name = "je-auto-control" -version = "0.0.226" +version = "0.0.227" source = { editable = "." } dependencies = [ { name = "cryptography", marker = "platform_machine != 'ARM64' or sys_platform != 'win32'" }, @@ -509,7 +509,7 @@ requires-dist = [ { name = "av", marker = "extra == 'webrtc'", specifier = ">=14.0.0" }, { name = "babel", marker = "extra == 'locale'", specifier = ">=2.12" }, { name = "boto3", marker = "extra == 's3'", specifier = ">=1.34" }, - { name = "cryptography", marker = "platform_machine != 'ARM64' or sys_platform != 'win32'", specifier = ">=48.0.1" }, + { name = "cryptography", marker = "platform_machine != 'ARM64' or sys_platform != 'win32'", specifier = ">=50.0.0" }, { name = "defusedxml", specifier = "==0.7.1" }, { name = "fastapi", marker = "extra == 'signaling'", specifier = ">=0.115" }, { name = "je-open-cv", marker = "platform_machine != 'ARM64' or sys_platform != 'win32'", specifier = "==0.0.22" }, From 440f3fb59e7874b37e6162489b829c3e08183496 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 23:53:52 +0800 Subject: [PATCH 07/26] Refuse unlisted packages by default at the package gate The warning release has been out for two releases, so AC_add_package_to_executor and AC_add_package_to_callback_executor now refuse a package nobody allowed instead of importing it with a DeprecationWarning. Hosts with no Python of their own allow packages with JE_AUTOCONTROL_ALLOWED_PACKAGES (read where the shared manager is created, so every entry point honours it) or je_auto_control run --allow-package; executor.set_allow_arbitrary_packages(True) remains the explicit opt-out. --- docs/source/API/utils/package_manager.rst | 21 ++-- .../callback_function_doc.rst | 2 + docs/source/Eng/doc/cli/cli_doc.rst | 8 ++ .../keyword_and_executor_doc.rst | 25 +++-- .../callback_function_doc.rst | 2 + docs/source/Zh/doc/cli/cli_doc.rst | 14 +++ .../keyword_and_executor_doc.rst | 21 ++-- je_auto_control/cli.py | 19 ++++ .../utils/executor/action_executor.py | 6 +- .../package_manager/package_manager_class.py | 64 ++++++++---- test/unit_test/headless/test_package_gate.py | 98 ++++++++++++++++++- 11 files changed, 234 insertions(+), 46 deletions(-) diff --git a/docs/source/API/utils/package_manager.rst b/docs/source/API/utils/package_manager.rst index 1b211d46d..4a8e7681f 100644 --- a/docs/source/API/utils/package_manager.rst +++ b/docs/source/API/utils/package_manager.rst @@ -25,6 +25,8 @@ PackageManager ``event_dict``. :param str package: Package name to load. + :raises AutoControlExecuteActionException: the package gate refused the package + (see ``allow_packages``); nothing is imported. Functions are added with the naming convention ``package_function``. For example, ``time.sleep`` becomes ``time_sleep``. @@ -47,15 +49,19 @@ PackageManager .. method:: allow_packages(*packages) - Adds packages, and their submodules, to the package gate's allowlist. + Adds packages, and their submodules, to the package gate's allowlist. The allowlist starts + from the comma-separated names in the ``JE_AUTOCONTROL_ALLOWED_PACKAGES`` environment + variable, read when the manager is created; ``je_auto_control run --allow-package NAME`` + adds to it for one run. Raises ``AutoControlExecuteActionException`` for a name that is not + a dotted module name. .. method:: set_allow_arbitrary_packages(enabled) - Allows (``True``) or refuses (``False``) packages outside the allowlist. Until either switch - is called, any package loads with a ``DeprecationWarning``. ``add_package_to_executor`` and - ``add_package_to_callback_executor`` check the gate before importing and raise - ``AutoControlExecuteActionException`` for a refused package. The ``Executor`` has the same two - static methods; neither is an ``AC_*`` command. + Allows (``True``) or refuses (``False``, the default) packages outside the allowlist. + ``add_package_to_executor`` and ``add_package_to_callback_executor`` check the gate before + importing and raise ``AutoControlExecuteActionException`` for a refused package, so with + nothing configured no package loads. The ``Executor`` has the same two static methods; + neither is an ``AC_*`` command. .. method:: add_package_to_target(package, target) @@ -70,6 +76,9 @@ PackageManager from je_auto_control import package_manager + # Nothing loads until it is allowed (the package gate) + package_manager.allow_packages("os") + # Add 'os' module to the executor package_manager.add_package_to_executor("os") diff --git a/docs/source/Eng/doc/callback_function/callback_function_doc.rst b/docs/source/Eng/doc/callback_function/callback_function_doc.rst index 8572d9536..bff0da577 100644 --- a/docs/source/Eng/doc/callback_function/callback_function_doc.rst +++ b/docs/source/Eng/doc/callback_function/callback_function_doc.rst @@ -56,6 +56,8 @@ Load external package functions into the callback executor: from je_auto_control import package_manager + # The package gate refuses a package that has not been allowed + package_manager.allow_packages("time") # Add all functions from the 'time' module package_manager.add_package_to_callback_executor("time") diff --git a/docs/source/Eng/doc/cli/cli_doc.rst b/docs/source/Eng/doc/cli/cli_doc.rst index 49397d343..808f10132 100644 --- a/docs/source/Eng/doc/cli/cli_doc.rst +++ b/docs/source/Eng/doc/cli/cli_doc.rst @@ -21,11 +21,19 @@ Run a script python -m je_auto_control.cli run script.json python -m je_auto_control.cli run script.json --var count=10 --var name=alice python -m je_auto_control.cli run script.json --dry-run + python -m je_auto_control.cli run script.json --allow-package time --allow-package my_plugins ``--var name=value`` is parsed as JSON when the value parses, otherwise it is treated as a plain string. ``--dry-run`` records every action through the executor without invoking any side effects. +``--allow-package NAME`` (repeatable) puts a package, and its submodules, on +the package gate's allowlist for this run: ``AC_add_package_to_executor`` and +``AC_add_package_to_callback_executor`` refuse every package that has not +been allowed. The ``JE_AUTOCONTROL_ALLOWED_PACKAGES`` environment variable +(comma-separated names) does the same for every entry point, including the +legacy flags below, ``start-server``, ``start-rest`` and the MCP server. + List scheduler jobs ------------------- diff --git a/docs/source/Eng/doc/keyword_and_executor/keyword_and_executor_doc.rst b/docs/source/Eng/doc/keyword_and_executor/keyword_and_executor_doc.rst index 79d2ad399..b7c0660e0 100644 --- a/docs/source/Eng/doc/keyword_and_executor/keyword_and_executor_doc.rst +++ b/docs/source/Eng/doc/keyword_and_executor/keyword_and_executor_doc.rst @@ -83,20 +83,31 @@ Extending the Executor You can dynamically load external Python packages into the executor: -The package gate decides which packages may load. ``AC_add_package_to_executor`` can import ``os`` or -``subprocess`` for any action list, so the host program lists what it needs: +The package gate decides which packages may load. ``AC_add_package_to_executor`` could import ``os`` or +``subprocess`` for any action list, so **no package loads unless it has been allowed**; a package that is +not on the allowlist is refused before it is imported, and its action fails with +``AutoControlExecuteActionException``. A listed package also allows its submodules. There are three ways +to allow one: .. code-block:: python from je_auto_control import executor executor.allow_packages("time") # these, and their submodules - executor.set_allow_arbitrary_packages(False) # refuse everything else before importing it -Neither switch is an ``AC_*`` command, so an action list cannot open its own gate; a refused package fails -its action with ``AutoControlExecuteActionException``. Until the host calls either switch, any package still -loads but raises a ``DeprecationWarning``; a future release will refuse packages outside the allowlist by -default. +.. code-block:: bash + + # every entry point: both CLIs, the socket / REST / MCP servers, the scheduler + JE_AUTOCONTROL_ALLOWED_PACKAGES=time,my_plugins je_auto_control start-server + + # one run of the CLI; the flag may be repeated + je_auto_control run script.json --allow-package time --allow-package my_plugins + +``JE_AUTOCONTROL_ALLOWED_PACKAGES`` is a comma-separated list read once, when the process starts; an entry +that is not a dotted module name is ignored and logged. ``executor.set_allow_arbitrary_packages(True)`` +opens the gate for every package, which is what releases before this one did by default (they loaded any +package and raised a ``DeprecationWarning``). None of these is an ``AC_*`` command, so an action list +cannot open its own gate. .. code-block:: python diff --git a/docs/source/Zh/doc/callback_function/callback_function_doc.rst b/docs/source/Zh/doc/callback_function/callback_function_doc.rst index 947015700..587e0f61a 100644 --- a/docs/source/Zh/doc/callback_function/callback_function_doc.rst +++ b/docs/source/Zh/doc/callback_function/callback_function_doc.rst @@ -55,6 +55,8 @@ from je_auto_control import package_manager + # 套件閘門會拒絕沒有被放行的套件 + package_manager.allow_packages("time") # 載入 time 模組的所有函式 package_manager.add_package_to_callback_executor("time") diff --git a/docs/source/Zh/doc/cli/cli_doc.rst b/docs/source/Zh/doc/cli/cli_doc.rst index 3ba01d663..e76b14f70 100644 --- a/docs/source/Zh/doc/cli/cli_doc.rst +++ b/docs/source/Zh/doc/cli/cli_doc.rst @@ -33,6 +33,20 @@ AutoControl 可以直接從命令列執行自動化腳本。 python -m je_auto_control --execute_str '[["AC_screenshot", {"file_path": "test.png"}]]' +放行動作檔要載入的套件 +====================== + +``AC_add_package_to_executor`` 與 ``AC_add_package_to_callback_executor`` 會拒絕沒有被放行的套件。 +從命令列放行的方式有兩種：環境變數 ``JE_AUTOCONTROL_ALLOWED_PACKAGES``（以逗號分隔的套件名稱，含子模組）， +對上面的旗標、``start-server``、``start-rest`` 與 MCP server 等所有入口都適用；以及 ``je_auto_control run`` +的 ``--allow-package NAME``（可重複），只對該次執行有效。 + +.. code-block:: bash + + JE_AUTOCONTROL_ALLOWED_PACKAGES=time,my_plugins python -m je_auto_control -e "path/to/actions.json" + + python -m je_auto_control.cli run script.json --allow-package time --allow-package my_plugins + 建立專案範本 ============ diff --git a/docs/source/Zh/doc/keyword_and_executor/keyword_and_executor_doc.rst b/docs/source/Zh/doc/keyword_and_executor/keyword_and_executor_doc.rst index 9eb2bba3e..ac87b7f88 100644 --- a/docs/source/Zh/doc/keyword_and_executor/keyword_and_executor_doc.rst +++ b/docs/source/Zh/doc/keyword_and_executor/keyword_and_executor_doc.rst @@ -82,19 +82,28 @@ JSON 陣列（關鍵字），由執行者解析並執行。 你可以動態載入外部 Python 套件到執行者中： -哪些套件可以載入由套件閘門決定。``AC_add_package_to_executor`` 能替任何動作清單匯入 ``os`` 或 -``subprocess``，所以由宿主程式列出它需要的套件： +哪些套件可以載入由套件閘門決定。``AC_add_package_to_executor`` 原本能替任何動作清單匯入 ``os`` 或 +``subprocess``，所以 **沒有被放行的套件一律不載入**：不在允許清單上的套件在匯入前就被拒絕，該動作以 +``AutoControlExecuteActionException`` 失敗。列出一個套件也同時放行它的子模組。放行的方式有三種： .. code-block:: python from je_auto_control import executor executor.allow_packages("time") # 這些套件與其子模組 - executor.set_allow_arbitrary_packages(False) # 其他套件在匯入前就拒絕 -這兩個開關都不是 ``AC_*`` 命令，所以動作清單不能自己打開閘門；被拒絕的套件會讓該動作以 -``AutoControlExecuteActionException`` 失敗。宿主程式呼叫任一個開關之前，任何套件仍會載入，但會發出 -``DeprecationWarning``；之後的版本會預設拒絕允許清單以外的套件。 +.. code-block:: bash + + # 所有入口都適用：兩個 CLI、socket／REST／MCP server、排程器 + JE_AUTOCONTROL_ALLOWED_PACKAGES=time,my_plugins je_auto_control start-server + + # 只對 CLI 的這一次執行；旗標可以重複 + je_auto_control run script.json --allow-package time --allow-package my_plugins + +``JE_AUTOCONTROL_ALLOWED_PACKAGES`` 是以逗號分隔的清單，只在行程啟動時讀一次；不是模組名稱（以點分隔的識別字） +的項目會被略過並記錄到日誌。``executor.set_allow_arbitrary_packages(True)`` 會放行所有套件，也就是這一版之前的 +預設行為（當時任何套件都會載入，只發出 ``DeprecationWarning``）。以上都不是 ``AC_*`` 命令，所以動作清單不能 +自己打開閘門。 .. code-block:: python diff --git a/je_auto_control/cli.py b/je_auto_control/cli.py index dcc851a7b..ed6e99c65 100644 --- a/je_auto_control/cli.py +++ b/je_auto_control/cli.py @@ -6,6 +6,7 @@ Usage:: je_auto_control run script.json [--var x=10 --var y=20] [--dry-run] + [--allow-package NAME ...] je_auto_control validate script.json # alias: lint je_auto_control list-commands [--filter mouse] [--json] je_auto_control fmt script.json [--check] @@ -49,6 +50,19 @@ def _parse_vars(pairs: Optional[Sequence[str]]) -> Dict[str, object]: return resolved +def _allow_packages(names: Optional[Sequence[str]]) -> None: + """Put each ``--allow-package`` name on the package gate's allowlist.""" + if not names: + return + from je_auto_control.utils.package_manager.package_manager_class import ( + is_package_name, package_manager, + ) + for name in names: + if not is_package_name(name): + raise SystemExit(f"--allow-package must be a package name; got {name!r}") + package_manager.allow_packages(*names) + + def cmd_run(args: argparse.Namespace) -> int: from je_auto_control.utils.executor.action_executor import ( execute_action, execute_action_with_vars, recorded_failures, @@ -57,6 +71,7 @@ def cmd_run(args: argparse.Namespace) -> int: from je_auto_control.utils.json.json_file import read_executable_action_json actions = read_executable_action_json(args.script) variables = _parse_vars(args.var) + _allow_packages(args.allow_package) reset_recorded_failures() if args.dry_run: from je_auto_control.utils.executor.action_executor import executor @@ -254,6 +269,10 @@ def build_parser() -> argparse.ArgumentParser: help="name=value override; may be repeated") p_run.add_argument("--dry-run", action="store_true", help="record actions without calling them") + p_run.add_argument("--allow-package", action="append", metavar="NAME", + help="package AC_add_package_to_executor may load (submodules " + "included); may be repeated. JE_AUTOCONTROL_ALLOWED_PACKAGES " + "lists them for every entry point") p_run.set_defaults(func=cmd_run) for name in ("validate", "lint"): diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index b9a7bf614..73741b222 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -8058,10 +8058,10 @@ def __init__(self): @staticmethod def set_allow_arbitrary_packages(enabled: bool) -> None: """ - Allow (True) or refuse (False) ``AC_add_package_to_executor`` / + Allow (True) or refuse (False, the default) ``AC_add_package_to_executor`` / ``AC_add_package_to_callback_executor`` for packages outside the allowlist. Python only, - never an action command, so an action list cannot open its own gate. Until it is called, - any package loads with a ``DeprecationWarning``. + never an action command, so an action list cannot open its own gate. The allowlist is + ``allow_packages`` plus the names in ``JE_AUTOCONTROL_ALLOWED_PACKAGES``. """ package_manager.set_allow_arbitrary_packages(enabled) diff --git a/je_auto_control/utils/package_manager/package_manager_class.py b/je_auto_control/utils/package_manager/package_manager_class.py index 8c64fd375..ffdbbe662 100644 --- a/je_auto_control/utils/package_manager/package_manager_class.py +++ b/je_auto_control/utils/package_manager/package_manager_class.py @@ -1,6 +1,6 @@ import importlib +import os import re -import warnings from importlib.util import find_spec from inspect import getmembers, isfunction, isbuiltin, isclass from types import ModuleType @@ -10,8 +10,31 @@ from je_auto_control.utils.logging.logging_instance import autocontrol_logger _PACKAGE_NAME_RE = re.compile(r"^[A-Za-z_]\w*(\.[A-Za-z_]\w*)*$") -# warnings.warn -> _check_allowed -> add_package_to_* -> the command's caller -_GATE_WARNING_STACKLEVEL = 3 + +#: Comma-separated package names the gate allows, for hosts with no Python of +#: their own to call ``executor.allow_packages``: the CLIs, the socket / REST / +#: MCP servers and the scheduler. Read once, when a manager is created. +ALLOWED_PACKAGES_ENV = "JE_AUTOCONTROL_ALLOWED_PACKAGES" + + +def is_package_name(name: object) -> bool: + """Whether ``name`` is a dotted Python module name, the only thing the gate lists.""" + return isinstance(name, str) and bool(_PACKAGE_NAME_RE.match(name)) + + +def _packages_from_environment() -> set[str]: + """The names in ``JE_AUTOCONTROL_ALLOWED_PACKAGES``; what is not a module name is skipped.""" + allowed: set[str] = set() + for raw in os.environ.get(ALLOWED_PACKAGES_ENV, "").split(","): + name = raw.strip() + if not name: + continue + if is_package_name(name): + allowed.add(name) + else: + # Importing the package must not fail on a typo in the environment. + autocontrol_logger.error("%s: ignored %r, not a package name", ALLOWED_PACKAGES_ENV, name) + return allowed class PackageManager: @@ -26,15 +49,15 @@ def __init__(self): self.installed_package_dict: dict[str, ModuleType] = {} self.executor = None self.callback_executor = None - # Package gate (workspace X-12). None = not configured: any package still loads, with a - # DeprecationWarning. False = only ``allowed_packages``; True = any package, silently. - self.allow_arbitrary_packages: Optional[bool] = None - self.allowed_packages: set[str] = set() + # Package gate (workspace X-12). False (the default) = only ``allowed_packages``; + # True = any package. The allowlist starts from JE_AUTOCONTROL_ALLOWED_PACKAGES. + self.allow_arbitrary_packages: bool = False + self.allowed_packages: set[str] = _packages_from_environment() def set_allow_arbitrary_packages(self, enabled: bool) -> None: """ 設定是否允許載入允許清單以外的套件 - Allow (True) or refuse (False) packages outside :attr:`allowed_packages`. + Allow (True) or refuse (False, the default) packages outside :attr:`allowed_packages`. Deliberately not an ``AC_*`` command: an action list must not open its own gate. """ self.allow_arbitrary_packages = bool(enabled) @@ -43,7 +66,13 @@ def allow_packages(self, *packages: str) -> None: """ 把套件加入允許清單（連同其子模組） Add packages to the allowlist; a listed package also allows its submodules. + + :raises AutoControlExecuteActionException: a name is not a dotted module name (none is added) """ + for package in packages: + if not is_package_name(package): + raise AutoControlExecuteActionException( + f"cannot allow {package!r}: not a package name") self.allowed_packages.update(packages) def _is_allowlisted(self, package: str) -> bool: @@ -54,19 +83,14 @@ def _check_allowed(self, package: object) -> None: """Refuse ``package`` before it is imported, unless the gate lets it through.""" if isinstance(package, str) and self._is_allowlisted(package): return - if self.allow_arbitrary_packages is True: + if self.allow_arbitrary_packages: return - if self.allow_arbitrary_packages is False: - raise AutoControlExecuteActionException( - f"package {package!r} is not allowed; the host must call " - "executor.allow_packages(...) or executor.set_allow_arbitrary_packages(True)" - ) - warnings.warn( - f"loading package {package!r} that is not on the allowlist; a future release will refuse " - "it by default. Call executor.allow_packages(...) for the packages you load, or " - "executor.set_allow_arbitrary_packages(True) to keep loading any package.", - DeprecationWarning, - stacklevel=_GATE_WARNING_STACKLEVEL, + raise AutoControlExecuteActionException( + f"package {package!r} is not allowed. To allow it: list it in the " + f"{ALLOWED_PACKAGES_ENV} environment variable (comma-separated), pass " + "--allow-package NAME to `je_auto_control run`, or call " + "executor.allow_packages(...) from Python; " + "executor.set_allow_arbitrary_packages(True) allows every package." ) def check_package(self, package: str) -> Optional[ModuleType]: diff --git a/test/unit_test/headless/test_package_gate.py b/test/unit_test/headless/test_package_gate.py index cf0600049..974d7b75a 100644 --- a/test/unit_test/headless/test_package_gate.py +++ b/test/unit_test/headless/test_package_gate.py @@ -1,4 +1,8 @@ """The package gate in front of AC_add_package_to_executor (workspace X-12).""" +import json +import os +import subprocess # nosec B404 # reason: runs this interpreter on a fixed -c snippet +import sys import types import warnings from unittest.mock import patch @@ -25,11 +29,95 @@ def shared_gate(): manager.allow_arbitrary_packages, manager.allowed_packages = saved[0], set(saved[1]) -def test_unconfigured_gate_still_loads_but_warns(): +ENV = "JE_AUTOCONTROL_ALLOWED_PACKAGES" + + +@pytest.fixture(autouse=True) +def _no_inherited_allowlist(monkeypatch): + monkeypatch.delenv(ENV, raising=False) + + +def test_unconfigured_gate_refuses_before_importing(): manager = _manager() - with pytest.warns(DeprecationWarning, match="not on the allowlist"): - manager.add_package_to_executor("json") + assert manager.allow_arbitrary_packages is False + with patch.object(package_manager_class.importlib, "import_module") as importer: + with warnings.catch_warnings(): + warnings.simplefilter("error") + with pytest.raises(AutoControlExecuteActionException) as refused: + manager.add_package_to_executor("json") + importer.assert_not_called() + assert manager.executor.event_dict == {} + # The message names every way to allow the package. + for way in (ENV, "--allow-package", "executor.allow_packages", + "executor.set_allow_arbitrary_packages(True)"): + assert way in str(refused.value) + + +def test_environment_variable_allowlists_packages_and_submodules(monkeypatch): + assert package_manager_class.ALLOWED_PACKAGES_ENV == ENV + monkeypatch.setenv(ENV, " json , ,collections.abc,not a name,") + manager = _manager() + assert manager.allowed_packages == {"json", "collections.abc"} + assert manager.allow_arbitrary_packages is False + manager.add_package_to_executor("json") + manager.add_package_to_executor("json.decoder") assert "json_dumps" in manager.executor.event_dict + assert "json.decoder_JSONDecoder" in manager.executor.event_dict + with pytest.raises(AutoControlExecuteActionException): + manager.add_package_to_executor("collections") + with pytest.raises(AutoControlExecuteActionException): + manager.add_package_to_executor("os") + + +def test_environment_variable_reaches_the_gate_every_entry_point_shares(tmp_path): + """The executor and the callback executor register this one instance's methods.""" + from je_auto_control.utils.callback.callback_function_executor import callback_executor + shared = package_manager_class.package_manager + for table in (executor.event_dict, callback_executor.event_dict): + for name in ("AC_add_package_to_executor", "AC_add_package_to_callback_executor"): + assert table[name].__self__ is shared + code = ("import json; from je_auto_control.utils.package_manager.package_manager_class " + "import package_manager as m; " + "print(json.dumps([sorted(m.allowed_packages), m.allow_arbitrary_packages]))") + env = dict(os.environ, **{ENV: "time,my_plugins"}) + # The child imports the tree under test, wherever pytest was started from. + tree = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(package_manager_class.__file__))))) + env["PYTHONPATH"] = os.pathsep.join(filter(None, [tree, env.get("PYTHONPATH")])) + done = subprocess.run( # nosec B603 # reason: fixed argv, this interpreter + [sys.executable, "-c", code], env=env, cwd=tmp_path, + capture_output=True, text=True, timeout=120, check=True) + assert json.loads(done.stdout.strip().splitlines()[-1]) == [["my_plugins", "time"], False] + + +def test_allow_packages_rejects_what_is_not_a_package_name(): + manager = _manager() + for bad in ("", "os; rm", "a..b", 3): + with pytest.raises(AutoControlExecuteActionException): + manager.allow_packages("json", bad) + assert manager.allowed_packages == set() + + +def test_cli_run_allow_package_opens_the_gate_for_those_names(shared_gate, tmp_path, capsys): + from je_auto_control.cli import main + script = tmp_path / "s.json" + script.write_text(json.dumps([["AC_add_package_to_executor", {"package": "json"}]]), + encoding="utf-8") + executor.event_dict.pop("json_dumps", None) + try: + assert main(["run", str(script)]) == 1 + assert "--allow-package" in capsys.readouterr().out + assert "json_dumps" not in executor.event_dict + assert main(["run", str(script), "--allow-package", "json", + "--allow-package", "collections"]) == 0 + assert "json_dumps" in executor.event_dict + assert {"json", "collections"} <= shared_gate.allowed_packages + assert shared_gate.allow_arbitrary_packages is False + with pytest.raises(SystemExit): + main(["run", str(script), "--allow-package", "os;rm"]) + finally: + for name in [name for name in executor.event_dict if name.startswith("json_")]: + del executor.event_dict[name] def test_closed_gate_refuses_before_importing(): @@ -74,6 +162,9 @@ def test_open_gate_loads_anything_without_warning(): def test_executor_configures_the_shared_gate(shared_gate): + assert shared_gate.allow_arbitrary_packages is False # the default + executor.set_allow_arbitrary_packages(True) # the explicit opt-out + assert shared_gate.allow_arbitrary_packages is True executor.set_allow_arbitrary_packages(False) executor.allow_packages("my_company_helpers") assert shared_gate.allow_arbitrary_packages is False @@ -87,7 +178,6 @@ def test_no_action_command_can_open_the_gate(): def test_refusal_reaches_the_action_record(shared_gate): - executor.set_allow_arbitrary_packages(False) record = executor.execute_action([["AC_add_package_to_executor", {"package": "os"}]]) assert any("not allowed" in str(value) for value in record.values()) assert "os_system" not in executor.event_dict From cb7e57b36f737fe96b18688a8db69db283f6cbf7 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 23:56:19 +0800 Subject: [PATCH 08/26] Give every top-level run its own variable scope execute_action_with_vars seeded the module executor and never cleared it, and REST, MCP, the socket server, the scheduler, triggers, hotkeys, ChatOps and voice commands all ran on that executor, so one caller's ${user} resolved in the next caller's run instead of failing as unknown. execution_scope() binds a fresh VariableScope to the module executor for one run (contextvars, restored in finally); those entry points open one. Direct executor.execute_action() calls keep the process scope, which the GUI's Variables tab relies on. AC_run_dag ran its nodes on the module executor from pool threads, so inside an AC_parallel branch they read the wrong scope; the runner now captures the calling executor and scope. A parallel branch starts from a copy of its parent's variables instead of an empty scope. --- .../Eng/doc/new_features/new_features_doc.rst | 36 +++ .../Eng/doc/new_features/v4_features_doc.rst | 2 + .../Zh/doc/new_features/new_features_doc.rst | 33 +++ .../Zh/doc/new_features/v4_features_doc.rst | 2 +- je_auto_control/__init__.py | 3 +- je_auto_control/utils/chatops/handlers.py | 4 +- je_auto_control/utils/dag/runner.py | 31 +- .../utils/executor/action_executor.py | 40 ++- .../utils/executor/flow_control.py | 11 +- .../utils/mcp_server/tools/_base.py | 13 +- .../utils/rest_api/rest_handlers.py | 10 +- .../utils/run_history/run_outcome.py | 9 +- je_auto_control/utils/script_vars/__init__.py | 5 +- .../utils/script_vars/execution.py | 62 ++++ je_auto_control/utils/script_vars/scope.py | 8 +- .../auto_control_socket_server.py | 5 +- je_auto_control/utils/voice/voice_router.py | 4 +- .../test_execution_scope_isolation.py | 278 ++++++++++++++++++ .../headless/test_r3_executor_containment.py | 10 +- 19 files changed, 538 insertions(+), 28 deletions(-) create mode 100644 je_auto_control/utils/script_vars/execution.py create mode 100644 test/unit_test/headless/test_execution_scope_isolation.py diff --git a/docs/source/Eng/doc/new_features/new_features_doc.rst b/docs/source/Eng/doc/new_features/new_features_doc.rst index 08cd73f77..957bf8b2e 100644 --- a/docs/source/Eng/doc/new_features/new_features_doc.rst +++ b/docs/source/Eng/doc/new_features/new_features_doc.rst @@ -367,6 +367,42 @@ so nested ``body`` / ``then`` / ``else`` lists keep their placeholders and re-bind each time they execute — letting ``AC_for_each`` iterate over a list while the body sees the current item. +Variable scope per run +---------------------- + +Variables live for one top-level run, not for the life of the process. +``execute_action_with_vars``, a REST ``/execute`` or ``/execute_file`` +request, an MCP tool call, a socket-server command, a scheduler job, a +trigger / hotkey / webhook / e-mail firing, a ChatOps ``/run`` and a voice +command each open a fresh ``VariableScope``. ``AC_set_var``, loop variables +and macro parameters work as before inside that run and are gone when it +ends, so a later run's ``${user}`` fails with ``Unknown variable`` instead of +reading what an earlier caller set. To carry a value from one run to the +next, pass it in again (``execute_action_with_vars(actions, variables)``, a +webhook payload) or keep it outside the scope (a file, the secrets vault). + +Plain Python calls are unchanged: ``executor.execute_action(...)`` and the +module-level ``execute_action(...)`` use the module executor's own scope, +which lasts as long as the process -- this is the scope the GUI's Variables +tab shows and scripts run from the GUI use. Wrap such calls in +``execution_scope`` to isolate them the same way:: + + import je_auto_control as ac + + with ac.execution_scope({"user": "alice"}) as scope: + ac.execute_action([["AC_set_var", {"name": "n", "value": 1}]]) + ac.execute_action([["AC_inc_var", {"name": "n"}]]) + scope.get_value("n") # 2 + # "user" and "n" are gone here + +The binding is per thread: concurrent server requests never see each other's +variables, and the previous scope is restored when the block ends, error or +not. Nested action lists (``AC_circuit_call``, ``AC_bulkhead_run``, +``AC_run_chaos``, ``AC_run_dag`` and the like) run in the scope of the list +that called them, also inside an ``AC_parallel`` branch; a branch starts from +a copy of its parent's variables and its own writes stay in the branch. +An executor you construct yourself (``Executor()``) always owns its scope. + :: import je_auto_control as ac diff --git a/docs/source/Eng/doc/new_features/v4_features_doc.rst b/docs/source/Eng/doc/new_features/v4_features_doc.rst index 378ba0571..e54c56bae 100644 --- a/docs/source/Eng/doc/new_features/v4_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v4_features_doc.rst @@ -68,6 +68,8 @@ Flow control & variables * **In-process parallel** — ``AC_parallel`` runs branch action lists concurrently, each on a fresh isolated executor so branches never race on shared variables (the in-process complement to the cross-host DAG). + A branch starts from a copy of the caller's variables; what it sets stays + in that branch. * **Performance-budget assertion** — ``assert_duration(action, max_ms)`` / ``AC_assert_duration`` fails a block that takes longer than the budget — a latency-regression guard bridging the profiler and the diff --git a/docs/source/Zh/doc/new_features/new_features_doc.rst b/docs/source/Zh/doc/new_features/new_features_doc.rst index 0961cce4a..5abbecdc9 100644 --- a/docs/source/Zh/doc/new_features/new_features_doc.rst +++ b/docs/source/Zh/doc/new_features/new_features_doc.rst @@ -343,6 +343,39 @@ executor 現在改成「每次呼叫」才解析 ``${var}`` placeholder（不會 placeholder，每次重複執行時重新繫結 — 因此 ``AC_for_each`` 走訪 list 時，body 內看到的就是當前的元素。 +每次執行各自的變數範圍 +---------------------- + +變數的壽命是一次頂層執行，而不是整個行程。``execute_action_with_vars``、 +REST 的 ``/execute`` 與 ``/execute_file`` 請求、一次 MCP 工具呼叫、socket +server 的一則指令、排程工作、trigger / hotkey / webhook / e-mail 的一次觸發、 +ChatOps 的 ``/run`` 以及語音指令，各自開一個新的 ``VariableScope``。 +``AC_set_var``、迴圈變數與巨集參數在該次執行內照舊運作，執行結束即消失， +所以下一次執行裡的 ``${user}`` 會以 ``Unknown variable`` 失敗，而不是讀到 +前一個呼叫者設的值。要把值帶到下一次執行，請重新傳入 +（``execute_action_with_vars(actions, variables)``、webhook payload）， +或存在範圍之外（檔案、secrets vault）。 + +一般的 Python 呼叫不變：``executor.execute_action(...)`` 與模組層級的 +``execute_action(...)`` 使用模組 executor 自己的範圍，與行程同壽命——GUI 的 +Variables 分頁顯示的、以及從 GUI 執行的腳本用的就是這個範圍。要讓這類呼叫 +同樣隔離，包在 ``execution_scope`` 裡:: + + import je_auto_control as ac + + with ac.execution_scope({"user": "alice"}) as scope: + ac.execute_action([["AC_set_var", {"name": "n", "value": 1}]]) + ac.execute_action([["AC_inc_var", {"name": "n"}]]) + scope.get_value("n") # 2 + # 離開後 "user" 與 "n" 都不存在 + +這個綁定以執行緒為單位：同時進行的伺服器請求互相看不到對方的變數，區塊結束時 +（不論是否出錯）會還原先前的範圍。巢狀動作清單（``AC_circuit_call``、 +``AC_bulkhead_run``、``AC_run_chaos``、``AC_run_dag`` 等）跑在呼叫它的清單的 +範圍裡，在 ``AC_parallel`` 分支內也一樣；分支從父層變數的一份複本開始， +自己寫入的值留在分支內。自行建立的 executor（``Executor()``）永遠擁有自己的範圍。 + + :: import je_auto_control as ac diff --git a/docs/source/Zh/doc/new_features/v4_features_doc.rst b/docs/source/Zh/doc/new_features/v4_features_doc.rst index 2a010e192..857aad962 100644 --- a/docs/source/Zh/doc/new_features/v4_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v4_features_doc.rst @@ -59,7 +59,7 @@ Builder 項目。視覺與視窗功能的 geometry / IO 操作皆可注入,因 呼叫端同名的變數會恢復原值。 * **同進程平行** — ``AC_parallel`` 讓多個分支動作清單並行執行,各自在 獨立的全新 executor 上,因此分支不會在共享變數上互相 race(跨主機 DAG - 的同進程版)。 + 的同進程版)。分支從呼叫端變數的一份複本開始;分支內設定的值只留在該分支。 * **效能預算斷言** — ``assert_duration(action, max_ms)`` / ``AC_assert_duration`` 在區塊耗時超過預算時判失敗——銜接 profiler 與 斷言 DSL 的延遲回歸守門。 diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 0117874ed..a14962684 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -1170,6 +1170,7 @@ ScheduledJob, Scheduler, default_scheduler, ) # Script variables (headless) +from je_auto_control.utils.script_vars.execution import execution_scope from je_auto_control.utils.script_vars.interpolate import ( interpolate_actions, interpolate_value, load_vars_from_json, ) @@ -1332,7 +1333,7 @@ def start_autocontrol_gui(*args, **kwargs): "Scheduler", "ScheduledJob", "default_scheduler", # Script variables "interpolate_actions", "interpolate_value", "load_vars_from_json", - "VariableScope", + "VariableScope", "execution_scope", # Watchers "MouseWatcher", "PixelWatcher", "LogTail", # Window manager diff --git a/je_auto_control/utils/chatops/handlers.py b/je_auto_control/utils/chatops/handlers.py index 357190c13..b918a6d49 100644 --- a/je_auto_control/utils/chatops/handlers.py +++ b/je_auto_control/utils/chatops/handlers.py @@ -59,7 +59,9 @@ def cmd_run(argv: List[str], context: Dict[str, Any]) -> CommandResult: root = _require_script_root(context) script_path = _resolve_script(root, argv[0]) from je_auto_control.utils.executor.action_executor import execute_files - result = execute_files([str(script_path)]) + from je_auto_control.utils.script_vars.execution import execution_scope + with execution_scope(): # one /run, one variable scope + result = execute_files([str(script_path)]) return CommandResult( text=f"ran {script_path.name}: {len(result)} action(s) executed", metadata={"script": str(script_path), "results": _safe(result)}, diff --git a/je_auto_control/utils/dag/runner.py b/je_auto_control/utils/dag/runner.py index 680a676cf..48443b76f 100644 --- a/je_auto_control/utils/dag/runner.py +++ b/je_auto_control/utils/dag/runner.py @@ -92,7 +92,7 @@ def run_dag(definition: Any, and every pending node is ``skipped`` with the error ``"stopped"``. """ dag = _coerce_definition(definition) - local = local_runner or _default_local_runner + local = local_runner or _current_run_local_runner() remote = remote_runner or _default_remote_runner if stop_event is not None: local = _stoppable(local, stop_event) @@ -291,15 +291,38 @@ def _ancestor_index(dag: DagDefinition) -> Dict[str, Set[str]]: return ancestors -def _default_local_runner(node: DagNode, _definition: DagDefinition) -> Any: - from je_auto_control.utils.executor.action_executor import executor +def _current_run_local_runner() -> NodeRunner: + """A local runner bound to the run that called :func:`run_dag`. + + Nodes run on pool threads, where neither the running executor nor the + run's variable scope is visible: a DAG started inside an ``AC_parallel`` + branch (or an isolated top-level run) ran its nodes on the module + executor's process scope, so ``${var}`` read some other run's value. + Both are captured here, on the calling thread. + """ + from je_auto_control.utils.executor.action_executor import _running_executor + from je_auto_control.utils.script_vars.execution import bound_scope + runner = _running_executor() + scope = runner.variables + + def run(node: DagNode, definition: DagDefinition) -> Any: + with bound_scope(scope): + return _default_local_runner(node, definition, runner) + return run + + +def _default_local_runner(node: DagNode, _definition: DagDefinition, + runner: Any = None) -> Any: + """Run a local node's actions on ``runner`` (default: the module executor).""" from je_auto_control.utils.json.json_file import read_executable_action_json + if runner is None: + from je_auto_control.utils.executor.action_executor import executor as runner # raise_on_error=True: by default a failed action is only recorded, so # a node whose actions all failed counted as succeeded and its # dependants ran anyway. actions = (list(node.actions) if node.actions is not None else read_executable_action_json(str(node.action_file))) - return executor.execute_action(actions, raise_on_error=True) + return runner.execute_action(actions, raise_on_error=True) def _default_remote_runner(node: DagNode, diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index b9a7bf614..531621bd3 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -60,6 +60,7 @@ from je_auto_control.utils.script_vars.interpolate import ( interpolate_value, ) +from je_auto_control.utils.script_vars.execution import current_scope, execution_scope from je_auto_control.utils.script_vars.scope import VariableScope from je_auto_control.utils.http_client.http_client import http_request from je_auto_control.utils.generate_report.generate_html_report import generate_html, generate_html_report @@ -7161,6 +7162,30 @@ class Executor: "AC_admin_broadcast_execute")}, } + #: Whether :attr:`variables` follows the current ``execution_scope``. Only + #: the module-level executor does: it is the one object every server + #: request shares. A private Executor (an AC_parallel branch, a device + #: matrix runner, a resumable run) owns its scope outright. + _run_scoped = False + + @property + def variables(self) -> VariableScope: + """The scope ``${var}`` and the variable commands read and write. + + On the module executor inside an ``execution_scope`` that is the + run's own scope; otherwise it is this executor's own, which lives as + long as the executor does. + """ + if self._run_scoped: + scope = current_scope() + if scope is not None: + return scope + return self._own_variables + + @variables.setter + def variables(self, scope: VariableScope) -> None: + self._own_variables = scope + def __init__(self): self._block_commands = BLOCK_COMMANDS self.variables = VariableScope() @@ -8345,6 +8370,7 @@ def _count_recorded_failure() -> None: # === 全域 Executor 實例 Global Executor Instance === executor = Executor() +executor._run_scoped = True # the one executor every entry point shares package_manager.executor = executor @@ -8372,7 +8398,15 @@ def execute_files(execute_files_list: list) -> List[Dict[str, str]]: def execute_action_with_vars(action_list: list, variables: dict ) -> Dict[str, str]: - """Seed ``variables`` into the runtime scope and execute. + """Run ``action_list`` in a fresh variable scope seeded with ``variables``. + + The scope belongs to this run: ``AC_set_var``, loop variables and macro + parameters work as usual inside it and are gone when it returns, so the + next run's ``${name}`` fails with ``Unknown variable`` instead of quietly + reading this caller's value. Plain ``executor.execute_action(...)`` keeps + the executor's process-lifetime scope; wrap it in + :func:`~je_auto_control.utils.script_vars.execution.execution_scope` to + isolate it the same way. Interpolation happens at dispatch time through the executor's runtime resolver, which defers nested action bodies (loops/branches/try). Doing a @@ -8383,5 +8417,5 @@ def execute_action_with_vars(action_list: list, variables: dict landed in logs and record keys. Seeding the scope and letting the runtime resolver interpolate per action fixes both. """ - executor.variables.update_many(variables) - return executor.execute_action(action_list) + with execution_scope(variables): + return executor.execute_action(action_list) diff --git a/je_auto_control/utils/executor/flow_control.py b/je_auto_control/utils/executor/flow_control.py index 9bc76e463..8de99111b 100644 --- a/je_auto_control/utils/executor/flow_control.py +++ b/je_auto_control/utils/executor/flow_control.py @@ -459,6 +459,10 @@ def __init__(self, executor: Any, branches: list) -> None: # parent's live command/macro maps concurrently. self._event_dict = dict(executor.event_dict) self._macros = dict(executor.macros) + # Each branch starts from what the parent could see when AC_parallel + # ran -- a branch used to start empty, so ${var} set before the + # block was unknown inside it -- and writes to its own copy. + self._variables = executor.variables.as_dict() self._strict = getattr(executor_module._STRICT_BODIES, "value", False) self._macro_depth = getattr(_MACRO_DEPTH, "value", 0) self.results: list = [None] * len(branches) @@ -477,6 +481,7 @@ def _branch_executor(self) -> Any: for name, handler in self._event_dict.items(): branch_executor.event_dict.setdefault(name, handler) branch_executor.macros.update(self._macros) + branch_executor.variables.update_many(self._variables) return branch_executor def run_branch(self, index: int, branch: Any) -> None: @@ -519,8 +524,10 @@ def exec_parallel(executor: Any, args: Mapping[str, Any]) -> Dict[str, Any]: """Run each branch action list concurrently on its own isolated executor. ``branches`` is a list of action lists (or a JSON string of one). Each - branch runs on a fresh :class:`Executor` with a separate variable scope, - so concurrent branches never race on shared state. Custom commands + branch runs on a fresh :class:`Executor` whose variable scope is a fork of + the caller's: it starts with the variables visible when ``AC_parallel`` + ran, and what a branch sets stays in that branch, so concurrent branches + never race on shared state and nothing flows back. Custom commands (registered via ``add_command_to_executor``) and ``AC_define_macro`` macros are copied from the parent so a branch recognises them — otherwise a branch's fresh executor only has the stock command set and rejects them diff --git a/je_auto_control/utils/mcp_server/tools/_base.py b/je_auto_control/utils/mcp_server/tools/_base.py index e4b836821..7bb26ed63 100644 --- a/je_auto_control/utils/mcp_server/tools/_base.py +++ b/je_auto_control/utils/mcp_server/tools/_base.py @@ -108,10 +108,17 @@ def invoke(self, arguments: Dict[str, Any], ctx: Any = None) -> Any: for the active call, which lets them report progress and observe cooperative cancellation. Handlers that do not declare ``ctx`` see the original behaviour unchanged. + + Every call runs in its own variable scope: a tool that executes + actions (``ac_execute_actions``, the circuit / bulkhead / saga + tools) used to leave its variables in the module executor for the + next call, possibly another client's. """ - if ctx is not None and _handler_accepts_ctx(self.handler): - return self.handler(ctx=ctx, **arguments) - return self.handler(**arguments) + from je_auto_control.utils.script_vars.execution import execution_scope + with execution_scope(): + if ctx is not None and _handler_accepts_ctx(self.handler): + return self.handler(ctx=ctx, **arguments) + return self.handler(**arguments) @lru_cache(maxsize=512) diff --git a/je_auto_control/utils/rest_api/rest_handlers.py b/je_auto_control/utils/rest_api/rest_handlers.py index 055d81069..2623f1cee 100644 --- a/je_auto_control/utils/rest_api/rest_handlers.py +++ b/je_auto_control/utils/rest_api/rest_handlers.py @@ -18,6 +18,7 @@ AutoControlException, AutoControlJsonActionException, ) from je_auto_control.utils.logging.logging_instance import autocontrol_logger +from je_auto_control.utils.script_vars.execution import execution_scope @dataclass @@ -174,7 +175,8 @@ def _execute_strict(actions: Any) -> HandlerResult: """ from je_auto_control.utils.executor.action_executor import executor try: - result = executor.execute_action(actions, raise_on_error=True) + with execution_scope(): + result = executor.execute_action(actions, raise_on_error=True) except Exception as error: # noqa: BLE001 # pylint: disable=broad-except # reason: REST boundary; the failure is the response autocontrol_logger.info("rest execute stopped on a failed action: %r", error) return 200, {"ok": False, "error": f"{type(error).__name__}: {error}"} @@ -198,7 +200,8 @@ def handle_execute(ctx: RouteContext) -> HandlerResult: return _execute_strict(actions) try: from je_auto_control.utils.executor.action_executor import execute_action - result = execute_action(actions) + with execution_scope(): # one request, one variable scope + result = execute_action(actions) except Exception as error: # noqa: BLE001 # pylint: disable=broad-except # reason: REST boundary must always return JSON, never drop the HTTP response autocontrol_logger.error("rest execute failed: %r", error) return 500, {"error": "execute_action failed"} @@ -213,7 +216,8 @@ def handle_execute_file(ctx: RouteContext) -> HandlerResult: return 400, {"error": "missing 'path' field"} try: from je_auto_control.utils.executor.action_executor import execute_files - result = execute_files([path]) + with execution_scope(): + result = execute_files([path]) except (AutoControlActionException, AutoControlActionNullException, AutoControlJsonActionException) as error: # The caller chose the path, so an unreadable file or an action list diff --git a/je_auto_control/utils/run_history/run_outcome.py b/je_auto_control/utils/run_history/run_outcome.py index d36e326c2..e0872acd5 100644 --- a/je_auto_control/utils/run_history/run_outcome.py +++ b/je_auto_control/utils/run_history/run_outcome.py @@ -6,6 +6,9 @@ e-mail triggers recorded such runs as succeeded and took no error snapshot. They run their actions through :func:`run_counting_failures`, which reads the per-thread failure count ``je_auto_control run`` already uses for its exit code. + +Each such run also gets its own variable scope: a job's ``AC_set_var`` used to +stay in the module executor for whichever job, trigger or hotkey fired next. """ from typing import Any, Callable @@ -13,12 +16,14 @@ def run_counting_failures(run: Callable[[], Any]) -> Any: - """Call ``run()``; raise if an action it executed was recorded as failed.""" + """Call ``run()`` in a fresh variable scope; raise if an action it ran failed.""" from je_auto_control.utils.executor.action_executor import ( recorded_failures, reset_recorded_failures, ) + from je_auto_control.utils.script_vars.execution import execution_scope reset_recorded_failures() - result = run() + with execution_scope(): + result = run() failures = recorded_failures() if failures: raise AutoControlActionException( diff --git a/je_auto_control/utils/script_vars/__init__.py b/je_auto_control/utils/script_vars/__init__.py index 9ac72755b..b29256480 100644 --- a/je_auto_control/utils/script_vars/__init__.py +++ b/je_auto_control/utils/script_vars/__init__.py @@ -1,10 +1,11 @@ """Variable interpolation for action JSON scripts.""" +from je_auto_control.utils.script_vars.execution import execution_scope from je_auto_control.utils.script_vars.interpolate import ( interpolate_actions, interpolate_value, load_vars_from_json, ) from je_auto_control.utils.script_vars.scope import VariableScope __all__ = [ - "VariableScope", "interpolate_actions", "interpolate_value", - "load_vars_from_json", + "VariableScope", "execution_scope", "interpolate_actions", + "interpolate_value", "load_vars_from_json", ] diff --git a/je_auto_control/utils/script_vars/execution.py b/je_auto_control/utils/script_vars/execution.py new file mode 100644 index 000000000..a43df3a5a --- /dev/null +++ b/je_auto_control/utils/script_vars/execution.py @@ -0,0 +1,62 @@ +"""The variable scope of the current top-level run. + +The module-level executor is one object per process, and so was its +:class:`VariableScope`: what one REST, MCP or socket request (or one +``execute_action_with_vars`` call) set stayed there for the next, whose +``${user}`` then resolved to the previous caller's value instead of failing +with ``Unknown variable``. + +:func:`execution_scope` opens a fresh scope for one run and makes it the +module executor's scope for as long as the ``with`` block lasts. The binding +is a :class:`contextvars.ContextVar`, so it is per thread (and per asyncio +task): two server threads running at once each see their own scope, and a +thread started inside the block does not inherit it. Code that hands work to +another thread on behalf of the run -- the DAG runner's pool -- captures the +scope first and re-binds it there with :func:`bound_scope`. + +Outside any ``execution_scope`` the module executor falls back to its own +process-lifetime scope, which is what plain ``executor.execute_action(...)`` +calls and the GUI's Variables tab use. +""" +from contextlib import contextmanager +from contextvars import ContextVar +from typing import Any, Iterator, Mapping, Optional + +from je_auto_control.utils.script_vars.scope import VariableScope + +_ACTIVE: ContextVar[Optional[VariableScope]] = ContextVar( + "je_auto_control_execution_scope", default=None) + + +def current_scope() -> Optional[VariableScope]: + """The scope bound by the enclosing :func:`execution_scope`, else ``None``.""" + return _ACTIVE.get() + + +@contextmanager +def bound_scope(scope: VariableScope) -> Iterator[VariableScope]: + """Make an existing ``scope`` the current run's on this thread; restore on exit.""" + token = _ACTIVE.set(scope) + try: + yield scope + finally: + _ACTIVE.reset(token) + + +@contextmanager +def execution_scope(variables: Optional[Mapping[str, Any]] = None + ) -> Iterator[VariableScope]: + """Run the block in a fresh variable scope seeded with ``variables``. + + Everything the module executor runs inside the block -- including nested + bodies, macros and ``AC_set_var`` -- reads and writes this scope, and it + is dropped when the block ends: the previous binding (an enclosing + ``execution_scope``, or the process scope) is restored even on error. + A ``variables`` mapping with an invalid name raises ``ValueError`` before + anything is bound. + """ + scope = VariableScope() + if variables: + scope.update_many(variables) + with bound_scope(scope): + yield scope diff --git a/je_auto_control/utils/script_vars/scope.py b/je_auto_control/utils/script_vars/scope.py index a88bdfba8..2aedfae21 100644 --- a/je_auto_control/utils/script_vars/scope.py +++ b/je_auto_control/utils/script_vars/scope.py @@ -6,12 +6,18 @@ ``for_each`` items. ``VariableScope`` is a thin mutable container the executor exposes to flow-control commands so those commands can read and write the same bag the runtime interpolator consults. + +A scope lasts one top-level run when the run is opened through +:func:`~je_auto_control.utils.script_vars.execution.execution_scope` (as +``execute_action_with_vars`` and every server entry point do); a plain +``executor.execute_action(...)`` uses the executor's own scope, which lasts +as long as the executor. """ from typing import Any, Dict, Iterator, Mapping, MutableMapping, Optional class VariableScope(MutableMapping[str, Any]): - """Mutable mapping of script variables shared across action execution.""" + """Mutable mapping of the script variables one run's actions share.""" __slots__ = ("_vars",) diff --git a/je_auto_control/utils/socket_server/auto_control_socket_server.py b/je_auto_control/utils/socket_server/auto_control_socket_server.py index 58710e2c7..4866c0c13 100644 --- a/je_auto_control/utils/socket_server/auto_control_socket_server.py +++ b/je_auto_control/utils/socket_server/auto_control_socket_server.py @@ -5,6 +5,7 @@ from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.executor.action_executor import execute_action from je_auto_control.utils.logging.logging_instance import autocontrol_logger +from je_auto_control.utils.script_vars.execution import execution_scope _RECV_CHUNK_BYTES = 8192 # Safety cap so a client that never sends the newline terminator can't make us @@ -107,7 +108,9 @@ def handle(self) -> None: else: try: execute_str = json.loads(command_string) - for execute_return in execute_action(execute_str).values(): + with execution_scope(): # one command, one variable scope + execute_record = execute_action(execute_str) + for execute_return in execute_record.values(): socket.sendall(str(execute_return).encode("utf-8")) socket.sendall("\n".encode("utf-8")) socket.sendall("Return_Data_Over_JE".encode("utf-8")) diff --git a/je_auto_control/utils/voice/voice_router.py b/je_auto_control/utils/voice/voice_router.py index 24375111f..882c5c840 100644 --- a/je_auto_control/utils/voice/voice_router.py +++ b/je_auto_control/utils/voice/voice_router.py @@ -85,7 +85,9 @@ def listen_once(self, recognizer: Callable[[], str], def _default_runner(actions: List[Any]) -> Any: from je_auto_control.utils.executor.action_executor import execute_action - return execute_action(actions) + from je_auto_control.utils.script_vars.execution import execution_scope + with execution_scope(): # one utterance, one variable scope + return execute_action(actions) default_voice_router = VoiceRouter() diff --git a/test/unit_test/headless/test_execution_scope_isolation.py b/test/unit_test/headless/test_execution_scope_isolation.py new file mode 100644 index 000000000..26595ab90 --- /dev/null +++ b/test/unit_test/headless/test_execution_scope_isolation.py @@ -0,0 +1,278 @@ +"""Each top-level run gets its own variable scope (2026-09-24 audit, plan A3). + +``execute_action_with_vars`` seeded its variables into the module executor and +never removed them, and REST, MCP and the socket server ran on that same +executor: the next run's ``${user}`` quietly resolved to the previous caller's +value instead of failing with ``Unknown variable``. ``AC_run_dag`` ran its +nodes on the module executor from pool threads, so inside an ``AC_parallel`` +branch they read the parent's variables rather than the branch's. + +Nothing here touches the real mouse, keyboard or screen: the only commands +run are variable commands and a probe registered for the test. +""" +import json +import socket +import threading +import types + +import pytest + +from je_auto_control.utils.executor.action_executor import ( + Executor, execute_action, execute_action_with_vars, executor, +) +from je_auto_control.utils.script_vars import VariableScope, execution_scope +from je_auto_control.utils.script_vars.execution import current_scope + +PROBE = "AC_scope_probe" +_SET_USER = ["AC_set_var", {"name": "user", "value": "alice"}] +_READ_USER = [PROBE, {"value": "${user}"}] + + +@pytest.fixture +def seen(): + """Register a probe command on the module executor; restore it afterwards.""" + values = [] + saved = executor.variables.as_dict() + executor.variables.clear() + executor.event_dict[PROBE] = lambda value=None: values.append(value) or value + yield values + executor.event_dict.pop(PROBE, None) + executor.variables.clear() + executor.variables.update_many(saved) + + +def _failed(record): + return [value for value in record.values() if "Unknown variable" in str(value)] + + +# --- the defect ------------------------------------------------------------- + +def test_top_level_vars_do_not_leak(seen): + first = execute_action_with_vars([_READ_USER], {"user": "alice"}) + second = execute_action_with_vars([_READ_USER], {}) + assert seen == ["alice"] + assert not _failed(first) + second_run_unknown_variable = bool(_failed(second)) + assert second_run_unknown_variable is True + assert "user" not in executor.variables + + +def test_set_var_loop_and_macro_variables_end_with_the_run(seen): + execute_action_with_vars([ + _SET_USER, + ["AC_for_each", {"items": [1, 2], "as": "item", "body": [[PROBE, {"value": "${item}"}]]}], + ["AC_define_macro", {"name": "scope_probe_macro", "params": ["label"], + "body": [[PROBE, {"value": "${label}"}]]}], + ["AC_call_macro", {"name": "scope_probe_macro", "args": {"label": "m"}}], + ], {}) + executor.macros.pop("scope_probe_macro", None) + assert seen == [1, 2, "m"] + assert executor.variables.as_dict() == {} + + +def test_set_var_still_works_within_one_run(seen): + record = execute_action_with_vars([_SET_USER, _READ_USER], {}) + assert seen == ["alice"] + assert not _failed(record) + + +def test_the_seed_still_rejects_an_invalid_name(seen): + with pytest.raises(ValueError): + execute_action_with_vars([_READ_USER], {"": 1}) + assert current_scope() is None + + +# --- the context manager ---------------------------------------------------- + +def test_execution_scope_is_restored_after_an_error(seen): + with pytest.raises(RuntimeError): + with execution_scope({"user": "alice"}) as scope: + assert isinstance(scope, VariableScope) + assert executor.variables is scope + raise RuntimeError("boom") + assert current_scope() is None + assert "user" not in executor.variables + + +def test_a_nested_scope_gives_the_outer_one_back(seen): + with execution_scope({"user": "outer"}) as outer: + with execution_scope({"user": "inner"}): + execute_action([_READ_USER]) + assert executor.variables is outer + execute_action([_READ_USER]) + assert seen == ["inner", "outer"] + + +def test_concurrent_runs_keep_their_own_variables(seen): + barrier = threading.Barrier(2, timeout=10) + executor.event_dict["AC_scope_meet"] = lambda: barrier.wait() and None + records = {} + + def run(name): + records[name] = execute_action_with_vars( + [["AC_scope_meet"], _READ_USER], {"user": name}) + + threads = [threading.Thread(target=run, args=(name,)) for name in ("left", "right")] + try: + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=20) + finally: + executor.event_dict.pop("AC_scope_meet", None) + assert sorted(seen) == ["left", "right"] + assert "user" not in executor.variables + + +# --- what stays as it was --------------------------------------------------- + +def test_direct_calls_on_the_module_executor_keep_the_process_scope(seen): + executor.execute_action([_SET_USER], raise_on_error=True) + execute_action([_READ_USER]) + assert seen == ["alice"] + assert executor.variables.get("user") == "alice" + + +def test_a_private_executor_keeps_its_own_scope_inside_a_run(seen): + private = Executor() + with execution_scope({"user": "run"}): + private.variables.set("user", "private") + assert executor.variables.get("user") == "run" + assert private.variables.get("user") == "private" + + +# --- server and daemon entry points ----------------------------------------- + +def _rest(body): + from je_auto_control.utils.rest_api.rest_handlers import handle_execute + return handle_execute(types.SimpleNamespace( + body=body, query={}, headers={}, path="/execute")) + + +@pytest.mark.parametrize("strict", [False, True]) +def test_rest_execute_does_not_leak(seen, strict): + status, _body = _rest({"actions": [_SET_USER], "raise_on_error": strict}) + assert status == 200 + _status, body = _rest({"actions": [_READ_USER], "raise_on_error": strict}) + assert "Unknown variable" in json.dumps(body) + assert seen == [] + assert "user" not in executor.variables + + +def test_rest_execute_file_does_not_leak(seen, tmp_path): + from je_auto_control.utils.rest_api.rest_handlers import handle_execute_file + script = tmp_path / "set_user.json" + script.write_text(json.dumps([_SET_USER]), encoding="utf-8") + status, _body = handle_execute_file(types.SimpleNamespace( + body={"path": str(script)}, query={}, headers={}, path="/execute_file")) + assert status == 200 + assert "user" not in executor.variables + + +def test_mcp_tool_calls_do_not_leak(seen, tmp_path): + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + tools = {tool.name: tool for tool in build_default_tool_registry()} + tools["ac_execute_actions"].invoke({"actions": [_SET_USER]}) + record = tools["ac_execute_actions"].invoke({"actions": [_READ_USER]}) + assert _failed(record) + script = tmp_path / "set_user.json" + script.write_text(json.dumps([_SET_USER]), encoding="utf-8") + tools["ac_execute_action_file"].invoke({"file_path": str(script)}) + assert seen == [] + assert "user" not in executor.variables + + +def test_socket_commands_do_not_leak(seen): + from je_auto_control.utils.socket_server import auto_control_socket_server as module + server = module.start_autocontrol_socket_server("127.0.0.1", 0) + try: + replies = [_socket_exchange(server, actions) + for actions in ([_SET_USER], [_READ_USER])] + finally: + server.shutdown() + server.server_close() + assert "Unknown variable" in replies[1] + assert seen == [] + assert "user" not in executor.variables + + +def _socket_exchange(server, actions): + with socket.create_connection(server.server_address, timeout=10) as sock: + sock.sendall((json.dumps(actions) + "\n").encode("utf-8")) + reply = b"" + while b"Return_Data_Over_JE" not in reply: + chunk = sock.recv(65536) + if not chunk: + break + reply += chunk + return reply.decode("utf-8") + + +def test_scheduler_trigger_and_hotkey_runs_do_not_leak(seen): + from je_auto_control.utils.run_history.run_outcome import run_counting_failures + run_counting_failures(lambda: execute_action([_SET_USER])) + with pytest.raises(Exception, match="action"): + run_counting_failures(lambda: execute_action([_READ_USER])) + assert seen == [] + assert "user" not in executor.variables + + +def test_chatops_run_does_not_leak(seen, tmp_path): + from je_auto_control.utils.chatops.handlers import cmd_run + (tmp_path / "set_user.json").write_text(json.dumps([_SET_USER]), encoding="utf-8") + cmd_run(["set_user.json"], {"script_root": str(tmp_path)}) + assert "user" not in executor.variables + + +def test_voice_commands_do_not_leak(seen): + from je_auto_control.utils.voice.voice_router import _default_runner + _default_runner([_SET_USER]) + assert "user" not in executor.variables + + +# --- nested helpers inside AC_parallel -------------------------------------- + +def _circuit(name): + return ["AC_circuit_call", {"name": f"scope-{name}", "actions": [[PROBE, {"value": "${who}"}]]}] + + +def _bulkhead(name): + return ["AC_bulkhead_run", {"name": f"scope-{name}", "max_concurrent": 1, + "actions": [[PROBE, {"value": "${who}"}]]}] + + +def _chaos(_name): + return ["AC_run_chaos", {"spec": {"title": "scope", "method": [ + {"name": "probe", "action": [[PROBE, {"value": "${who}"}]]}]}}] + + +def _dag(_name): + return ["AC_run_dag", {"definition": {"nodes": [ + {"id": "a", "actions": [[PROBE, {"value": "${who}"}]]}]}}] + + +@pytest.mark.parametrize("helper", [_circuit, _bulkhead, _chaos, _dag]) +def test_parallel_nested_helpers_use_branch_scope(seen, helper): + branches = [[["AC_set_var", {"name": "who", "value": name}], helper(name)] + for name in ("left", "right")] + execute_action_with_vars([["AC_parallel", {"branches": branches}]], {"who": "parent"}) + branch_results = list(seen) + assert sorted(branch_results) == ["left", "right"] + + +@pytest.mark.parametrize("helper", [_circuit, _bulkhead, _chaos, _dag]) +def test_nested_helpers_use_the_run_scope_at_top_level(seen, helper): + execute_action_with_vars([helper("top")], {"who": "run"}) + assert seen == ["run"] + assert "who" not in executor.variables + + +def test_a_parallel_branch_forks_the_parent_scope(seen): + record = execute_action_with_vars([ + ["AC_parallel", {"branches": [ + [[PROBE, {"value": "${who}"}], ["AC_set_var", {"name": "who", "value": "branch"}]], + ]}], + [PROBE, {"value": "${who}"}], + ], {"who": "parent"}) + assert not _failed(record) + assert seen == ["parent", "parent"] diff --git a/test/unit_test/headless/test_r3_executor_containment.py b/test/unit_test/headless/test_r3_executor_containment.py index 708858c1e..0ae1ef3f9 100644 --- a/test/unit_test/headless/test_r3_executor_containment.py +++ b/test/unit_test/headless/test_r3_executor_containment.py @@ -92,12 +92,16 @@ def test_execute_action_with_vars_defers_loop_body(): The eager pre-pass previously resolved ${item} against the seed mapping — where it does not exist — and raised before execution. """ - execute_action_with_vars( + # Read back inside the run: its scope ends with it, so nothing is left in + # the module executor to inspect afterwards. + record = execute_action_with_vars( [["AC_for_each", {"items": ["a", "b"], "as": "item", "body": [["AC_set_var", - {"name": "r3_last", "value": "${item}"}]]}]], + {"name": "r3_last", "value": "${item}"}]]}], + ["AC_get_var", {"name": "r3_last"}]], {"seed_only": 1}) - assert executor.variables.get_value("r3_last") == "b" + assert list(record.values())[-1] == "b" + assert "r3_last" not in executor.variables def test_parallel_branch_sees_custom_command(): From fe599ea7a5e15635a52fdee7457cd8183fe33592 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 23:57:14 +0800 Subject: [PATCH 09/26] Carry the click count on macOS mouse events so clicks=2 is a double-click macOS applications read kCGMouseEventClickState off the event instead of timing two clicks, and the osx backend never wrote it, so click_mouse(clicks=2) arrived as two single clicks. The n-th click of a run now sets the field to n on both press and release, unless the interval is longer than the system double-click interval. A single click and every other platform make the same backend call as before. --- docs/source/API/wrapper/mouse.rst | 7 +- docs/source/Eng/doc/mouse/mouse_doc.rst | 5 +- docs/source/Zh/doc/mouse/mouse_doc.rst | 4 +- je_auto_control/osx/mouse/osx_mouse.py | 76 ++++- je_auto_control/wrapper/auto_control_mouse.py | 27 +- je_auto_control/wrapper/backend_contract.py | 23 +- .../headless/test_click_mouse_multi.py | 7 +- .../headless/test_osx_mouse_click_state.py | 304 ++++++++++++++++++ 8 files changed, 425 insertions(+), 28 deletions(-) create mode 100644 test/unit_test/headless/test_osx_mouse_click_state.py diff --git a/docs/source/API/wrapper/mouse.rst b/docs/source/API/wrapper/mouse.rst index a1de4c4a1..73a9b66ad 100644 --- a/docs/source/API/wrapper/mouse.rst +++ b/docs/source/API/wrapper/mouse.rst @@ -103,8 +103,11 @@ click_mouse Presses and releases a mouse button at the specified position, ``clicks`` times, ``interval`` seconds apart; ``clicks=2`` double-clicks. Windows and X11 recognise a double-click from the timing and distance of the clicks, so - keep ``interval`` under the system double-click time. On macOS the clicks - arrive as separate single clicks. + keep ``interval`` under the system double-click time. macOS applications + read a click count carried by the event instead, so there the n-th click + has its click-state field (``kCGMouseEventClickState``) set to n; when + ``interval`` is longer than the system double-click interval every click is + sent as a single click. :param mouse_keycode: Mouse button name (e.g., ``"mouse_left"``). :type mouse_keycode: int or str diff --git a/docs/source/Eng/doc/mouse/mouse_doc.rst b/docs/source/Eng/doc/mouse/mouse_doc.rst index 6880c342a..95e347cb0 100644 --- a/docs/source/Eng/doc/mouse/mouse_doc.rst +++ b/docs/source/Eng/doc/mouse/mouse_doc.rst @@ -54,7 +54,10 @@ Press and immediately release a mouse button: Windows and X11 recognise a double-click from the timing and distance of the clicks, so keep ``interval`` under the system double-click time (500 ms by -default on Windows). On macOS the clicks arrive as separate single clicks. +default on Windows). macOS applications read a click count carried by the +event instead, so there the n-th click has its click-state field +(``kCGMouseEventClickState``) set to n; when ``interval`` is longer than the +system double-click interval every click is sent as a single click. Position ======== diff --git a/docs/source/Zh/doc/mouse/mouse_doc.rst b/docs/source/Zh/doc/mouse/mouse_doc.rst index af5530a80..676d90cb6 100644 --- a/docs/source/Zh/doc/mouse/mouse_doc.rst +++ b/docs/source/Zh/doc/mouse/mouse_doc.rst @@ -52,7 +52,9 @@ AutoControl 提供模擬滑鼠操作的功能，包括點擊、定位、捲動 click_mouse("mouse_left", x=500, y=300, clicks=2, interval=0.06) Windows 與 X11 依兩次點擊的時間差與位移判定雙擊，所以 ``interval`` 要小於系統的雙擊判定時間 -（Windows 預設 500 毫秒）。macOS 上這些點擊會被當成各自獨立的單擊。 +（Windows 預設 500 毫秒）。macOS 的應用程式讀的是事件上帶的點擊次數，所以在 macOS 上第 n 次點擊的 +點擊次數欄位（``kCGMouseEventClickState``）會設成 n；``interval`` 比系統的雙擊間隔還長時，每一下都當成 +單擊送出。 游標位置 ======== diff --git a/je_auto_control/osx/mouse/osx_mouse.py b/je_auto_control/osx/mouse/osx_mouse.py index 8a11248d3..ddec885e5 100644 --- a/je_auto_control/osx/mouse/osx_mouse.py +++ b/je_auto_control/osx/mouse/osx_mouse.py @@ -48,7 +48,8 @@ def position() -> Tuple[int, int]: return int(loc.x), int(height - loc.y) -def mouse_event(event: int, x: int, y: int, mouse_button: int) -> None: +def mouse_event(event: int, x: int, y: int, mouse_button: int, + click_count: int = 1) -> None: """ Create and post a mouse event 建立並送出滑鼠事件 @@ -57,9 +58,53 @@ def mouse_event(event: int, x: int, y: int, mouse_button: int) -> None: :param x: X coordinate X 座標 :param y: Y coordinate Y 座標 :param mouse_button: Mouse button code 滑鼠按鍵代碼 + :param click_count: 這是連續第幾次點擊 Which click of a run this is (>= 1) """ + Quartz.CGEventPost( + Quartz.kCGHIDEventTap, + _build_mouse_event(event, x, y, mouse_button, click_count)) + + +def _build_mouse_event(event: int, x: int, y: int, mouse_button: int, + click_count: int = 1): + """ + The Quartz event ``mouse_event`` posts, not yet posted + 建立（但不送出）滑鼠事件 + + macOS 的應用程式不從兩次點擊的時間差判定雙擊，而是讀事件上的點擊次數欄位 + （``kCGMouseEventClickState``）。這裡原本從不寫它，所以連點兩次只是兩次 + 單擊；``click_count`` 大於 1 時寫進去，等於 1 時事件與以前完全相同。 + A macOS application does not infer a double-click from the timing of two + clicks: it reads the click-state field of the event. This never wrote it, + so two clicks were two single clicks. A ``click_count`` above 1 is written + into the field; at 1 the event is exactly the one this always built. + """ + count = _click_count(click_count) curr_event = Quartz.CGEventCreateMouseEvent(None, event, (x, y), mouse_button) - Quartz.CGEventPost(Quartz.kCGHIDEventTap, curr_event) + if count != 1: + Quartz.CGEventSetIntegerValueField( + curr_event, Quartz.kCGMouseEventClickState, count) + return curr_event + + +def _click_count(click_count: int) -> int: + """``click_count`` as an int >= 1, else raise ``AutoControlMouseException``.""" + if isinstance(click_count, bool) or not isinstance(click_count, int) or click_count < 1: + raise AutoControlMouseException( + f"click_count must be an integer >= 1, got {click_count!r}") + return click_count + + +def double_click_interval() -> float: + """ + The longest gap, in seconds, the system still counts as one multi-click + 系統仍視為連擊的最長間隔（秒） + + 使用者可在系統設定調整；兩次點擊隔得比這更久，就是兩次獨立的單擊。 + User-adjustable in System Settings; two clicks further apart than this + are two separate single clicks. + """ + return float(Quartz.NSEvent.doubleClickInterval()) def set_position(x: int, y: int) -> None: @@ -73,7 +118,7 @@ def set_position(x: int, y: int) -> None: mouse_event(Quartz.kCGEventMouseMoved, x, y, 0) -def press_mouse(x: int, y: int, mouse_button: int) -> None: +def press_mouse(x: int, y: int, mouse_button: int, click_count: int = 1) -> None: """ Press mouse button 模擬按下滑鼠按鍵 @@ -81,18 +126,19 @@ def press_mouse(x: int, y: int, mouse_button: int) -> None: :param x: X coordinate X 座標 :param y: Y coordinate Y 座標 :param mouse_button: Mouse button code 滑鼠按鍵代碼 + :param click_count: 這是連續第幾次點擊 Which click of a run this is (>= 1) """ if mouse_button == osx_mouse_left: - mouse_event(Quartz.kCGEventLeftMouseDown, x, y, Quartz.kCGMouseButtonLeft) + mouse_event(Quartz.kCGEventLeftMouseDown, x, y, Quartz.kCGMouseButtonLeft, click_count) elif mouse_button == osx_mouse_middle: - mouse_event(Quartz.kCGEventOtherMouseDown, x, y, Quartz.kCGMouseButtonCenter) + mouse_event(Quartz.kCGEventOtherMouseDown, x, y, Quartz.kCGMouseButtonCenter, click_count) elif mouse_button == osx_mouse_right: - mouse_event(Quartz.kCGEventRightMouseDown, x, y, Quartz.kCGMouseButtonRight) + mouse_event(Quartz.kCGEventRightMouseDown, x, y, Quartz.kCGMouseButtonRight, click_count) else: # nothing was posted, and the wrapper reported the press done raise AutoControlMouseException(f"unknown mouse button {mouse_button!r}") -def release_mouse(x: int, y: int, mouse_button: int) -> None: +def release_mouse(x: int, y: int, mouse_button: int, click_count: int = 1) -> None: """ Release mouse button 模擬釋放滑鼠按鍵 @@ -100,18 +146,20 @@ def release_mouse(x: int, y: int, mouse_button: int) -> None: :param x: X coordinate X 座標 :param y: Y coordinate Y 座標 :param mouse_button: Mouse button code 滑鼠按鍵代碼 + :param click_count: 這是連續第幾次點擊，與對應的按下相同 + Which click of a run this is; the same number as the matching press """ if mouse_button == osx_mouse_left: - mouse_event(Quartz.kCGEventLeftMouseUp, x, y, Quartz.kCGMouseButtonLeft) + mouse_event(Quartz.kCGEventLeftMouseUp, x, y, Quartz.kCGMouseButtonLeft, click_count) elif mouse_button == osx_mouse_middle: - mouse_event(Quartz.kCGEventOtherMouseUp, x, y, Quartz.kCGMouseButtonCenter) + mouse_event(Quartz.kCGEventOtherMouseUp, x, y, Quartz.kCGMouseButtonCenter, click_count) elif mouse_button == osx_mouse_right: - mouse_event(Quartz.kCGEventRightMouseUp, x, y, Quartz.kCGMouseButtonRight) + mouse_event(Quartz.kCGEventRightMouseUp, x, y, Quartz.kCGMouseButtonRight, click_count) else: raise AutoControlMouseException(f"unknown mouse button {mouse_button!r}") -def click_mouse(x: int, y: int, mouse_button: int) -> None: +def click_mouse(x: int, y: int, mouse_button: int, click_count: int = 1) -> None: """ Perform mouse click (press + release) 模擬滑鼠點擊（按下 + 釋放） @@ -119,10 +167,12 @@ def click_mouse(x: int, y: int, mouse_button: int) -> None: :param x: X coordinate X 座標 :param y: Y coordinate Y 座標 :param mouse_button: Mouse button code 滑鼠按鍵代碼 + :param click_count: 這是連續第幾次點擊；雙擊的第二下傳 2 + Which click of a run this is; the second click of a double-click is 2 """ - press_mouse(x, y, mouse_button) + press_mouse(x, y, mouse_button, click_count) time.sleep(0.001) # 小延遲確保事件正確送出 - release_mouse(x, y, mouse_button) + release_mouse(x, y, mouse_button, click_count) def scroll(scroll_value: int) -> None: diff --git a/je_auto_control/wrapper/auto_control_mouse.py b/je_auto_control/wrapper/auto_control_mouse.py index f6f23e15a..e829976fc 100644 --- a/je_auto_control/wrapper/auto_control_mouse.py +++ b/je_auto_control/wrapper/auto_control_mouse.py @@ -280,6 +280,24 @@ def _click_plan(clicks: object, interval: object) -> Tuple[int, float]: return count, pause +def _darwin_click(x: int, y: int, keycode: MouseKeycode, index: int, + pause: float) -> None: + """Send click ``index`` (0-based) of a run to the macOS backend. + + macOS applications read the click count off the event rather than timing + two clicks, so the n-th click carries n. A pause longer than the system + double-click interval is not a multi-click anywhere, so those clicks stay + single ones -- and a single click is the three-argument call this always + made. Does nothing off macOS, where the field does not exist. + """ + if sys.platform != "darwin": + return + if not index or (pause and pause > mouse.double_click_interval()): + mouse.click_mouse(x, y, keycode) + else: + mouse.click_mouse(x, y, keycode, index + 1) + + def click_mouse(mouse_keycode: int | str, x: int | None = None, y: int | None = None, clicks: int = 1, interval: float = 0.0) -> Tuple[MouseKeycode, int, int]: @@ -290,8 +308,11 @@ def click_mouse(mouse_keycode: int | str, x: int | None = None, All clicks land on the same point, ``interval`` seconds apart. Windows and X11 recognise a double-click from the timing and distance of two clicks, so keep ``interval`` under the system double-click time (500 ms by - default on Windows). macOS apps read a click count this backend does not - set, so there the clicks arrive as separate single clicks. + default on Windows). macOS apps read a click count carried by the event + instead, so there the n-th click of the run has its click-state field + (``kCGMouseEventClickState``) set to n -- unless ``interval`` is longer + than the system double-click interval, in which case every click is sent + as a single click, as the other platforms would treat it. :param mouse_keycode: 滑鼠按鍵代碼 Mouse keycode :param x: X 座標 X position @@ -318,7 +339,7 @@ def click_mouse(mouse_keycode: int | str, x: int | None = None, # strings, so the int never matches any branch and the click is # silently dropped with no exception. if sys.platform == "darwin": - mouse.click_mouse(x, y, keycode) + _darwin_click(x, y, keycode, index, pause) else: mouse.click_mouse(keycode, x, y) record_action_to_list("click_mouse", param) diff --git a/je_auto_control/wrapper/backend_contract.py b/je_auto_control/wrapper/backend_contract.py index f91b348b8..69f3f7721 100644 --- a/je_auto_control/wrapper/backend_contract.py +++ b/je_auto_control/wrapper/backend_contract.py @@ -178,14 +178,25 @@ def position(self) -> Tuple[int, int]: def set_position(self, x: int, y: int) -> None: """Move the cursor to a point on the desktop.""" - def press_mouse(self, x: int, y: int, mouse_button: int) -> None: - """Hold a button down at a point.""" + def press_mouse(self, x: int, y: int, mouse_button: int, + click_count: int = 1) -> None: + """Hold a button down at a point; ``click_count`` as in ``click_mouse``.""" - def release_mouse(self, x: int, y: int, mouse_button: int) -> None: - """Let a button up at a point.""" + def release_mouse(self, x: int, y: int, mouse_button: int, + click_count: int = 1) -> None: + """Let a button up at a point; ``click_count`` as in ``click_mouse``.""" - def click_mouse(self, x: int, y: int, mouse_button: int) -> None: - """Press and release a button at a point.""" + def click_mouse(self, x: int, y: int, mouse_button: int, + click_count: int = 1) -> None: + """Press and release a button at a point. + + ``click_count`` is which click of a run this is. Applications here + read it off the event instead of timing two clicks, so the second + click of a double-click has to say 2. + """ + + def double_click_interval(self) -> float: + """Seconds within which the system still counts clicks as one run.""" def scroll(self, scroll_value: int) -> None: """Turn the wheel; one axis, so the sign carries the direction.""" diff --git a/test/unit_test/headless/test_click_mouse_multi.py b/test/unit_test/headless/test_click_mouse_multi.py index 646cfabd7..44697d7cb 100644 --- a/test/unit_test/headless/test_click_mouse_multi.py +++ b/test/unit_test/headless/test_click_mouse_multi.py @@ -68,11 +68,14 @@ def test_an_integer_string_is_accepted(env): def test_macos_binds_every_click_xy_first(env, monkeypatch): calls = [] + # The first click is the three-argument call it always was; the second + # also says it is the second, which macOS applications read off the + # event (test_osx_mouse_click_state.py). monkeypatch.setattr(auto_control_mouse, "mouse", types.SimpleNamespace( - click_mouse=lambda x, y, mouse_button: calls.append((x, y, mouse_button)))) + click_mouse=lambda *args: calls.append(args))) monkeypatch.setattr(sys, "platform", "darwin") auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=2) - assert calls == [(5, 6, 1), (5, 6, 1)] + assert calls == [(5, 6, 1), (5, 6, 1, 2)] @pytest.mark.parametrize("clicks", [0, -1, True, False, 1.5, 2.0, "x", None]) diff --git a/test/unit_test/headless/test_osx_mouse_click_state.py b/test/unit_test/headless/test_osx_mouse_click_state.py new file mode 100644 index 000000000..a40d212f4 --- /dev/null +++ b/test/unit_test/headless/test_osx_mouse_click_state.py @@ -0,0 +1,304 @@ +"""macOS multi-click: the click count rides on the event, not on the clock. + +Windows and X11 recognise a double-click from two clicks close together in +time and space, so `click_mouse(clicks=2)` only had to click twice. A macOS +application reads the count off the event instead -- the +`kCGMouseEventClickState` field -- and `osx_mouse.py` never wrote it, so two +clicks arrived as two single clicks whatever the interval. + +Three layers, tested separately: + +* the osx backend, loaded here against a fake `Quartz` so it runs on every + CI square, not only the Darwin ones (the module refuses to import off + macOS, so `sys.platform` is patched for the load and the module is loaded + under a private name that never enters `sys.modules`); +* the wrapper, which numbers the clicks on macOS and nowhere else; +* the real framework, on the macOS squares only: an event is built and read + back, never posted. That holds the backend to real API names and to the + field actually taking the value. It does **not** show that an application + treats the result as a double-click; nothing here posts an event. + +No Qt imports. +""" +from __future__ import annotations + +import importlib.util +import pathlib +import sys +import types + +import pytest + +import je_auto_control +from je_auto_control.utils.exception.exceptions import AutoControlMouseException +from je_auto_control.wrapper import auto_control_mouse + +_OSX_MOUSE = (pathlib.Path(je_auto_control.__file__).parent + / "osx" / "mouse" / "osx_mouse.py") +_OSX_VK = "je_auto_control.osx.core.utils.osx_vk" + +#: The Quartz names `osx_mouse.py` uses to build and post a button event. +QUARTZ_MOUSE_NAMES = ( + "CGEventCreateMouseEvent", "CGEventSetIntegerValueField", "CGEventPost", + "kCGHIDEventTap", "kCGMouseEventClickState", "kCGEventMouseMoved", + "kCGEventLeftMouseDown", "kCGEventLeftMouseUp", + "kCGEventRightMouseDown", "kCGEventRightMouseUp", + "kCGEventOtherMouseDown", "kCGEventOtherMouseUp", + "kCGMouseButtonLeft", "kCGMouseButtonRight", "kCGMouseButtonCenter", + "NSEvent", +) + + +class _FakeQuartz(types.ModuleType): + """Builds events as dicts and records what was posted.""" + + def __init__(self) -> None: + super().__init__("Quartz") + self.posted = [] + self.double_click_seconds = 0.5 + for name in QUARTZ_MOUSE_NAMES: + if name.startswith("k"): + setattr(self, name, name) + self.NSEvent = types.SimpleNamespace( + doubleClickInterval=lambda: self.double_click_seconds) + + def CGEventCreateMouseEvent(self, source, kind, point, button): # noqa: N802 # reason: the Quartz name + return {"source": source, "kind": kind, "point": point, + "button": button, "fields": {}} + + def CGEventSetIntegerValueField(self, event, field, value): # noqa: N802 # reason: the Quartz name + event["fields"][field] = value + + def CGEventPost(self, tap, event): # noqa: N802 # reason: the Quartz name + self.posted.append((tap, event)) + + +@pytest.fixture +def osx(monkeypatch): + """`osx_mouse` bound to a fake Quartz, leaving `sys.modules` as it was.""" + quartz = _FakeQuartz() + monkeypatch.setattr(sys, "platform", "darwin") + monkeypatch.setitem(sys.modules, "Quartz", quartz) + had_vk = _OSX_VK in sys.modules + spec = importlib.util.spec_from_file_location("_osx_mouse_under_test", + _OSX_MOUSE) + module = importlib.util.module_from_spec(spec) + try: + spec.loader.exec_module(module) + # The module's own name, not `time.sleep` itself: the real module + # object is shared with everything else in the process. + monkeypatch.setattr(module, "time", types.SimpleNamespace( + sleep=lambda _seconds: None)) + yield types.SimpleNamespace(mouse=module, quartz=quartz) + finally: + if not had_vk: + # Imported for the first time by the load above, under a patched + # platform. Left behind it would make `import ...osx_vk` succeed + # on Windows and Linux for the rest of the session. + sys.modules.pop(_OSX_VK, None) + + +def _states(quartz): + return [(event["kind"], event["fields"].get("kCGMouseEventClickState")) + for _tap, event in quartz.posted] + + +# --- the backend -------------------------------------------------------------- + +def test_a_single_click_builds_the_event_it_always_did(osx): + # No field is written at all, so the default path is unchanged rather + # than "changed to something believed equivalent". + osx.mouse.click_mouse(10, 20, "Left") + assert osx.quartz.posted == [ + ("kCGHIDEventTap", {"source": None, "kind": "kCGEventLeftMouseDown", + "point": (10, 20), "button": "kCGMouseButtonLeft", + "fields": {}}), + ("kCGHIDEventTap", {"source": None, "kind": "kCGEventLeftMouseUp", + "point": (10, 20), "button": "kCGMouseButtonLeft", + "fields": {}}), + ] + + +def test_the_second_click_carries_two_on_both_the_press_and_the_release(osx): + # An application reads the count on the down *and* the up; a release + # still saying 1 ends a double-click as a single one. + osx.mouse.click_mouse(10, 20, "Left", 2) + assert _states(osx.quartz) == [("kCGEventLeftMouseDown", 2), + ("kCGEventLeftMouseUp", 2)] + + +@pytest.mark.parametrize("button,down,up", [ + ("Left", "kCGEventLeftMouseDown", "kCGEventLeftMouseUp"), + ("Right", "kCGEventRightMouseDown", "kCGEventRightMouseUp"), + ("Middle", "kCGEventOtherMouseDown", "kCGEventOtherMouseUp"), +]) +def test_every_button_carries_the_count(osx, button, down, up): + osx.mouse.press_mouse(1, 2, button, 3) + osx.mouse.release_mouse(1, 2, button, 3) + assert _states(osx.quartz) == [(down, 3), (up, 3)] + + +def test_press_and_release_default_to_a_single_click(osx): + osx.mouse.press_mouse(1, 2, "Left") + osx.mouse.release_mouse(1, 2, "Left") + assert _states(osx.quartz) == [("kCGEventLeftMouseDown", None), + ("kCGEventLeftMouseUp", None)] + + +def test_moving_the_cursor_writes_no_click_state(osx): + osx.mouse.set_position(5, 6) + [(_tap, event)] = osx.quartz.posted + assert event["fields"] == {} + + +@pytest.mark.parametrize("count", [0, -1, True, 1.5, "2", None]) +def test_a_click_count_that_is_not_a_positive_integer_posts_nothing(osx, + count): + with pytest.raises(AutoControlMouseException, match="click_count"): + osx.mouse.click_mouse(1, 2, "Left", count) + assert osx.quartz.posted == [] + + +def test_an_unknown_button_is_still_refused_with_a_count(osx): + with pytest.raises(AutoControlMouseException, match="unknown mouse button"): + osx.mouse.press_mouse(1, 2, 99, 2) + assert osx.quartz.posted == [] + + +def test_the_double_click_interval_is_the_systems_own(osx): + osx.quartz.double_click_seconds = 0.25 + assert osx.mouse.double_click_interval() == pytest.approx(0.25) + + +def test_loading_the_backend_here_leaves_no_macos_module_behind(osx): + assert "_osx_mouse_under_test" not in sys.modules + assert "je_auto_control.osx.mouse.osx_mouse" not in sys.modules or ( + sys.modules["je_auto_control.osx.mouse.osx_mouse"] is not osx.mouse) + + +# --- the wrapper -------------------------------------------------------------- + +class _DarwinBackend: + """The macOS seam: point first, then the button, then the count.""" + + def __init__(self, interval=0.5): + self.clicks = [] + self.interval = interval + self.interval_reads = 0 + + def click_mouse(self, x, y, mouse_button, click_count=1): + self.clicks.append((x, y, mouse_button, click_count)) + + def double_click_interval(self): + self.interval_reads += 1 + return self.interval + + +@pytest.fixture +def wrapper(monkeypatch): + backend = _DarwinBackend() + sleeps = [] + monkeypatch.setattr(auto_control_mouse, "mouse", backend) + monkeypatch.setattr(auto_control_mouse, "time", + types.SimpleNamespace(sleep=sleeps.append)) + monkeypatch.setattr(auto_control_mouse, "mouse_keys_table", + {"mouse_left": "Left"}) + monkeypatch.setattr(auto_control_mouse, "record_action_to_list", + lambda *_args, **_kwargs: None) + monkeypatch.setattr(sys, "platform", "darwin") + return types.SimpleNamespace(backend=backend, sleeps=sleeps) + + +def test_a_double_click_on_macos_numbers_its_clicks(wrapper): + auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=2) + assert wrapper.backend.clicks == [(5, 6, "Left", 1), (5, 6, "Left", 2)] + + +def test_a_triple_click_counts_to_three(wrapper): + auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=3, interval=0.05) + assert [click[3] for click in wrapper.backend.clicks] == [1, 2, 3] + assert wrapper.sleeps == [0.05, 0.05] + + +def test_the_system_interval_is_only_read_when_there_is_a_pause_to_judge( + wrapper): + auto_control_mouse.click_mouse("mouse_left", 5, 6) + auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=2) + assert wrapper.backend.interval_reads == 0 + auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=2, interval=0.1) + assert wrapper.backend.interval_reads == 1 + + +def test_clicks_further_apart_than_a_double_click_stay_single(wrapper): + # `clicks=3, interval=2` is "click three times", not a triple-click: no + # platform reads it as one, and a text view sent a count of 3 would + # select the paragraph. + wrapper.backend.interval = 0.5 + auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=3, interval=2) + assert [click[3] for click in wrapper.backend.clicks] == [1, 1, 1] + + +def test_an_interval_exactly_at_the_limit_still_counts(wrapper): + wrapper.backend.interval = 0.5 + auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=2, interval=0.5) + assert [click[3] for click in wrapper.backend.clicks] == [1, 2] + + +@pytest.mark.parametrize("platform", ["win32", "linux"]) +def test_no_other_platform_is_passed_a_count(wrapper, monkeypatch, platform): + calls = [] + monkeypatch.setattr(auto_control_mouse, "mouse", types.SimpleNamespace( + click_mouse=lambda *args: calls.append(args))) + monkeypatch.setattr(sys, "platform", platform) + auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=2) + assert calls == [("Left", 5, 6), ("Left", 5, 6)], ( + "three positional arguments, and no double_click_interval lookup " + "on a backend that has none" + ) + + +def test_the_wrapper_and_the_backend_agree_end_to_end(osx, monkeypatch): + # The two halves joined: what the wrapper sends is what the backend the + # fake Quartz is bound to accepts, with nothing renumbered in between. + monkeypatch.setattr(auto_control_mouse, "mouse", osx.mouse) + monkeypatch.setattr(auto_control_mouse, "mouse_keys_table", + {"mouse_left": "Left"}) + monkeypatch.setattr(auto_control_mouse, "record_action_to_list", + lambda *_args, **_kwargs: None) + auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=2) + assert _states(osx.quartz) == [ + ("kCGEventLeftMouseDown", None), ("kCGEventLeftMouseUp", None), + ("kCGEventLeftMouseDown", 2), ("kCGEventLeftMouseUp", 2), + ] + + +# --- the real framework, on the macOS squares --------------------------------- + +def _real_quartz(): + if sys.platform != "darwin": + pytest.skip("pyobjc is a macOS-only dependency") + return pytest.importorskip("Quartz") + + +@pytest.mark.parametrize("name", QUARTZ_MOUSE_NAMES) +def test_every_quartz_name_the_mouse_backend_uses_exists(name): + assert hasattr(_real_quartz(), name), f"Quartz.{name}" + + +def test_the_real_double_click_interval_is_a_positive_number(): + _real_quartz() + from je_auto_control.osx.mouse import osx_mouse + assert osx_mouse.double_click_interval() > 0 + + +@pytest.mark.parametrize("count", [2, 3]) +def test_a_real_event_takes_the_click_count_it_is_given(count): + # Built and read back, never posted: this moves no cursor and clicks + # nothing, and it says only that the field holds the number -- not that + # an application acts on it. + quartz = _real_quartz() + from je_auto_control.osx.mouse import osx_mouse + event = osx_mouse._build_mouse_event( + quartz.kCGEventLeftMouseDown, 10, 10, quartz.kCGMouseButtonLeft, count) + assert quartz.CGEventGetIntegerValueField( + event, quartz.kCGMouseEventClickState) == count From 0909604c8dd31268d122d60db2b635a4e3668a7f Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 23:57:14 +0800 Subject: [PATCH 10/26] Look a macOS window up by id so a minimized one can be listed and restored Every single-window lookup went through Quartz's on-screen list, which a minimized window is not in: after minimize() the window vanished from list_windows and restore() raised the Accessibility refusal on a Mac that had granted it. _info_for now asks with kCGWindowListOptionIncludingWindow, list_windows appends the off-screen windows whose accessibility element reports AXMinimized (as the Windows backend keeps iconified windows), and is_minimized no longer answers True for a window Quartz does not know. --- .../wrapper/window_backends/macos_backend.py | 134 ++++++++++-- test/unit_test/headless/_pyobjc_stub.py | 55 ++++- .../headless/test_pyobjc_stub_names.py | 13 +- .../headless/test_window_backend_macos.py | 199 +++++++++++++++++- .../test_window_backend_macos_real.py | 153 ++++++++++++++ 5 files changed, 517 insertions(+), 37 deletions(-) create mode 100644 test/unit_test/headless/test_window_backend_macos_real.py diff --git a/je_auto_control/wrapper/window_backends/macos_backend.py b/je_auto_control/wrapper/window_backends/macos_backend.py index 1aaa5d990..b8f087df4 100644 --- a/je_auto_control/wrapper/window_backends/macos_backend.py +++ b/je_auto_control/wrapper/window_backends/macos_backend.py @@ -3,9 +3,9 @@ Reading and acting are two different APIs on macOS, with two different permission stories, and this backend needs both: -* **Quartz** (``CGWindowListCopyWindowInfo``) enumerates every on-screen - window with its id, title, owning pid and bounds. It needs no grant, so - listing, rectangles and ownership work out of the box. +* **Quartz** (``CGWindowListCopyWindowInfo``) enumerates windows with their + id, title, owning pid and bounds. It needs no grant, so listing, rectangles + and ownership work out of the box. * **The accessibility API** is the only way to *move*, *close*, *minimise* or *raise* someone else's window. It is gated by TCC: the user grants Accessibility to the interpreter, and until they do every action silently @@ -17,6 +17,16 @@ ``CGWindowID`` is matched to its ``AXUIElement`` by owner, title and frame. That is what the two APIs give us to work with; the alternative is a private symbol (``_AXUIElementGetWindow``) this project will not depend on. + +**A minimised window is off screen, not gone.** Quartz's on-screen list leaves +it out, so one window is always looked up by id +(``kCGWindowListOptionIncludingWindow``), which answers for it wherever it is. +Looking it up in the on-screen list instead made ``minimize`` a one-way trip: +the window vanished from ``list_windows`` and ``restore`` blamed a missing +Accessibility grant that had in fact been given. ``list_windows`` includes +minimised windows, as the Windows backend's does, but Quartz cannot tell a +minimised window from the many other things an application keeps off screen, +so that half asks the accessibility API and is empty without the grant. """ import sys from typing import Any, List, Optional, Tuple @@ -61,27 +71,78 @@ def _window_info(self) -> List[dict]: found = Quartz.CGWindowListCopyWindowInfo(options, Quartz.kCGNullWindowID) return list(found or []) + def _off_screen_info(self, on_screen: List[dict]) -> List[dict]: + """Application-layer windows Quartz knows of that are not on screen. + + Far more than the minimised ones: hidden applications, other Spaces + and windows an application built and never showed are all here. + """ + import Quartz + + shown = {_number(info) for info in on_screen} + options = (Quartz.kCGWindowListOptionAll + | Quartz.kCGWindowListExcludeDesktopElements) + found = Quartz.CGWindowListCopyWindowInfo(options, Quartz.kCGNullWindowID) + return [info for info in (found or []) + if _number(info) and _number(info) not in shown + and _is_application_layer(info)] + + def _minimized_info(self, on_screen: List[dict]) -> List[dict]: + """The off-screen windows whose accessibility element says minimised. + + One ``AXWindows`` query per owning process, not per window. Without + the Accessibility grant no element is found and this is empty, which + leaves the listing at what is on screen. + """ + import Quartz + + minimized = [] + elements: dict = {} + for info in self._off_screen_info(on_screen): + pid = int(info.get(Quartz.kCGWindowOwnerPID, 0) or 0) + if not pid: + continue + if pid not in elements: + elements[pid] = self._ax_windows_for(pid) + window = _match_info(elements[pid], info) + if window is not None and _ax_minimized(window): + minimized.append(info) + return minimized + def list_windows(self) -> List[Tuple[int, str]]: import Quartz + on_screen = self._window_info() windows = [] - for info in self._window_info(): + # On-screen windows first, in Quartz's front-to-back order; minimised + # ones are behind everything, so they follow. + for info in on_screen + self._minimized_info(on_screen): # Layer 0 is the ordinary application layer. Menu bars, the Dock # and status items live above it and are not windows a caller # means when they say "the Safari window". - if int(info.get(Quartz.kCGWindowLayer, 0) or 0) != 0: + if not _is_application_layer(info): continue - number = int(info.get(Quartz.kCGWindowNumber, 0) or 0) + number = _number(info) if not number: continue windows.append((number, str(info.get(Quartz.kCGWindowName, "") or ""))) return windows def _info_for(self, window_id: int) -> Optional[dict]: + """Quartz's description of one window, on screen or not, or None. + + Asked for by id rather than searched for in the on-screen list, which + is what lets a minimised window be found again and restored. + """ import Quartz - for info in self._window_info(): - if int(info.get(Quartz.kCGWindowNumber, 0) or 0) == int(window_id): + wanted = int(window_id) + if wanted <= 0: + return None + found = Quartz.CGWindowListCopyWindowInfo( + Quartz.kCGWindowListOptionIncludingWindow, wanted) + for info in found or []: + if _number(info) == wanted: return info return None @@ -150,11 +211,7 @@ def _ax_window(self, window_id: int): pid = int(info.get(Quartz.kCGWindowOwnerPID, 0) or 0) if not pid: return None - bounds = info.get(Quartz.kCGWindowBounds) or {} - return _best_match( - self._ax_windows_for(pid), - (int(bounds.get("X", 0)), int(bounds.get("Y", 0))), - str(info.get(Quartz.kCGWindowName, "") or "")) + return _match_info(self._ax_windows_for(pid), info) def _require_ax_window(self, window_id: int, operation: str): window = self._ax_window(window_id) @@ -163,16 +220,17 @@ def _require_ax_window(self, window_id: int, operation: str): return window def is_minimized(self, window_id: int) -> bool: - import ApplicationServices as ax + import Quartz window = self._ax_window(window_id) - if window is None: - # A minimised window is not in the on-screen list at all, so - # failing to find it is itself the answer here. - return self._info_for(window_id) is None - _error, value = ax.AXUIElementCopyAttributeValue( - window, "AXMinimized", None) - return bool(value) + if window is not None: + return _ax_minimized(window) + # No element to ask -- the Accessibility grant is missing, or the + # match failed. Quartz still knows whether the window is on screen, + # which is the nearest answer it has; a window it has never heard of + # is not minimised, it is gone. + info = self._info_for(window_id) + return info is not None and not info.get(Quartz.kCGWindowIsOnscreen) def set_foreground(self, window_id: int) -> None: import AppKit @@ -256,6 +314,40 @@ def move(self, window_id: int, x: int, y: int, # Refusing says so; a "success" that focused something else would not. +def _number(info: dict) -> int: + """The ``CGWindowID`` in a Quartz window-info dictionary, or 0.""" + import Quartz + + return int(info.get(Quartz.kCGWindowNumber, 0) or 0) + + +def _is_application_layer(info: dict) -> bool: + """Whether a Quartz window sits on layer 0, where ordinary windows live.""" + import Quartz + + return int(info.get(Quartz.kCGWindowLayer, 0) or 0) == 0 + + +def _ax_minimized(window: Any) -> bool: + """Whether an accessibility window reports itself minimised.""" + import ApplicationServices as ax + + _error, value = ax.AXUIElementCopyAttributeValue( + window, "AXMinimized", None) + return bool(value) + + +def _match_info(candidates: list, info: dict): + """The accessibility window among ``candidates`` that ``info`` describes.""" + import Quartz + + bounds = info.get(Quartz.kCGWindowBounds) or {} + return _best_match( + candidates, + (int(bounds.get("X", 0)), int(bounds.get("Y", 0))), + str(info.get(Quartz.kCGWindowName, "") or "")) + + def _best_match(candidates: list, wanted_origin: Tuple[int, int], wanted_title: str): """Pick the accessibility window that is the Quartz window described. diff --git a/test/unit_test/headless/_pyobjc_stub.py b/test/unit_test/headless/_pyobjc_stub.py index b370ab523..8d79ae059 100644 --- a/test/unit_test/headless/_pyobjc_stub.py +++ b/test/unit_test/headless/_pyobjc_stub.py @@ -30,12 +30,13 @@ #: Quartz's window-info dictionary keys, spelled as pyobjc spells them. WINDOW_KEYS = ( "kCGWindowNumber", "kCGWindowName", "kCGWindowLayer", - "kCGWindowOwnerPID", "kCGWindowBounds", + "kCGWindowOwnerPID", "kCGWindowBounds", "kCGWindowIsOnscreen", ) #: The rest of the surface the backend names, by module. QUARTZ_NAMES = WINDOW_KEYS + ( "kCGWindowListOptionOnScreenOnly", "kCGWindowListExcludeDesktopElements", + "kCGWindowListOptionAll", "kCGWindowListOptionIncludingWindow", "kCGNullWindowID", "CGWindowListCopyWindowInfo", "CGPoint", "CGSize", ) @@ -57,6 +58,14 @@ AX_SUCCESS = 0 AX_FAILURE = -25200 # kAXErrorCannotComplete, in spirit +#: `CGWindowListOption` values. Unlike the info-dictionary keys these are real +#: numbers, because the stub has to *read* them to decide which windows a +#: query sees -- that is the whole of the minimised-window defect. +OPTION_ALL = 0 +OPTION_ON_SCREEN_ONLY = 1 +OPTION_INCLUDING_WINDOW = 8 +OPTION_EXCLUDE_DESKTOP = 16 + def window_info(number: int, *, name: str = "", layer: int = 0, pid: int = 0, bounds=None) -> dict: @@ -77,7 +86,11 @@ def window_info(number: int, *, name: str = "", layer: int = 0, class AXElement: """An accessibility element: attributes, and what was done to it.""" - def __init__(self, **attributes) -> None: + def __init__(self, quartz_id=None, **attributes) -> None: + # The `CGWindowID` this element is the accessibility face of. Only a + # test that needs minimising to take the window off screen sets it; + # the backend never sees it, as it never sees the real bridge either. + self.quartz_id = quartz_id self.attributes = dict(attributes) self.actions = [] self.assignments = [] @@ -113,14 +126,38 @@ def __init__(self, windows=None, ax_windows=None, frontmost_pid=None, # pid lookup and the activation, which is its own branch. self.running_pids = (set() if running_pids is None else set(running_pids)) + # Window numbers Quartz knows but does not have on screen: minimised + # ones, and anything a test parks here to stand for a hidden window. + self.off_screen = set() self.list_options = [] self.activated = [] self.ax_list_error = AX_SUCCESS # -- Quartz -- def copy_window_info(self, options, relative_to): + """Answer as Quartz does for the three option shapes the backend uses. + + On-screen-only leaves out what is off screen, including-window answers + for exactly the id asked about wherever it is, and "all" returns the + lot. `kCGWindowIsOnscreen` is present only on the windows it is true + for, which is how the real dictionaries spell it. + """ self.list_options.append((options, relative_to)) - return list(self.windows) + if options & OPTION_INCLUDING_WINDOW: + chosen = [info for info in self.windows + if info.get("kCGWindowNumber") == relative_to] + elif options & OPTION_ON_SCREEN_ONLY: + chosen = [info for info in self.windows + if info.get("kCGWindowNumber") not in self.off_screen] + else: + chosen = list(self.windows) + return [self._described(info) for info in chosen] + + def _described(self, info): + described = dict(info) + if info.get("kCGWindowNumber") not in self.off_screen: + described["kCGWindowIsOnscreen"] = True + return described # -- ApplicationServices -- def ax_application(self, pid): @@ -144,6 +181,12 @@ def ax_set_attribute(self, element, attribute, value): if element.set_error: return element.set_error element.attributes[attribute] = value + if attribute == "AXMinimized" and element.quartz_id is not None: + # What the Dock does: the window leaves the screen, and comes back. + if value: + self.off_screen.add(element.quartz_id) + else: + self.off_screen.discard(element.quartz_id) return AX_SUCCESS def ax_perform_action(self, element, action): @@ -164,8 +207,10 @@ def install(monkeypatch, world: World) -> World: quartz = types.ModuleType("Quartz") for name in WINDOW_KEYS: setattr(quartz, name, name) - quartz.kCGWindowListOptionOnScreenOnly = 1 - quartz.kCGWindowListExcludeDesktopElements = 16 + quartz.kCGWindowListOptionAll = OPTION_ALL + quartz.kCGWindowListOptionOnScreenOnly = OPTION_ON_SCREEN_ONLY + quartz.kCGWindowListOptionIncludingWindow = OPTION_INCLUDING_WINDOW + quartz.kCGWindowListExcludeDesktopElements = OPTION_EXCLUDE_DESKTOP quartz.kCGNullWindowID = 0 quartz.CGWindowListCopyWindowInfo = world.copy_window_info quartz.CGPoint = AXPoint diff --git a/test/unit_test/headless/test_pyobjc_stub_names.py b/test/unit_test/headless/test_pyobjc_stub_names.py index 6688c03e0..c4d1f5189 100644 --- a/test/unit_test/headless/test_pyobjc_stub_names.py +++ b/test/unit_test/headless/test_pyobjc_stub_names.py @@ -44,9 +44,14 @@ def test_every_accessibility_name_the_stub_answers_for_exists(name): def test_the_window_list_options_are_the_flags_the_backend_composes(): - # These two are the only Quartz numbers the backend does arithmetic on: - # it ORs them into the argument of CGWindowListCopyWindowInfo. + # These are the only Quartz numbers the backend does arithmetic on -- it + # ORs them into the argument of CGWindowListCopyWindowInfo -- and the only + # ones the stub reads, to decide which windows a query sees. import Quartz - assert Quartz.kCGWindowListOptionOnScreenOnly == 1 - assert Quartz.kCGWindowListExcludeDesktopElements == 16 + assert Quartz.kCGWindowListOptionAll == objc_stub.OPTION_ALL + assert Quartz.kCGWindowListOptionOnScreenOnly == objc_stub.OPTION_ON_SCREEN_ONLY + assert (Quartz.kCGWindowListOptionIncludingWindow + == objc_stub.OPTION_INCLUDING_WINDOW) + assert (Quartz.kCGWindowListExcludeDesktopElements + == objc_stub.OPTION_EXCLUDE_DESKTOP) assert Quartz.kCGNullWindowID == 0 diff --git a/test/unit_test/headless/test_window_backend_macos.py b/test/unit_test/headless/test_window_backend_macos.py index f5c7e6210..492a0d20c 100644 --- a/test/unit_test/headless/test_window_backend_macos.py +++ b/test/unit_test/headless/test_window_backend_macos.py @@ -88,11 +88,16 @@ def test_the_backend_names_both_apis_it_uses(backend): # --- listing ------------------------------------------------------------------ -def test_listing_asks_quartz_for_on_screen_windows_only(backend, world): +def test_listing_asks_quartz_for_what_is_on_screen_and_then_for_the_rest( + backend, world): + # Two questions, in this order: the on-screen list is the one that comes + # back front-to-back, so it sets the order; the full list is only mined + # for windows the first one left out. backend.list_windows() - [(options, relative_to)] = world.list_options - assert options == 1 | 16, "on-screen only, excluding desktop elements" - assert relative_to == 0, "kCGNullWindowID" + assert world.list_options == [ + (1 | 16, 0), # on-screen only, excluding desktop elements + (0 | 16, 0), # everything, excluding desktop elements + ], "both relative to kCGNullWindowID" def test_listing_keeps_the_order_quartz_gives(backend, world): @@ -288,9 +293,22 @@ def test_a_window_the_element_says_is_not_minimised_is_not(monkeypatch, assert backend.is_minimized(7) is False -def test_a_window_that_is_not_on_screen_at_all_is_minimised(backend): - # A minimised window is absent from the on-screen list, so failing to - # find it is itself the answer rather than an error. +def test_a_window_quartz_has_never_heard_of_is_not_minimised(backend): + # This used to answer True, on the reasoning that a minimised window is + # absent from the on-screen list. It is -- but so is a window that was + # closed, and now that one window is looked up by id the two can be told + # apart. The Windows backend answers False for a dead handle too. + assert backend.is_minimized(7) is False + + +def test_an_off_screen_window_with_no_element_reads_as_minimised(monkeypatch, + on_darwin): + # Without the Accessibility grant there is no element to ask, and Quartz + # only knows "not on screen". That is the nearest answer available, and + # it is the one that makes focus_window try a restore first. + world = _mac_with_window(ax_windows=[]) + world.off_screen.add(7) + backend = _build(monkeypatch, world) assert backend.is_minimized(7) is True @@ -490,3 +508,170 @@ def test_input_to_an_unfocused_window_is_refused_not_faked(backend, call): # focus is. A "success" that clicked somewhere else is worse than a no. with pytest.raises(AutoControlUnsupportedOperationException): call(backend) + + +# --- a minimised window is off screen, not gone ------------------------------- +# +# The defect these pin: every lookup of one window went through the on-screen +# list, which a minimised window is not in. `minimize(7)` succeeded, and from +# then on the window was missing from `list_windows` and `restore(7)` raised +# the "grant Accessibility" refusal on a Mac where it had been granted. + +def _linked_window(number=7, *, origin=(10, 20), title="Editor"): + """An accessibility window whose minimising the stub's Quartz notices.""" + element = AXElement(quartz_id=number) + element.attributes["AXPosition"] = ("point", objc_stub.AXPoint(*origin)) + element.attributes["AXTitle"] = title + return element + + +def test_one_window_is_asked_for_by_id_not_searched_for_on_screen(backend, + world): + world.windows = [window_info(7, pid=501)] + backend.window_process_id(7) + assert world.list_options == [(objc_stub.OPTION_INCLUDING_WINDOW, 7)] + + +@pytest.mark.parametrize("window_id", [0, -1]) +def test_a_null_window_id_is_not_put_to_quartz(backend, world, window_id): + # kCGNullWindowID with the including-window option is not a question + # about a window, so it is answered here rather than asked. + assert backend.window_process_id(window_id) == 0 + assert world.list_options == [] + + +def test_a_minimised_window_can_be_restored(monkeypatch, on_darwin): + element = _linked_window() + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + assert backend.minimize(7) is True + assert 7 in world.off_screen, "the stub took it off screen, as the Dock does" + backend.restore(7) + assert element.attributes["AXMinimized"] is False + assert 7 not in world.off_screen + + +def test_a_minimised_window_still_reports_its_owner_and_rectangle( + monkeypatch, on_darwin): + world = _mac_with_window(ax_windows=[_linked_window()]) + backend = _build(monkeypatch, world) + backend.minimize(7) + assert backend.window_process_id(7) == 501 + assert backend.window_rect(7) == (10, 20, 310, 420) + + +def test_a_minimised_window_says_it_is_minimised(monkeypatch, on_darwin): + world = _mac_with_window(ax_windows=[_linked_window()]) + backend = _build(monkeypatch, world) + assert backend.is_minimized(7) is False + backend.minimize(7) + assert backend.is_minimized(7) is True + backend.restore(7) + assert backend.is_minimized(7) is False + + +def test_a_minimised_window_stays_in_the_listing(monkeypatch, on_darwin): + # What the Windows backend does: EnumWindows + IsWindowVisible keeps an + # iconified window, so a title search still finds it. + world = _mac_with_window(ax_windows=[_linked_window()]) + backend = _build(monkeypatch, world) + backend.minimize(7) + assert backend.list_windows() == [(7, "Editor")] + + +def test_minimised_windows_list_after_the_ones_on_screen(monkeypatch, + on_darwin): + # Front-most first is the contract; a minimised window is in front of + # nothing, wherever Quartz happens to put it in the full list. + world = World(windows=[ + window_info(7, name="Minimised", pid=501, bounds=(10, 20, 300, 400)), + window_info(3, name="Front", pid=501, bounds=(0, 0, 50, 50)), + window_info(4, name="Back", pid=99, bounds=(5, 5, 50, 50)), + ]) + world.ax_windows = {501: [_linked_window(title="Minimised")]} + backend = _build(monkeypatch, world) + backend.minimize(7) + assert backend.list_windows() == [(3, "Front"), (4, "Back"), + (7, "Minimised")] + + +def test_an_off_screen_window_that_is_not_minimised_is_not_listed( + monkeypatch, on_darwin): + # Quartz's full list is mostly not windows anyone minimised: hidden + # applications, other Spaces, windows built and never shown. Only the + # accessibility element can say which is which. + hidden = _linked_window() + hidden.attributes["AXMinimized"] = False + world = _mac_with_window(ax_windows=[hidden]) + world.off_screen.add(7) + backend = _build(monkeypatch, world) + assert backend.list_windows() == [] + + +def test_an_off_screen_window_with_no_element_is_not_listed(monkeypatch, + on_darwin): + # No Accessibility grant: nothing can vouch for the window, and listing + # every off-screen surface would bury the real ones. + world = _mac_with_window(ax_windows=[]) + world.off_screen.add(7) + backend = _build(monkeypatch, world) + assert backend.list_windows() == [] + + +def test_off_screen_windows_above_the_application_layer_are_not_listed( + monkeypatch, on_darwin): + element = _linked_window() + element.attributes["AXMinimized"] = True + world = World(windows=[window_info(7, name="Editor", pid=501, layer=25, + bounds=(10, 20, 300, 400))]) + world.ax_windows = {501: [element]} + world.off_screen.add(7) + backend = _build(monkeypatch, world) + assert backend.list_windows() == [] + + +def test_an_off_screen_window_with_no_owner_is_not_listed(monkeypatch, + on_darwin): + world = World(windows=[window_info(7, pid=0)]) + world.off_screen.add(7) + backend = _build(monkeypatch, world) + assert backend.list_windows() == [] + + +def test_each_application_is_asked_for_its_windows_once(monkeypatch, + on_darwin): + # An AX round trip per off-screen window would be dozens for one browser. + first = _linked_window(7, origin=(10, 20), title="One") + second = _linked_window(8, origin=(30, 40), title="Two") + for element in (first, second): + element.attributes["AXMinimized"] = True + world = World(windows=[ + window_info(7, name="One", pid=501, bounds=(10, 20, 1, 1)), + window_info(8, name="Two", pid=501, bounds=(30, 40, 1, 1)), + ]) + world.ax_windows = {501: [first, second]} + world.off_screen.update({7, 8}) + asked = [] + real = world.ax_application + world.ax_application = lambda pid: asked.append(pid) or real(pid) + backend = _build(monkeypatch, world) + assert backend.list_windows() == [(7, "One"), (8, "Two")] + assert asked == [501] + + +def test_focusing_a_minimised_window_by_title_restores_and_raises_it( + monkeypatch, on_darwin): + # The wrapper's whole path: find by title, see it is minimised, restore, + # raise. Before the fix it stopped at the first step -- "no window + # matches" -- for a window the same process had just minimised. + from je_auto_control.wrapper import auto_control_window + + element = _linked_window() + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + monkeypatch.setattr(auto_control_window, "get_backend", lambda: backend) + assert auto_control_window.minimize_window_by_title("Editor") is True + assert 7 in world.off_screen + assert auto_control_window.focus_window("Editor") == 7 + assert element.attributes["AXMinimized"] is False + assert element.actions == ["AXRaise"] diff --git a/test/unit_test/headless/test_window_backend_macos_real.py b/test/unit_test/headless/test_window_backend_macos_real.py new file mode 100644 index 000000000..b32ae56af --- /dev/null +++ b/test/unit_test/headless/test_window_backend_macos_real.py @@ -0,0 +1,153 @@ +"""Minimise a real macOS window and bring it back, on a real window server. + +`test_window_backend_macos.py` holds the backend to a stand-in for pyobjc, +and a stand-in can only agree with what its author believed about Quartz. +The defect this file exists for was exactly such a belief: a minimised window +is missing from Quartz's *on-screen* list, the backend looked every window up +there, and so `minimize` was a one-way trip -- `restore` raised the "grant +Accessibility" refusal on a Mac where it had been granted. + +So this opens a window of its own, in a child process, and drives it through +the real frameworks. The child owns the window because the accessibility API +talks to an application through its run loop, and a test process blocked in +an assertion is not running one. + +**Where it runs.** Only on macOS, and only when asked: on CI (`CI` is set by +GitHub Actions, whose `macos-14` runners were measured to grant Accessibility +-- see `test/verify/macos_verify.py`) or with `AUTOCONTROL_REAL_WINDOW_TEST=1`. +It puts a window on the screen, and a developer running the headless suite on +their own Mac has not asked for that. + +**What is a skip and what is a failure.** Everything needed to *reach* the +question is a skip with its reason: no window server, no Accessibility grant, +a window that would not minimise. Once a window has been minimised through +this backend, failing to find it, list it or restore it is the defect, and +fails. + +No Qt imports. +""" +from __future__ import annotations + +import os +import subprocess # nosec B404 # reason: starts this interpreter with a fixed script to own a test window +import sys +import time + +import pytest + +pytestmark = pytest.mark.skipif( + sys.platform != "darwin" + or not (os.environ.get("CI") + or os.environ.get("AUTOCONTROL_REAL_WINDOW_TEST")), + reason="opens a real window: macOS only, on CI or with " + "AUTOCONTROL_REAL_WINDOW_TEST=1", +) + +_WIDTH, _HEIGHT = 320, 240 + +#: The whole child: one titled, miniaturisable window and a run loop. +_CHILD = f""" +import AppKit +app = AppKit.NSApplication.sharedApplication() +app.setActivationPolicy_(AppKit.NSApplicationActivationPolicyRegular) +style = (AppKit.NSWindowStyleMaskTitled | AppKit.NSWindowStyleMaskClosable + | AppKit.NSWindowStyleMaskMiniaturizable) +window = AppKit.NSWindow.alloc().initWithContentRect_styleMask_backing_defer_( + ((200, 200), ({_WIDTH}, {_HEIGHT})), style, + AppKit.NSBackingStoreBuffered, False) +window.setTitle_("AutoControl minimise test") +window.makeKeyAndOrderFront_(None) +app.activateIgnoringOtherApps_(True) +app.run() +""" + +_POLL_S = 0.1 + + +def _wait_for(condition, timeout_s: float): + """Poll `condition` until it is truthy; its last value either way.""" + deadline = time.monotonic() + timeout_s + while True: + value = condition() + if value or time.monotonic() >= deadline: + return value + time.sleep(_POLL_S) + + +def _on_screen_ids(backend, pid: int) -> list: + import Quartz + + return [int(info.get(Quartz.kCGWindowNumber, 0) or 0) + for info in backend._window_info() + if int(info.get(Quartz.kCGWindowOwnerPID, 0) or 0) == pid + and int(info.get(Quartz.kCGWindowLayer, 0) or 0) == 0] + + +@pytest.fixture +def owned_window(): + """`(backend, window_id, pid)` for a window this test may do anything to.""" + from je_auto_control.wrapper.window_backends.macos_backend import ( + MacOSWindowBackend, + ) + + backend = MacOSWindowBackend() + if not backend.available: + pytest.skip("the macOS window backend is unavailable here") + child = subprocess.Popen( # nosec B603 # reason: this interpreter, fixed argv, no shell + [sys.executable, "-c", _CHILD], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + try: + ids = _wait_for(lambda: _on_screen_ids(backend, child.pid), 15.0) + if not ids: + pytest.skip("no window appeared: no window server in this session") + window_id = ids[0] + if not _wait_for(lambda: backend._ax_window(window_id), 5.0): + pytest.skip("no accessibility element: Accessibility not granted") + yield backend, window_id, child.pid + finally: + child.terminate() + try: + child.wait(timeout=5) + except subprocess.TimeoutExpired: + child.kill() + child.wait(timeout=5) + + +def test_a_really_minimised_window_is_found_listed_and_restored(owned_window): + backend, window_id, pid = owned_window + + if not backend.minimize(window_id): + pytest.skip("the accessibility API refused to minimise the window") + if _wait_for(lambda: window_id not in _on_screen_ids(backend, pid), + 5.0) is not True: + pytest.skip("the window never left the screen after minimising") + + # From here on the window is minimised by this backend, so every answer + # below is the contract and not a precondition. + assert backend.window_process_id(window_id) == pid, ( + "a minimised window must still be found by id") + assert backend.window_rect(window_id) is not None + assert backend.is_minimized(window_id) is True + + backend.restore(window_id) # raised the Accessibility refusal before + assert _wait_for(lambda: window_id in _on_screen_ids(backend, pid), + 5.0), "restore did not bring the window back on screen" + assert backend.is_minimized(window_id) is False + + +def test_a_really_minimised_window_stays_in_the_listing(owned_window): + # Separate from the test above on purpose: this half depends on matching + # an off-screen Quartz window to its accessibility element, which is the + # part of the fix a stand-in is least able to vouch for. + backend, window_id, pid = owned_window + + assert window_id in [number for number, _title in backend.list_windows()] + if not backend.minimize(window_id): + pytest.skip("the accessibility API refused to minimise the window") + if _wait_for(lambda: window_id not in _on_screen_ids(backend, pid), + 5.0) is not True: + pytest.skip("the window never left the screen after minimising") + + listed = [number for number, _title in backend.list_windows()] + assert window_id in listed, ( + "list_windows must include a minimised window, as on Windows") From f99b2a2809622a70c5e0b31bd645b9507f196a3e Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 23:58:38 +0800 Subject: [PATCH 11/26] Record who opened each remote-desktop host and viewer so panels stop cutting each other's sessions Quick Connect, the viewer tab and AC_remote_connect wrote the same registry slot and each cleared it first. A connect on one side cut the other's session without telling it: its popup stayed on the last frame and its Disconnect then ended a session it never opened. The registry now records an owner per occupant, panels read and close only what they opened, and a replaced owner is told so it closes its window. The script-side commands keep acting on the active host or viewer whoever opened it. --- .../Eng/doc/mcp_server/mcp_server_doc.rst | 4 +- .../Eng/doc/new_features/new_features_doc.rst | 40 +- .../Zh/doc/mcp_server/mcp_server_doc.rst | 4 +- .../Zh/doc/new_features/new_features_doc.rst | 36 +- .../gui/language_wrapper/english.py | 1 + .../gui/language_wrapper/japanese.py | 1 + .../language_wrapper/simplified_chinese.py | 1 + .../language_wrapper/traditional_chinese.py | 1 + .../gui/remote_desktop/_helpers.py | 24 ++ .../gui/remote_desktop/connection_screen.py | 63 ++- .../gui/remote_desktop/host_panel.py | 32 +- .../gui/remote_desktop/viewer_panel.py | 43 +- .../utils/remote_desktop/registry.py | 326 ++++++++++++--- test/unit_test/headless/test_rd_gui_audit.py | 3 +- .../headless/test_rd_webrtc_audit.py | 6 +- .../headless/test_remote_registry_owners.py | 377 ++++++++++++++++++ .../test_remote_registry_owners_gui.py | 353 ++++++++++++++++ 17 files changed, 1203 insertions(+), 112 deletions(-) create mode 100644 test/unit_test/headless/test_remote_registry_owners.py create mode 100644 test/unit_test/headless/test_remote_registry_owners_gui.py diff --git a/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst b/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst index bf0622cda..0eb27b014 100644 --- a/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst +++ b/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst @@ -75,7 +75,9 @@ Remote desktop (TCP host + viewer registry) ``ac_remote_host_status``, ``ac_remote_viewer_connect``, ``ac_remote_viewer_disconnect``, ``ac_remote_viewer_status``, ``ac_remote_viewer_send_input``. These wrap the same singleton - registry the GUI's Remote Desktop tab uses, so a model can spin + registry the GUI's Remote Desktop tab uses and act on its active host + or viewer whoever opened it (the status results name that ``owner``; + a GUI panel whose session a tool replaces or ends closes its window), so a model can spin up a host (``token``, ``bind``, ``port``, ``fps``, ``quality``, ``host_id``), open a viewer to another machine, query status, and forward mouse / keyboard / type / hotkey actions through the diff --git a/docs/source/Eng/doc/new_features/new_features_doc.rst b/docs/source/Eng/doc/new_features/new_features_doc.rst index 08cd73f77..753529d89 100644 --- a/docs/source/Eng/doc/new_features/new_features_doc.rst +++ b/docs/source/Eng/doc/new_features/new_features_doc.rst @@ -525,13 +525,45 @@ Action-JSON commands (use the singleton in AC_start_remote_host # token, bind, port, fps, quality, region AC_stop_remote_host - AC_remote_host_status # → {running, port, connected_clients} + AC_remote_host_status # → {running, port, connected_clients, host_id, owner} AC_remote_connect # host, port, token, timeout AC_remote_disconnect - AC_remote_viewer_status # → {connected} + AC_remote_viewer_status # → {connected, host_id, owner} AC_remote_send_input # action: {...} +**Who owns the host and the viewer.** The registry holds one host and one +viewer per transport (TCP, WebSocket), and records the *owner* that opened +each: ``"script"`` for these commands, the ``AC_ws_*`` ones and the MCP +``ac_remote_*`` tools, and a token of its own (``quick-connect#1``, +``viewer-tab#2``, ``host-tab#3``) for each GUI panel. The ``*_status`` +results name it under ``owner`` (``None`` when nothing is active). + +- A script that uses only these commands behaves as it always did: a second + ``AC_remote_connect`` replaces the first, ``AC_remote_disconnect`` ends it. +- In a process that also runs the GUI, the commands still act on *the* + active host or viewer of that transport, whoever opened it: + ``AC_remote_viewer_status`` and ``AC_remote_send_input`` see a session a + panel opened, ``AC_remote_connect`` replaces it and + ``AC_remote_disconnect`` / ``AC_stop_remote_host`` end it. The panel is + told, closes its remote-screen window and returns to idle. +- A panel only reads, drives and disconnects the viewer it opened itself. + *Connect* on one panel still replaces the other panel's session on the + same transport — there is one viewer per transport — but the replaced + panel closes its window instead of freezing on the last frame, and its + *Disconnect* no longer reaches the session that replaced it. Starting a + host no longer disconnects any viewer. *Stop* on either host surface + stops the host shown as running, whoever started it. + +From Python the same model is ``remote_desktop_registry.adopt(slot, +resource, owner, on_displaced)``, ``owned(slot, owner)``, ``release(slot, +owner)``, ``evict(slot, by)`` and ``owner_of(slot)``, with ``slot`` one of +``"host"``, ``"viewer"``, ``"ws_host"``, ``"ws_viewer"`` and owner tokens +from ``je_auto_control.utils.remote_desktop.registry.new_owner(label)``. +``stop_host`` / ``disconnect_viewer`` (and the ``ws`` pair) take an optional +``owner=`` that limits them to that owner's host or viewer. +``on_displaced(slot, by)`` runs on the thread that did the replacing. + GUI: **Remote Desktop** tab opens to the **Quick Connect** screen (AnyDesk-style) by default — huge Host ID on one side, a single input that accepts ``host:port``, ``ws://``, ``wss://``, or a 9-digit Host @@ -1020,6 +1052,10 @@ clicking through the GUI: "action": "type", "text": "hello", }) +The MCP tools are the same ``"script"`` owner as the ``AC_remote_*`` +commands: they act on the active host or viewer whoever opened it, and a +GUI panel whose session they replace or end is told and closes its window. + The status / observer tools (``ac_remote_host_status``, ``ac_remote_viewer_status``) are read-only and survive the MCP server's ``--readonly`` filter; everything that mutates state is diff --git a/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst b/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst index 9ca641b7f..a1787eae1 100644 --- a/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst +++ b/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst @@ -74,7 +74,9 @@ list-changed 通知與 elicitation。 ``ac_remote_host_status``、``ac_remote_viewer_connect``、 ``ac_remote_viewer_disconnect``、``ac_remote_viewer_status``、 ``ac_remote_viewer_send_input``。這組工具直接包裝 GUI 的「遠端 - 桌面」分頁所用的 process-global registry,模型可以代為啟動 host + 桌面」分頁所用的 process-global registry,作用在目前那一個 host 或 + viewer,不論是誰開的(狀態結果以 ``owner`` 回報;連線被工具取代或結束的 + GUI 面板會關掉自己的視窗),模型可以代為啟動 host (``token``、``bind``、``port``、``fps``、``quality``、 ``host_id``)、連線 viewer 至另一台主機、查詢狀態,並透過目前的 viewer 將滑鼠 / 鍵盤 / type / hotkey 動作轉送給遠端 host。狀態 diff --git a/docs/source/Zh/doc/new_features/new_features_doc.rst b/docs/source/Zh/doc/new_features/new_features_doc.rst index 0961cce4a..075733ead 100644 --- a/docs/source/Zh/doc/new_features/new_features_doc.rst +++ b/docs/source/Zh/doc/new_features/new_features_doc.rst @@ -490,13 +490,41 @@ Action-JSON 指令（使用 :mod:`utils.remote_desktop.registry` 的單例）:: AC_start_remote_host # token, bind, port, fps, quality, region AC_stop_remote_host - AC_remote_host_status # → {running, port, connected_clients} + AC_remote_host_status # → {running, port, connected_clients, host_id, owner} AC_remote_connect # host, port, token, timeout AC_remote_disconnect - AC_remote_viewer_status # → {connected} + AC_remote_viewer_status # → {connected, host_id, owner} AC_remote_send_input # action: {...} +**host 與 viewer 由誰擁有。** registry 每種傳輸（TCP、WebSocket）各保存一個 +host 與一個 viewer，並記錄是誰開的（owner）：這些指令、``AC_ws_*`` 指令與 MCP +的 ``ac_remote_*`` 工具是 ``"script"``，每個 GUI 面板則有自己的代號 +（``quick-connect#1``、``viewer-tab#2``、``host-tab#3``）。``*_status`` +的結果以 ``owner`` 欄位回報（沒有連線時為 ``None``）。 + +- 只用這些指令的腳本行為與以往完全相同：第二次 ``AC_remote_connect`` + 取代第一次，``AC_remote_disconnect`` 中斷它。 +- 在同時跑 GUI 的行程裡，這些指令仍然作用在該傳輸\ *目前那一個* host 或 + viewer，不論是誰開的：``AC_remote_viewer_status`` 與 + ``AC_remote_send_input`` 看得到面板開的連線，``AC_remote_connect`` + 會取代它，``AC_remote_disconnect``／``AC_stop_remote_host`` 會結束它。 + 面板會收到通知，關掉自己的遠端畫面視窗並回到閒置狀態。 +- 面板只讀取、操作、中斷自己開的 viewer。在一個面板按 *連線* 仍會取代另一個 + 面板在同一傳輸上的連線（每種傳輸只有一個 viewer），但被取代的面板會關掉 + 視窗，不再停在最後一格畫面，它的 *中斷* 也不會再切到取代它的那條連線。 + 啟動 host 不再中斷任何 viewer。兩個 host 介面的 *停止* 都會停掉顯示為 + 執行中的那個 host，不論是誰啟動的。 + +Python 端對應的介面是 ``remote_desktop_registry.adopt(slot, resource, owner, +on_displaced)``、``owned(slot, owner)``、``release(slot, owner)``、 +``evict(slot, by)`` 與 ``owner_of(slot)``；``slot`` 為 ``"host"``、 +``"viewer"``、``"ws_host"``、``"ws_viewer"`` 之一，owner 代號由 +``je_auto_control.utils.remote_desktop.registry.new_owner(label)`` 產生。 +``stop_host``／``disconnect_viewer``（以及 ``ws`` 那一組）多了可選的 +``owner=``，只在該 owner 擁有時才動作。``on_displaced(slot, by)`` +在執行取代動作的那個執行緒上被呼叫。 + GUI：\ **Remote Desktop**\ 分頁預設打開的是 **快速連線** （AnyDesk 風格）— 一邊是超大本機 Host ID，另一邊一個輸入框接受 ``host:port``、 @@ -952,6 +980,10 @@ registry 包成工具,工廠函式為 "action": "type", "text": "hello", }) +MCP 工具與 ``AC_remote_*`` 指令同屬 ``"script"`` 這個 owner：它們作用在 +目前那一個 host 或 viewer，不論是誰開的；被它們取代或結束連線的 GUI 面板會 +收到通知並關掉自己的視窗。 + 狀態類工具(``ac_remote_host_status``、 ``ac_remote_viewer_status``)為唯讀,可以通過 MCP server 的 ``--readonly`` 過濾;會修改狀態的工具都正確帶上 diff --git a/je_auto_control/gui/language_wrapper/english.py b/je_auto_control/gui/language_wrapper/english.py index a5ccc74ca..6bc8d1ebe 100644 --- a/je_auto_control/gui/language_wrapper/english.py +++ b/je_auto_control/gui/language_wrapper/english.py @@ -1297,6 +1297,7 @@ "rd_viewer_status_connected": "Connected — receiving frames", "rd_viewer_status_idle": "Not connected", "rd_viewer_error": "Remote desktop error", + "rd_viewer_displaced": "This session was closed: another connection took its place.", "rd_remote_screen_title": "Remote Desktop — Live Session", "rd_remote_screen_title_with_id": "Remote Desktop — {host_id}", diff --git a/je_auto_control/gui/language_wrapper/japanese.py b/je_auto_control/gui/language_wrapper/japanese.py index 5e7faea6b..0740bfc51 100644 --- a/je_auto_control/gui/language_wrapper/japanese.py +++ b/je_auto_control/gui/language_wrapper/japanese.py @@ -1184,6 +1184,7 @@ "rd_viewer_status_connected": "接続中 — フレーム受信中", "rd_viewer_status_idle": "未接続", "rd_viewer_error": "リモートデスクトップエラー", + "rd_viewer_displaced": "このセッションは終了しました。別の接続に置き換えられました。", "rd_remote_screen_title": "リモートデスクトップ — ライブセッション", "rd_remote_screen_title_with_id": "リモートデスクトップ — {host_id}", diff --git a/je_auto_control/gui/language_wrapper/simplified_chinese.py b/je_auto_control/gui/language_wrapper/simplified_chinese.py index 424def77e..d002ab239 100644 --- a/je_auto_control/gui/language_wrapper/simplified_chinese.py +++ b/je_auto_control/gui/language_wrapper/simplified_chinese.py @@ -1163,6 +1163,7 @@ "rd_viewer_status_connected": "已连接 — 正在接收画面", "rd_viewer_status_idle": "未连接", "rd_viewer_error": "远程桌面错误", + "rd_viewer_displaced": "此会话已关闭：已被另一个连接取代。", "rd_remote_screen_title": "远程桌面 — 实时会话", "rd_remote_screen_title_with_id": "远程桌面 — {host_id}", diff --git a/je_auto_control/gui/language_wrapper/traditional_chinese.py b/je_auto_control/gui/language_wrapper/traditional_chinese.py index db29e507d..046af99cd 100644 --- a/je_auto_control/gui/language_wrapper/traditional_chinese.py +++ b/je_auto_control/gui/language_wrapper/traditional_chinese.py @@ -1164,6 +1164,7 @@ "rd_viewer_status_connected": "已連線 — 正在接收畫面", "rd_viewer_status_idle": "尚未連線", "rd_viewer_error": "遠端桌面錯誤", + "rd_viewer_displaced": "此工作階段已關閉：已被另一個連線取代。", "rd_remote_screen_title": "遠端桌面 — 即時連線", "rd_remote_screen_title_with_id": "遠端桌面 — {host_id}", diff --git a/je_auto_control/gui/remote_desktop/_helpers.py b/je_auto_control/gui/remote_desktop/_helpers.py index 33d6501ad..8c46dba2a 100644 --- a/je_auto_control/gui/remote_desktop/_helpers.py +++ b/je_auto_control/gui/remote_desktop/_helpers.py @@ -201,6 +201,30 @@ def set_body_layout(self, layout) -> None: self._body.setLayout(layout) +def displaced_notifier(panel: QWidget) -> Callable[[str, str], None]: + """Return the registry ``on_displaced`` callback for ``panel``. + + The registry calls it on whichever thread replaced the panel's host or + viewer - an executor or MCP thread as easily as the GUI thread - and may + do so long after the panel is gone. It emits ``panel._displaced(slot, + by)``, which Qt queues to the GUI thread when the caller is another one, + holds the panel only weakly, and does nothing once it is destroyed. + """ + ref = weakref.ref(panel) + + def notify(slot: str, by: str) -> None: + target = ref() + if target is None: + return + try: + target._displaced.emit(slot, by) # noqa: SLF001 # reason: the panel's own signal + except RuntimeError: + # The Qt object was deleted while Python still held the wrapper. + return + + return notify + + def _short_fp(fp: Optional[str]) -> str: if not fp: return "" diff --git a/je_auto_control/gui/remote_desktop/connection_screen.py b/je_auto_control/gui/remote_desktop/connection_screen.py index 47ad18b86..9c17e1c28 100644 --- a/je_auto_control/gui/remote_desktop/connection_screen.py +++ b/je_auto_control/gui/remote_desktop/connection_screen.py @@ -27,7 +27,8 @@ from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui.remote_desktop._helpers import ( - _StatusBadge, _build_verifying_client_context, _t, wire_remote_input, + _StatusBadge, _build_verifying_client_context, _t, displaced_notifier, + wire_remote_input, ) from je_auto_control.gui.remote_desktop.remote_screen_window import ( RemoteScreenWindow, @@ -44,7 +45,9 @@ ConnectTarget, UnresolvableTargetError, parse_target, ) from je_auto_control.utils.remote_desktop.host_id import format_host_id -from je_auto_control.utils.remote_desktop.registry import registry +from je_auto_control.utils.remote_desktop.registry import ( + SLOT_HOST, SLOT_VIEWER, SLOT_WS_VIEWER, new_owner, registry, +) from je_auto_control.utils.remote_desktop.wake_on_lan import ( send_magic_packet, ) @@ -114,10 +117,14 @@ class QuickConnectScreen(TranslatableMixin, QWidget): _frame_arrived = Signal(object) _error_arrived = Signal(str) _cursor_moved = Signal(int, int) + # Another owner took one of this screen's registry slots. + _displaced = Signal(str, str) def __init__(self, parent: Optional[QWidget] = None) -> None: super().__init__(parent) self._tr_init() + self._owner = new_owner("quick-connect") + self._displaced.connect(self._on_displaced) self._host_id_label = QLabel("---") self._host_id_label.setStyleSheet(_HOST_ID_CSS) self._host_id_label.setAlignment(Qt.AlignmentFlag.AlignCenter) @@ -280,8 +287,7 @@ def _start_hosting(self) -> None: if not token: self._generate_token() token = self._host_token.text().strip() - registry.disconnect_viewer() - registry.stop_host() + registry.evict(SLOT_HOST, by=self._owner) try: host = RemoteDesktopHost( token=token, bind="127.0.0.1", port=0, @@ -292,7 +298,7 @@ def _start_hosting(self) -> None: except (OSError, ValueError, RuntimeError) as error: QMessageBox.warning(self, _t("rd_quick_start_host"), str(error)) return - registry._host = host # noqa: SLF001 centralised lifecycle ownership + registry.adopt(SLOT_HOST, host, self._owner, displaced_notifier(self)) self._refresh_status() def _host_approval_callback(self, pending: PendingViewer): @@ -353,8 +359,9 @@ def _show_approval_dialog(self, request: _ApprovalRequest) -> None: request.event.set() def _stop_hosting(self) -> None: + # Stops the host the badge shows, whoever started it; see _HostPanel._stop. try: - registry.stop_host() + registry.evict(SLOT_HOST, by=self._owner) except (OSError, RuntimeError) as error: QMessageBox.warning(self, _t("rd_quick_stop_host"), str(error)) return @@ -414,8 +421,26 @@ def _dispatch_target(self, target: ConnectTarget, token: str) -> None: self, _t("rd_quick_connect_btn"), _t("rd_quick_bad_target"), ) + def _take_slot(self, slot: str) -> None: + """Clear ``slot`` for a new session and end this screen's other one.""" + other = SLOT_WS_VIEWER if slot == SLOT_VIEWER else SLOT_VIEWER + registry.release(other, self._owner) + registry.evict(slot, by=self._owner) + + def _own_viewer(self): + """The viewer this screen opened, or None once it is gone or replaced.""" + return (registry.owned(SLOT_VIEWER, self._owner) + or registry.owned(SLOT_WS_VIEWER, self._owner)) + + def _on_displaced(self, slot: str, _by: str) -> None: + """GUI thread: another panel or a script took one of this screen's slots.""" + if slot != SLOT_HOST and self._own_viewer() is None: + self._pending_frame = None + self._close_screen_window() + self._refresh_status() + def _do_tcp_connect(self, host: str, port: int, token: str) -> None: - registry.disconnect_viewer() + self._take_slot(SLOT_VIEWER) try: viewer = RemoteDesktopViewer( host=host, port=port, token=token, @@ -429,7 +454,7 @@ def _do_tcp_connect(self, host: str, port: int, token: str) -> None: except (OSError, RuntimeError, ValueError, AutoControlException) as error: QMessageBox.warning(self, _t("rd_quick_connect_btn"), str(error)) return - registry._viewer = viewer # noqa: SLF001 centralised lifecycle ownership + registry.adopt(SLOT_VIEWER, viewer, self._owner, displaced_notifier(self)) self._remember_tcp(host, port) self._open_screen_window(f"{host}:{port}") self._refresh_status() @@ -438,7 +463,7 @@ def _do_ws_connect(self, target: ConnectTarget, token: str) -> None: host = target.host or "" port = target.port or 0 path = target.path or "/" - registry.disconnect_ws_viewer() + self._take_slot(SLOT_WS_VIEWER) # wss:// was dialled as plain ws://: the session went unencrypted to # a host the operator took for TLS, and a real TLS host was unreachable. ssl_context = _build_verifying_client_context() if target.kind == "wss" else None @@ -454,7 +479,7 @@ def _do_ws_connect(self, target: ConnectTarget, token: str) -> None: except (OSError, RuntimeError, ValueError, AutoControlException) as error: QMessageBox.warning(self, _t("rd_quick_connect_btn"), str(error)) return - registry._ws_viewer = viewer # noqa: SLF001 centralised lifecycle ownership + registry.adopt(SLOT_WS_VIEWER, viewer, self._owner, displaced_notifier(self)) scheme = "wss" if target.kind == "wss" else "ws" self._remember_url(f"{scheme}://{host}:{port}{path}") self._open_screen_window(f"{scheme}://{host}:{port}") @@ -476,10 +501,9 @@ def _handoff_to_webrtc(self, host_id: str, token: str) -> None: self.webrtc_handoff_requested.emit(host_id, token) def _disconnect(self) -> None: - # Both transports may be live; clear whichever slot was filled - # so the operator does not need to remember which they used. - registry.disconnect_viewer() - registry.disconnect_ws_viewer() + # Either transport, and only a session this screen opened. + registry.release(SLOT_VIEWER, self._owner) + registry.release(SLOT_WS_VIEWER, self._owner) self._close_screen_window() self._refresh_status() @@ -523,7 +547,7 @@ def _open_screen_window(self, title: str) -> None: def _send_input(self, action: dict) -> None: """Forward one input action from the popup to the live viewer.""" - viewer = registry.viewer or registry._ws_viewer # noqa: SLF001 + viewer = self._own_viewer() if viewer is None or not viewer.connected: return try: @@ -533,7 +557,7 @@ def _send_input(self, action: dict) -> None: def _on_files_dropped(self, paths) -> None: """Upload each dropped file to the host's home directory.""" - viewer = registry.viewer or registry._ws_viewer # noqa: SLF001 + viewer = self._own_viewer() if viewer is None or not viewer.connected: return for path in paths: @@ -561,7 +585,7 @@ def _close_screen_window(self) -> None: def _on_window_closed(self) -> None: # Either transport: closing a ws:// popup left its session running. - if registry.viewer is not None or registry._ws_viewer is not None: # noqa: SLF001 + if self._own_viewer() is not None: self._disconnect() # --- recent connections ------------------------------------------ @@ -692,8 +716,9 @@ def _refresh_host_status(self) -> None: ) def _refresh_viewer_status(self) -> None: - # A ws:// session read as disconnected: only the TCP slot was asked. - if registry.viewer_status()["connected"] or registry.ws_viewer_status()["connected"]: + # Only this screen's own session, on either transport. + viewer = self._own_viewer() + if viewer is not None and viewer.connected: self._viewer_badge.set_state("live", _t("rd_quick_connected")) else: self._viewer_badge.set_state( diff --git a/je_auto_control/gui/remote_desktop/host_panel.py b/je_auto_control/gui/remote_desktop/host_panel.py index e1209b46c..17cb1cec5 100644 --- a/je_auto_control/gui/remote_desktop/host_panel.py +++ b/je_auto_control/gui/remote_desktop/host_panel.py @@ -3,7 +3,7 @@ import ssl from typing import Optional -from PySide6.QtCore import QTimer +from PySide6.QtCore import QTimer, Signal from PySide6.QtGui import QGuiApplication, QImage from PySide6.QtWidgets import ( QCheckBox, QComboBox, QFileDialog, QGroupBox, QHBoxLayout, QLabel, @@ -12,7 +12,7 @@ from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui.remote_desktop._helpers import ( - _CollapsibleSection, _StatusBadge, _t, + _CollapsibleSection, _StatusBadge, _t, displaced_notifier, ) from je_auto_control.gui.remote_desktop.frame_display import _FrameDisplay from je_auto_control.utils.remote_desktop import ( @@ -22,7 +22,9 @@ AudioCaptureConfig, is_audio_backend_available, ) from je_auto_control.utils.remote_desktop.host_id import format_host_id -from je_auto_control.utils.remote_desktop.registry import registry +from je_auto_control.utils.remote_desktop.registry import ( + SLOT_HOST, new_owner, registry, +) class _HostPanel(TranslatableMixin, QWidget): @@ -30,9 +32,14 @@ class _HostPanel(TranslatableMixin, QWidget): _PREVIEW_INTERVAL_MS = 250 # 4 fps preview is enough to confirm liveness + # Another owner replaced or stopped the host this panel started. + _displaced = Signal(str, str) + def __init__(self, parent: Optional[QWidget] = None) -> None: super().__init__(parent) self._tr_init() + self._owner = new_owner("host-tab") + self._displaced.connect(self._on_displaced) self._host_id_label = QLabel("---") self._host_id_label.setStyleSheet( "font-family: 'Consolas', 'Menlo', 'Courier New', monospace; " @@ -286,8 +293,9 @@ def _start(self) -> None: host_cls = (WebSocketDesktopHost if transport == "WebSocket" else RemoteDesktopHost) bind = self._bind.text().strip() or "127.0.0.1" - registry.disconnect_viewer() - registry.stop_host() + # Whoever started the running host is told it was replaced. Viewers + # are left alone: they belong to other panels or to a script. + registry.evict(SLOT_HOST, by=self._owner) try: host = host_cls( token=token, @@ -304,7 +312,7 @@ def _start(self) -> None: except (OSError, ValueError, RuntimeError) as error: QMessageBox.warning(self, _t("rd_host_start"), str(error)) return - registry._host = host # noqa: SLF001 centralised lifecycle ownership + registry.adopt(SLOT_HOST, host, self._owner, displaced_notifier(self)) # The transport a viewer picks: with a certificate, TCP is TLS and # WebSocket is WSS, which the share text used to call TCP / WebSocket. if ssl_context is not None: @@ -313,11 +321,21 @@ def _start(self) -> None: self._refresh_status() def _stop(self) -> None: + # The badge shows the machine's host whoever started it, so Stop stops + # that one: a visible "running" with a Stop that does nothing would be + # worse on a remote-access surface. Its owner is told. try: - registry.stop_host() + registry.evict(SLOT_HOST, by=self._owner) except (OSError, RuntimeError) as error: QMessageBox.warning(self, _t("rd_host_stop"), str(error)) return + self._shared = None + self._refresh_status() + + def _on_displaced(self, _slot: str, _by: str) -> None: + """GUI thread: the host this panel started was replaced or stopped.""" + if registry.owned(SLOT_HOST, self._owner) is None: + self._shared = None self._refresh_status() def _refresh_status(self) -> None: diff --git a/je_auto_control/gui/remote_desktop/viewer_panel.py b/je_auto_control/gui/remote_desktop/viewer_panel.py index 78b30ce33..e61d777dd 100644 --- a/je_auto_control/gui/remote_desktop/viewer_panel.py +++ b/je_auto_control/gui/remote_desktop/viewer_panel.py @@ -15,7 +15,8 @@ from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui.remote_desktop._helpers import ( _CollapsibleSection, _StatusBadge, _build_insecure_client_context, - _build_verifying_client_context, _t, wire_remote_input, + _build_verifying_client_context, _t, displaced_notifier, + wire_remote_input, ) from je_auto_control.gui.remote_desktop.remote_screen_window import ( RemoteScreenWindow, @@ -30,7 +31,9 @@ from je_auto_control.utils.remote_desktop.host_id import ( HostIdError, parse_host_id, ) -from je_auto_control.utils.remote_desktop.registry import registry +from je_auto_control.utils.remote_desktop.registry import ( + SLOT_VIEWER, new_owner, registry, +) class _ViewerPanel(TranslatableMixin, QWidget): @@ -42,10 +45,15 @@ class _ViewerPanel(TranslatableMixin, QWidget): _clipboard_signal = Signal(str, object) _file_progress_signal = Signal(str, int, int) _file_complete_signal = Signal(str, bool, str, str) + # Another owner took the viewer slot; emitted from that owner's thread. + _displaced = Signal(str, str) def __init__(self, parent: Optional[QWidget] = None) -> None: super().__init__(parent) self._tr_init() + # This panel's name in the registry: it only ever reads, drives and + # disconnects the viewer it opened itself. + self._owner = new_owner("viewer-tab") self._host_field = QLineEdit("127.0.0.1") self._port = QSpinBox() # 0 is "not entered yet", which _connect refuses; with a minimum of 1 @@ -195,6 +203,7 @@ def _wire_signals(self) -> None: self._clipboard_signal.connect(self._on_clipboard_main) self._file_progress_signal.connect(self._on_file_progress_main) self._file_complete_signal.connect(self._on_file_complete_main) + self._displaced.connect(self._on_displaced) # Input-forwarding signals come from the popup window (see # _ensure_screen_window). They aren't wired here because the # window is created lazily on connect. @@ -220,7 +229,7 @@ def _connect(self) -> None: viewer_cls = (WebSocketDesktopViewer if transport in ("WebSocket", "WSS") else RemoteDesktopViewer) - registry.disconnect_viewer() + registry.evict(SLOT_VIEWER, by=self._owner) try: viewer = viewer_cls( host=host, port=port, token=token, @@ -246,7 +255,7 @@ def _connect(self) -> None: except (OSError, RuntimeError, ValueError, AutoControlException) as error: QMessageBox.warning(self, _t("rd_viewer_connect"), str(error)) return - registry._viewer = viewer # noqa: SLF001 centralised lifecycle ownership + registry.adopt(SLOT_VIEWER, viewer, self._owner, displaced_notifier(self)) self._connected = True self._start_audio_player_if_requested() # AnyDesk-style: open the live screen in its own window so the @@ -294,8 +303,19 @@ def _stop_audio_player(self) -> None: except (OSError, RuntimeError): pass + def _own_viewer(self): + """The viewer this panel opened, or None once it is gone or replaced.""" + return registry.owned(SLOT_VIEWER, self._owner) + + def _on_displaced(self, _slot: str, _by: str) -> None: + """GUI thread: another panel or a script took this panel's session.""" + if self._own_viewer() is not None: + return # reconnected since; the notice is about the old session + self._disconnect() + self._status.setText(_t("rd_viewer_displaced")) + def _disconnect(self) -> None: - registry.disconnect_viewer() + registry.release(SLOT_VIEWER, self._owner) self._stop_audio_player() self._connected = False self._close_screen_window() @@ -342,7 +362,8 @@ def _on_screen_window_closed(self) -> None: self._disconnect() def _refresh_status(self) -> None: - live = self._connected and registry.viewer_status()["connected"] + viewer = self._own_viewer() + live = self._connected and viewer is not None and viewer.connected if live: self._badge.set_state("live", _t("rd_badge_live")) else: @@ -424,7 +445,7 @@ def _on_file_complete_main(self, transfer_id: str, success: bool, # --- input forwarding --------------------------------------------- def _send(self, action: dict) -> None: - viewer = registry.viewer + viewer = self._own_viewer() if viewer is None or not viewer.connected: return try: @@ -435,7 +456,7 @@ def _send(self, action: dict) -> None: # --- clipboard / file transfer (viewer -> host) ------------------- def _push_clipboard_to_host(self) -> None: - viewer = registry.viewer + viewer = self._own_viewer() if viewer is None or not viewer.connected: QMessageBox.warning(self, _t("rd_viewer_push_clipboard"), _t("rd_viewer_status_idle")) @@ -453,7 +474,7 @@ def _push_clipboard_to_host(self) -> None: self._status.setText(_t("rd_clipboard_sent")) def _on_send_file_clicked(self) -> None: - viewer = registry.viewer + viewer = self._own_viewer() if viewer is None or not viewer.connected: QMessageBox.warning(self, _t("rd_viewer_send_file"), _t("rd_viewer_status_idle")) @@ -466,7 +487,7 @@ def _on_send_file_clicked(self) -> None: self._upload_file(source) def _on_files_dropped(self, paths) -> None: - viewer = registry.viewer + viewer = self._own_viewer() if viewer is None or not viewer.connected: return for path in paths: @@ -482,7 +503,7 @@ def _upload_file(self, source_path: str) -> None: ) if not ok or not dest: return - viewer = registry.viewer + viewer = self._own_viewer() if viewer is None: return thread = _FileSendThread(viewer, source_path, dest, self) diff --git a/je_auto_control/utils/remote_desktop/registry.py b/je_auto_control/utils/remote_desktop/registry.py index 4fffddbeb..941f56c85 100644 --- a/je_auto_control/utils/remote_desktop/registry.py +++ b/je_auto_control/utils/remote_desktop/registry.py @@ -8,10 +8,35 @@ (``WebSocketDesktop*``) and WebRTC (``WebRTCDesktop*``); each has its own host + viewer slot so JSON scripts can stand up, e.g., a TCP host and a WebRTC viewer in the same process if they want to. + +Ownership. The TCP and WebSocket slots are shared with the GUI panels, so +each occupant is recorded with the *owner* that put it there: + +* ``AC_remote_*`` / ``AC_ws_*`` commands, the MCP ``ac_remote_*`` tools and + any caller of the plain ``start_*`` / ``connect_*`` / ``stop_*`` / + ``disconnect_*`` methods are the :data:`SCRIPT_OWNER`. They keep their + documented meaning: they act on *the* active host or viewer of that + transport whoever opened it, and the ``*_status`` dictionaries say who + that is under ``"owner"``. +* A GUI panel takes a token from :func:`new_owner`, hands its host or + viewer over with :meth:`_RemoteDesktopRegistry.adopt`, reads it back with + :meth:`_RemoteDesktopRegistry.owned` and closes it with + :meth:`_RemoteDesktopRegistry.release` - all three ignore a slot some + other owner holds. + +A slot still holds one occupant. When another owner replaces or closes it +(:meth:`_RemoteDesktopRegistry.adopt`, :meth:`_RemoteDesktopRegistry.evict`, +or the script-side methods above), the previous owner's ``on_displaced`` +callback is called with ``(slot, by)`` so it can drop its window and state. +The callback runs on the thread that did the replacing. """ +import itertools import ssl -from typing import Any, Callable, Dict, Optional, Sequence +import threading +from typing import Any, Callable, Dict, NamedTuple, Optional, Sequence +from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.remote_desktop.host import RemoteDesktopHost from je_auto_control.utils.remote_desktop.viewer import RemoteDesktopViewer from je_auto_control.utils.remote_desktop.ws_host import WebSocketDesktopHost @@ -21,6 +46,36 @@ FrameCallback = Callable[[bytes], None] ErrorCallback = Callable[[Exception], None] +#: ``on_displaced(slot, by)``: ``slot`` is the slot that was taken away and +#: ``by`` the owner that took it. +DisplacedCallback = Callable[[str, str], None] + +#: Owner recorded for everything the executor commands and MCP tools open. +SCRIPT_OWNER = "script" + +SLOT_HOST = "host" +SLOT_VIEWER = "viewer" +SLOT_WS_HOST = "ws_host" +SLOT_WS_VIEWER = "ws_viewer" +_SLOT_ATTRS = { + SLOT_HOST: "_host", SLOT_VIEWER: "_viewer", + SLOT_WS_HOST: "_ws_host", SLOT_WS_VIEWER: "_ws_viewer", +} +_HOST_SLOTS = (SLOT_HOST, SLOT_WS_HOST) +_owner_ids = itertools.count(1) + + +def new_owner(label: str) -> str: + """Return an owner token unique in this process, e.g. ``"viewer-tab#3"``.""" + return f"{label}#{next(_owner_ids)}" + + +class _Claim(NamedTuple): + """Who put ``resource`` in a slot and how to tell them it is gone.""" + + resource: Any + owner: str + on_displaced: Optional[DisplacedCallback] def _load_webrtc_classes(): @@ -50,6 +105,162 @@ def __init__(self) -> None: self._ws_viewer: Optional[WebSocketDesktopViewer] = None self._webrtc_host: Optional[Any] = None # WebRTCDesktopHost self._webrtc_viewer: Optional[Any] = None # WebRTCDesktopViewer + self._claims: Dict[str, _Claim] = {} + self._lock = threading.Lock() + + # ------------------------------------------------------------------ + # Ownership of the TCP / WebSocket slots + # ------------------------------------------------------------------ + + @staticmethod + def _attr(slot: str) -> str: + try: + return _SLOT_ATTRS[slot] + except KeyError: + raise AutoControlException( + f"unknown remote desktop slot {slot!r}; " + f"expected one of {sorted(_SLOT_ATTRS)}" + ) from None + + def _claim_of(self, slot: str) -> Optional[_Claim]: + """Return the slot's occupant with its owner; call with the lock held. + + An occupant nobody adopted (a test that set the attribute directly) + belongs to :data:`SCRIPT_OWNER`. + """ + resource = getattr(self, self._attr(slot)) + if resource is None: + return None + claim = self._claims.get(slot) + if claim is not None and claim.resource is resource: + return claim + return _Claim(resource, SCRIPT_OWNER, None) + + def _take(self, slot: str, + only_owner: Optional[str] = None) -> Optional[_Claim]: + """Empty ``slot`` and return what it held, or None if left alone.""" + with self._lock: + claim = self._claim_of(slot) + if claim is None: + return None + if only_owner is not None and claim.owner != only_owner: + return None + setattr(self, self._attr(slot), None) + self._claims.pop(slot, None) + return claim + + def _shut(self, slot: str, claim: _Claim, timeout: float, by: str) -> None: + """Stop a displaced occupant, then tell its owner unless it asked.""" + try: + if slot in _HOST_SLOTS: + claim.resource.stop(timeout=timeout) + else: + claim.resource.disconnect(timeout=timeout) + finally: + if claim.owner != by: + self._notify(slot, claim, by) + + @staticmethod + def _notify(slot: str, claim: _Claim, by: str) -> None: + callback = claim.on_displaced + if callback is None: + return + try: + callback(slot, by) + except Exception as error: # noqa: BLE001 # pylint: disable=broad-except # reason: a displaced owner's callback must not undo the new owner's connect + autocontrol_logger.warning( + "remote desktop: %s owner %s failed handling displacement by %s: %r", + slot, claim.owner, by, error, + ) + + def adopt(self, slot: str, resource: Any, owner: str, + on_displaced: Optional[DisplacedCallback] = None, + timeout: float = 2.0) -> None: + """Put an already started host / connected viewer in ``slot`` for ``owner``. + + Whatever the slot held is stopped; its owner, when it is someone + else, has ``on_displaced(slot, owner)`` called. + """ + attr = self._attr(slot) + with self._lock: + previous = self._claim_of(slot) + setattr(self, attr, resource) + self._claims[slot] = _Claim(resource, owner, on_displaced) + if previous is not None and previous.resource is not resource: + self._shut(slot, previous, timeout, by=owner) + + def evict(self, slot: str, by: str, timeout: float = 2.0) -> bool: + """Close whatever ``slot`` holds, whoever owns it; True if it held anything. + + The owner is told through ``on_displaced(slot, by)`` unless it is + ``by`` itself. Panels call this before starting their own host or + viewer, so the old one has let go of its port or its seat first. + """ + claim = self._take(slot) + if claim is None: + return False + self._shut(slot, claim, timeout, by=by) + return True + + def release(self, slot: str, owner: str, timeout: float = 2.0) -> bool: + """Close ``slot`` only if ``owner`` holds it; True if it was closed.""" + claim = self._take(slot, only_owner=owner) + if claim is None: + return False + self._shut(slot, claim, timeout, by=owner) + return True + + def owner_of(self, slot: str) -> Optional[str]: + """Return the owner of ``slot``'s occupant, or None when it is empty.""" + with self._lock: + claim = self._claim_of(slot) + return None if claim is None else claim.owner + + def owned(self, slot: str, owner: str) -> Optional[Any]: + """Return ``slot``'s host / viewer if ``owner`` holds it, else None.""" + with self._lock: + claim = self._claim_of(slot) + if claim is None or claim.owner != owner: + return None + return claim.resource + + def _close(self, slot: str, owner: Optional[str], timeout: float) -> None: + if owner is None: + self.evict(slot, by=SCRIPT_OWNER, timeout=timeout) + else: + self.release(slot, owner, timeout=timeout) + + def _host_status(self, slot: str) -> Dict[str, Any]: + with self._lock: + claim = self._claim_of(slot) + if claim is None: + return { + "running": False, "port": 0, "connected_clients": 0, + "host_id": None, "owner": None, + } + host = claim.resource + return { + "running": host.is_running, + "port": host.port, + "connected_clients": host.connected_clients, + "host_id": host.host_id, + "owner": claim.owner, + } + + def _viewer_status(self, slot: str) -> Dict[str, Any]: + with self._lock: + claim = self._claim_of(slot) + if claim is None: + return {"connected": False, "host_id": None, "owner": None} + return { + "connected": claim.resource.connected, + "host_id": claim.resource.remote_host_id, + "owner": claim.owner, + } + + # ------------------------------------------------------------------ + # TCP transport + # ------------------------------------------------------------------ @property def host(self) -> Optional[RemoteDesktopHost]: @@ -78,29 +289,22 @@ def start_host(self, token: str, host_id=host_id, ssl_context=ssl_context, ) host.start() - self._host = host + self.adopt(SLOT_HOST, host, SCRIPT_OWNER) return self.host_status() - def stop_host(self, timeout: float = 2.0) -> Dict[str, Any]: - """Stop the active host (if any) and clear the slot.""" - if self._host is not None: - self._host.stop(timeout=timeout) - self._host = None + def stop_host(self, timeout: float = 2.0, + owner: Optional[str] = None) -> Dict[str, Any]: + """Stop the active host (if any) and clear the slot. + + With ``owner`` the host is stopped only if that owner started it. + Without, it is stopped whoever started it and that owner is told. + """ + self._close(SLOT_HOST, owner, timeout) return self.host_status() def host_status(self) -> Dict[str, Any]: - host = self._host - if host is None: - return { - "running": False, "port": 0, "connected_clients": 0, - "host_id": None, - } - return { - "running": host.is_running, - "port": host.port, - "connected_clients": host.connected_clients, - "host_id": host.host_id, - } + """Describe the active host; ``"owner"`` names who started it.""" + return self._host_status(SLOT_HOST) def connect_viewer(self, host: str, port: int, token: str, timeout: float = 5.0, @@ -127,30 +331,33 @@ def connect_viewer(self, host: str, port: int, token: str, server_hostname=server_hostname, ) viewer.connect(timeout=float(timeout)) - self._viewer = viewer + self.adopt(SLOT_VIEWER, viewer, SCRIPT_OWNER) return self.viewer_status() - def disconnect_viewer(self, timeout: float = 2.0) -> Dict[str, Any]: - """Disconnect the active viewer (if any) and clear the slot.""" - if self._viewer is not None: - self._viewer.disconnect(timeout=timeout) - self._viewer = None + def disconnect_viewer(self, timeout: float = 2.0, + owner: Optional[str] = None) -> Dict[str, Any]: + """Disconnect the active viewer (if any) and clear the slot. + + With ``owner`` the viewer is disconnected only if that owner opened + it. Without, it is disconnected whoever opened it and that owner is + told. + """ + self._close(SLOT_VIEWER, owner, timeout) return self.viewer_status() def viewer_status(self) -> Dict[str, Any]: - viewer = self._viewer - if viewer is None: - return {"connected": False, "host_id": None} - return { - "connected": viewer.connected, - "host_id": viewer.remote_host_id, - } + """Describe the active viewer; ``"owner"`` names who opened it.""" + return self._viewer_status(SLOT_VIEWER) def send_input(self, action: Dict[str, Any]) -> Dict[str, Any]: - """Forward ``action`` through the connected viewer, raise if offline.""" - if self._viewer is None or not self._viewer.connected: + """Forward ``action`` through the connected viewer, raise if offline. + + The viewer is the active one, whoever opened it. + """ + viewer = self._viewer + if viewer is None or not viewer.connected: raise ConnectionError("no remote viewer is connected") - self._viewer.send_input(action) + viewer.send_input(action) return {"sent": True} # ------------------------------------------------------------------ @@ -176,28 +383,18 @@ def start_ws_host(self, token: str, host_id=host_id, ssl_context=ssl_context, ) host.start() - self._ws_host = host + self.adopt(SLOT_WS_HOST, host, SCRIPT_OWNER) return self.ws_host_status() - def stop_ws_host(self, timeout: float = 2.0) -> Dict[str, Any]: - if self._ws_host is not None: - self._ws_host.stop(timeout=timeout) - self._ws_host = None + def stop_ws_host(self, timeout: float = 2.0, + owner: Optional[str] = None) -> Dict[str, Any]: + """Stop the WS host; with ``owner``, only if that owner started it.""" + self._close(SLOT_WS_HOST, owner, timeout) return self.ws_host_status() def ws_host_status(self) -> Dict[str, Any]: - host = self._ws_host - if host is None: - return { - "running": False, "port": 0, "connected_clients": 0, - "host_id": None, - } - return { - "running": host.is_running, - "port": host.port, - "connected_clients": host.connected_clients, - "host_id": host.host_id, - } + """Describe the active WS host; ``"owner"`` names who started it.""" + return self._host_status(SLOT_WS_HOST) def connect_ws_viewer(self, host: str, port: int, token: str, path: str = "/", @@ -219,28 +416,25 @@ def connect_ws_viewer(self, host: str, port: int, token: str, path=path, ) viewer.connect(timeout=float(timeout)) - self._ws_viewer = viewer + self.adopt(SLOT_WS_VIEWER, viewer, SCRIPT_OWNER) return self.ws_viewer_status() - def disconnect_ws_viewer(self, timeout: float = 2.0) -> Dict[str, Any]: - if self._ws_viewer is not None: - self._ws_viewer.disconnect(timeout=timeout) - self._ws_viewer = None + def disconnect_ws_viewer(self, timeout: float = 2.0, + owner: Optional[str] = None) -> Dict[str, Any]: + """Disconnect the WS viewer; with ``owner``, only if that owner opened it.""" + self._close(SLOT_WS_VIEWER, owner, timeout) return self.ws_viewer_status() def ws_viewer_status(self) -> Dict[str, Any]: - viewer = self._ws_viewer - if viewer is None: - return {"connected": False, "host_id": None} - return { - "connected": viewer.connected, - "host_id": viewer.remote_host_id, - } + """Describe the active WS viewer; ``"owner"`` names who opened it.""" + return self._viewer_status(SLOT_WS_VIEWER) def ws_send_input(self, action: Dict[str, Any]) -> Dict[str, Any]: - if self._ws_viewer is None or not self._ws_viewer.connected: + """Forward ``action`` through the active WS viewer, whoever opened it.""" + viewer = self._ws_viewer + if viewer is None or not viewer.connected: raise ConnectionError("no websocket viewer is connected") - self._ws_viewer.send_input(action) + viewer.send_input(action) return {"sent": True} # ------------------------------------------------------------------ diff --git a/test/unit_test/headless/test_rd_gui_audit.py b/test/unit_test/headless/test_rd_gui_audit.py index e6761b436..3a178fad7 100644 --- a/test/unit_test/headless/test_rd_gui_audit.py +++ b/test/unit_test/headless/test_rd_gui_audit.py @@ -64,8 +64,9 @@ def test_quick_connect_dials_wss_with_a_verifying_tls_context(qapp, monkeypatch) def test_the_quick_connect_popup_forwards_input(qapp, monkeypatch): viewer = _FakeViewer() - monkeypatch.setattr(registry, "_ws_viewer", viewer) + monkeypatch.setattr(registry, "_ws_viewer", None) # restored after the test screen = connection_screen.QuickConnectScreen() + registry.adopt("ws_viewer", viewer, screen._owner) # the session this screen opened screen._open_screen_window("desk") window = screen._screen_window window.mouse_pressed.emit(3, 4, "mouse_left") diff --git a/test/unit_test/headless/test_rd_webrtc_audit.py b/test/unit_test/headless/test_rd_webrtc_audit.py index 37deae6e9..619c572ea 100644 --- a/test/unit_test/headless/test_rd_webrtc_audit.py +++ b/test/unit_test/headless/test_rd_webrtc_audit.py @@ -174,8 +174,9 @@ def disconnect(self, timeout=2.0): def test_quick_connect_treats_a_ws_session_as_a_session(monkeypatch): viewer = _WsViewer() - monkeypatch.setattr(registry, "_ws_viewer", viewer) + monkeypatch.setattr(registry, "_ws_viewer", None) # restored after the test screen = connection_screen.QuickConnectScreen() + registry.adopt("ws_viewer", viewer, screen._owner) # the session this screen opened screen._refresh_viewer_status() assert screen._viewer_badge.text() == _t("rd_quick_connected") screen._on_window_closed() @@ -184,8 +185,9 @@ def test_quick_connect_treats_a_ws_session_as_a_session(monkeypatch): def test_a_session_error_ends_the_quick_connect_session(monkeypatch, qapp): viewer = _WsViewer() - monkeypatch.setattr(registry, "_ws_viewer", viewer) + monkeypatch.setattr(registry, "_ws_viewer", None) # restored after the test screen = connection_screen.QuickConnectScreen() + registry.adopt("ws_viewer", viewer, screen._owner) # the session this screen opened screen._open_screen_window("desk") screen._on_error("connection reset") assert viewer.disconnected and screen._screen_window is None diff --git a/test/unit_test/headless/test_remote_registry_owners.py b/test/unit_test/headless/test_remote_registry_owners.py new file mode 100644 index 000000000..43ce06ff4 --- /dev/null +++ b/test/unit_test/headless/test_remote_registry_owners.py @@ -0,0 +1,377 @@ +"""Who owns the remote-desktop registry's host and viewer slots (fakes, no Qt). + +Quick Connect, the viewer tab and ``AC_remote_connect`` wrote the same slot +and each cleared it first, so one side's connect cut the other's session +without telling it, and its Disconnect then cut a session it never opened. +""" +import importlib +import threading + +import pytest + +from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.executor.action_executor import executor +from je_auto_control.utils.remote_desktop.registry import ( + SCRIPT_OWNER, SLOT_HOST, SLOT_VIEWER, SLOT_WS_HOST, SLOT_WS_VIEWER, + _RemoteDesktopRegistry, new_owner, +) + +# The package rebinds its ``registry`` attribute to the singleton, so the +# module itself has to be asked for by name. +registry_module = importlib.import_module("je_auto_control.utils.remote_desktop.registry") + +VIEWER_SLOTS = (SLOT_VIEWER, SLOT_WS_VIEWER) +HOST_SLOTS = (SLOT_HOST, SLOT_WS_HOST) +ALL_SLOTS = VIEWER_SLOTS + HOST_SLOTS + + +class _Viewer: + remote_host_id = "123456789" + + def __init__(self, **kwargs): + self.kwargs, self.connected, self.sent = kwargs, False, [] + + def connect(self, timeout=5.0): + self.connected = True + + def disconnect(self, timeout=2.0): + self.connected = False + + def send_input(self, action): + self.sent.append(action) + + +class _Host: + port, connected_clients, host_id = 4321, 0, "987654321" + + def __init__(self, **kwargs): + self.kwargs, self.is_running = kwargs, False + + def start(self): + self.is_running = True + + def stop(self, timeout=2.0): + self.is_running = False + + +def _live(slot): + """A started fake of the kind ``slot`` holds.""" + if slot in HOST_SLOTS: + made = _Host() + made.start() + else: + made = _Viewer() + made.connect() + return made + + +def _is_up(resource): + return getattr(resource, "is_running", None) or getattr(resource, "connected", False) + + +@pytest.fixture +def reg(monkeypatch): + monkeypatch.setattr(registry_module, "RemoteDesktopViewer", _Viewer) + monkeypatch.setattr(registry_module, "WebSocketDesktopViewer", _Viewer) + monkeypatch.setattr(registry_module, "RemoteDesktopHost", _Host) + monkeypatch.setattr(registry_module, "WebSocketDesktopHost", _Host) + return _RemoteDesktopRegistry() + + +class _Panel: + """A GUI panel as the registry sees it: an owner token and a callback.""" + + def __init__(self, label="panel"): + self.owner = new_owner(label) + self.displaced = [] + + def on_displaced(self, slot, by): + self.displaced.append((slot, by)) + + +_SCRIPT_OPEN = { + SLOT_VIEWER: lambda reg: reg.connect_viewer("h", 1, "t"), + SLOT_WS_VIEWER: lambda reg: reg.connect_ws_viewer("h", 1, "t"), + SLOT_HOST: lambda reg: reg.start_host("t"), + SLOT_WS_HOST: lambda reg: reg.start_ws_host("t"), +} +_SCRIPT_CLOSE = { + SLOT_VIEWER: "disconnect_viewer", SLOT_WS_VIEWER: "disconnect_ws_viewer", + SLOT_HOST: "stop_host", SLOT_WS_HOST: "stop_ws_host", +} +_STATUS = { + SLOT_VIEWER: "viewer_status", SLOT_WS_VIEWER: "ws_viewer_status", + SLOT_HOST: "host_status", SLOT_WS_HOST: "ws_host_status", +} + + +def _occupant(reg, slot): + return getattr(reg, "_" + slot) + + +# --- owner tokens ------------------------------------------------------------------------------------------ + +def test_owner_tokens_are_unique_and_never_the_script_owner(): + first, second = new_owner("viewer-tab"), new_owner("viewer-tab") + assert first != second and SCRIPT_OWNER not in (first, second) + assert first.startswith("viewer-tab#") + + +@pytest.mark.parametrize("call", ["owner_of", "owned", "evict", "release", "adopt"]) +def test_an_unknown_slot_is_refused(reg, call): + args = {"owner_of": (), "owned": ("me",), "evict": ("me",), "release": ("me",), + "adopt": (object(), "me")}[call] + with pytest.raises(AutoControlException): + getattr(reg, call)("webrtc", *args) + + +# --- a panel's own session --------------------------------------------------------------------------------- + +@pytest.mark.parametrize("slot", ALL_SLOTS) +def test_a_panel_reads_and_closes_what_it_adopted(reg, slot): + panel, made = _Panel(), _live(slot) + reg.adopt(slot, made, panel.owner, panel.on_displaced) + assert reg.owner_of(slot) == panel.owner + assert reg.owned(slot, panel.owner) is made + assert getattr(reg, _STATUS[slot])()["owner"] == panel.owner + assert reg.release(slot, panel.owner) is True + assert not _is_up(made) and reg.owner_of(slot) is None + assert panel.displaced == [] # it asked for this itself + + +@pytest.mark.parametrize("slot", ALL_SLOTS) +def test_a_panel_reconnecting_is_not_told_it_was_displaced(reg, slot): + panel, first, second = _Panel(), _live(slot), _live(slot) + reg.adopt(slot, first, panel.owner, panel.on_displaced) + assert reg.evict(slot, by=panel.owner) is True + reg.adopt(slot, second, panel.owner, panel.on_displaced) + reg.adopt(slot, _live(slot), panel.owner, panel.on_displaced) # replaced without evicting first + assert not _is_up(first) and not _is_up(second) + assert panel.displaced == [] + + +def test_adopting_the_same_object_again_does_not_close_it(reg): + panel, viewer = _Panel(), _live(SLOT_VIEWER) + reg.adopt(SLOT_VIEWER, viewer, panel.owner, panel.on_displaced) + reg.adopt(SLOT_VIEWER, viewer, panel.owner, panel.on_displaced) + assert viewer.connected and reg.owned(SLOT_VIEWER, panel.owner) is viewer + + +# --- panel against panel ------------------------------------------------------------------------------------ + +@pytest.mark.parametrize("slot", ALL_SLOTS) +def test_one_panel_cannot_see_or_close_another_panels_session(reg, slot): + first, second, made = _Panel("quick-connect"), _Panel("viewer-tab"), _live(slot) + reg.adopt(slot, made, first.owner, first.on_displaced) + assert reg.owned(slot, second.owner) is None + assert reg.release(slot, second.owner) is False + getattr(reg, _SCRIPT_CLOSE[slot])(owner=second.owner) + assert _is_up(made) and reg.owned(slot, first.owner) is made + assert first.displaced == [] + + +@pytest.mark.parametrize("slot", ALL_SLOTS) +@pytest.mark.parametrize("evict_first", [True, False]) +def test_a_panel_replaced_by_another_panel_is_told(reg, slot, evict_first): + first, second, old, new = _Panel("quick-connect"), _Panel("viewer-tab"), _live(slot), _live(slot) + reg.adopt(slot, old, first.owner, first.on_displaced) + if evict_first: + reg.evict(slot, by=second.owner) + reg.adopt(slot, new, second.owner, second.on_displaced) + assert not _is_up(old) and _is_up(new) + assert first.displaced == [(slot, second.owner)] and second.displaced == [] + # The replaced panel's Disconnect no longer reaches the new session. + assert reg.release(slot, first.owner) is False and _is_up(new) + assert reg.owned(slot, first.owner) is None + + +def test_the_two_viewer_transports_do_not_displace_each_other(reg): + tcp, ws = _Panel(), _Panel() + reg.adopt(SLOT_VIEWER, _live(SLOT_VIEWER), tcp.owner, tcp.on_displaced) + reg.adopt(SLOT_WS_VIEWER, _live(SLOT_WS_VIEWER), ws.owner, ws.on_displaced) + reg.evict(SLOT_WS_VIEWER, by=tcp.owner) + assert tcp.displaced == [] and ws.displaced == [(SLOT_WS_VIEWER, tcp.owner)] + assert reg.owned(SLOT_VIEWER, tcp.owner).connected + + +# --- script against panel ----------------------------------------------------------------------------------- + +@pytest.mark.parametrize("slot", ALL_SLOTS) +def test_a_script_connect_replaces_a_panels_session_and_tells_it(reg, slot): + panel, old = _Panel(), _live(slot) + reg.adopt(slot, old, panel.owner, panel.on_displaced) + status = _SCRIPT_OPEN[slot](reg) + assert status["owner"] == SCRIPT_OWNER and not _is_up(old) + assert panel.displaced == [(slot, SCRIPT_OWNER)] + assert reg.release(slot, panel.owner) is False and _is_up(_occupant(reg, slot)) + + +@pytest.mark.parametrize("slot", ALL_SLOTS) +def test_a_script_close_ends_a_panels_session_and_tells_it(reg, slot): + panel, made = _Panel(), _live(slot) + reg.adopt(slot, made, panel.owner, panel.on_displaced) + status = getattr(reg, _SCRIPT_CLOSE[slot])() + assert status["owner"] is None and not _is_up(made) + assert panel.displaced == [(slot, SCRIPT_OWNER)] + + +@pytest.mark.parametrize("slot", ALL_SLOTS) +def test_a_panel_replaces_a_scripts_session(reg, slot): + _SCRIPT_OPEN[slot](reg) + scripted, panel = _occupant(reg, slot), _Panel() + assert reg.owned(slot, panel.owner) is None + assert reg.release(slot, panel.owner) is False and _is_up(scripted) + reg.evict(slot, by=panel.owner) + reg.adopt(slot, _live(slot), panel.owner, panel.on_displaced) + assert not _is_up(scripted) and reg.owner_of(slot) == panel.owner + + +@pytest.mark.parametrize(("slot", "send"), [(SLOT_VIEWER, "send_input"), (SLOT_WS_VIEWER, "ws_send_input")]) +def test_script_input_goes_to_the_active_viewer_whoever_opened_it(reg, slot, send): + panel, viewer = _Panel(), _live(slot) + reg.adopt(slot, viewer, panel.owner, panel.on_displaced) + assert getattr(reg, send)({"action": "type", "text": "a"}) == {"sent": True} + assert viewer.sent == [{"action": "type", "text": "a"}] + reg.release(slot, panel.owner) + with pytest.raises(ConnectionError): + getattr(reg, send)({"action": "type", "text": "b"}) + + +# --- script alone: unchanged --------------------------------------------------------------------------------- + +@pytest.mark.parametrize("slot", ALL_SLOTS) +def test_a_script_only_session_behaves_as_before(reg, slot): + first_status = _SCRIPT_OPEN[slot](reg) + first = _occupant(reg, slot) + assert first_status["owner"] == SCRIPT_OWNER and _is_up(first) + _SCRIPT_OPEN[slot](reg) # a second connect replaces the first + second = _occupant(reg, slot) + assert second is not first and not _is_up(first) and _is_up(second) + closed = getattr(reg, _SCRIPT_CLOSE[slot])() + assert not _is_up(second) and _occupant(reg, slot) is None + assert closed["owner"] is None and not (closed.get("connected") or closed.get("running")) + getattr(reg, _SCRIPT_CLOSE[slot])() # closing nothing is still fine + + +def test_status_keeps_its_old_keys(reg): + assert reg.viewer_status() == {"connected": False, "host_id": None, "owner": None} + assert reg.host_status() == {"running": False, "port": 0, "connected_clients": 0, + "host_id": None, "owner": None} + reg.start_host("t") + reg.connect_viewer("h", 1, "t") + assert reg.host_status() == {"running": True, "port": 4321, "connected_clients": 0, + "host_id": "987654321", "owner": SCRIPT_OWNER} + assert reg.viewer_status() == {"connected": True, "host_id": "123456789", "owner": SCRIPT_OWNER} + + +def test_an_occupant_set_on_the_attribute_counts_as_the_scripts(reg): + panel, stale, injected = _Panel(), _live(SLOT_VIEWER), _live(SLOT_VIEWER) + reg.adopt(SLOT_VIEWER, stale, panel.owner, panel.on_displaced) + reg._viewer = injected # what older tests and callers do + assert reg.owner_of(SLOT_VIEWER) == SCRIPT_OWNER + assert reg.owned(SLOT_VIEWER, panel.owner) is None + reg.disconnect_viewer() + assert not injected.connected and panel.displaced == [] + + +# --- notification delivery -------------------------------------------------------------------------------- + +def test_a_failing_displaced_callback_does_not_undo_the_new_connect(reg): + def explode(slot, by): + raise RuntimeError("the panel is gone") + reg.adopt(SLOT_VIEWER, _live(SLOT_VIEWER), new_owner("panel"), explode) + assert reg.connect_viewer("h", 1, "t")["connected"] is True + + +def test_the_owner_is_told_even_when_closing_its_session_raises(reg): + class Stuck(_Viewer): + def disconnect(self, timeout=2.0): + raise OSError("socket already gone") + panel = _Panel() + reg.adopt(SLOT_VIEWER, Stuck(), panel.owner, panel.on_displaced) + with pytest.raises(OSError): + reg.disconnect_viewer() + assert panel.displaced == [(SLOT_VIEWER, SCRIPT_OWNER)] and reg.viewer is None + + +def test_a_callback_may_call_back_into_the_registry(reg): + seen = [] + owner = new_owner("panel") + reg.adopt(SLOT_VIEWER, _live(SLOT_VIEWER), owner, + lambda slot, by: seen.append((reg.owner_of(slot), reg.owned(slot, owner)))) + done = threading.Thread(target=reg.connect_viewer, args=("h", 1, "t"), daemon=True) + done.start() + done.join(5.0) + assert not done.is_alive(), "the callback ran under the registry lock" + # Told while the slot is empty: the old viewer is closed before the new one dials. + assert seen == [(None, None)] and reg.owner_of(SLOT_VIEWER) == SCRIPT_OWNER + + +# --- through the executor and the MCP handlers ---------------------------------------------------------------- + +@pytest.fixture +def shared(monkeypatch): + """The process-wide registry, emptied, building fakes.""" + shared_registry = registry_module.registry + for name in ("RemoteDesktopViewer", "WebSocketDesktopViewer"): + monkeypatch.setattr(registry_module, name, _Viewer) + for name in ("RemoteDesktopHost", "WebSocketDesktopHost"): + monkeypatch.setattr(registry_module, name, _Host) + for attr in ("_host", "_viewer", "_ws_host", "_ws_viewer"): + monkeypatch.setattr(shared_registry, attr, None) + monkeypatch.setattr(shared_registry, "_claims", {}) + return shared_registry + + +def _run(command, params=None): + action = [command] if params is None else [command, params] + return next(iter(executor.execute_action([action]).values())) + + +def test_ac_remote_commands_alone_work_as_before(shared): + connect = {"host": "h", "port": 1, "token": "t"} + assert _run("AC_start_remote_host", {"token": "t"})["running"] is True + assert _run("AC_remote_connect", connect)["connected"] is True + first = shared.viewer + assert _run("AC_remote_connect", connect)["connected"] is True + assert shared.viewer is not first and not first.connected + assert _run("AC_remote_send_input", {"action": {"action": "type", "text": "x"}}) == {"sent": True} + assert _run("AC_remote_viewer_status")["owner"] == SCRIPT_OWNER + assert _run("AC_remote_disconnect")["connected"] is False + assert _run("AC_stop_remote_host")["running"] is False + assert shared.viewer is None and shared.host is None + + +def test_ac_remote_commands_see_and_end_a_panels_session(shared): + panel, viewer, host = _Panel(), _live(SLOT_VIEWER), _live(SLOT_HOST) + shared.adopt(SLOT_VIEWER, viewer, panel.owner, panel.on_displaced) + shared.adopt(SLOT_HOST, host, panel.owner, panel.on_displaced) + assert _run("AC_remote_viewer_status") == { + "connected": True, "host_id": "123456789", "owner": panel.owner} + assert _run("AC_remote_host_status")["owner"] == panel.owner + _run("AC_remote_send_input", {"action": {"action": "type", "text": "x"}}) + assert viewer.sent == [{"action": "type", "text": "x"}] + _run("AC_remote_disconnect") + _run("AC_stop_remote_host") + assert not viewer.connected and not host.is_running + assert panel.displaced == [(SLOT_VIEWER, SCRIPT_OWNER), (SLOT_HOST, SCRIPT_OWNER)] + + +def test_ac_ws_commands_tell_a_replaced_panel(shared): + panel, viewer = _Panel(), _live(SLOT_WS_VIEWER) + shared.adopt(SLOT_WS_VIEWER, viewer, panel.owner, panel.on_displaced) + assert _run("AC_ws_connect", {"host": "h", "port": 1, "token": "t"})["owner"] == SCRIPT_OWNER + assert not viewer.connected and panel.displaced == [(SLOT_WS_VIEWER, SCRIPT_OWNER)] + assert _run("AC_ws_disconnect")["connected"] is False + + +def test_the_mcp_tools_share_the_script_owner(shared): + from je_auto_control.utils.mcp_server.tools import _handlers_remote as handlers + panel, viewer = _Panel(), _live(SLOT_VIEWER) + shared.adopt(SLOT_VIEWER, viewer, panel.owner, panel.on_displaced) + assert handlers.remote_viewer_status()["owner"] == panel.owner + assert handlers.remote_viewer_connect("h", 1, "t")["owner"] == SCRIPT_OWNER + assert panel.displaced == [(SLOT_VIEWER, SCRIPT_OWNER)] + assert handlers.remote_viewer_disconnect()["connected"] is False + assert handlers.remote_host_start("t")["owner"] == SCRIPT_OWNER + assert handlers.remote_host_stop()["running"] is False diff --git a/test/unit_test/headless/test_remote_registry_owners_gui.py b/test/unit_test/headless/test_remote_registry_owners_gui.py new file mode 100644 index 000000000..d053903bf --- /dev/null +++ b/test/unit_test/headless/test_remote_registry_owners_gui.py @@ -0,0 +1,353 @@ +"""Remote-desktop panels and their own registry slots (offscreen Qt, fakes only). + +Each panel treated the registry's one viewer as its own: a connect elsewhere +left its popup on the last frame, its badge live and its Disconnect pointed at +the other side's session. No test here opens a socket or shows a window. +""" +import gc +import os +import threading + +import pytest + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) + +from PySide6.QtCore import QCoreApplication, QEvent, QThread # noqa: E402 +from PySide6.QtWidgets import QApplication, QMessageBox # noqa: E402 + +from je_auto_control.gui.remote_desktop import connection_screen, host_panel, viewer_panel # noqa: E402 +from je_auto_control.gui.remote_desktop._helpers import _t, displaced_notifier # noqa: E402 +from je_auto_control.gui.remote_desktop.remote_screen_window import RemoteScreenWindow # noqa: E402 +from je_auto_control.utils.remote_desktop.address_book import AddressBook # noqa: E402 +from je_auto_control.utils.remote_desktop.connect_coordinator import parse_target # noqa: E402 +from je_auto_control.utils.remote_desktop.registry import ( # noqa: E402 + SCRIPT_OWNER, SLOT_HOST, SLOT_VIEWER, SLOT_WS_VIEWER, registry, +) + + +class _Viewer: + remote_host_id = None + + def __init__(self, **kwargs): + self.kwargs, self.connected, self.sent = kwargs, False, [] + + def connect(self, timeout=5.0): + self.connected = True + + def disconnect(self, timeout=2.0): + self.connected = False + + def send_input(self, action): + self.sent.append(action) + + def set_file_receiver(self, receiver): + self.receiver = receiver + + +class _Host: + port, connected_clients, host_id = 4321, 0, "123456789" + + def __init__(self, **kwargs): + self.kwargs, self.is_running = kwargs, False + + def start(self): + self.is_running = True + + def stop(self, timeout=2.0): + self.is_running = False + + def latest_frame(self): + return None + + +class _Window(RemoteScreenWindow): + """The popup, never shown: these tests put nothing on a screen.""" + + def show(self): + return None + + def raise_(self): + return None + + def activateWindow(self): # noqa: N802 # reason: Qt override + return None + + +@pytest.fixture(autouse=True) +def qapp(monkeypatch, tmp_path): + app = QApplication.instance() or QApplication([]) + messages = [] + # Every box a slot may open is replaced: a modal exec() would block the run. + for box in ("warning", "information", "question", "critical"): + monkeypatch.setattr(QMessageBox, box, lambda *args: messages.append(args[-1])) + for attr in ("_host", "_viewer", "_ws_host", "_ws_viewer"): + monkeypatch.setattr(registry, attr, None) + monkeypatch.setattr(registry, "_claims", {}) + for module in (connection_screen, viewer_panel): + monkeypatch.setattr(module, "RemoteDesktopViewer", _Viewer) + monkeypatch.setattr(module, "WebSocketDesktopViewer", _Viewer) + monkeypatch.setattr(module, "RemoteScreenWindow", _Window) + monkeypatch.setattr(connection_screen, "RemoteDesktopHost", _Host) + monkeypatch.setattr(host_panel, "RemoteDesktopHost", _Host) + monkeypatch.setattr(host_panel, "WebSocketDesktopHost", _Host) + monkeypatch.setattr(viewer_panel, "is_audio_backend_available", lambda: False) + monkeypatch.setattr(host_panel, "is_audio_backend_available", lambda: False) + book = AddressBook(tmp_path / "book.json") # never the operator's own address book + monkeypatch.setattr(connection_screen, "default_address_book", lambda: book) + app.messages = messages + yield app + + +def _quick(target="tcp"): + screen = connection_screen.QuickConnectScreen() + address = {"tcp": "desk:5555", "ws": "ws://desk:8080/"}[target] + screen._dispatch_target(parse_target(address), "tok") + return screen + + +def _tab(transport="TCP"): + panel = viewer_panel._ViewerPanel() + panel._host_field.setText("desk") + panel._port.setValue(5555) + panel._token.setText("tok") + panel._transport.setCurrentText(transport) + panel._connect() + return panel + + +def _script_connect(monkeypatch): + registry_module = __import__("importlib").import_module(registry.__class__.__module__) + monkeypatch.setattr(registry_module, "RemoteDesktopViewer", _Viewer) + monkeypatch.setattr(registry_module, "RemoteDesktopHost", _Host) + return registry_module + + +def _idle(panel): + return panel._screen_window is None and panel._own_viewer() is None + + +# --- each panel owns what it opened ---------------------------------------------------------------------------- + +def test_each_panel_registers_under_its_own_owner(): + first, second = _quick(), connection_screen.QuickConnectScreen() + assert first._owner != second._owner + assert registry.owner_of(SLOT_VIEWER) == first._owner + assert first._own_viewer().connected and first._screen_window is not None + assert first._viewer_badge.text() == _t("rd_quick_connected") + # A second Quick Connect screen does not take the first one's session for its own. + second._refresh_status() + assert second._own_viewer() is None + assert second._viewer_badge.text() == _t("rd_quick_disconnected") + + +def test_quick_connect_disconnect_leaves_the_viewer_tabs_session_alone(): + tab = _tab() + screen = connection_screen.QuickConnectScreen() + screen._disconnect() + screen._on_window_closed() + screen._on_error("stale error from an earlier session") + assert tab._own_viewer().connected and tab._screen_window is not None + + +def test_the_viewer_tabs_disconnect_leaves_quick_connects_session_alone(): + screen = _quick() + tab = viewer_panel._ViewerPanel() + tab._disconnect() + tab._on_error_main("stale error") + assert screen._own_viewer().connected and screen._screen_window is not None + + +def test_a_panel_only_drives_its_own_viewer(): + screen = _quick() + theirs = screen._own_viewer() + tab = viewer_panel._ViewerPanel() + tab._send({"action": "type", "text": "x"}) + tab._push_clipboard_to_host() + assert theirs.sent == [] + screen._send_input({"action": "type", "text": "y"}) + assert theirs.sent == [{"action": "type", "text": "y"}] + + +# --- being replaced -------------------------------------------------------------------------------------------- + +def test_the_viewer_tab_connecting_closes_quick_connects_session(): + screen = _quick() + old = screen._own_viewer() + tab = _tab() + assert not old.connected and _idle(screen) + assert screen._viewer_badge.text() == _t("rd_quick_disconnected") + # Its Disconnect no longer reaches the session that replaced it. + screen._disconnect() + assert tab._own_viewer().connected and tab._screen_window is not None + + +def test_quick_connect_connecting_closes_the_viewer_tabs_session(): + tab = _tab() + old = tab._own_viewer() + screen = _quick() + assert not old.connected and _idle(tab) and tab._connected is False + assert tab._status.text() == _t("rd_viewer_displaced") + assert tab._badge.text() == _t("rd_badge_idle") + tab._disconnect() + assert screen._own_viewer().connected + + +def test_a_ws_quick_connect_session_survives_a_tcp_connect_elsewhere(): + screen = _quick("ws") + tab = _tab() + assert registry.owner_of(SLOT_WS_VIEWER) == screen._owner + assert screen._own_viewer().connected and screen._screen_window is not None + assert tab._own_viewer().connected + + +def test_quick_connect_changing_transport_ends_its_previous_session(): + screen = _quick("ws") + first = screen._own_viewer() + screen._dispatch_target(parse_target("desk:5555"), "tok") + assert not first.connected and registry.owner_of(SLOT_WS_VIEWER) is None + assert registry.owner_of(SLOT_VIEWER) == screen._owner and screen._screen_window is not None + + +def test_a_panel_reconnecting_keeps_its_window(): + tab = _tab() + first = tab._own_viewer() + tab._connect() + assert not first.connected and tab._own_viewer().connected + assert tab._screen_window is not None and tab._status.text() != _t("rd_viewer_displaced") + + +def test_a_late_notice_about_an_old_session_does_not_close_the_new_one(): + tab = _tab() + tab._on_displaced(SLOT_VIEWER, SCRIPT_OWNER) # queued before the reconnect, delivered after + assert tab._own_viewer().connected and tab._screen_window is not None + screen = _quick("ws") + screen._on_displaced(SLOT_VIEWER, SCRIPT_OWNER) + assert screen._own_viewer().connected and screen._screen_window is not None + + +def test_a_script_connect_closes_the_panels_session(monkeypatch): + _script_connect(monkeypatch) + tab = _tab() + registry.connect_viewer("desk", 5555, "tok") + assert _idle(tab) and tab._status.text() == _t("rd_viewer_displaced") + assert registry.viewer_status()["owner"] == SCRIPT_OWNER + tab._disconnect() + assert registry.viewer_status()["connected"] is True + + +def test_a_script_disconnect_closes_the_panels_session(): + screen = _quick() + registry.disconnect_viewer() + assert _idle(screen) and screen._viewer_badge.text() == _t("rd_quick_disconnected") + + +def test_a_panel_takes_over_a_scripts_session(monkeypatch): + _script_connect(monkeypatch) + registry.connect_viewer("desk", 5555, "tok") + scripted = registry.viewer + screen = connection_screen.QuickConnectScreen() + screen._refresh_status() + assert screen._viewer_badge.text() == _t("rd_quick_disconnected") # not its session + screen._disconnect() + assert scripted.connected + screen._dispatch_target(parse_target("desk:5555"), "tok") + assert not scripted.connected and registry.owner_of(SLOT_VIEWER) == screen._owner + + +# --- hosts ----------------------------------------------------------------------------------------------------- + +def _host_tab(): + panel = host_panel._HostPanel() + panel._token.setText("tok") + panel._start() + return panel + + +def test_starting_a_host_leaves_every_viewer_alone(): + tab, screen = _tab(), _quick("ws") + _host_tab() + connection_screen.QuickConnectScreen()._start_hosting() + assert tab._own_viewer().connected and screen._own_viewer().connected + + +def test_quick_connect_hosting_replaces_the_host_tabs_host_and_tells_it(qapp): + tab = _host_tab() + first = registry.host + assert tab._shared is not None and registry.owner_of(SLOT_HOST) == tab._owner + screen = connection_screen.QuickConnectScreen() + screen._start_hosting() + assert not first.is_running and registry.owner_of(SLOT_HOST) == screen._owner + assert tab._shared is None # no share text for a host it does not run + tab._copy_share_text() + assert qapp.messages == [_t("rd_host_copy_share_unavailable")] and registry.host.is_running + + +def test_stop_stops_the_host_on_show_and_tells_its_owner(): + tab = _host_tab() + running = registry.host + screen = connection_screen.QuickConnectScreen() + assert screen._host_id_label.text() != "---" # it shows the host as running + screen._stop_hosting() + assert not running.is_running and registry.host is None and tab._shared is None + + +def test_a_script_stopping_the_host_clears_the_panels_share_text(): + tab = _host_tab() + registry.stop_host() + assert tab._shared is None and registry.host_status()["running"] is False + + +# --- delivery -------------------------------------------------------------------------------------------------- + +def _pump_until(app, predicate, rounds=200): + for _ in range(rounds): + app.processEvents() + if predicate(): + return True + QThread.msleep(5) + return predicate() + + +def test_a_notice_from_another_thread_is_handled_on_the_gui_thread(qapp, monkeypatch): + tab = _tab() + handled = [] + original = tab._on_displaced + monkeypatch.setattr(tab, "_on_displaced", lambda slot, by: ( + handled.append(threading.current_thread()), original(slot, by))) + tab._displaced.disconnect() + tab._displaced.connect(tab._on_displaced) + worker = threading.Thread(target=registry.disconnect_viewer, daemon=True) + worker.start() + worker.join(5.0) + assert handled == [] and tab._screen_window is not None # nothing touched off the GUI thread + assert _pump_until(qapp, lambda: bool(handled)) + assert handled == [threading.main_thread()] and _idle(tab) + + +def test_a_notice_for_a_destroyed_panel_is_dropped(qapp): + import shiboken6 + tab = _tab() + notify = displaced_notifier(tab) + registry.release(SLOT_VIEWER, tab._owner) + tab._close_screen_window() + shiboken6.delete(tab) # the Qt object goes, the wrapper stays + notify(SLOT_VIEWER, SCRIPT_OWNER) + del tab + gc.collect() + notify(SLOT_VIEWER, SCRIPT_OWNER) # and with the wrapper collected too + QCoreApplication.sendPostedEvents(None, QEvent.Type.DeferredDelete) + qapp.processEvents() + + +def test_a_session_outliving_its_panel_can_still_be_replaced(qapp, monkeypatch): + _script_connect(monkeypatch) + tab = _tab() + orphan = registry.viewer + tab._close_screen_window() + tab.deleteLater() + del tab + gc.collect() + QCoreApplication.sendPostedEvents(None, QEvent.Type.DeferredDelete) + assert registry.connect_viewer("desk", 5555, "tok")["connected"] is True + assert not orphan.connected From d15d610a9c0800e4e7400b30d62565bb15111fc7 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:00:22 +0800 Subject: [PATCH 12/26] Pair USB passthrough replies with their request by an echoed request id Replies were matched by kind only, so a reply that arrived after its request timed out completed the next request of that kind: open(bbbb) bound the claim opened for aaaa, and a bulk read returned the previous read's data. The viewer now sends an optional request_id in every request and the host echoes it in every reply, so a late reply is discarded by its tombstone and nothing else is. A host that does not echo ids cannot be disambiguated, and dropping the next reply would lose a correct answer whenever the late one never comes, so after a timeout the viewer refuses further requests of that kind with UsbClientDesynchronized instead of guessing. --- .../usb_passthrough_design.rst | 78 +++ .../usb_passthrough_operator_guide.rst | 13 + .../usb_passthrough_design.rst | 68 ++ .../usb_passthrough_operator_guide.rst | 10 + je_auto_control/utils/usb/__init__.py | 6 +- .../utils/usb/passthrough/__init__.py | 6 +- .../utils/usb/passthrough/protocol.py | 44 ++ .../utils/usb/passthrough/session.py | 82 ++- .../utils/usb/passthrough/viewer_client.py | 540 ++++++++++------ .../headless/test_usb_request_correlation.py | 586 ++++++++++++++++++ 10 files changed, 1216 insertions(+), 217 deletions(-) create mode 100644 test/unit_test/headless/test_usb_request_correlation.py diff --git a/docs/source/Eng/doc/operations_layer/usb_passthrough_design.rst b/docs/source/Eng/doc/operations_layer/usb_passthrough_design.rst index 9f02d0acb..541afbed2 100644 --- a/docs/source/Eng/doc/operations_layer/usb_passthrough_design.rst +++ b/docs/source/Eng/doc/operations_layer/usb_passthrough_design.rst @@ -153,6 +153,84 @@ transfers share one already-authenticated channel instead of coupling in a second REST transport, and ACL filtering reuses the same logic as the claim decision. +Request identity +---------------- + +Replies were originally paired with requests by kind alone: the one +pending OPEN, the one pending LIST, the one pending exchange on a +``claim_id``. A reply that arrived after its request had timed out was +therefore handed to the *next* request of that kind -- ``open(bbbb)`` +bound the claim the host had opened for ``aaaa``, and a bulk read +returned the previous read's data. The host accepts a ``timeout_ms`` of +up to 60 s while the viewer gives up after 10 s by default, so this +happened in ordinary use. + +Every JSON payload may now carry one extra, optional key:: + + "request_id": "" + +- **Viewer → host.** The viewer generates the id and adds it to the + payload of every request: ``OPEN``, ``RESUME``, ``LIST``, ``CLOSE``, + ``CTRL``, ``BULK``, ``INT``. ``LIST`` and ``CLOSE`` used to have an + empty payload; they now send ``{"request_id": "..."}``. +- **Host → viewer.** The host copies the id, unchanged, into every reply + to that request: ``OPENED``, ``LIST``, ``CLOSED``, the ``CTRL`` / + ``BULK`` / ``INT`` reply, and ``ERROR`` (including the "rate limited" + ERROR of a locked-out peer). A fragmented reply carries it once, in the + reassembled JSON. +- ``CREDIT`` never carries an id. A credit is a grant on the claim, not + an answer, and the credit that follows a late reply is still valid. +- The frame header is unchanged, and the id is opaque to the host. + +The viewer hands a reply to the request whose id it carries. A request +that times out leaves a *tombstone*; the reply that later matches it is +discarded -- that reply and no other. A late ``OPENED`` that succeeded +means the host holds a device for a caller who already got a timeout, so +the viewer sends ``CLOSE`` for that claim. (A late reply to ``RESUME`` +is left alone: the claim may have been resumed again and be in use.) + +Compatibility: + +================ ======================================================= +Viewer / host Behaviour +================ ======================================================= +new / new Paired by id. A timeout costs only the timed-out + request; the claim and the client stay usable. +old / new The viewer sends no id, so the host echoes none: every + payload is byte-for-byte what it was. Paired by kind, + with the original defect. +new / old The host ignores the unknown key and echoes nothing; + replies are paired by kind. After a timeout the viewer + stops instead of guessing (below). +================ ======================================================= + +Against a host that does not echo ids there is no way to tell a late +reply from the answer to the next request. Discarding "the next reply" +is not an option: if the host never sent the late one, that throws away +a correct answer. So once a request has timed out the viewer refuses +further requests of that kind with ``UsbClientDesynchronized``: + +- a transfer timeout makes that claim unusable (``ClientHandle.reusable`` + is ``False``). ``close()`` still works -- ``CLOSED`` is a kind no late + transfer reply can be mistaken for -- and opening the device again + gives a clean claim. Other claims are unaffected; +- an ``OPEN`` / ``RESUME`` timeout blocks further ``open`` / ``resume`` + on that client (``UsbPassthroughClient.reusable`` is ``False``), and a + ``LIST`` timeout blocks further ``list_devices``. Reconnect the channel + and use a new client. + +The viewer learns that the host is current from the first reply that +echoes one of its ids (``UsbPassthroughClient.peer_echoes_request_ids``). +Until then a timeout is treated the cautious way; if the late reply then +arrives with its id, the block is lifted. + +Two smaller pairing faults were fixed with this. An ``ERROR`` on +``claim_id`` 0 -- the answer to a refused ``OPEN`` or a failed ``LIST`` +-- had no request to go to and the call timed out; with an id it fails +the call it answers. And the tail of a reassembled message dropped for +exceeding the 2 MiB cap was parsed as a message of its own; it is now +skipped through its EOF frame. + Backpressure ------------ diff --git a/docs/source/Eng/doc/operations_layer/usb_passthrough_operator_guide.rst b/docs/source/Eng/doc/operations_layer/usb_passthrough_operator_guide.rst index 82bc172b7..bfb33fc12 100644 --- a/docs/source/Eng/doc/operations_layer/usb_passthrough_operator_guide.rst +++ b/docs/source/Eng/doc/operations_layer/usb_passthrough_operator_guide.rst @@ -242,6 +242,15 @@ Errors: The most common case is *denied by ACL policy* — go check the prompt dialog or the ACL rule on the host. - ``UsbClientClosed`` — the client or its handle was already shut down. +- ``UsbClientDesynchronized`` — an earlier request timed out and the + host does not echo request ids (it predates them), so a late reply + could not be told from the next one. Nothing was sent. After a + transfer timeout, ``close()`` the handle and ``open`` the device + again; after an ``open`` / ``resume`` / ``list_devices`` timeout, + reconnect the channel and use a new client. ``handle.reusable`` and + ``client.reusable`` report the state without raising. Against a + current host a timeout costs only the request that timed out. See + *Request identity* in the design document. Troubleshooting matrix @@ -262,6 +271,10 @@ Symptom Likely cause / fix Transfer ``UsbClientTimeout`` Host process is busy or the WebRTC channel is broken. Inspect the *Packet Inspector* tab for RTT / packet loss. +``UsbClientDesynchronized`` A request timed out against a host too old to + echo request ids. Close the handle and open the + device again (transfer), or reconnect (open / + list). Upgrading the host removes the limit. After OPEN, host's keyboard stops working Linux: a HID device was claimed and ``usbhid`` was detached. The driver re-attaches on CLOSE; if not, ``udevadm trigger`` to recover. diff --git a/docs/source/Zh/doc/operations_layer/usb_passthrough_design.rst b/docs/source/Zh/doc/operations_layer/usb_passthrough_design.rst index a9ba8933b..c30fc96e9 100644 --- a/docs/source/Zh/doc/operations_layer/usb_passthrough_design.rst +++ b/docs/source/Zh/doc/operations_layer/usb_passthrough_design.rst @@ -138,6 +138,74 @@ viewer 端用 ``UsbPassthroughClient.list_devices()`` 取得。讓列舉與傳 共用同一條已通過 auth gate 的 channel，避免再耦合一層 REST transport， 也讓 ACL 過濾與 claim 決策走同一份邏輯。 +請求識別 +-------- + +回覆原本只依種類配對：等待中的那一個 OPEN、那一個 LIST、某個 +``claim_id`` 上的那一次交換。請求逾時後才到的回覆因此會交給同種類的 +*下一個*\ 請求——``open(bbbb)`` 綁到 host 為 ``aaaa`` 開的 claim，bulk +讀取拿到上一次讀取的資料。host 接受最長 60 秒的 ``timeout_ms``\ ， +viewer 預設 10 秒就放棄，所以正常使用就會遇到。 + +現在每個 JSON payload 都可以多帶一個選用的 key:: + + "request_id": "<字串，1..64 個字元>" + +- **viewer → host。** 編號由 viewer 產生，加在每個請求的 payload 裡： + ``OPEN``\ 、``RESUME``\ 、``LIST``\ 、``CLOSE``\ 、``CTRL``\ 、 + ``BULK``\ 、``INT``\ 。``LIST`` 與 ``CLOSE`` 原本 payload 是空的， + 現在送 ``{"request_id": "..."}``\ 。 +- **host → viewer。** host 把編號原樣放進該請求的每一個回覆： + ``OPENED``\ 、``LIST``\ 、``CLOSED``\ 、``CTRL`` / ``BULK`` / + ``INT`` 的回覆，以及 ``ERROR``\ （包含被鎖定時的「rate limited」 + ERROR）。分片的回覆只帶一次，在重組後的 JSON 裡。 +- ``CREDIT`` 不帶編號。credit 是對 claim 的授權而不是回覆，遲到回覆 + 後面跟著的 credit 仍然有效。 +- frame header 沒有變動；host 不解讀編號的內容。 + +viewer 把回覆交給編號相符的請求。逾時的請求會留下一個 *tombstone*\ ； +之後與它相符的回覆會被丟棄——只丟那一個。遲到但成功的 ``OPENED`` +代表 host 為一個已經收到逾時的呼叫端佔著裝置，所以 viewer 會對那個 +claim 送 ``CLOSE``\ 。（``RESUME`` 的遲到回覆不處理：那個 claim 可能 +已經再次 resume 並在使用中。） + +相容性： + +================ ======================================================= +viewer / host 行為 +================ ======================================================= +新 / 新 依編號配對。逾時只損失逾時的那個請求，claim 與 client + 都能繼續使用。 +舊 / 新 viewer 不送編號，host 就不帶回：每個 payload 與以前 + 逐位元組相同。依種類配對，原本的缺陷仍在。 +新 / 舊 host 忽略不認得的 key，也不帶回編號；回覆依種類配對。 + 逾時之後 viewer 會停下來而不是猜（見下）。 +================ ======================================================= + +對不帶回編號的 host，無法分辨遲到的回覆與下一個請求的回覆。「丟掉 +下一個回覆」不可行：如果 host 根本沒送出那個遲到的回覆，丟掉的就是 +正確的回覆。所以請求逾時之後，viewer 會以 ``UsbClientDesynchronized`` +拒絕同種類的後續請求： + +- 傳輸逾時讓該 claim 不能再用（``ClientHandle.reusable`` 為 + ``False``\ ）。``close()`` 仍然可用——遲到的傳輸回覆不會被誤認成 + ``CLOSED``\ ——重新 open 裝置就得到乾淨的 claim。其他 claim 不受 + 影響； +- ``OPEN`` / ``RESUME`` 逾時會擋住該 client 之後的 ``open`` / + ``resume``\ （``UsbPassthroughClient.reusable`` 為 ``False``\ ）， + ``LIST`` 逾時會擋住之後的 ``list_devices``\ 。請重新連線 channel + 並使用新的 client。 + +viewer 從第一個帶回自己編號的回覆得知 host 是新版 +（``UsbPassthroughClient.peer_echoes_request_ids``\ ）。在那之前逾時 +一律以保守方式處理；之後若遲到的回覆帶著編號到達，就解除限制。 + +同時修正了兩個較小的配對問題。``claim_id`` 為 0 的 ``ERROR``\ —— +被拒絕的 ``OPEN`` 或失敗的 ``LIST`` 的回覆——原本沒有請求可以交付， +呼叫會逾時；帶編號之後會讓對應的呼叫失敗。另外，因超過 2 MiB 上限 +而被丟棄的重組訊息，其尾段原本會被當成獨立的訊息解析；現在會一路 +略過到它的 EOF frame。 + Backpressure ------------- diff --git a/docs/source/Zh/doc/operations_layer/usb_passthrough_operator_guide.rst b/docs/source/Zh/doc/operations_layer/usb_passthrough_operator_guide.rst index e7d38804e..50fade05a 100644 --- a/docs/source/Zh/doc/operations_layer/usb_passthrough_operator_guide.rst +++ b/docs/source/Zh/doc/operations_layer/usb_passthrough_operator_guide.rst @@ -226,6 +226,13 @@ Open + transfer 情境是 *denied by ACL policy* — 去看 host 端的 prompt 對話框或 ACL 規則。 - ``UsbClientClosed`` — client 或其 handle 已 shutdown。 +- ``UsbClientDesynchronized`` — 先前有請求逾時，而 host 不帶回請求 + 編號（版本早於此欄位），遲到的回覆與下一個回覆無法分辨。此時不會 + 送出任何東西。傳輸逾時後，對 handle 呼叫 ``close()`` 再重新 + ``open`` 裝置；``open`` / ``resume`` / ``list_devices`` 逾時後， + 重新連線 channel 並使用新的 client。``handle.reusable`` 與 + ``client.reusable`` 可在不拋例外的情況下查詢狀態。對新版 host， + 逾時只損失逾時的那個請求。見設計文件的\ *請求識別*\ 。 疑難排解對照表 @@ -244,6 +251,9 @@ transfer 上 ``credit exhausted`` viewer 送的 frame 超過 host ``i ``initial_credits``\ 。 Transfer ``UsbClientTimeout`` host 程序忙或 WebRTC channel 壞了。看 *Packet Inspector* 分頁的 RTT / 封包遺失。 +``UsbClientDesynchronized`` 請求對不帶回請求編號的舊版 host 逾時。關閉 handle + 再重新 open 裝置（傳輸），或重新連線（open / + list）。升級 host 即可解除此限制。 OPEN 後 host 鍵盤停止運作 Linux：HID 裝置被 claim 且 ``usbhid`` 被 detach。 CLOSE 時 driver 會重新 attach；如果沒有，用 ``udevadm trigger`` 救回。 diff --git a/je_auto_control/utils/usb/__init__.py b/je_auto_control/utils/usb/__init__.py index f1a6ce91e..9600e38ed 100644 --- a/je_auto_control/utils/usb/__init__.py +++ b/je_auto_control/utils/usb/__init__.py @@ -2,7 +2,8 @@ from je_auto_control.utils.usb.passthrough import ( AclRule, ClientHandle, FakeUsbBackend, Frame, LibusbBackend, LoopbackTransport, MAX_PAYLOAD_BYTES, Opcode, ProtocolError, - SessionError, UsbAcl, UsbBackend, UsbClientClosed, UsbClientError, + SessionError, UsbAcl, UsbBackend, UsbClientClosed, + UsbClientDesynchronized, UsbClientError, UsbClientTimeout, UsbHandle, UsbLoopback, UsbPassthroughClient, UsbPassthroughSession, decode_frame, default_acl_path, default_passthrough_backend, enable_usb_passthrough, encode_frame, @@ -26,7 +27,8 @@ "enable_usb_passthrough", "encode_frame", "fragment_payload", "is_usb_passthrough_enabled", "LoopbackTransport", "UsbLoopback", # Viewer client (round 40) - "ClientHandle", "UsbClientClosed", "UsbClientError", "UsbClientTimeout", + "ClientHandle", "UsbClientClosed", "UsbClientDesynchronized", + "UsbClientError", "UsbClientTimeout", "UsbPassthroughClient", # Phase 2d ACL (round 41) "AclRule", "UsbAcl", "default_acl_path", diff --git a/je_auto_control/utils/usb/passthrough/__init__.py b/je_auto_control/utils/usb/passthrough/__init__.py index 58eed401f..cca9cc564 100644 --- a/je_auto_control/utils/usb/passthrough/__init__.py +++ b/je_auto_control/utils/usb/passthrough/__init__.py @@ -39,7 +39,8 @@ SessionError, UsbPassthroughSession, ) from je_auto_control.utils.usb.passthrough.viewer_client import ( - ClientHandle, UsbClientClosed, UsbClientError, UsbClientTimeout, + ClientHandle, UsbClientClosed, UsbClientDesynchronized, UsbClientError, + UsbClientTimeout, UsbPassthroughClient, ) from je_auto_control.utils.usb.passthrough.webrtc_channel import ( @@ -57,7 +58,8 @@ "decode_frame", "encode_frame", "fragment_payload", "MAX_PAYLOAD_BYTES", "SessionError", "UsbPassthroughSession", - "ClientHandle", "UsbClientClosed", "UsbClientError", "UsbClientTimeout", + "ClientHandle", "UsbClientClosed", "UsbClientDesynchronized", + "UsbClientError", "UsbClientTimeout", "UsbPassthroughClient", "UsbChannelClient", "UsbChannelHost", "AclRule", "UsbAcl", "default_acl_path", diff --git a/je_auto_control/utils/usb/passthrough/protocol.py b/je_auto_control/utils/usb/passthrough/protocol.py index 0f05192fc..d20c57519 100644 --- a/je_auto_control/utils/usb/passthrough/protocol.py +++ b/je_auto_control/utils/usb/passthrough/protocol.py @@ -12,13 +12,28 @@ sender writes one frame per ``send()`` call. The 16 KiB payload cap keeps message sizes well under the recommended SCTP boundary. +Request identity. Every payload that is a JSON object may carry an +optional ``"request_id"`` string (1..64 characters). The viewer generates +it and puts it in each request (OPEN, RESUME, LIST, CLOSE, CTRL, BULK, +INT); the host copies it unchanged into every reply to that request +(OPENED, LIST, CLOSED, CTRL, BULK, INT, ERROR), so the viewer pairs a +reply with the request that asked for it rather than with whichever +request of that kind is waiting. CREDIT frames never carry one: a credit +is a grant on the claim, not an answer. The field is optional in both +directions -- a host that receives no id (or one it cannot echo) replies +exactly as it did before the field existed, and a viewer that receives +no id falls back to pairing by kind. + This module is pure data — no I/O, no asyncio, no peer connection. """ from __future__ import annotations import enum +import json import struct from dataclasses import dataclass +from typing import Any, Optional + from je_auto_control.utils.exception.exceptions import AutoControlException @@ -26,6 +41,9 @@ HEADER_BYTES = struct.calcsize(_HEADER_FORMAT) MAX_PAYLOAD_BYTES = 16 * 1024 FLAG_EOF = 0x01 +#: JSON key of the optional request identity, and the longest id a host echoes. +REQUEST_ID_KEY = "request_id" +MAX_REQUEST_ID_CHARS = 64 class Opcode(enum.IntEnum): @@ -126,8 +144,34 @@ def decode_frame(data: bytes) -> Frame: return Frame(op=op, flags=flags, claim_id=claim_id, payload=payload) +def valid_request_id(value: Any) -> bool: + """True if ``value`` is a request id a peer may send and echo.""" + return isinstance(value, str) and 0 < len(value) <= MAX_REQUEST_ID_CHARS + + +def request_id_of(payload: bytes) -> Optional[str]: + """The ``request_id`` of a JSON payload, or ``None`` if it has none. + + Never raises: an empty payload, one that is not a JSON object, or an + id that is not a 1..64 character string all mean "no id", which is + what a peer older than the field sends. + """ + if not payload: + return None + try: + decoded = json.loads(bytes(payload).decode("utf-8")) + except ValueError: # UnicodeDecodeError and JSONDecodeError both are + return None + if not isinstance(decoded, dict): + return None + value = decoded.get(REQUEST_ID_KEY) + return value if valid_request_id(value) else None + + __all__ = [ "Frame", "Opcode", "ProtocolError", "decode_frame", "encode_frame", "fragment_payload", + "request_id_of", "valid_request_id", "MAX_PAYLOAD_BYTES", "HEADER_BYTES", "FLAG_EOF", + "REQUEST_ID_KEY", "MAX_REQUEST_ID_CHARS", ] diff --git a/je_auto_control/utils/usb/passthrough/session.py b/je_auto_control/utils/usb/passthrough/session.py index 034ad44a0..dd41d392e 100644 --- a/je_auto_control/utils/usb/passthrough/session.py +++ b/je_auto_control/utils/usb/passthrough/session.py @@ -54,6 +54,14 @@ alongside every transfer reply so a well-behaved peer never stalls. A peer that exhausts its budget gets ERROR("credit exhausted") and is expected to wait for CREDIT before retrying. + +Request identity: a request payload may carry ``"request_id": "<1..64 +chars>"`` next to the fields above (LIST and CLOSE, which have no other +fields, then send ``{"request_id": "..."}``). Every reply to that request +-- OPENED, LIST, CLOSED, the transfer reply, or ERROR -- carries the same +value under the same key; CREDIT never does. A request without a usable +id is answered exactly as before, so a viewer older than the field sees +no change. See ``protocol`` for the rule in full. """ from __future__ import annotations @@ -70,7 +78,8 @@ from je_auto_control.utils.usb.passthrough.acl import UsbAcl, normalize_usb_id from je_auto_control.utils.usb.passthrough.backend import UsbBackend, UsbHandle from je_auto_control.utils.usb.passthrough.protocol import ( - Frame, Opcode, fragment_payload, + MAX_PAYLOAD_BYTES, REQUEST_ID_KEY, Frame, Opcode, fragment_payload, + request_id_of, ) @@ -192,16 +201,24 @@ def is_locked_out(self) -> bool: def handle_frame(self, frame: Frame) -> List[Frame]: """Process one frame, with abuse tracking, and return replies.""" + request_id = request_id_of(frame.payload) if self._abuse.is_locked(): - return [_error_frame(frame.claim_id, "rate limited; locked out")] - replies = self._dispatch(frame) + return [_with_request_id( + _error_frame(frame.claim_id, "rate limited; locked out"), request_id)] + replies = [_with_request_id(reply, request_id) + for reply in self._dispatch(frame, request_id)] if _is_misbehaviour(replies) and self._abuse.record_strike(): self._audit("usb_rate_limited", "?", "?", None, detail="viewer locked out for repeated failures") return replies - def _dispatch(self, frame: Frame) -> List[Frame]: - """Route one incoming frame; return zero or more reply frames.""" + def _dispatch(self, frame: Frame, request_id: Optional[str] = None) -> List[Frame]: + """Route one incoming frame; return zero or more reply frames. + + Replies that may be fragmented (LIST, transfers) are built with + ``request_id`` already inside; the single-frame ones are stamped + by :meth:`handle_frame`. + """ if frame.op == Opcode.OPEN: return [self._handle_open(frame)] if frame.op == Opcode.RESUME: @@ -209,16 +226,16 @@ def _dispatch(self, frame: Frame) -> List[Frame]: if frame.op == Opcode.CLOSE: return [self._handle_close(frame)] if frame.op == Opcode.CTRL: - return self._handle_transfer(frame, _control_handler) + return self._handle_transfer(frame, _control_handler, request_id) if frame.op == Opcode.BULK: - return self._handle_transfer(frame, _bulk_handler) + return self._handle_transfer(frame, _bulk_handler, request_id) if frame.op == Opcode.INT: - return self._handle_transfer(frame, _interrupt_handler) + return self._handle_transfer(frame, _interrupt_handler, request_id) if frame.op == Opcode.CREDIT: self._handle_credit(frame) return [] if frame.op == Opcode.LIST: - return self._handle_list(frame) + return self._handle_list(frame, request_id) if frame.op in (Opcode.OPENED, Opcode.CLOSED, Opcode.ERROR): # Responses we don't expect to receive on the host side here. return [] @@ -226,7 +243,8 @@ def _dispatch(self, frame: Frame) -> List[Frame]: # --- LIST --------------------------------------------------------------- - def _handle_list(self, frame: Frame) -> List[Frame]: + def _handle_list(self, frame: Frame, + request_id: Optional[str] = None) -> List[Frame]: """Enumerate backend devices the ACL would not outright deny. Resolves open question 3: the device list rides the same ``usb`` @@ -248,7 +266,7 @@ def _handle_list(self, frame: Frame) -> List[Frame]: for dev in devices if self._list_visible(dev.vendor_id, dev.product_id, dev.serial) ] - payload = _encode_json_payload({"devices": visible}) + payload = _encode_json_payload(_identified({"devices": visible}, request_id)) return fragment_payload(Opcode.LIST, frame.claim_id, payload) def _list_visible(self, vendor_id: str, product_id: str, @@ -444,7 +462,7 @@ def _handle_close(self, frame: Frame) -> Frame: def _handle_transfer(self, frame: Frame, dispatcher: Callable[[UsbHandle, Dict[str, Any]], bytes], - ) -> List[Frame]: + request_id: Optional[str] = None) -> List[Frame]: with self._lock: claim = self._claims.get(int(frame.claim_id)) if claim is None: @@ -464,14 +482,14 @@ def _handle_transfer(self, frame: Frame, try: result_bytes = dispatcher(handle, request) except Exception as error: # noqa: BLE001 # pylint: disable=broad-except # reason: backends raise their own error types - reply_payload = _encode_json_payload( - {"ok": False, "error": str(error)}, - ) + reply_payload = _encode_json_payload(_identified( + {"ok": False, "error": str(error)}, request_id, + )) else: - reply_payload = _encode_json_payload({ + reply_payload = _encode_json_payload(_identified({ "ok": True, "data": base64.b64encode(result_bytes).decode("ascii"), - }) + }, request_id)) # Fragment so an oversize IN transfer (open question 2) spans # multiple EOF-terminated frames instead of breaching the cap. frames = fragment_payload( @@ -626,6 +644,36 @@ def _error_frame(claim_id: int, message: str) -> Frame: ) +#: Replies that are always one frame, so the id can be added after the fact. +_SINGLE_FRAME_REPLIES = (Opcode.OPENED, Opcode.CLOSED, Opcode.ERROR) + + +def _identified(body: Dict[str, Any], request_id: Optional[str]) -> Dict[str, Any]: + """``body`` with the request's id added, if the request had one.""" + if request_id is not None: + body[REQUEST_ID_KEY] = request_id + return body + + +def _with_request_id(frame: Frame, request_id: Optional[str]) -> Frame: + """Echo ``request_id`` in a single-frame reply; other frames pass through. + + CREDIT is a grant rather than a reply and never carries an id; LIST and + transfer replies got theirs before they were fragmented. + """ + if request_id is None or frame.op not in _SINGLE_FRAME_REPLIES: + return frame + try: + body = _decode_json_payload(frame.payload) + except ValueError: + return frame + payload = _encode_json_payload(_identified(body, request_id)) + if len(payload) > MAX_PAYLOAD_BYTES: + return frame # an unidentified reply still beats none at all + return Frame(op=frame.op, flags=frame.flags, claim_id=frame.claim_id, + payload=payload) + + def _encode_json_payload(obj: object) -> bytes: return json.dumps(obj, ensure_ascii=False).encode("utf-8") diff --git a/je_auto_control/utils/usb/passthrough/viewer_client.py b/je_auto_control/utils/usb/passthrough/viewer_client.py index 662da3705..55e06df5c 100644 --- a/je_auto_control/utils/usb/passthrough/viewer_client.py +++ b/je_auto_control/utils/usb/passthrough/viewer_client.py @@ -29,19 +29,40 @@ * ``UsbClientTimeout`` — peer did not reply within the timeout. * ``UsbClientError`` — peer replied with ``{ok: false}`` or ERROR. * ``UsbClientClosed`` — the client (or its handle) was shut down. +* ``UsbClientDesynchronized`` — an earlier request timed out against a + host that does not echo request ids; see below. + +Pairing replies with requests. Every request carries a ``request_id`` +the host echoes (see ``protocol``), and a reply is handed to the request +with that id. A reply whose request has already timed out matches a +tombstone and is discarded -- that reply and no other. + +A host older than the field echoes nothing, and its replies are paired by +kind (OPEN / LIST / claim) as they always were. That is only safe while +no request has timed out: afterwards a reply could be the late one or the +answer to the next request, and nothing on the wire says which. Dropping +"the next reply" would throw away a correct answer whenever the host +never sent the late one, so the client does not guess. It refuses further +requests of that kind with :class:`UsbClientDesynchronized` -- a claim +after a transfer timeout (close the handle and open the device again), +OPEN/RESUME or LIST after theirs (reconnect the channel). Until the first +echoed id arrives the peer's age is unknown and a timeout is treated the +cautious way; the first echoed id lifts it. """ from __future__ import annotations import base64 import json +import secrets import threading -from dataclasses import dataclass -from typing import Any, Callable, Dict, List, Optional +from collections import OrderedDict +from dataclasses import dataclass, field +from typing import Any, Callable, Dict, List, Optional, Set, Tuple, Union from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.usb.passthrough.protocol import ( - FLAG_EOF, Frame, Opcode, + FLAG_EOF, REQUEST_ID_KEY, Frame, Opcode, ProtocolError, valid_request_id, ) #: Largest reassembled message: a 1 MiB endpoint transfer (the host's @@ -53,6 +74,17 @@ _DEFAULT_CREDIT_TIMEOUT_S = 30.0 _INITIAL_CREDIT_GUESS = 16 _CLIENT_SHUT_DOWN_MSG = "client is shut down" +#: Where a request waits: one OPEN/RESUME, one LIST, and one exchange per +#: claim id may be outstanding at a time. +_Slot = Union[str, int] +_OPEN_SLOT = "open" +_LIST_SLOT = "list" +#: Replies a host may split across frames, and those that are always one. +_FRAGMENTED_REPLIES = (Opcode.LIST, Opcode.CTRL, Opcode.BULK, Opcode.INT) +_SINGLE_FRAME_REPLIES = (Opcode.OPENED, Opcode.CLOSED, Opcode.ERROR) +#: Ids of timed-out requests kept so their late replies are recognised. +#: Forgetting one is harmless: a reply with an unknown id is dropped too. +_MAX_TOMBSTONES = 256 class UsbClientError(AutoControlException): @@ -67,21 +99,59 @@ class UsbClientClosed(UsbClientError): """The client / handle was shut down before a reply arrived.""" +class UsbClientDesynchronized(UsbClientError): + """A timed-out request left replies impossible to pair; reconnect. + + Raised instead of sending when an earlier request of the same kind + timed out and the host does not echo request ids: its late reply and + the answer to a new request would be indistinguishable. A claim is + recovered by closing its handle and opening the device again; OPEN / + RESUME and LIST by reconnecting the channel with a new client. + """ + + @dataclass class _PendingRequest: """One outstanding viewer→host request awaiting a reply. - The reply is stored as ``(reply_op, reply_payload)`` rather than a + The reply is kept as its decoded JSON body rather than a :class:`Frame` because a reassembled payload (open question 2) may exceed the per-frame cap that :class:`Frame` enforces. """ + send_op: Opcode expected_op: Opcode - event: threading.Event + slot: _Slot + event: threading.Event = field(default_factory=threading.Event) + request_id: str = "" reply_op: Optional[Opcode] = None - reply_payload: bytes = b"" + reply_body: Dict[str, Any] = field(default_factory=dict) cancelled: bool = False + @property + def label(self) -> str: + return self.send_op.name + + @property + def claim_id(self) -> int: + return self.slot if isinstance(self.slot, int) else 0 + + def timeout_message(self) -> str: + """What :class:`UsbClientTimeout` says when this request expires.""" + if isinstance(self.slot, int): + return f"{self.label} timed out for claim {self.slot}" + return f"{self.label} timed out" + + +def _desynchronized_message(slot: _Slot) -> str: + reason = ("an earlier request timed out and the host does not echo request " + "ids, so its late reply cannot be told from the next one") + if isinstance(slot, int): + return (f"claim {slot} cannot be reused: {reason}. " + "Close this handle and open the device again.") + return (f"no further {slot.upper()} on this client: {reason}. " + "Reconnect the USB channel and use a new client.") + # --------------------------------------------------------------------------- # ClientHandle — what the user actually drives once they hold a claim @@ -118,6 +188,16 @@ def closed(self) -> bool: with self._lock: return self._closed + @property + def reusable(self) -> bool: + """False once the handle is closed or its claim is desynchronised. + + A transfer that timed out against a host that does not echo request + ids leaves the claim unable to pair replies; further transfers raise + :class:`UsbClientDesynchronized`. :meth:`close` still works. + """ + return not self.closed and self._client.slot_reusable(self._claim_id) + def control_transfer(self, *, bm_request_type: int, b_request: int, w_value: int = 0, w_index: int = 0, data: bytes = b"", length: int = 0, @@ -189,15 +269,23 @@ def __init__( self._reply_timeout = float(reply_timeout_s) self._credit_timeout = float(credit_timeout_s) self._lock = threading.Lock() - self._pending: Dict[int, _PendingRequest] = {} + # One request per slot (pairing by kind, for hosts without ids)... + self._pending: Dict[_Slot, _PendingRequest] = {} + # ...and the same requests by the id a current host echoes. + self._by_id: Dict[str, _PendingRequest] = {} + self._tombstones: "OrderedDict[str, Opcode]" = OrderedDict() + self._desynced: Set[_Slot] = set() + self._peer_echoes_ids = False + self._id_prefix = secrets.token_hex(4) + self._id_counter = 0 self._credits: Dict[int, int] = {} self._credit_events: Dict[int, threading.Event] = {} self._claim_locks: Dict[int, threading.Lock] = {} - self._open_pending: Optional[_PendingRequest] = None - self._list_pending: Optional[_PendingRequest] = None # Reassembly buffers for fragmented replies, keyed by claim_id # (open question 2). LIST uses claim_id 0. self._reasm: Dict[int, bytearray] = {} + # Claims whose current message overflowed: drop through its EOF. + self._reasm_skip: Set[int] = set() self._initial_credit_guess = max(1, int(initial_credit_guess)) self._closed = False @@ -208,104 +296,105 @@ def shutdown(self) -> None: with self._lock: self._closed = True pending: List[_PendingRequest] = list(self._pending.values()) - if self._open_pending is not None: - pending.append(self._open_pending) - if self._list_pending is not None: - pending.append(self._list_pending) self._pending.clear() - self._open_pending = None - self._list_pending = None + self._by_id.clear() + self._tombstones.clear() self._reasm.clear() + self._reasm_skip.clear() credit_events = list(self._credit_events.values()) + for request in pending: + request.cancelled = True for request in pending: - request.cancelled = True request.event.set() for event in credit_events: event.set() + @property + def peer_echoes_request_ids(self) -> bool: + """True once the host has echoed a request id this client issued.""" + with self._lock: + return self._peer_echoes_ids + + @property + def reusable(self) -> bool: + """False once the client is shut down or can no longer OPEN / RESUME. + + See :class:`UsbClientDesynchronized`; a claim's own state is on + :attr:`ClientHandle.reusable`. + """ + with self._lock: + return not self._closed and _OPEN_SLOT not in self._desynced + + def slot_reusable(self, slot: _Slot) -> bool: + """Whether requests may still go out on a claim id, ``"open"`` or ``"list"``.""" + with self._lock: + return slot not in self._desynced + # --- Inbound transport entry point -------------------------------------- def feed_frame(self, frame: Frame) -> None: """Hand a frame received from the transport to the client.""" - if frame.op == Opcode.OPENED: - self._on_opened(frame) - return - if frame.op == Opcode.CLOSED: - self._complete_pending(frame.claim_id, frame.payload, Opcode.CLOSED) - return if frame.op == Opcode.CREDIT: self._on_credit(frame) return - if frame.op in (Opcode.CTRL, Opcode.BULK, Opcode.INT): - assembled = self._reassemble(frame) - if assembled is not None: - self._complete_pending(frame.claim_id, assembled, frame.op) - return - if frame.op == Opcode.LIST: - self._on_list(frame) - return - if frame.op == Opcode.ERROR: - self._on_error(frame) + if frame.op in _FRAGMENTED_REPLIES: + payload = self._reassemble(frame) + if payload is None: + return + elif frame.op in _SINGLE_FRAME_REPLIES: + payload = bytes(frame.payload) + else: + autocontrol_logger.debug( + "passthrough client: ignoring incoming opcode %s", frame.op, + ) return - autocontrol_logger.debug( - "passthrough client: ignoring incoming opcode %s", frame.op, - ) + self._route_reply(frame.op, int(frame.claim_id), payload) def _reassemble(self, frame: Frame) -> Optional[bytes]: """Buffer a fragment; return the full payload once EOF arrives. A message may not grow past :data:`_MAX_REASSEMBLED_BYTES`: a host - sending fragments without EOF grew the buffer without limit. + sending fragments without EOF grew the buffer without limit. The + rest of a message dropped that way is skipped through its EOF -- + its tail used to be parsed as a message of its own. Nothing else + resets a buffer: a request that times out leaves its half-received + reply to finish and be discarded whole. """ cid = int(frame.claim_id) + is_last = bool(frame.flags & FLAG_EOF) with self._lock: + if cid in self._reasm_skip: + if is_last: + self._reasm_skip.discard(cid) + return None buffer = self._reasm.setdefault(cid, bytearray()) if len(buffer) + len(frame.payload) > _MAX_REASSEMBLED_BYTES: self._reasm.pop(cid, None) + if not is_last: + self._reasm_skip.add(cid) autocontrol_logger.warning( "passthrough client: message on claim %d exceeds %d bytes; dropped", cid, _MAX_REASSEMBLED_BYTES) return None buffer.extend(frame.payload) - if not (frame.flags & FLAG_EOF): + if not is_last: return None - full = bytes(buffer) self._reasm.pop(cid, None) - return full + return bytes(buffer) # --- Outbound: open / close --------------------------------------------- def open(self, *, vendor_id: str, product_id: str, serial: Optional[str] = None) -> ClientHandle: - request = _PendingRequest( - expected_op=Opcode.OPENED, event=threading.Event(), - ) - with self._lock: - if self._closed: - raise UsbClientClosed(_CLIENT_SHUT_DOWN_MSG) - if self._open_pending is not None: - raise UsbClientError("another open is in progress") - self._open_pending = request + """Claim a device on the host; block until it answers.""" body: Dict[str, Any] = { "vendor_id": vendor_id, "product_id": product_id, } if serial is not None: body["serial"] = serial - self._send(Frame(op=Opcode.OPEN, - payload=json.dumps(body).encode("utf-8"))) - if not request.event.wait(timeout=self._reply_timeout): - with self._lock: - if self._open_pending is request: - self._open_pending = None - raise UsbClientTimeout("OPEN timed out") - if request.cancelled: - raise UsbClientClosed("client shut down before OPEN reply") - if request.reply_op is None: - raise UsbClientError("event signalled without a reply") - body = _decode_json(request.reply_payload) - if not body.get("ok"): - raise UsbClientError(body.get("error", "open failed")) - return self._bind_claim(body) + request = _PendingRequest(Opcode.OPEN, Opcode.OPENED, _OPEN_SLOT) + self._round_trip(request, body, busy="another open is in progress") + return self._bind_claim(_accepted_body(request, "open failed")) def resume(self, resume_token: str) -> ClientHandle: """Re-bind a claim after a reconnect using a token from ``open``. @@ -315,36 +404,15 @@ def resume(self, resume_token: str) -> ClientHandle: for the same ``claim_id``; raises :class:`UsbClientError` if the token is unknown or expired. """ - request = _PendingRequest( - expected_op=Opcode.OPENED, event=threading.Event(), - ) - with self._lock: - if self._closed: - raise UsbClientClosed(_CLIENT_SHUT_DOWN_MSG) - if self._open_pending is not None: - raise UsbClientError("another open is in progress") - self._open_pending = request - self._send(Frame( - op=Opcode.RESUME, - payload=json.dumps({"resume_token": resume_token}).encode("utf-8"), - )) - if not request.event.wait(timeout=self._reply_timeout): - with self._lock: - if self._open_pending is request: - self._open_pending = None - raise UsbClientTimeout("RESUME timed out") - if request.cancelled: - raise UsbClientClosed("client shut down before RESUME reply") - body = _decode_json(request.reply_payload) - if not body.get("ok"): - raise UsbClientError(body.get("error", "resume failed")) - return self._bind_claim(body) + request = _PendingRequest(Opcode.RESUME, Opcode.OPENED, _OPEN_SLOT) + self._round_trip(request, {"resume_token": resume_token}, + busy="another open is in progress") + return self._bind_claim(_accepted_body(request, "resume failed")) def _bind_claim(self, body: Dict[str, Any]) -> ClientHandle: - try: - claim_id = int(body["claim_id"]) - except (KeyError, TypeError, ValueError) as error: - raise UsbClientError(f"host reply has no valid claim_id: {body!r}") from error + claim_id = _claim_id_in(body) + if claim_id is None: + raise UsbClientError(f"host reply has no valid claim_id: {body!r}") with self._lock: self._credits[claim_id] = self._initial_credit_guess self._credit_events[claim_id] = threading.Event() @@ -356,117 +424,207 @@ def list_devices(self) -> List[Dict[str, Any]]: Blocks until the host replies. Returns a list of dicts with ``vendor_id`` / ``product_id`` / ``serial`` / ``bus_location``. """ - request = _PendingRequest( - expected_op=Opcode.LIST, event=threading.Event(), - ) - with self._lock: - if self._closed: - raise UsbClientClosed(_CLIENT_SHUT_DOWN_MSG) - if self._list_pending is not None: - raise UsbClientError("another list is in progress") - self._list_pending = request - self._send(Frame(op=Opcode.LIST)) - if not request.event.wait(timeout=self._reply_timeout): - with self._lock: - if self._list_pending is request: - self._list_pending = None - self._reasm.pop(0, None) - raise UsbClientTimeout("LIST timed out") - if request.cancelled: - raise UsbClientClosed("client shut down before LIST reply") - body = _decode_json(request.reply_payload) - devices = body.get("devices") + request = _PendingRequest(Opcode.LIST, Opcode.LIST, _LIST_SLOT) + self._round_trip(request, {}, busy="another list is in progress") + if request.reply_op == Opcode.ERROR: + raise UsbClientError(request.reply_body.get("error", "host ERROR")) + devices = request.reply_body.get("devices") return list(devices) if isinstance(devices, list) else [] def _exchange_close(self, claim_id: int) -> None: - request = _PendingRequest( - expected_op=Opcode.CLOSED, event=threading.Event(), - ) + request = _PendingRequest(Opcode.CLOSE, Opcode.CLOSED, int(claim_id)) with self._claim_lock(claim_id): - self._round_trip(claim_id, request, - Frame(op=Opcode.CLOSE, claim_id=int(claim_id)), "CLOSE") + self._round_trip(request, {}) self._forget_claim(claim_id) def _claim_lock(self, claim_id: int) -> threading.Lock: - """One exchange per claim at a time: replies carry only the claim id.""" + """One exchange per claim at a time: an old host's replies carry only the claim id.""" with self._lock: return self._claim_locks.setdefault(int(claim_id), threading.Lock()) - def _round_trip(self, claim_id: int, request: "_PendingRequest", - frame: Frame, label: str) -> None: - """Register ``request``, send ``frame`` and wait for its reply. + def _round_trip(self, request: "_PendingRequest", body: Dict[str, Any], + *, busy: Optional[str] = None) -> None: + """Register ``request``, send it with its id, and wait for the reply. The entry is removed on every failure (a failed send or a missing - credit left it behind), and only if it is still this request's. + credit left it behind). On return the request holds a reply. """ - cid = int(claim_id) - with self._lock: - if self._closed: - raise UsbClientClosed(_CLIENT_SHUT_DOWN_MSG) - self._pending[cid] = request + self._register(request, busy) try: - self._consume_credit(cid) - self._send(frame) + if isinstance(request.slot, int): + self._consume_credit(request.slot) + payload = dict(body) + payload[REQUEST_ID_KEY] = request.request_id + self._send(Frame(op=request.send_op, claim_id=request.claim_id, + payload=json.dumps(payload).encode("utf-8"))) except BaseException: - self._drop_pending(cid, request) + with self._lock: + self._unregister_locked(request) raise - if not request.event.wait(timeout=self._reply_timeout): - self._drop_pending(cid, request) - raise UsbClientTimeout(f"{label} timed out for claim {cid}") + if not request.event.wait(timeout=self._reply_timeout) and self._expire(request): + raise UsbClientTimeout(request.timeout_message()) if request.cancelled: - raise UsbClientClosed(f"client shut down before {label} reply") + raise UsbClientClosed(f"client shut down before {request.label} reply") + if request.reply_op is None: + raise UsbClientError("event signalled without a reply") - def _drop_pending(self, claim_id: int, request: "_PendingRequest") -> None: + def _register(self, request: "_PendingRequest", busy: Optional[str]) -> None: + """Give ``request`` its id and its slot, or refuse to send it.""" with self._lock: - if self._pending.get(claim_id) is request: - self._pending.pop(claim_id, None) + if self._closed: + raise UsbClientClosed(_CLIENT_SHUT_DOWN_MSG) + # CLOSE is exempt: its CLOSED reply is a kind no late transfer + # reply can be mistaken for, and the device must be released. + if request.send_op != Opcode.CLOSE and request.slot in self._desynced: + raise UsbClientDesynchronized(_desynchronized_message(request.slot)) + if busy is not None and request.slot in self._pending: + raise UsbClientError(busy) + request.request_id = self._next_request_id_locked() + self._pending[request.slot] = request + self._by_id[request.request_id] = request + + def _next_request_id_locked(self) -> str: + self._id_counter += 1 + return f"{self._id_prefix}-{self._id_counter}" + + def _unregister_locked(self, request: "_PendingRequest") -> None: + if self._pending.get(request.slot) is request: + self._pending.pop(request.slot, None) + if self._by_id.get(request.request_id) is request: + self._by_id.pop(request.request_id, None) + + def _expire(self, request: "_PendingRequest") -> bool: + """Give up on ``request``; False if its reply arrived just in time. + + A host that echoes ids only needs the tombstone. One that does not + (or has not shown that it does) could deliver the late reply to the + slot's next request, so the slot stops taking requests instead. + """ + with self._lock: + if self._by_id.get(request.request_id) is not request: + return False + self._unregister_locked(request) + self._bury_locked(request.request_id, request.send_op) + if not self._peer_echoes_ids: + self._desynced.add(request.slot) + return True + + def _bury_locked(self, request_id: str, send_op: Opcode) -> None: + self._tombstones[request_id] = send_op + while len(self._tombstones) > _MAX_TOMBSTONES: + self._tombstones.popitem(last=False) # --- Outbound: transfers ------------------------------------------------ def _exchange_transfer(self, claim_id: int, op: Opcode, body: Dict[str, Any]) -> bytes: - request = _PendingRequest(expected_op=op, event=threading.Event()) - frame = Frame(op=op, claim_id=int(claim_id), - payload=json.dumps(body).encode("utf-8")) + request = _PendingRequest(op, op, int(claim_id)) # Serialised per claim: a second transfer overwrote the first's # pending entry, and one caller received the other's data. with self._claim_lock(claim_id): - self._round_trip(claim_id, request, frame, op.name) - if request.reply_op is None: - raise UsbClientError("event signalled without a reply") - if request.reply_op == Opcode.ERROR: - err = _decode_json(request.reply_payload).get("error", "host ERROR") - raise UsbClientError(err) - body = _decode_json(request.reply_payload) - if not body.get("ok"): - raise UsbClientError(body.get("error", "transfer failed")) + self._round_trip(request, body) + reply = _accepted_body(request, "transfer failed") try: - return base64.b64decode(body.get("data") or "", validate=True) + return base64.b64decode(reply.get("data") or "", validate=True) except (TypeError, ValueError) as error: # binascii.Error is a ValueError raise UsbClientError(f"host sent undecodable transfer data: {error}") from error # --- Inbound dispatch helpers ------------------------------------------ - def _on_opened(self, frame: Frame) -> None: + def _route_reply(self, op: Opcode, claim_id: int, payload: bytes) -> None: + """Hand a complete reply to the request it answers, or drop it.""" + body = _decode_json(payload) + request_id = body.get(REQUEST_ID_KEY) + orphan: Optional[int] = None with self._lock: - request = self._open_pending - self._open_pending = None + if isinstance(request_id, str) and valid_request_id(request_id): + request, orphan = self._take_by_id_locked(request_id, op, body) + else: + request = self._take_by_kind_locked(op, claim_id, payload) + if request is not None: + request.reply_op = op + request.reply_body = body if request is not None: - request.reply_op = frame.op - request.reply_payload = frame.payload request.event.set() + if orphan is not None: + self._release_orphan(orphan) - def _on_list(self, frame: Frame) -> None: - assembled = self._reassemble(frame) - if assembled is None: - return - with self._lock: - request = self._list_pending - self._list_pending = None + def _take_by_id_locked(self, request_id: str, op: Opcode, body: Dict[str, Any], + ) -> Tuple[Optional[_PendingRequest], Optional[int]]: + """The request ``request_id`` names, plus a claim to release if any.""" + request = self._by_id.get(request_id) + if request is None: + return None, self._discard_late_locked(request_id, op, body) + self._peer_echoes_ids_locked() + if op not in (request.expected_op, Opcode.ERROR): + autocontrol_logger.warning( + "passthrough client: %s reply to a %s request dropped", + op.name, request.label) + return None, None + self._unregister_locked(request) + return request, None + + def _discard_late_locked(self, request_id: str, op: Opcode, + body: Dict[str, Any]) -> Optional[int]: + """Drop a reply nobody is waiting for; return a claim it leaves open. + + A late OPENED that succeeded means the host holds a device for a + caller that already got a timeout. A late RESUME is left alone: the + claim it names may have been resumed again and be in use. + """ + sent_as = self._tombstones.pop(request_id, None) + if sent_as is None: + autocontrol_logger.warning( + "passthrough client: %s reply with unknown request id dropped", op.name) + return None + self._peer_echoes_ids_locked() + autocontrol_logger.debug( + "passthrough client: late reply to timed-out %s discarded", sent_as.name) + if sent_as == Opcode.OPEN and op == Opcode.OPENED and body.get("ok"): + return _claim_id_in(body) + return None + + def _peer_echoes_ids_locked(self) -> None: + """The host echoed an id of ours: every reply is identifiable.""" + self._peer_echoes_ids = True + self._desynced.clear() + + def _take_by_kind_locked(self, op: Opcode, claim_id: int, + payload: bytes) -> Optional[_PendingRequest]: + """Pair a reply that has no id the way hosts without ids require.""" + slot: _Slot = claim_id + if op == Opcode.OPENED: + slot = _OPEN_SLOT + elif op == Opcode.LIST: + slot = _LIST_SLOT + request = self._pending.get(slot) + if op == Opcode.ERROR: + self._reasm.pop(claim_id, None) + if request is None: + autocontrol_logger.warning( + "passthrough client: unsolicited ERROR for claim %s: %s", + claim_id, payload[:200]) + elif request is not None and request.expected_op != op: + return None if request is not None: - request.reply_op = Opcode.LIST - request.reply_payload = assembled - request.event.set() + self._unregister_locked(request) + return request + + def _release_orphan(self, claim_id: int) -> None: + """CLOSE a claim whose OPEN had already timed out; nobody holds it.""" + with self._lock: + if self._closed: + return + request_id = self._next_request_id_locked() + self._bury_locked(request_id, Opcode.CLOSE) # its CLOSED is unwanted + try: + self._send(Frame( + op=Opcode.CLOSE, claim_id=claim_id, + payload=json.dumps({REQUEST_ID_KEY: request_id}).encode("utf-8"))) + except (UsbClientError, ProtocolError) as error: + autocontrol_logger.warning( + "passthrough client: could not release claim %s left by a late " + "OPENED: %r", claim_id, error) def _on_credit(self, frame: Frame) -> None: try: @@ -485,35 +643,6 @@ def _on_credit(self, frame: Frame) -> None: if event is not None: event.set() - def _on_error(self, frame: Frame) -> None: - # An unsolicited ERROR — route to whichever pending request matches - # the claim_id; if none, log and drop. - with self._lock: - request = self._pending.pop(int(frame.claim_id), None) - self._reasm.pop(int(frame.claim_id), None) - if request is None: - autocontrol_logger.warning( - "passthrough client: unsolicited ERROR for claim %s: %s", - frame.claim_id, frame.payload[:200], - ) - return - request.reply_op = frame.op - request.reply_payload = frame.payload - request.event.set() - - def _complete_pending(self, claim_id: int, payload: bytes, - expected_op: Opcode) -> None: - with self._lock: - request = self._pending.get(int(claim_id)) - if request is None: - return - if request.expected_op != expected_op: - return - self._pending.pop(int(claim_id), None) - request.reply_op = expected_op - request.reply_payload = payload - request.event.set() - # --- Credit helpers ---------------------------------------------------- def _consume_credit(self, claim_id: int) -> None: @@ -550,7 +679,7 @@ def credits_remaining(self, claim_id: int) -> int: def pending_count(self) -> int: with self._lock: - return len(self._pending) + (1 if self._open_pending else 0) + return sum(1 for slot in self._pending if slot != _LIST_SLOT) # --- Internal ---------------------------------------------------------- @@ -582,6 +711,25 @@ def _endpoint_request(*, endpoint: int, direction: str, data: bytes, return body +def _accepted_body(request: _PendingRequest, default_error: str) -> Dict[str, Any]: + """The reply's body, or :class:`UsbClientError` if the host refused.""" + body = request.reply_body + if request.reply_op == Opcode.ERROR: + raise UsbClientError(body.get("error", "host ERROR")) + if not body.get("ok"): + raise UsbClientError(body.get("error", default_error)) + return body + + +def _claim_id_in(body: Dict[str, Any]) -> Optional[int]: + """The ``claim_id`` of an OPENED body if it is a usable one.""" + try: + claim_id = int(body["claim_id"]) + except (KeyError, TypeError, ValueError): + return None + return claim_id if 0 <= claim_id <= 0xFFFF else None + + def _decode_json(payload: bytes) -> Dict[str, Any]: if not payload: return {} @@ -595,6 +743,6 @@ def _decode_json(payload: bytes) -> Dict[str, Any]: __all__ = [ - "ClientHandle", "UsbClientClosed", "UsbClientError", "UsbClientTimeout", - "UsbPassthroughClient", + "ClientHandle", "UsbClientClosed", "UsbClientDesynchronized", + "UsbClientError", "UsbClientTimeout", "UsbPassthroughClient", ] diff --git a/test/unit_test/headless/test_usb_request_correlation.py b/test/unit_test/headless/test_usb_request_correlation.py new file mode 100644 index 000000000..43dcbe022 --- /dev/null +++ b/test/unit_test/headless/test_usb_request_correlation.py @@ -0,0 +1,586 @@ +"""USB passthrough replies are paired with the request that asked for them. + +Replies used to be matched by kind only (OPEN / LIST / claim), so a reply +that arrived after its request had timed out completed the *next* request +of that kind: ``open(bbbb)`` bound the claim opened for ``aaaa``, and a +bulk read returned the previous read's data. + +The wire is a fake: frames the client sends are captured in a list and the +test plays the host by hand, which is the only way to deliver a reply late. +""" +import base64 +import json +import threading + +import pytest + +from je_auto_control.utils.usb.passthrough import ( + FLAG_EOF, Frame, MAX_PAYLOAD_BYTES, Opcode, UsbClientError, + UsbClientTimeout, UsbPassthroughClient, UsbPassthroughSession, + fragment_payload, viewer_client, +) +from je_auto_control.utils.usb.passthrough.backend import ( + BackendDevice, FakeUsbBackend, +) + +_SAMPLE = BackendDevice(vendor_id="1050", product_id="0407", serial="ABC123") +_WAIT_S = 5.0 +#: Replies a host may split across frames; the last one carries FLAG_EOF. +_FRAGMENTED = (Opcode.LIST, Opcode.CTRL, Opcode.BULK, Opcode.INT) + + +class _Wire: + """Captures what the client sends; the test feeds the replies.""" + + def __init__(self, *, timeout_s: float = 0.15) -> None: + self.sent = [] + self._cond = threading.Condition() + self.client = UsbPassthroughClient( + send_frame=self._send, reply_timeout_s=timeout_s, + ) + + def _send(self, frame: Frame) -> None: + with self._cond: + self.sent.append(frame) + self._cond.notify_all() + + def wait_sent(self, count: int) -> Frame: + """Block until the client has sent ``count`` frames; return the last.""" + with self._cond: + if not self._cond.wait_for(lambda: len(self.sent) >= count, _WAIT_S): + raise AssertionError(f"client sent {len(self.sent)} frames, wanted {count}") + return self.sent[count - 1] + + def patient(self) -> None: + """Later requests must not time out while the test is feeding frames.""" + self.client._reply_timeout = _WAIT_S # noqa: SLF001 # reason: the first request needs a short timeout, the rest a long one + + +class _Call: + """Runs a blocking client call on a thread and keeps its outcome.""" + + def __init__(self, function) -> None: + self.result = None + self.error = None + self._thread = threading.Thread(target=self._run, args=(function,), daemon=True) + self._thread.start() + + def _run(self, function) -> None: + try: + self.result = function() + except Exception as error: # noqa: BLE001 # reason: the test asserts on whatever the call raised + self.error = error + + def finish(self): + self._thread.join(_WAIT_S) + assert not self._thread.is_alive(), "client call never returned" + if self.error is not None: + raise self.error + return self.result + + +def _body(frame: Frame) -> dict: + return json.loads(frame.payload.decode("utf-8")) if frame.payload else {} + + +def _payload(body: dict, request: Frame = None) -> bytes: + """A reply payload; echoes ``request``'s id the way a current host does.""" + body = dict(body) + request_id = _body(request).get("request_id") if request is not None else None + if request_id is not None: + body["request_id"] = request_id + return json.dumps(body).encode("utf-8") + + +def _reply(op: Opcode, claim_id: int, body: dict, request: Frame = None) -> Frame: + flags = FLAG_EOF if op in _FRAGMENTED else 0 + return Frame(op=op, flags=flags, claim_id=claim_id, payload=_payload(body, request)) + + +def _data_body(data: bytes) -> dict: + return {"ok": True, "data": base64.b64encode(data).decode("ascii")} + + +def _data_reply(op: Opcode, claim_id: int, data: bytes, request: Frame = None) -> Frame: + return _reply(op, claim_id, _data_body(data), request) + + +def _big_reply(claim_id: int, data: bytes, request: Frame) -> list: + """A BULK reply too large for one frame, split the way the host splits it.""" + frames = fragment_payload(Opcode.BULK, claim_id, _payload(_data_body(data), request)) + assert len(frames) > 1 + return frames + + +def _current_peer(wire: "_Wire") -> None: + """Answer one LIST with its id echoed, so the client knows the host is current.""" + sent_before = len(wire.sent) + call = _Call(wire.client.list_devices) + request = wire.wait_sent(sent_before + 1) + wire.client.feed_frame(_reply(Opcode.LIST, 0, {"devices": []}, request)) + call.finish() + + +def _opened(wire: _Wire, claim_id: int, *, echo: bool = True): + """Open a claim on ``wire``; the reply echoes the id unless told not to.""" + sent_before = len(wire.sent) + call = _Call(lambda: wire.client.open(vendor_id="1050", product_id="0407")) + request = wire.wait_sent(sent_before + 1) + wire.client.feed_frame(_reply( + Opcode.OPENED, claim_id, {"ok": True, "claim_id": claim_id}, + request if echo else None)) + return call.finish() + + +def _timed_out_bulk(wire: _Wire, handle) -> Frame: + """Issue a bulk read nobody answers; return the request that timed out.""" + sent_before = len(wire.sent) + with pytest.raises(UsbClientTimeout): + handle.bulk_transfer(endpoint=0x81, direction="in", length=8) + return wire.sent[sent_before] + + +# --- the two failures the audit reproduced --------------------------------- + + +def test_late_open_does_not_complete_next_open(): + wire = _Wire() + _current_peer(wire) + with pytest.raises(UsbClientTimeout): + wire.client.open(vendor_id="aaaa", product_id="0001") + first = wire.sent[1] + wire.patient() + + call = _Call(lambda: wire.client.open(vendor_id="bbbb", product_id="0002")) + second = wire.wait_sent(3) + wire.client.feed_frame(_reply(Opcode.OPENED, 1, {"ok": True, "claim_id": 1}, first)) + assert wire.client.pending_count() == 1, "the late OPENED completed the next open" + + wire.client.feed_frame(_reply(Opcode.OPENED, 2, {"ok": True, "claim_id": 2}, second)) + assert call.finish().claim_id == 2 + + +def test_late_transfer_does_not_complete_next_transfer(): + wire = _Wire() + handle = _opened(wire, 1) + first = _timed_out_bulk(wire, handle) + wire.patient() + + call = _Call(lambda: handle.bulk_transfer(endpoint=0x81, direction="in", length=8)) + second = wire.wait_sent(3) + wire.client.feed_frame(_data_reply(Opcode.BULK, 1, b"stale", first)) + assert wire.client.pending_count() == 1, "the late BULK completed the next transfer" + + wire.client.feed_frame(_data_reply(Opcode.BULK, 1, b"fresh", second)) + assert call.finish() == b"fresh" + + +def test_late_list_does_not_complete_next_list(): + wire = _Wire() + _current_peer(wire) + with pytest.raises(UsbClientTimeout): + wire.client.list_devices() + first = wire.sent[1] + wire.patient() + + call = _Call(wire.client.list_devices) + second = wire.wait_sent(3) + wire.client.feed_frame(_reply(Opcode.LIST, 0, {"devices": [{"vendor_id": "old"}]}, first)) + wire.client.feed_frame(_reply(Opcode.LIST, 0, {"devices": [{"vendor_id": "new"}]}, second)) + assert call.finish() == [{"vendor_id": "new"}] + + +# --- a peer that echoes ids: only the expired reply is discarded ----------- + + +def test_every_request_carries_a_distinct_string_id(): + wire = _Wire() + handle = _opened(wire, 1) + for _ in range(3): + _timed_out_bulk(wire, handle) + ids = [_body(frame)["request_id"] for frame in wire.sent] + assert all(isinstance(value, str) and value for value in ids) + assert len(set(ids)) == len(ids) + + +def test_claim_stays_usable_after_timeout_when_peer_echoes_ids(): + wire = _Wire() + handle = _opened(wire, 1) # the OPENED echoed an id: the peer is current + _timed_out_bulk(wire, handle) + assert wire.client.peer_echoes_request_ids is True + assert handle.reusable is True + + +def test_late_open_reply_releases_the_claim_nobody_holds(): + """The host opened the device for a caller that has already given up.""" + wire = _Wire() + with pytest.raises(UsbClientTimeout): + wire.client.open(vendor_id="aaaa", product_id="0001") + wire.client.feed_frame(_reply( + Opcode.OPENED, 7, {"ok": True, "claim_id": 7}, wire.sent[0])) + assert wire.client.reusable is True # the echoed id showed the host is current + close = wire.wait_sent(2) + assert (close.op, close.claim_id) == (Opcode.CLOSE, 7) + # Its CLOSED is expected by nobody and must be dropped quietly. + wire.client.feed_frame(_reply(Opcode.CLOSED, 7, {"ok": True}, close)) + assert wire.client.pending_count() == 0 + + +def test_late_resume_reply_does_not_close_the_claim(): + """A resumed claim may be in use again by the time the late reply lands.""" + wire = _Wire() + with pytest.raises(UsbClientTimeout): + wire.client.resume("token") + wire.client.feed_frame(_reply( + Opcode.OPENED, 7, {"ok": True, "claim_id": 7}, wire.sent[0])) + assert len(wire.sent) == 1 + + +def test_late_fragmented_reply_is_discarded_whole(): + wire = _Wire() + handle = _opened(wire, 1) + first = _timed_out_bulk(wire, handle) + wire.patient() + stale_frames = _big_reply(1, b"\x01" * (MAX_PAYLOAD_BYTES * 2), first) + + call = _Call(lambda: handle.bulk_transfer(endpoint=0x81, direction="in", length=8)) + second = wire.wait_sent(3) + for frame in stale_frames: + wire.client.feed_frame(frame) + assert wire.client.pending_count() == 1 + for frame in _big_reply(1, b"\x02" * (MAX_PAYLOAD_BYTES * 2), second): + wire.client.feed_frame(frame) + assert call.finish() == b"\x02" * (MAX_PAYLOAD_BYTES * 2) + + +def test_reply_half_received_at_the_timeout_is_still_discarded_whole(): + """The buffer is not reset by the timeout, so the tail is not a new message.""" + wire = _Wire(timeout_s=0.5) + handle = _opened(wire, 1) + sent_before = len(wire.sent) + call = _Call(lambda: handle.bulk_transfer(endpoint=0x81, direction="in", length=8)) + first = wire.wait_sent(sent_before + 1) + stale_frames = _big_reply(1, b"\x01" * (MAX_PAYLOAD_BYTES * 2), first) + wire.client.feed_frame(stale_frames[0]) + with pytest.raises(UsbClientTimeout): + call.finish() + wire.patient() + + call = _Call(lambda: handle.bulk_transfer(endpoint=0x81, direction="in", length=8)) + second = wire.wait_sent(sent_before + 2) + for frame in stale_frames[1:]: + wire.client.feed_frame(frame) + assert wire.client.pending_count() == 1 + wire.client.feed_frame(_data_reply(Opcode.BULK, 1, b"fresh", second)) + assert call.finish() == b"fresh" + + +def test_late_error_does_not_fail_the_next_transfer(): + wire = _Wire() + handle = _opened(wire, 1) + first = _timed_out_bulk(wire, handle) + wire.patient() + + call = _Call(lambda: handle.bulk_transfer(endpoint=0x81, direction="in", length=8)) + second = wire.wait_sent(3) + wire.client.feed_frame(_reply(Opcode.ERROR, 1, {"error": "credit exhausted"}, first)) + assert wire.client.pending_count() == 1, "the late ERROR failed the next transfer" + wire.client.feed_frame(_data_reply(Opcode.BULK, 1, b"fresh", second)) + assert call.finish() == b"fresh" + + +def test_error_with_an_id_fails_the_open_it_answers(): + """A locked-out host answers OPEN with ERROR on claim 0; it used to time out.""" + wire = _Wire(timeout_s=_WAIT_S) + call = _Call(lambda: wire.client.open(vendor_id="1050", product_id="0407")) + request = wire.wait_sent(1) + wire.client.feed_frame(_reply( + Opcode.ERROR, 0, {"error": "rate limited; locked out"}, request)) + with pytest.raises(UsbClientError, match="rate limited"): + call.finish() + + +def test_credit_from_a_late_reply_is_still_granted(): + """CREDIT is a grant on the claim, not a reply to one request.""" + wire = _Wire() + handle = _opened(wire, 1) + before = wire.client.credits_remaining(1) + first = _timed_out_bulk(wire, handle) + assert wire.client.credits_remaining(1) == before - 1 + wire.client.feed_frame(_data_reply(Opcode.BULK, 1, b"stale", first)) + wire.client.feed_frame(Frame(op=Opcode.CREDIT, claim_id=1, payload=b'{"credits": 1}')) + assert wire.client.credits_remaining(1) == before + + +def test_reply_with_the_wrong_kind_for_its_id_is_dropped(): + wire = _Wire() + handle = _opened(wire, 1) + wire.patient() + call = _Call(lambda: handle.bulk_transfer(endpoint=0x81, direction="in", length=8)) + request = wire.wait_sent(2) + wire.client.feed_frame(_data_reply(Opcode.CTRL, 1, b"wrong", request)) + assert wire.client.pending_count() == 1 + wire.client.feed_frame(_data_reply(Opcode.BULK, 1, b"right", request)) + assert call.finish() == b"right" + + +def test_oversize_message_is_skipped_through_its_last_fragment(): + """The tail of a dropped message used to be parsed as a message of its own.""" + wire = _Wire(timeout_s=_WAIT_S) + handle = _opened(wire, 1) + call = _Call(lambda: handle.bulk_transfer(endpoint=0x81, direction="in", length=8)) + request = wire.wait_sent(2) + chunk = b"x" * MAX_PAYLOAD_BYTES + limit = viewer_client._MAX_REASSEMBLED_BYTES # noqa: SLF001 # reason: the cap is the thing under test + for _ in range(limit // MAX_PAYLOAD_BYTES + 1): + wire.client.feed_frame(Frame(op=Opcode.BULK, claim_id=1, payload=chunk)) + wire.client.feed_frame(Frame(op=Opcode.BULK, claim_id=1, flags=FLAG_EOF, payload=b"tail")) + assert wire.client.pending_count() == 1, "the tail completed the request" + wire.client.feed_frame(_data_reply(Opcode.BULK, 1, b"next", request)) + assert call.finish() == b"next" + + +# --- a peer that does not echo ids: stop, do not guess --------------------- + + +def test_legacy_timeout_requires_reconnect(): + wire = _Wire() + handle = _opened(wire, 1, echo=False) + _timed_out_bulk(wire, handle) + assert wire.client.peer_echoes_request_ids is False + assert handle.reusable is False + + sent = len(wire.sent) + with pytest.raises(viewer_client.UsbClientDesynchronized, match="claim 1"): + handle.bulk_transfer(endpoint=0x81, direction="in", length=8) + assert len(wire.sent) == sent, "a request went out on a desynchronised claim" + # The late reply has nobody to mislead any more. + wire.client.feed_frame(_data_reply(Opcode.BULK, 1, b"stale")) + assert wire.client.pending_count() == 0 + + +def test_legacy_desynchronised_claim_can_still_be_closed(): + wire = _Wire() + handle = _opened(wire, 1, echo=False) + _timed_out_bulk(wire, handle) + wire.patient() + call = _Call(handle.close) + close = wire.wait_sent(3) + assert close.op == Opcode.CLOSE + wire.client.feed_frame(_data_reply(Opcode.BULK, 1, b"stale")) # wrong kind for CLOSE + assert wire.client.pending_count() == 1 + wire.client.feed_frame(_reply(Opcode.CLOSED, 1, {"ok": True})) + call.finish() + assert handle.closed + + +def test_legacy_timeout_on_one_claim_leaves_the_others_usable(): + wire = _Wire() + broken = _opened(wire, 1, echo=False) + healthy = _opened(wire, 2, echo=False) + _timed_out_bulk(wire, broken) + wire.patient() + assert healthy.reusable is True + call = _Call(lambda: healthy.bulk_transfer(endpoint=0x81, direction="in", length=8)) + wire.wait_sent(4) + wire.client.feed_frame(_data_reply(Opcode.BULK, 2, b"fine")) + assert call.finish() == b"fine" + + +def test_legacy_open_timeout_blocks_further_opens(): + wire = _Wire() + with pytest.raises(UsbClientTimeout): + wire.client.open(vendor_id="aaaa", product_id="0001") + assert wire.client.reusable is False + with pytest.raises(viewer_client.UsbClientDesynchronized, match="[Rr]econnect"): + wire.client.open(vendor_id="bbbb", product_id="0002") + with pytest.raises(viewer_client.UsbClientDesynchronized): + wire.client.resume("token") + assert len(wire.sent) == 1 + # The late, id-less OPENED is dropped rather than handed to anyone. + wire.client.feed_frame(_reply(Opcode.OPENED, 1, {"ok": True, "claim_id": 1})) + assert wire.client.reusable is False + + +def test_legacy_list_timeout_blocks_further_lists(): + wire = _Wire() + with pytest.raises(UsbClientTimeout): + wire.client.list_devices() + with pytest.raises(viewer_client.UsbClientDesynchronized): + wire.client.list_devices() + + +def test_desynchronised_error_is_a_client_error(): + from je_auto_control.utils.exception.exceptions import AutoControlException + assert issubclass(viewer_client.UsbClientDesynchronized, UsbClientError) + assert issubclass(viewer_client.UsbClientDesynchronized, AutoControlException) + + +def test_peer_of_unknown_age_is_cleared_once_it_echoes_an_id(): + """The very first request timed out, so nothing says yet what the peer is.""" + wire = _Wire() + with pytest.raises(UsbClientTimeout): + wire.client.open(vendor_id="aaaa", product_id="0001") + assert wire.client.reusable is False + wire.client.feed_frame(_reply( + Opcode.OPENED, 0, {"ok": False, "error": "no device"}, wire.sent[0])) + assert wire.client.peer_echoes_request_ids is True + assert wire.client.reusable is True + assert _opened(wire, 3).claim_id == 3 + + +def test_reply_with_an_id_nobody_issued_proves_nothing(): + wire = _Wire() + with pytest.raises(UsbClientTimeout): + wire.client.open(vendor_id="aaaa", product_id="0001") + wire.client.feed_frame(Frame( + op=Opcode.OPENED, claim_id=1, + payload=b'{"ok": true, "claim_id": 1, "request_id": "not-ours"}')) + assert wire.client.peer_echoes_request_ids is False + assert wire.client.reusable is False + + +# --- the host -------------------------------------------------------------- + + +def _session() -> UsbPassthroughSession: + return UsbPassthroughSession(FakeUsbBackend(devices=[_SAMPLE])) + + +def _request(op: Opcode, claim_id: int = 0, **body) -> Frame: + return Frame(op=op, claim_id=claim_id, payload=json.dumps(body).encode("utf-8")) + + +def _joined(replies, op: Opcode) -> dict: + return json.loads(b"".join(r.payload for r in replies if r.op == op).decode("utf-8")) + + +_OPEN_BODY = {"vendor_id": "1050", "product_id": "0407", "serial": "ABC123"} +_CTRL_BODY = {"bm_request_type": 0xC0, "b_request": 6, "length": 18} + + +def test_host_echoes_the_id_on_every_reply_kind(): + session = _session() + opened = _joined(session.handle_frame( + _request(Opcode.OPEN, request_id="r-open", **_OPEN_BODY)), Opcode.OPENED) + assert (opened["ok"], opened["request_id"]) == (True, "r-open") + claim = opened["claim_id"] + + resumed = _joined(session.handle_frame(_request( + Opcode.RESUME, resume_token=opened["resume_token"], request_id="r-res")), Opcode.OPENED) + assert resumed["request_id"] == "r-res" + + listed = _joined(session.handle_frame( + _request(Opcode.LIST, request_id="r-list")), Opcode.LIST) + assert listed["request_id"] == "r-list" and len(listed["devices"]) == 1 + + for op in (Opcode.CTRL, Opcode.BULK, Opcode.INT): + body = _CTRL_BODY if op == Opcode.CTRL else { + "endpoint": 0x81, "direction": "in", "length": 4} + replies = session.handle_frame(_request(op, claim, request_id=f"r-{op.name}", **body)) + assert _joined(replies, op)["request_id"] == f"r-{op.name}" + + unknown = _joined(session.handle_frame( + _request(Opcode.BULK, 99, request_id="r-err", endpoint=1, direction="in")), Opcode.ERROR) + assert unknown["request_id"] == "r-err" + + closed = _joined(session.handle_frame( + _request(Opcode.CLOSE, claim, request_id="r-close")), Opcode.CLOSED) + assert closed == {"ok": True, "request_id": "r-close"} + + +def test_host_does_not_put_an_id_on_credit_frames(): + session = _session() + claim = _joined(session.handle_frame( + _request(Opcode.OPEN, request_id="a", **_OPEN_BODY)), Opcode.OPENED)["claim_id"] + replies = session.handle_frame(_request(Opcode.CTRL, claim, request_id="b", **_CTRL_BODY)) + assert _joined(replies, Opcode.CREDIT) == {"credits": 1} + + +def test_host_reply_to_a_request_without_an_id_is_unchanged(): + """An older viewer must see exactly the payloads it always saw.""" + session = _session() + opened = _joined(session.handle_frame(_request(Opcode.OPEN, **_OPEN_BODY)), Opcode.OPENED) + assert set(opened) == {"ok", "claim_id", "resume_token"} + claim = opened["claim_id"] + assert set(_joined(session.handle_frame(Frame(op=Opcode.LIST)), Opcode.LIST)) == {"devices"} + transfer = session.handle_frame(_request(Opcode.CTRL, claim, **_CTRL_BODY)) + assert set(_joined(transfer, Opcode.CTRL)) == {"ok", "data"} + assert _joined(session.handle_frame( + Frame(op=Opcode.CLOSE, claim_id=claim)), Opcode.CLOSED) == {"ok": True} + assert set(_joined(session.handle_frame( + Frame(op=Opcode.CLOSE, claim_id=claim)), Opcode.ERROR)) == {"error"} + + +@pytest.mark.parametrize("bad_id", [7, "", "x" * 65, None, ["a"], {"a": 1}, True]) +def test_host_ignores_an_id_it_cannot_echo(bad_id): + session = _session() + opened = _joined(session.handle_frame( + _request(Opcode.OPEN, request_id=bad_id, **_OPEN_BODY)), Opcode.OPENED) + assert opened["ok"] is True and "request_id" not in opened + + +def test_host_echoes_the_id_when_the_peer_is_locked_out(): + session = _session() + for _ in range(40): + session.handle_frame(Frame(op=Opcode.CLOSE, claim_id=99)) + assert session.is_locked_out() + reply = _joined(session.handle_frame( + _request(Opcode.OPEN, request_id="r", **_OPEN_BODY)), Opcode.ERROR) + assert reply["request_id"] == "r" and "locked out" in reply["error"] + + +def test_host_echoes_the_id_on_a_fragmented_reply(): + session = _session() + claim = _joined(session.handle_frame( + _request(Opcode.OPEN, request_id="a", **_OPEN_BODY)), Opcode.OPENED)["claim_id"] + replies = session.handle_frame(_request( + Opcode.BULK, claim, request_id="big", endpoint=0x81, direction="in", + length=MAX_PAYLOAD_BYTES * 2)) + data_frames = [r for r in replies if r.op == Opcode.BULK] + assert len(data_frames) > 1 + assert _joined(replies, Opcode.BULK)["request_id"] == "big" + + +# --- both ends together, across versions ----------------------------------- + + +class _Routed: + """A real session behind the client; ``legacy_host`` strips echoed ids.""" + + def __init__(self, *, legacy_host: bool) -> None: + self.session = _session() + self._legacy_host = legacy_host + self.client = UsbPassthroughClient(send_frame=self._send, reply_timeout_s=_WAIT_S) + + def _send(self, frame: Frame) -> None: + if self._legacy_host: + frame = _without_id(frame) # an old host never reads the field + threading.Thread(target=self._deliver, args=(frame,), daemon=True).start() + + def _deliver(self, frame: Frame) -> None: + for reply in self.session.handle_frame(frame): + self.client.feed_frame(reply) + + +def _without_id(frame: Frame) -> Frame: + body = _body(frame) + body.pop("request_id", None) + payload = json.dumps(body).encode("utf-8") if body else b"" + return Frame(op=frame.op, flags=frame.flags, claim_id=frame.claim_id, payload=payload) + + +@pytest.mark.parametrize("legacy_host", [False, True]) +def test_full_exchange_against_current_and_legacy_host(legacy_host): + routed = _Routed(legacy_host=legacy_host) + assert len(routed.client.list_devices()) == 1 + handle = routed.client.open(vendor_id="1050", product_id="0407", serial="ABC123") + assert isinstance(handle.control_transfer( + bm_request_type=0xC0, b_request=6, length=18), bytes) + assert isinstance(handle.bulk_transfer( + endpoint=0x81, direction="in", length=MAX_PAYLOAD_BYTES * 2), bytes) + assert routed.client.resume(handle.resume_token).claim_id == handle.claim_id + handle.close() + assert routed.session.active_claim_count == 0 + assert routed.client.peer_echoes_request_ids is (not legacy_host) + routed.client.shutdown() From eeb60e2842e6bcb0b0f995bbfa1bbd33cf6b3a01 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:01:45 +0800 Subject: [PATCH 13/26] Report window focus, show and z-order failures and stop layouts drifting Windows' foreground lock refuses a background process often, and focus_window, show_window and the z-order driver reported success anyway, so the caller's next keystrokes went to whatever window the user had active. focus_window now confirms the window is in front and raises otherwise; the others return what Win32 answered. post_key posts a printable character as WM_CHAR alone and other keys with a real lParam: the old trio was translated into three characters by the target. post_key_to_window resolves enter/esc through the platform's own key names. Cloaked and zero-area windows are no longer listed. A saved layout stores the rectangle MoveWindow positions, so restoring no longer moves a window 7 px per round, and snap/grid/cascade use the work area. wait_for_window clamps its poll and never sleeps past the timeout. --- .../Eng/doc/new_features/v4_features_doc.rst | 19 +- .../Zh/doc/new_features/v4_features_doc.rst | 15 +- .../utils/window_capture/window_capture.py | 115 ++++- .../utils/window_zorder/window_zorder.py | 5 +- .../windows/window/windows_window_manage.py | 147 +++++- .../wrapper/auto_control_window.py | 60 ++- .../wrapper/window_backends/base.py | 8 +- .../window_backends/windows_backend.py | 4 +- .../headless/test_a11y_window_audit.py | 3 +- .../headless/test_window_backends.py | 4 +- .../test_window_lifecycle_contract.py | 464 ++++++++++++++++++ test/unit_test/headless/test_window_manage.py | 3 + 12 files changed, 775 insertions(+), 72 deletions(-) create mode 100644 test/unit_test/headless/test_window_lifecycle_contract.py diff --git a/docs/source/Eng/doc/new_features/v4_features_doc.rst b/docs/source/Eng/doc/new_features/v4_features_doc.rst index 378ba0571..44fe68347 100644 --- a/docs/source/Eng/doc/new_features/v4_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v4_features_doc.rst @@ -111,15 +111,22 @@ Window management ================= * **Per-window capture** — ``capture_window(title, output_path)`` - resolves a window's geometry by title (Win32 ``GetWindowRect``) and - screenshots exactly its bounds. ``AC_capture_window``. + resolves a window's geometry by title (its visible frame, as DWM reports + it) and screenshots exactly its bounds, on whichever monitor the window + is. ``AC_capture_window``. * **Layout save / restore** — ``save_window_layout(path)`` snapshots every window's position to JSON; ``restore_window_layout(path)`` moves them all - back (handy for test setup / teardown). ``AC_save_window_layout`` / - ``AC_restore_window_layout``. + back (handy for test setup / teardown). The saved rectangle is the one + ``MoveWindow`` positions (Win32 ``GetWindowRect``), so a window comes back + exactly where it was. A layout file saved by an earlier version holds the + visible frame instead and restores 7 px to the right and 14 x 7 px smaller; + save it again. ``AC_save_window_layout`` / ``AC_restore_window_layout``. * **Snap / tile** — ``snap_window(title, "left")`` moves a window to a - screen half (left / right / top / bottom), a quarter (the four corners), - or ``"max"``. ``AC_snap_window``. + half (left / right / top / bottom), a quarter (the four corners), or + ``"max"`` of the primary monitor's *work area* — the screen minus the + taskbar — so the bottom of a snapped window is not under the taskbar. + ``arrange_grid`` and ``arrange_cascade`` lay out in the same area. + ``AC_snap_window``. File security & safety diff --git a/docs/source/Zh/doc/new_features/v4_features_doc.rst b/docs/source/Zh/doc/new_features/v4_features_doc.rst index 2a010e192..ea337fb8e 100644 --- a/docs/source/Zh/doc/new_features/v4_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v4_features_doc.rst @@ -95,13 +95,18 @@ Builder 項目。視覺與視窗功能的 geometry / IO 操作皆可注入,因 ======== * **單一視窗擷取** — ``capture_window(title, output_path)`` 以標題解析視窗 - geometry(Win32 ``GetWindowRect``)並精確擷取其範圍。``AC_capture_window``。 + geometry(DWM 回報的可見外框)並精確擷取其範圍,視窗在哪個螢幕都可以。 + ``AC_capture_window``。 * **版面儲存 / 還原** — ``save_window_layout(path)`` 把每個視窗的位置快照 成 JSON;``restore_window_layout(path)`` 再把它們全部移回(方便測試 - setup / teardown)。``AC_save_window_layout`` / ``AC_restore_window_layout``。 -* **貼齊 / 平鋪** — ``snap_window(title, "left")`` 把視窗移到螢幕一半 - (left / right / top / bottom)、四分之一(四個角)或 ``"max"``。 - ``AC_snap_window``。 + setup / teardown)。存下的是 ``MoveWindow`` 定位的那個矩形(Win32 + ``GetWindowRect``),所以視窗會回到原本的位置。舊版存的版面檔記的是可見 + 外框,還原會右移 7 px、縮小 14 x 7 px,請重新儲存。 + ``AC_save_window_layout`` / ``AC_restore_window_layout``。 +* **貼齊 / 平鋪** — ``snap_window(title, "left")`` 把視窗移到主螢幕\ *工作區*\ + (螢幕扣掉工作列)的一半(left / right / top / bottom)、四分之一(四個角) + 或 ``"max"``,貼齊後視窗底部不會被工作列蓋住。``arrange_grid`` 與 + ``arrange_cascade`` 也排在同一個區域。``AC_snap_window``。 檔案安全 diff --git a/je_auto_control/utils/window_capture/window_capture.py b/je_auto_control/utils/window_capture/window_capture.py index f497daa6c..a8dc15586 100644 --- a/je_auto_control/utils/window_capture/window_capture.py +++ b/je_auto_control/utils/window_capture/window_capture.py @@ -5,9 +5,18 @@ bounds, plus save every window's position and move them all back later (handy for test setup / teardown). -Window geometry is read per-platform — on Windows via the Win32 -``GetWindowRect`` API; other platforms return ``None`` for now. The -geometry / capture / list / move operations are all injectable so the +Window geometry is read per-platform — on Windows via Win32; other +platforms return ``None`` for now. Two rectangles are in play there and they +are not interchangeable: a capture wants the *visible* frame (DWM's extended +frame bounds), a saved layout wants ``GetWindowRect``, because that is the +rectangle ``MoveWindow`` positions. It includes the invisible resize borders +of Windows 10 / 11, so restoring the visible frame through ``MoveWindow`` +moved every window 7 px right and shrank it by 14 x 7 px on each round. + +Snap, grid and cascade lay windows out in the primary monitor's work area +(the screen minus the taskbar), not the whole screen. + +The geometry / capture / list / move operations are all injectable so the logic is fully unit-testable without real windows. GUI-free. """ import json @@ -21,6 +30,7 @@ WindowLister = Callable[[], List[Tuple[int, str]]] WindowMover = Callable[[str, int, int, int, int], bool] SizeProvider = Callable[[], Tuple[int, int]] +AreaProvider = Callable[[], Tuple[int, int, int, int]] def get_window_geometry(title: str, @@ -58,6 +68,33 @@ def _win32_geometry(hwnd: int) -> Optional[Rect]: _DWMWA_EXTENDED_FRAME_BOUNDS = 9 +_SPI_GETWORKAREA = 0x0030 + + +def _win32_window_rect(hwnd: int) -> Optional[Rect]: + """``GetWindowRect`` as ``(x, y, width, height)``: what ``MoveWindow`` takes back.""" + import ctypes + from ctypes import wintypes + rect = wintypes.RECT() + user32 = ctypes.windll.user32 # type: ignore[attr-defined] # reason: win32-only ctypes + if user32.IsIconic(hwnd) or not user32.GetWindowRect(hwnd, ctypes.byref(rect)): + # A minimized window sits at (-32000, -32000); saving that would + # "restore" it off every screen. + return None + return (rect.left, rect.top, + rect.right - rect.left, rect.bottom - rect.top) + + +def _win32_work_area() -> Optional[Rect]: + """The primary monitor's work area as ``(x, y, width, height)``, or ``None``.""" + import ctypes + from ctypes import wintypes + rect = wintypes.RECT() + user32 = ctypes.windll.user32 # type: ignore[attr-defined] # reason: win32-only ctypes + if not user32.SystemParametersInfoW(_SPI_GETWORKAREA, 0, ctypes.byref(rect), 0): + return None + width, height = rect.right - rect.left, rect.bottom - rect.top + return (rect.left, rect.top, width, height) if width > 0 and height > 0 else None def _default_capture(output_path: str, rect: Rect) -> None: @@ -106,6 +143,11 @@ def save_window_layout(path: Optional[Union[str, Path]] = None, *, is given, also writes it as JSON for a later :func:`restore_window_layout`. Windows with no readable geometry are skipped. + + The rectangle is the one ``MoveWindow`` positions (``GetWindowRect``), so + saving and restoring leaves a window exactly where it was. A layout file + written before this held the visible frame instead and still restores + 7 px off; save it again. """ layout: List[Dict[str, Any]] = [] for hwnd, title in (lister or _default_lister)(): @@ -138,7 +180,8 @@ def _default_mover(title: str, x: int, y: int, def _handle_geometry(hwnd: int) -> Optional[Rect]: - return _win32_geometry(int(hwnd)) if sys.platform == "win32" else None + """The rectangle a layout stores for ``hwnd``: the one the mover takes.""" + return _win32_window_rect(int(hwnd)) if sys.platform == "win32" else None def _exact_title_mover() -> WindowMover: @@ -185,7 +228,8 @@ def restore_window_layout(layout: Union[List[Dict[str, Any]], str, Path], *, return restored -def _snap_rect(position: str, width: int, height: int) -> Rect: +def _snap_rect(position: str, width: int, height: int, + origin: Tuple[int, int] = (0, 0)) -> Rect: half_w = width // 2 half_h = height // 2 regions = { @@ -205,7 +249,7 @@ def _snap_rect(position: str, width: int, height: int) -> Rect: f"unknown snap position {position!r}; " f"expected one of {sorted(regions)}", ) - return rect + return (origin[0] + rect[0], origin[1] + rect[1], rect[2], rect[3]) def _default_screen_size() -> Tuple[int, int]: @@ -214,17 +258,46 @@ def _default_screen_size() -> Tuple[int, int]: return (int(size[0]), int(size[1])) +def _default_work_area() -> Rect: + """Where windows may be laid out: the work area, else the whole screen.""" + area = _win32_work_area() if sys.platform == "win32" else None + if area is not None: + return area + width, height = _default_screen_size() + return (0, 0, width, height) + + +def _layout_area(screen_size: Optional[SizeProvider], + work_area: Optional[AreaProvider]) -> Rect: + """The ``(x, y, width, height)`` to lay out in, from whichever provider was given. + + ``screen_size`` only knows a size, so it means "this size, at the origin"; + with neither, the work area is asked for. + """ + if work_area is not None: + x, y, width, height = work_area() + return (int(x), int(y), int(width), int(height)) + if screen_size is not None: + width, height = screen_size() + return (0, 0, int(width), int(height)) + return _default_work_area() + + def snap_window(title: str, position: str = "left", *, mover: Optional[WindowMover] = None, - screen_size: Optional[SizeProvider] = None) -> bool: - """Move/resize the window matching ``title`` to a screen region. + screen_size: Optional[SizeProvider] = None, + work_area: Optional[AreaProvider] = None) -> bool: + """Move/resize the window matching ``title`` to a region of the work area. ``position`` is one of left / right / top / bottom / top-left / top-right / bottom-left / bottom-right / max. Returns ``True`` when the - window moved. The size provider and mover are injectable for tests. + window moved. The region is cut from the work area (the screen minus the + taskbar), so the bottom of a snapped window is no longer under the + taskbar. ``work_area`` (``() -> (x, y, width, height)``), ``screen_size`` + and the mover are injectable for tests. """ - width, height = (screen_size or _default_screen_size)() - x, y, w, h = _snap_rect(position, int(width), int(height)) + left, top, width, height = _layout_area(screen_size, work_area) + x, y, w, h = _snap_rect(position, width, height, (left, top)) return (mover or _default_mover)(title, x, y, w, h) @@ -253,37 +326,39 @@ def _grid_shape(count: int, rows: Optional[int], def arrange_grid(titles: List[str], *, rows: Optional[int] = None, cols: Optional[int] = None, gap: int = 0, mover: Optional[WindowMover] = None, - screen_size: Optional[SizeProvider] = None) -> int: + screen_size: Optional[SizeProvider] = None, + work_area: Optional[AreaProvider] = None) -> int: """Tile the given window ``titles`` into a grid; return the count moved. ``rows`` / ``cols`` default to a near-square auto-shape for the number of - windows; ``gap`` spaces the cells. The mover and size provider are injectable - for tests. Windows beyond the grid capacity are left untouched. + windows; ``gap`` spaces the cells. The grid fills the work area (the screen + minus the taskbar). The mover and the size / work-area providers are + injectable for tests. Windows beyond the grid capacity are left untouched. """ from je_auto_control.utils.window_layout import grid_rects titles = list(titles) if not titles: return 0 - width, height = (screen_size or _default_screen_size)() grid_rows, grid_cols = _grid_shape(len(titles), rows, cols) - rects = grid_rects((0, 0, int(width), int(height)), grid_rows, grid_cols, + rects = grid_rects(_layout_area(screen_size, work_area), grid_rows, grid_cols, gap=int(gap)) return _move_into(titles, rects, mover or _default_mover) def arrange_cascade(titles: List[str], *, offset: int = 30, mover: Optional[WindowMover] = None, - screen_size: Optional[SizeProvider] = None) -> int: + screen_size: Optional[SizeProvider] = None, + work_area: Optional[AreaProvider] = None) -> int: """Cascade the given window ``titles`` diagonally; return the count moved. Each window is ``offset`` pixels down-right of the previous, sized to 60% of - the work area and clamped on-screen. The mover and size provider are injectable. + the work area and clamped inside it. The mover and the size / work-area + providers are injectable. """ from je_auto_control.utils.window_layout import cascade_rects titles = list(titles) if not titles: return 0 - width, height = (screen_size or _default_screen_size)() - rects = cascade_rects((0, 0, int(width), int(height)), len(titles), + rects = cascade_rects(_layout_area(screen_size, work_area), len(titles), offset=int(offset)) return _move_into(titles, rects, mover or _default_mover) diff --git a/je_auto_control/utils/window_zorder/window_zorder.py b/je_auto_control/utils/window_zorder/window_zorder.py index f7b0726f4..e62ee7f6a 100644 --- a/je_auto_control/utils/window_zorder/window_zorder.py +++ b/je_auto_control/utils/window_zorder/window_zorder.py @@ -47,8 +47,9 @@ def _default_driver(title: str, action: str) -> bool: if hit is None: return False from je_auto_control.windows.window import windows_window_manage as wm - wm.set_window_position(int(hit[0]), plan_zorder(action)["insert_after"]) - return True + # SetWindowPos's own answer: reporting True regardless told callers a + # refused request (a window of a higher-integrity process) had worked. + return bool(wm.set_window_position(int(hit[0]), plan_zorder(action)["insert_after"])) def set_topmost(title: str, on: bool = True, *, diff --git a/je_auto_control/windows/window/windows_window_manage.py b/je_auto_control/windows/window/windows_window_manage.py index 7b2917e4b..9f3ef7132 100644 --- a/je_auto_control/windows/window/windows_window_manage.py +++ b/je_auto_control/windows/window/windows_window_manage.py @@ -6,7 +6,7 @@ from ctypes import ( # type: ignore[attr-defined] # reason: win32-only ctypes WINFUNCTYPE, byref, create_unicode_buffer, wintypes, ) -from typing import List, Optional, Tuple +from typing import Any, List, Optional, Tuple # 相容用途：舊版本從這個模組匯出共用的 user32。 # Compatibility: older code imported the shared user32 from this module. @@ -67,9 +67,65 @@ _user32.GetWindowThreadProcessId.argtypes = [wintypes.HWND, ctypes.POINTER(wintypes.DWORD)] _user32.GetWindowThreadProcessId.restype = wintypes.DWORD +_user32.IsWindow.argtypes = [wintypes.HWND] +_user32.IsWindow.restype = wintypes.BOOL +_user32.MapVirtualKeyW.argtypes = [wintypes.UINT, wintypes.UINT] +_user32.MapVirtualKeyW.restype = wintypes.UINT + + +def _load_dwmapi() -> Any: + """The module's own dwmapi handle, or ``None`` where the DLL is missing. + + Without it nothing is treated as cloaked, rather than the whole module + failing to import on a stripped-down Windows image. + """ + try: + dwmapi = ctypes.WinDLL("dwmapi") # type: ignore[attr-defined] # reason: win32-only ctypes + except OSError: + return None + dwmapi.DwmGetWindowAttribute.argtypes = [ + wintypes.HWND, wintypes.DWORD, ctypes.c_void_p, wintypes.DWORD] + dwmapi.DwmGetWindowAttribute.restype = ctypes.c_long + return dwmapi + + +_dwmapi = _load_dwmapi() WM_CLOSE = 0x0010 SW_RESTORE = 9 +_DWMWA_CLOAKED = 14 + + +def is_window_cloaked(hwnd: int) -> bool: + """ + 視窗是否被 DWM 隱藏（cloak） + Whether DWM is hiding the window although ``IsWindowVisible`` says it shows + + UWP 的背景視窗（「設定」、「Windows 輸入體驗」）與其他虛擬桌面上的視窗都是 + 這種狀態：樣式上可見，畫面上沒有。 + Suspended UWP frames and windows on another virtual desktop are in this + state: visible by style, absent from the screen. + """ + if _dwmapi is None: + return False + cloaked = wintypes.DWORD(0) + result = _dwmapi.DwmGetWindowAttribute( + hwnd, _DWMWA_CLOAKED, byref(cloaked), ctypes.sizeof(cloaked)) + return result == 0 and cloaked.value != 0 + + +def _has_area(hwnd: int) -> bool: + """False only for a window whose rectangle is known and empty.""" + rect = wintypes.RECT() + if not _user32.GetWindowRect(hwnd, byref(rect)): + return True + return rect.right > rect.left and rect.bottom > rect.top + + +def _is_listable(hwnd: int) -> bool: + """A window a user could see: visible, not cloaked, not zero-sized.""" + return (bool(_user32.IsWindowVisible(hwnd)) and not is_window_cloaked(hwnd) + and _has_area(hwnd)) def get_all_window_hwnd() -> List[Tuple[int, str]]: @@ -81,12 +137,17 @@ def get_all_window_hwnd() -> List[Tuple[int, str]]: handle 都成了 `LP_c_long` 物件：`int(hwnd)` 會丟 `ValueError`，也沒辦法拿去 跟其他 Win32 呼叫組合，等於這份清單只能看不能用。 + 被 DWM cloak 的視窗與零面積的視窗不列出：它們通過 `IsWindowVisible`，但 + 畫面上沒有，`find_window` 選到之後的每個動作都落空。 + Cloaked and zero-area windows are left out: they pass ``IsWindowVisible`` + but are not on screen, and anything done to one of them does nothing. + :return: [(hwnd, window_title), ...] """ window_info: List[Tuple[int, str]] = [] def _foreach_window(hwnd, _l_param) -> bool: - if _user32.IsWindowVisible(hwnd): + if _is_listable(hwnd): length = _user32.GetWindowTextLengthW(hwnd) buff = create_unicode_buffer(length + 1) _user32.GetWindowTextW(hwnd, buff, length + 1) @@ -188,23 +249,29 @@ def destroy_window(hwnd: int) -> bool: return bool(_user32.DestroyWindow(hwnd)) -def set_foreground_window(hwnd: int) -> None: +def set_foreground_window(hwnd: int) -> bool: """ - 設定視窗為前景視窗 - Set window to foreground + 設定視窗為前景視窗；回傳 Windows 是否接受 + Set window to foreground; return whether Windows accepted the request + + 前景鎖常拒絕背景行程，那時這裡回 False。True 也只代表請求被接受，要確認 + 請比對 :func:`get_foreground_window`。 + The foreground lock refuses a background process often, and this is False + then. True means accepted, not done: compare + :func:`get_foreground_window` to be sure. """ - _user32.SetForegroundWindow(hwnd) + return bool(_user32.SetForegroundWindow(hwnd)) -def set_window_position(hwnd: int, position: int) -> None: +def set_window_position(hwnd: int, position: int) -> bool: """ - 設定視窗位置 (僅改變 Z-order，不改變大小與座標) - Set window position (only Z-order, no resize or move) + 設定視窗位置 (僅改變 Z-order，不改變大小與座標)；回傳是否成功 + Set window position (only Z-order, no resize or move); return success """ swp_no_size = 0x0001 swp_no_move = 0x0002 - _user32.SetWindowPos(hwnd, position, 0, 0, 0, 0, - swp_no_move | swp_no_size) + return bool(_user32.SetWindowPos(hwnd, position, 0, 0, 0, 0, + swp_no_move | swp_no_size)) #: SW_SHOWNORMAL, SW_SHOWMAXIMIZED, SW_SHOW, SW_RESTORE, SW_SHOWDEFAULT. The @@ -213,20 +280,29 @@ def set_window_position(hwnd: int, position: int) -> None: _ACTIVATING_SHOW_COMMANDS = frozenset({1, 3, 5, 9, 10}) -def show_window(hwnd: int, cmd_show: int) -> None: +def show_window(hwnd: int, cmd_show: int) -> bool: """ - 顯示或隱藏視窗 - Show or hide a window + 顯示或隱藏視窗；回傳是否成功 + Show or hide a window; return whether it worked + + `ShowWindow` 的回傳值是「先前是否可見」，不是成敗，所以成敗看的是：handle + 是不是一個視窗，以及（會啟用視窗的指令）Windows 是否接受前景請求。 + ``ShowWindow`` returns the previous visibility, not success, so success is + whether the handle is a window and, for a command that activates, whether + Windows accepted the foreground request. :param cmd_show: Win32 ShowWindow flag (e.g., 0=Hide, 1=Normal, 2=Minimized, 3=Maximized) """ if cmd_show < 0 or cmd_show > 11: # Win32 ShowWindow 常見範圍 cmd_show = 1 # 預設為 Normal + if not _user32.IsWindow(hwnd): + return False _user32.ShowWindow(hwnd, cmd_show) # 隱藏之後不該再把它拉到前景，那是自相矛盾的一組動作。 # Do not pull a window forward right after hiding it. if cmd_show in _ACTIVATING_SHOW_COMMANDS: - _user32.SetForegroundWindow(hwnd) + return bool(_user32.SetForegroundWindow(hwnd)) + return True def move_window(hwnd: int, x: int, y: int, width: int, height: int, @@ -332,24 +408,47 @@ def deepest_child_at(hwnd: int, x: int, y: int) -> int: return current +#: Keys the keyboard driver reports with the extended-key bit (lParam bit 24). +_EXTENDED_KEYS = frozenset({ + 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, # page up/down, end, home, arrows + 0x2D, 0x2E, 0x5B, 0x5C, 0x5D, 0x6F, 0x90, 0xA3, 0xA5, +}) +_KEY_EXTENDED = 1 << 24 +_KEY_UP_BITS = 0xC0000000 # previous state down + transition (released) + + +def _key_lparams(keycode: int) -> Tuple[int, int]: + """``(WM_KEYDOWN lParam, WM_KEYUP lParam)`` as a real key press carries them.""" + scan = int(_user32.MapVirtualKeyW(int(keycode), 0)) & 0xFF + base = 1 | (scan << 16) + if int(keycode) in _EXTENDED_KEYS: + base |= _KEY_EXTENDED + return base, base | _KEY_UP_BITS + + def post_key(hwnd: int, keycode: int, character: str = "") -> bool: """ 把一次按鍵投遞給視窗（不搶焦點）；回傳訊息是否都排進佇列 Post one key press to a window without focusing it - 可列印字元要送 `WM_CHAR`：控制項是靠它拿到文字的，只送 `WM_KEYDOWN` 對多數 - 編輯控制項不會產生任何字。 + 可列印字元**只**送 `WM_CHAR`：控制項是靠它拿到文字的。先前另外送 + `WM_KEYDOWN`／`WM_KEYUP`，目標的 `TranslateMessage` 又各產生一次 `WM_CHAR` + （`lParam=0` 的 `WM_KEYUP` 被當成按下），一個字打成三個。其他鍵送 + `WM_KEYDOWN`／`WM_KEYUP`，`lParam` 帶重複次數、掃描碼與放開旗標。 - A printable character also needs ``WM_CHAR``: edit controls take their text - from that message, so ``WM_KEYDOWN`` alone types nothing in most of them. + A printable character is posted as ``WM_CHAR`` alone: controls take their + text from it. Posting the key messages as well made the target's + ``TranslateMessage`` produce one more ``WM_CHAR`` from each (a ``WM_KEYUP`` + whose ``lParam`` is 0 reads as a press), so one character arrived three + times. Any other key is a ``WM_KEYDOWN`` / ``WM_KEYUP`` pair whose + ``lParam`` holds the repeat count, scan code and release bits. """ target = get_focused_control(hwnd) - posted = bool(_user32.PostMessageW(target, WM_KEYDOWN, int(keycode), 0)) if character: - posted = bool(_user32.PostMessageW( - target, WM_CHAR, ord(character[0]), 0)) and posted - posted = bool(_user32.PostMessageW(target, WM_KEYUP, int(keycode), 0)) and posted - return posted + return bool(_user32.PostMessageW(target, WM_CHAR, ord(character[0]), 1)) + down, up = _key_lparams(keycode) + posted = bool(_user32.PostMessageW(target, WM_KEYDOWN, int(keycode), down)) + return bool(_user32.PostMessageW(target, WM_KEYUP, int(keycode), up)) and posted def post_click(hwnd: int, button: str, x: int, y: int) -> bool: diff --git a/je_auto_control/wrapper/auto_control_window.py b/je_auto_control/wrapper/auto_control_window.py index a9e7de787..c413483d7 100644 --- a/je_auto_control/wrapper/auto_control_window.py +++ b/je_auto_control/wrapper/auto_control_window.py @@ -18,7 +18,10 @@ from je_auto_control.utils.exception.exceptions import AutoControlActionException from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.wrapper.window_backends import get_backend -from je_auto_control.utils.timeouts import deadline_after +from je_auto_control.utils.timeouts import clamp_poll_interval, deadline_after + +#: How long focus_window waits for the window to become the foreground one. +_FOCUS_SETTLE_S = 1.0 def list_windows(titled_only: bool = False) -> List[Tuple[int, str]]: @@ -52,8 +55,26 @@ def find_window(title_substring: str, return None +def _became_foreground(backend, hwnd: int, settle_s: float) -> bool: + """Poll until ``hwnd`` is the foreground window or ``settle_s`` runs out.""" + deadline = time.monotonic() + max(0.0, float(settle_s)) + while True: + if backend.foreground_window() == hwnd: + return True + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + time.sleep(min(0.05, remaining)) + + def focus_window(title_substring: str, case_sensitive: bool = False) -> int: - """Bring the first matching window to the foreground; return its hwnd.""" + """Bring the first matching window to the foreground; return its hwnd. + + Raises ``AutoControlActionException`` when the window is not the foreground + one afterwards. Windows' foreground lock refuses a background process + often, and reporting success then sends the caller's next keystrokes to + whatever window the user has active. + """ hit = find_window(title_substring, case_sensitive) if hit is None: raise AutoControlActionException( @@ -67,6 +88,10 @@ def focus_window(title_substring: str, case_sensitive: bool = False) -> int: if backend.is_minimized(hwnd): backend.restore(hwnd) backend.set_foreground(hwnd) + if not _became_foreground(backend, hwnd, _FOCUS_SETTLE_S): + raise AutoControlActionException( + f"focus_window: {title!r} (hwnd={hwnd}) did not become the " + f"foreground window; the system refused the request") autocontrol_logger.info("focused window hwnd=%s title=%r", hwnd, title) return hwnd @@ -75,17 +100,23 @@ def wait_for_window(title_substring: str, timeout: float = 10.0, poll: float = 0.5, case_sensitive: bool = False) -> int: - """Poll until a window with the given title appears; return its hwnd.""" - poll = max(0.05, float(poll)) + """Poll until a window with the given title appears; return its hwnd. + + ``poll`` is clamped to a finite interval and never sleeps past ``timeout``: + ``poll=30`` with ``timeout=1`` used to wait 30 seconds, and ``poll=inf`` + raised ``OverflowError``. + """ + poll = clamp_poll_interval(poll) deadline = deadline_after(time.monotonic(), timeout) # Look first, then check the clock: with timeout=0 the loop never ran. while True: hit = find_window(title_substring, case_sensitive) if hit is not None: return hit[0] - if time.monotonic() >= deadline: + remaining = deadline - time.monotonic() + if remaining <= 0: break - time.sleep(poll) + time.sleep(min(poll, remaining)) raise AutoControlActionException( f"wait_for_window timeout: {title_substring!r}" ) @@ -172,8 +203,14 @@ def _resolve_key(key: Union[int, str]) -> Tuple[int, str]: if isinstance(key, int): return int(key), "" name = str(key) + from je_auto_control.utils.cua_action.cua_action import resolve_key_name from je_auto_control.wrapper.platform_wrapper import keyboard_keys_table + # Exact name first, so a table that tells "A" from "a" keeps doing so; then + # the platform's own spelling ("enter" is "return" on Windows, "esc" is + # "escape"), which is how every other key-taking function resolves names. keycode = keyboard_keys_table.get(name) + if keycode is None: + keycode = keyboard_keys_table.get(resolve_key_name(name, keyboard_keys_table)) if keycode is None: raise AutoControlActionException(f"unknown key name: {name!r}") # A one-character key is text: edit controls take their content from @@ -278,10 +315,15 @@ def move_window_by_title(title_substring: str, x: int, y: int, def show_window_by_title(title_substring: str, cmd_show: int = 1, case_sensitive: bool = False) -> bool: - """Show or restore a window (``cmd_show`` follows Win32 ShowWindow).""" + """Show or restore a window (``cmd_show`` follows Win32 ShowWindow). + + ``False`` when nothing matched or the backend reports the request failed + (on Windows: the handle is no longer a window, or a command that activates + the window was refused the foreground). + """ hit = find_window(title_substring, case_sensitive) if hit is None: return False backend = get_backend() - backend.show(hit[0], int(cmd_show)) - return True + # A backend that cannot tell answers None, which is not a failure. + return backend.show(hit[0], int(cmd_show)) is not False diff --git a/je_auto_control/wrapper/window_backends/base.py b/je_auto_control/wrapper/window_backends/base.py index e812736ae..dd7c23014 100644 --- a/je_auto_control/wrapper/window_backends/base.py +++ b/je_auto_control/wrapper/window_backends/base.py @@ -96,8 +96,12 @@ def restore(self, window_id: int) -> None: """ self._unsupported("restore") - def show(self, window_id: int, cmd_show: int) -> None: - """Apply a platform show-state code (Win32 ``ShowWindow`` numbering).""" + def show(self, window_id: int, cmd_show: int) -> Optional[bool]: + """Apply a platform show-state code (Win32 ``ShowWindow`` numbering). + + ``False`` when the platform reports the request failed, ``True`` when + it reports success, ``None`` when it cannot tell. + """ self._unsupported("show") def close(self, window_id: int) -> bool: diff --git a/je_auto_control/wrapper/window_backends/windows_backend.py b/je_auto_control/wrapper/window_backends/windows_backend.py index 39e87c393..9772a0f6f 100644 --- a/je_auto_control/wrapper/window_backends/windows_backend.py +++ b/je_auto_control/wrapper/window_backends/windows_backend.py @@ -46,8 +46,8 @@ def set_foreground(self, window_id: int) -> None: def restore(self, window_id: int) -> None: self._wm.show_window(window_id, self._wm.SW_RESTORE) - def show(self, window_id: int, cmd_show: int) -> None: - self._wm.show_window(window_id, int(cmd_show)) + def show(self, window_id: int, cmd_show: int) -> Optional[bool]: + return bool(self._wm.show_window(window_id, int(cmd_show))) def close(self, window_id: int) -> bool: return self._wm.close_window(window_id) diff --git a/test/unit_test/headless/test_a11y_window_audit.py b/test/unit_test/headless/test_a11y_window_audit.py index 090a21bcc..104d0446c 100644 --- a/test/unit_test/headless/test_a11y_window_audit.py +++ b/test/unit_test/headless/test_a11y_window_audit.py @@ -59,7 +59,7 @@ def test_ocr_matching_normalises_unicode(): def test_a_saved_layout_reads_each_window_by_its_handle(monkeypatch): monkeypatch.setattr(sys, "platform", "win32") real = {1: (0, 0, 800, 600), 2: (900, 50, 400, 300)} - monkeypatch.setattr(window_capture, "_win32_geometry", lambda hwnd: real[hwnd]) + monkeypatch.setattr(window_capture, "_win32_window_rect", lambda hwnd: real[hwnd]) layout = window_capture.save_window_layout(lister=lambda: [(1, "Editor - a"), (2, "Editor")]) assert [(entry["title"], entry["x"]) for entry in layout] == [("Editor - a", 0), ("Editor", 900)] @@ -106,6 +106,7 @@ def test_show_window_foregrounds_only_for_activating_commands(monkeypatch, comma from je_auto_control.windows.window import windows_window_manage calls = [] fake = types.SimpleNamespace(ShowWindow=lambda hwnd, cmd: calls.append("show"), + IsWindow=lambda hwnd: 1, SetForegroundWindow=lambda hwnd: calls.append("foreground")) monkeypatch.setattr(windows_window_manage, "_user32", fake) windows_window_manage.show_window(1, command) diff --git a/test/unit_test/headless/test_window_backends.py b/test/unit_test/headless/test_window_backends.py index 94d9aa081..0a4d310b2 100644 --- a/test/unit_test/headless/test_window_backends.py +++ b/test/unit_test/headless/test_window_backends.py @@ -32,12 +32,13 @@ def __init__(self): self.windows = [(11, "Editor"), (12, " "), (13, "Browser")] self.rect = (10, 20, 110, 220) self.minimized = False + self.foreground = 13 def list_windows(self): return list(self.windows) def foreground_window(self): - return 13 + return self.foreground def window_rect(self, window_id): self.calls.append(("window_rect", window_id)) @@ -51,6 +52,7 @@ def is_minimized(self, window_id): def set_foreground(self, window_id): self.calls.append(("set_foreground", window_id)) + self.foreground = window_id def restore(self, window_id): self.calls.append(("restore", window_id)) diff --git a/test/unit_test/headless/test_window_lifecycle_contract.py b/test/unit_test/headless/test_window_lifecycle_contract.py new file mode 100644 index 000000000..db8f474c6 --- /dev/null +++ b/test/unit_test/headless/test_window_lifecycle_contract.py @@ -0,0 +1,464 @@ +"""Window management reports what happened, and leaves windows where they were. + +Every Win32 call here is a fake: nothing in this file focuses, moves, shows or +types into a real window. + +* ``post_key`` posted ``WM_KEYDOWN`` + ``WM_CHAR`` + ``WM_KEYUP`` for a printable + key. The target's ``TranslateMessage`` makes a ``WM_CHAR`` of its own from the + key-down, and another from a key-up whose ``lParam`` is 0 (no release bit), so + one character arrived three times. ``post_key_to_window(title, "enter")`` + raised ``unknown key name`` on Windows, whose table calls that key ``return``. +* ``focus_window`` / ``show_window`` / the z-order driver dropped the BOOL the + Win32 call returned and reported success when Windows had refused. +* ``list_windows`` listed windows DWM is cloaking and windows with no area. +* A saved layout held the visible frame and was restored through + ``MoveWindow``, which positions the larger rectangle around it: 7 px right and + 14 x 7 px smaller on every round. Snap / grid / cascade used the whole screen, + so the bottom rows sat under the taskbar. +* ``wait_for_window`` slept a whole ``poll`` past its timeout, and ``poll=inf`` + raised ``OverflowError``. +""" +import ctypes +import math +import sys +import types + +import pytest + +from je_auto_control.utils.exception.exceptions import AutoControlActionException +from je_auto_control.utils.window_capture import window_capture +from je_auto_control.utils.window_zorder import window_zorder +from je_auto_control.wrapper import auto_control_window as w + +_WINDOWS = sys.platform in ("win32", "cygwin", "msys") +windows_only = pytest.mark.skipif(not _WINDOWS, reason="the Win32 modules import on Windows only") + +WM_KEYDOWN, WM_KEYUP, WM_CHAR = 0x0100, 0x0101, 0x0102 +_SCAN_CODES = {0x0D: 0x1C, 0x1B: 0x01, 0x26: 0x48, 0x41: 0x1E} +_TEXT_KEYS = {0x41: "a"} + + +class _MessageQueue: + """``user32`` as far as ``post_key`` uses it, recording what was posted.""" + + def __init__(self): + self.messages = [] + + def PostMessageW(self, hwnd, message, w_param, l_param): # noqa: N802 # reason: Win32 name + self.messages.append((message, w_param, l_param)) + return 1 + + def MapVirtualKeyW(self, keycode, _kind): # noqa: N802 # reason: Win32 name + return _SCAN_CODES.get(keycode, 0) + + def GetWindowThreadProcessId(self, hwnd, _pid): # noqa: N802 # reason: Win32 name + return 0 # no GUI thread info: the target is hwnd itself + + +def _typed(messages): + """The text an edit control ends up with, as ``TranslateMessage`` would build it. + + A key message for a text key yields a ``WM_CHAR`` when its ``lParam`` says + the key is going down (bit 31 clear) — for ``WM_KEYUP`` too, which is how a + key-up posted with ``lParam=0`` typed a second copy. + """ + text = "" + for message, w_param, l_param in messages: + if message == WM_CHAR: + text += chr(w_param) + elif message in (WM_KEYDOWN, WM_KEYUP) and not l_param & 0x80000000: + text += _TEXT_KEYS.get(w_param, "") + return text + + +@pytest.fixture +def queue(monkeypatch): + from je_auto_control.windows.window import windows_window_manage as module + fake = _MessageQueue() + monkeypatch.setattr(module, "_user32", fake) + return fake + + +# --- post_key ---------------------------------------------------------------- + +@windows_only +def test_a_printable_key_is_typed_once(queue): + from je_auto_control.windows.window import windows_window_manage as module + assert module.post_key(5, 0x41, "a") is True + assert _typed(queue.messages) == "a" + assert [message for message, _w, _l in queue.messages] == [WM_CHAR] + + +@windows_only +def test_a_control_key_is_one_press_and_one_release(queue): + from je_auto_control.windows.window import windows_window_manage as module + assert module.post_key(5, 0x0D) is True + (down, down_key, down_l), (up, up_key, up_l) = queue.messages + assert (down, down_key, up, up_key) == (WM_KEYDOWN, 0x0D, WM_KEYUP, 0x0D) + assert down_l == 1 | (0x1C << 16), "repeat count 1 and the scan code" + assert up_l == 0xC0000001 | (0x1C << 16), "previous-state and transition bits set" + + +@windows_only +def test_a_virtual_key_that_is_text_is_typed_once_too(queue): + """An int key carries no character: the key-down types it, the key-up must not.""" + from je_auto_control.windows.window import windows_window_manage as module + module.post_key(5, 0x41) + assert _typed(queue.messages) == "a" + + +@windows_only +def test_an_arrow_key_carries_the_extended_bit(queue): + from je_auto_control.windows.window import windows_window_manage as module + module.post_key(5, 0x26) + assert all(l_param & (1 << 24) for _message, _w, l_param in queue.messages) + + +class _PostBackend: + def __init__(self): + self.posted = [] + + def list_windows(self): + return [(11, "Editor")] + + def post_key(self, window_id, keycode, character=""): + self.posted.append((window_id, keycode, character)) + return True + + +@pytest.mark.parametrize("name, expected", [ + ("enter", (11, 13, "")), ("return", (11, 13, "")), ("ENTER", (11, 13, "")), + ("esc", (11, 27, "")), ("escape", (11, 27, "")), ("a", (11, 65, "a")), +]) +def test_post_key_to_window_takes_every_spelling_of_a_key(monkeypatch, name, expected): + """The Windows table says ``return`` / ``escape``; ``enter`` / ``esc`` raised.""" + from je_auto_control.wrapper import platform_wrapper + backend = _PostBackend() + monkeypatch.setattr(w, "get_backend", lambda: backend) + monkeypatch.setattr(platform_wrapper, "keyboard_keys_table", + {"return": 13, "escape": 27, "a": 65}, raising=False) + assert w.post_key_to_window("Editor", name) is True + assert backend.posted == [expected] + + +def test_an_unknown_key_name_is_still_refused(monkeypatch): + from je_auto_control.wrapper import platform_wrapper + monkeypatch.setattr(w, "get_backend", _PostBackend) + monkeypatch.setattr(platform_wrapper, "keyboard_keys_table", {"return": 13}, raising=False) + with pytest.raises(AutoControlActionException, match="unknown key name"): + w.post_key_to_window("Editor", "no_such_key") + + +# --- focus / show / z-order -------------------------------------------------- + +class _FocusBackend: + """A desktop whose foreground lock refuses unless ``allow`` is set.""" + + def __init__(self, allow): + self.allow = allow + self.foreground = 99 + self.show_result = None + + def list_windows(self): + return [(11, "Editor")] + + def is_minimized(self, window_id): + return False + + def set_foreground(self, window_id): + if self.allow: + self.foreground = window_id + + def foreground_window(self): + return self.foreground + + def show(self, window_id, cmd_show): + return self.show_result + + +def test_foreground_failure_propagates(monkeypatch): + backend = _FocusBackend(allow=False) + monkeypatch.setattr(w, "get_backend", lambda: backend) + monkeypatch.setattr(w, "_FOCUS_SETTLE_S", 0.05) + with pytest.raises(AutoControlActionException, match="did not become the foreground"): + w.focus_window("Editor") + + +def test_focus_window_returns_the_handle_once_it_is_in_front(monkeypatch): + backend = _FocusBackend(allow=True) + monkeypatch.setattr(w, "get_backend", lambda: backend) + assert w.focus_window("Editor") == 11 + + +@pytest.mark.parametrize("backend_answer, expected", [(False, False), (True, True), (None, True)]) +def test_show_window_by_title_reports_the_backends_answer(monkeypatch, backend_answer, expected): + """``None`` is a backend that cannot tell, which is not a failure.""" + backend = _FocusBackend(allow=True) + backend.show_result = backend_answer + monkeypatch.setattr(w, "get_backend", lambda: backend) + assert w.show_window_by_title("Editor", 3) is expected + + +def _user32(**functions): + return types.SimpleNamespace(**functions) + + +@windows_only +@pytest.mark.parametrize("accepted", [0, 1]) +def test_set_foreground_window_returns_what_windows_said(monkeypatch, accepted): + from je_auto_control.windows.window import windows_window_manage as module + monkeypatch.setattr(module, "_user32", _user32(SetForegroundWindow=lambda hwnd: accepted)) + assert module.set_foreground_window(7) is bool(accepted) + + +@windows_only +def test_show_window_reports_a_dead_handle_and_a_refused_activation(monkeypatch): + from je_auto_control.windows.window import windows_window_manage as module + shown = [] + fake = _user32(IsWindow=lambda hwnd: 0, ShowWindow=lambda hwnd, cmd: shown.append(cmd), + SetForegroundWindow=lambda hwnd: 0) + monkeypatch.setattr(module, "_user32", fake) + assert module.show_window(7, 3) is False and shown == [], "not a window: nothing shown" + fake.IsWindow = lambda hwnd: 1 + assert module.show_window(7, 3) is False, "maximised but refused the foreground" + assert module.show_window(7, 6) is True, "minimising does not need the foreground" + assert shown == [3, 6] + + +@windows_only +@pytest.mark.parametrize("win32_result", [0, 1]) +def test_zorder_reports_what_set_window_pos_said(monkeypatch, win32_result): + from je_auto_control.windows.window import windows_window_manage as module + seen = [] + monkeypatch.setattr(module, "_user32", _user32( + SetWindowPos=lambda hwnd, after, *rest: seen.append((hwnd, after)) or win32_result)) + monkeypatch.setattr(w, "find_window", lambda title, case_sensitive=False: (11, "Editor")) + assert window_zorder.set_topmost("Editor") is bool(win32_result) + assert seen == [(11, -1)] + + +# --- listing ----------------------------------------------------------------- + +class _Desktop: + """``user32`` + ``dwmapi`` for window enumeration: ``hwnd -> (title, rect, cloaked)``.""" + + def __init__(self, windows): + self.windows = windows + + def EnumWindows(self, callback, l_param): # noqa: N802 # reason: Win32 name + for hwnd in self.windows: + callback(hwnd, l_param) + return 1 + + def IsWindowVisible(self, hwnd): # noqa: N802 # reason: Win32 name + return 1 + + def GetWindowTextLengthW(self, hwnd): # noqa: N802 # reason: Win32 name + return len(self.windows[hwnd][0]) + + def GetWindowTextW(self, hwnd, buffer, _size): # noqa: N802 # reason: Win32 name + buffer.value = self.windows[hwnd][0] + return len(buffer.value) + + def GetWindowRect(self, hwnd, reference): # noqa: N802 # reason: Win32 name + rect = reference._obj + rect.left, rect.top, rect.right, rect.bottom = self.windows[hwnd][1] + return 1 + + def DwmGetWindowAttribute(self, hwnd, _attribute, reference, _size): # noqa: N802 # reason: Win32 name + reference._obj.value = self.windows[hwnd][2] + return 0 + + +@windows_only +def test_cloaked_window_filtered(monkeypatch): + from je_auto_control.windows.window import windows_window_manage as module + desktop = _Desktop({ + 1: ("Editor", (0, 0, 800, 600), 0), + 2: ("Settings", (0, 0, 800, 600), 2), # DWM_CLOAKED_SHELL + 3: ("Windows Input Experience", (0, 0, 0, 0), 0), + 4: ("Browser", (-1920, 0, -1000, 500), 0), # on a monitor to the left + }) + monkeypatch.setattr(module, "_user32", desktop) + monkeypatch.setattr(module, "_dwmapi", desktop) + assert module.get_all_window_hwnd() == [(1, "Editor"), (4, "Browser")] + + +@windows_only +def test_listing_survives_a_system_without_dwm(monkeypatch): + from je_auto_control.windows.window import windows_window_manage as module + monkeypatch.setattr(module, "_user32", _Desktop({1: ("Editor", (0, 0, 800, 600), 9)})) + monkeypatch.setattr(module, "_dwmapi", None) + assert module.get_all_window_hwnd() == [(1, "Editor")] + + +# --- layout ------------------------------------------------------------------ + +_BORDER = 7 # the invisible resize border of a Windows 10 / 11 window + + +class _Window: + """One window as Win32 sees it: ``MoveWindow`` and ``GetWindowRect`` agree.""" + + def __init__(self, x, y, width, height): + self.rect = (x, y, width, height) + + def win32(self): + """``ctypes.windll`` reading this window: the DWM frame is inside the rect.""" + def get_window_rect(_hwnd, reference): + x, y, width, height = self.rect + rect = reference._obj + rect.left, rect.top, rect.right, rect.bottom = x, y, x + width, y + height + return 1 + + def frame_bounds(_hwnd, _attribute, reference, _size): + x, y, width, height = self.rect + rect = reference._obj + rect.left, rect.top = x + _BORDER, y + rect.right, rect.bottom = x + width - _BORDER, y + height - _BORDER + return 0 + + return types.SimpleNamespace( + user32=types.SimpleNamespace(IsIconic=lambda hwnd: 0, GetWindowRect=get_window_rect), + dwmapi=types.SimpleNamespace(DwmGetWindowAttribute=frame_bounds)) + + +@windows_only +def test_capture_does_not_move_window(monkeypatch): + """Saving and restoring a layout, three times over, leaves the window where it was.""" + window = _Window(100, 50, 800, 600) + monkeypatch.setattr(ctypes, "windll", window.win32()) + + def move(_title, x, y, width, height): + window.rect = (x, y, width, height) + return True + + for _round in range(3): + layout = window_capture.save_window_layout(lister=lambda: [(1, "Editor")]) + assert window_capture.restore_window_layout(layout, mover=move) == 1 + assert window.rect == (100, 50, 800, 600) + + +@windows_only +def test_a_window_capture_still_takes_the_visible_frame(monkeypatch): + monkeypatch.setattr(ctypes, "windll", _Window(100, 50, 800, 600).win32()) + assert window_capture._win32_geometry(1) == (107, 50, 786, 593) + assert window_capture._win32_window_rect(1) == (100, 50, 800, 600) + + +@windows_only +def test_a_minimised_window_is_not_saved(monkeypatch): + fake = _Window(-32000, -32000, 160, 28).win32() + fake.user32.IsIconic = lambda hwnd: 1 + monkeypatch.setattr(ctypes, "windll", fake) + assert window_capture.save_window_layout(lister=lambda: [(1, "Editor")]) == [] + + +def _moves(): + seen = [] + + def move(title, x, y, width, height): + seen.append((x, y, width, height)) + return True + + return seen, move + + +@pytest.fixture +def work_area(monkeypatch): + """A 1920 x 1080 screen with a 48 px taskbar along the bottom.""" + monkeypatch.setattr(window_capture, "_default_work_area", lambda: (0, 0, 1920, 1032)) + + +def test_snap_stays_above_the_taskbar(work_area): + seen, move = _moves() + assert window_capture.snap_window("Editor", "bottom", mover=move) + assert window_capture.snap_window("Editor", "max", mover=move) + assert seen == [(0, 516, 1920, 516), (0, 0, 1920, 1032)] + + +def test_snap_starts_at_the_work_areas_corner(): + """A taskbar on the left or top moves the origin, not just the size.""" + seen, move = _moves() + window_capture.snap_window("Editor", "left", mover=move, work_area=lambda: (60, 0, 1860, 1080)) + assert seen == [(60, 0, 930, 1080)] + + +def test_grid_and_cascade_stay_above_the_taskbar(work_area): + seen, move = _moves() + assert window_capture.arrange_grid(["a", "b", "c", "d"], mover=move) == 4 + assert window_capture.arrange_cascade(["a", "b"], mover=move) == 2 + assert max(y + height for _x, y, _width, height in seen) <= 1032 + assert (960, 516, 960, 516) in seen, "the bottom-right cell ends at the work area" + + +def test_an_injected_screen_size_still_means_that_size_at_the_origin(work_area): + seen, move = _moves() + window_capture.snap_window("Editor", "bottom", mover=move, screen_size=lambda: (1000, 800)) + assert seen == [(0, 400, 1000, 400)] + + +@windows_only +def test_the_work_area_comes_from_system_parameters_info(monkeypatch): + def system_parameters_info(action, _param, reference, _flags): + rect = reference._obj + rect.left, rect.top, rect.right, rect.bottom = 0, 0, 1920, 1032 + return 1 if action == 0x0030 else 0 + + fake = types.SimpleNamespace(user32=types.SimpleNamespace( + SystemParametersInfoW=system_parameters_info)) + monkeypatch.setattr(ctypes, "windll", fake) + monkeypatch.setattr(sys, "platform", "win32") + assert window_capture._default_work_area() == (0, 0, 1920, 1032) + + +def test_without_a_work_area_the_whole_screen_is_used(monkeypatch): + monkeypatch.setattr(sys, "platform", "linux") + monkeypatch.setattr(window_capture, "_default_screen_size", lambda: (1280, 720)) + assert window_capture._default_work_area() == (0, 0, 1280, 720) + + +# --- waiting ----------------------------------------------------------------- + +class _Clock: + """``time`` for the wrapper: sleeping advances it and nothing really waits.""" + + def __init__(self): + self.now = 100.0 + self.sleeps = [] + + def monotonic(self): + return self.now + + def sleep(self, seconds): + self.sleeps.append(seconds) + self.now += seconds + + +@pytest.fixture +def clock(monkeypatch): + fake = _Clock() + monkeypatch.setattr(w, "time", fake) + monkeypatch.setattr(w, "find_window", lambda title, case_sensitive=False: None) + return fake + + +def test_poll_is_bounded(clock): + """``poll=30`` with ``timeout=1`` slept 30 seconds before giving up.""" + with pytest.raises(AutoControlActionException, match="timeout"): + w.wait_for_window("never", timeout=1.0, poll=30) + assert sum(clock.sleeps) == pytest.approx(1.0) + assert max(clock.sleeps) <= 1.0 + + +def test_an_infinite_poll_does_not_overflow(clock): + with pytest.raises(AutoControlActionException, match="timeout"): + w.wait_for_window("never", timeout=2.0, poll=math.inf) + assert sum(clock.sleeps) == pytest.approx(2.0) + + +def test_a_short_poll_still_polls_at_its_own_pace(clock): + with pytest.raises(AutoControlActionException): + w.wait_for_window("never", timeout=1.0, poll=0.25) + assert clock.sleeps[0] == pytest.approx(0.25) + assert sum(clock.sleeps) == pytest.approx(1.0) diff --git a/test/unit_test/headless/test_window_manage.py b/test/unit_test/headless/test_window_manage.py index ae13b57e5..9bc132a21 100644 --- a/test/unit_test/headless/test_window_manage.py +++ b/test/unit_test/headless/test_window_manage.py @@ -115,6 +115,9 @@ def test_hiding_a_window_does_not_pull_it_to_the_foreground(monkeypatch): seen = [] class _FakeUser32: + def IsWindow(self, hwnd): # noqa: N802 # Win32 name + return 1 + def ShowWindow(self, hwnd, cmd): # noqa: N802 # Win32 name seen.append(("show", hwnd, cmd)) From a6677b05f4568907e613a1205899b41c20317567 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:01:45 +0800 Subject: [PATCH 14/26] Capture every monitor in the coordinates the mouse takes The process asked for system DPI awareness, so Windows virtualised any monitor scaled differently from the primary one: scaled coordinates and a blurred, resized capture. It now asks for per-monitor-v2 first and falls back, once, without raising when the awareness was already fixed. pil_screenshot's region path (and with it screenshot, AC_screenshot, keyword_screenshot and capture_window) cropped a primary-monitor capture, so a region on a monitor left of or above it was black; it is cut from the whole virtual desktop on Windows now. mark_screen renders on that desktop too and reports the image's origin. grab_logical clips a region to the desktop and rejects an empty one with a framework error instead of handing a matcher black padding. On macOS the frame is built per display from Quartz's bounds in points, scaled down and stitched, so a Retina capture is no longer twice the mouse's coordinates and other displays are included. Tested with fakes only; not run on Retina hardware. --- .../Eng/doc/new_features/v22_features_doc.rst | 6 +- docs/source/Eng/doc/screen/screen_doc.rst | 28 ++ .../Zh/doc/new_features/v22_features_doc.rst | 5 +- docs/source/Zh/doc/screen/screen_doc.rst | 22 + je_auto_control/gui/_screen_geometry.py | 9 +- .../utils/cv2_utils/region_capture.py | 26 +- je_auto_control/utils/cv2_utils/screenshot.py | 30 +- .../utils/monitor_layout/logical_frame.py | 105 ++++- .../utils/monitor_layout/macos_frame.py | 142 +++++++ .../utils/set_of_marks/set_of_marks.py | 32 +- .../windows/screen/win32_screen.py | 94 ++++- .../test_platform_coordinate_contract.py | 388 ++++++++++++++++++ 12 files changed, 852 insertions(+), 35 deletions(-) create mode 100644 je_auto_control/utils/monitor_layout/macos_frame.py create mode 100644 test/unit_test/headless/test_platform_coordinate_contract.py diff --git a/docs/source/Eng/doc/new_features/v22_features_doc.rst b/docs/source/Eng/doc/new_features/v22_features_doc.rst index 9b1607232..13c53a210 100644 --- a/docs/source/Eng/doc/new_features/v22_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v22_features_doc.rst @@ -46,6 +46,10 @@ Live "mark then click" loop ``mark_screen`` numbers the live accessibility elements (and optionally saves a numbered-box overlay screenshot), caching the marks; ``mark_click`` -resolves a number from that cache and clicks the element's centre. Exposed +resolves a number from that cache and clicks the element's centre. The +overlay covers every monitor, and the result then carries ``image_origin``, +the screen coordinate of the image's top-left pixel: marks stay in screen +coordinates, so a mark is drawn at ``bbox - image_origin`` (the origin is +negative when a monitor sits left of or above the primary one). Exposed as ``AC_mark_screen`` / ``AC_mark_click`` (and ``ac_mark_screen`` / ``ac_mark_click``). diff --git a/docs/source/Eng/doc/screen/screen_doc.rst b/docs/source/Eng/doc/screen/screen_doc.rst index 8a2f995ec..89544192a 100644 --- a/docs/source/Eng/doc/screen/screen_doc.rst +++ b/docs/source/Eng/doc/screen/screen_doc.rst @@ -19,6 +19,34 @@ Capture the current screen and save to a file: # Capture a specific region [x1, y1, x2, y2] screenshot("region.png", screen_region=[100, 100, 500, 400]) +``screen_region`` is in the coordinates the mouse takes. On Windows it may lie +on any monitor — a monitor left of or above the primary one has negative +coordinates, e.g. ``screen_region=[-1920, 0, -1720, 100]`` — and the image is +always ``x2 - x1`` by ``y2 - y1``: whatever part of the region is off the +desktop is black, and a region with none of it on a monitor raises +``AutoControlScreenException``. Without ``screen_region`` the primary monitor +is captured. + +Coordinates and DPI on Windows +------------------------------ + +``import je_auto_control`` makes the process **per-monitor DPI aware (v2)**, +falling back to system awareness on a Windows too old for it or in a process +whose awareness was already fixed (an embedding host, a manifest). Per-monitor +means every monitor is addressed in its own physical pixels: a 1920x1080 +monitor is 1920 wide to the mouse, to ``screenshot`` and to the window +functions whatever its scale setting, and its capture is not resized. + +Before this the process was *system* aware, and a monitor whose scale differs +from the primary monitor's was virtualised by Windows: it appeared +``primary scale / its scale`` times its real size (1536x864 for a 1920x1080 +monitor at 125% beside a 100% primary) and its capture was a blurred, resized +image. Coordinates recorded on such a monitor, and template images cut from +it, were taken in that virtualised space and have to be recorded again; a +point at offset ``(dx, dy)`` from that monitor's top-left corner is now at +``(dx, dy) * its scale / primary scale``. The primary monitor, and any monitor +at the primary's scale, are unchanged. + Screen Size =========== diff --git a/docs/source/Zh/doc/new_features/v22_features_doc.rst b/docs/source/Zh/doc/new_features/v22_features_doc.rst index b4b4c7f9c..29bd47fa5 100644 --- a/docs/source/Zh/doc/new_features/v22_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v22_features_doc.rst @@ -42,6 +42,9 @@ mark_click(3) # 點擊第 3 號標記 ``mark_screen`` 為即時 accessibility 元素標號(並可另存編號方框疊圖截圖), -並快取這些標記;``mark_click`` 從快取解析編號並點擊該元素中心。對應 +並快取這些標記;``mark_click`` 從快取解析編號並點擊該元素中心。疊圖涵蓋 +所有螢幕,此時結果另帶 ``image_origin``,也就是影像左上角像素的螢幕座標: +標記維持螢幕座標,畫在影像上的位置是 ``bbox - image_origin``(有螢幕位於 +主螢幕左側或上方時原點為負)。對應 ``AC_mark_screen`` / ``AC_mark_click``(以及 ``ac_mark_screen`` / ``ac_mark_click``)。 diff --git a/docs/source/Zh/doc/screen/screen_doc.rst b/docs/source/Zh/doc/screen/screen_doc.rst index 5b0ddff8f..eaa391fb6 100644 --- a/docs/source/Zh/doc/screen/screen_doc.rst +++ b/docs/source/Zh/doc/screen/screen_doc.rst @@ -19,6 +19,28 @@ AutoControl 提供截圖與取得螢幕資訊的功能。 # 擷取特定區域 [x1, y1, x2, y2] screenshot("region.png", screen_region=[100, 100, 500, 400]) +``screen_region`` 用的是滑鼠座標。在 Windows 上它可以落在任何一個螢幕——位於 +主螢幕左側或上方的螢幕座標是負的，例如 ``screen_region=[-1920, 0, -1720, 100]`` +——回傳的影像一律是 ``x2 - x1`` 乘 ``y2 - y1``：區域超出桌面的部分是黑色，整個 +區域都不在任何螢幕上則丟出 ``AutoControlScreenException``。不帶 +``screen_region`` 時擷取主螢幕。 + +Windows 的座標與 DPI +-------------------- + +``import je_auto_control`` 會把行程設成 **per-monitor DPI 感知（v2）**；Windows +版本太舊，或行程的感知已經被決定（嵌入的 host 程式、manifest）時，退回系統感知。 +per-monitor 的意思是每個螢幕都用自己的實體像素：一個 1920x1080 的螢幕，不論縮放 +設定是多少，對滑鼠、對 ``screenshot``、對視窗函式都是 1920 寬，擷取到的影像也 +不會被縮放。 + +在此之前行程是\ *系統*\ 感知，縮放比例與主螢幕不同的螢幕會被 Windows 虛擬化： +它看起來是實際大小的「主螢幕縮放 ÷ 該螢幕縮放」倍（主螢幕 100%、旁邊 125% 的 +1920x1080 螢幕會變成 1536x864），截到的是縮過的模糊影像。在這種螢幕上錄下的 +座標、從它裁出來的樣板影像，都是在虛擬化的空間裡取得的，必須重錄；距離該螢幕 +左上角 ``(dx, dy)`` 的點，現在位於 ``(dx, dy) × 該螢幕縮放 ÷ 主螢幕縮放``。 +主螢幕以及縮放與主螢幕相同的螢幕不受影響。 + 螢幕尺寸 ======== diff --git a/je_auto_control/gui/_screen_geometry.py b/je_auto_control/gui/_screen_geometry.py index ba600b161..eadcc3dae 100644 --- a/je_auto_control/gui/_screen_geometry.py +++ b/je_auto_control/gui/_screen_geometry.py @@ -3,9 +3,12 @@ Qt keeps each screen's top-left corner the same in both and scales within the screen by its device pixel ratio. Measured on Windows in a per-monitor-aware process: a 125% screen at (1920, -164) is 1536x864 in Qt and 1920x1080 -natively. (``import je_auto_control`` makes a process system-DPI-aware, and -then Qt reports a ratio of 1.0 there, in the same virtualised space as the -cursor and ``mss``.) On macOS the capture APIs and the pointer take points, +natively. ``import je_auto_control`` makes the process per-monitor-v2 aware, +which is also Qt 6's own default, so that is the case here: the cursor, ``mss`` +and the capture all work in each screen's physical pixels and Qt's ratio is +the real one. (In a process something else made system-DPI-aware first, Qt +reports 1.0 on such a screen and everything shares the virtualised space; the +same arithmetic holds.) On macOS the capture APIs and the pointer take points, which is Qt's logical unit, so no scaling applies. """ import sys diff --git a/je_auto_control/utils/cv2_utils/region_capture.py b/je_auto_control/utils/cv2_utils/region_capture.py index 927913e65..23d68b87a 100644 --- a/je_auto_control/utils/cv2_utils/region_capture.py +++ b/je_auto_control/utils/cv2_utils/region_capture.py @@ -17,12 +17,33 @@ from __future__ import annotations import sys -from typing import TYPE_CHECKING, Optional, Sequence +from typing import TYPE_CHECKING, Any, Optional, Sequence if TYPE_CHECKING: from PIL import Image +def grab_windows_region(left: int, top: int, right: int, bottom: int, + grabber: Optional[Any] = None) -> Image.Image: + """The Windows capture of ``[left, top, right, bottom]``, on any monitor. + + Always ``right - left`` by ``bottom - top``: ``grab_logical`` clips the + region to the desktop, and the part that is off screen is padded back in + black here, so a point in the image is still ``(left, top)`` plus its + position. A region with none of it on screen raises + ``AutoControlScreenException``. + """ + from je_auto_control.utils.monitor_layout.logical_frame import grab_logical + width, height = right - left, bottom - top + image, origin_x, origin_y = grab_logical((left, top, width, height), grabber=grabber) + if image.size == (width, height): + return image + from PIL import Image + canvas = Image.new(image.mode, (width, height)) + canvas.paste(image, (origin_x - left, origin_y - top)) + return canvas + + def grab_screen_region(region: Optional[Sequence[int]] = None) -> Image.Image: """Return the screen inside ``[left, top, right, bottom]`` as a PIL image. @@ -36,8 +57,7 @@ def grab_screen_region(region: Optional[Sequence[int]] = None) -> Image.Image: _validate_region(list(region)) left, top, right, bottom = (int(value) for value in region) if sys.platform.startswith("win"): - from je_auto_control.utils.monitor_layout.logical_frame import grab_logical - return grab_logical((left, top, right - left, bottom - top))[0] + return grab_windows_region(left, top, right, bottom) if sys.platform == "darwin": from je_auto_control.utils.cv2_utils.screen_grabber import image_grabber return image_grabber().grab(bbox=(left, top, right, bottom), scale_down=True) diff --git a/je_auto_control/utils/cv2_utils/screenshot.py b/je_auto_control/utils/cv2_utils/screenshot.py index 649269f79..6ea52b5cd 100644 --- a/je_auto_control/utils/cv2_utils/screenshot.py +++ b/je_auto_control/utils/cv2_utils/screenshot.py @@ -1,6 +1,7 @@ from __future__ import annotations -from typing import TYPE_CHECKING, List, Optional +import sys +from typing import TYPE_CHECKING, Any, List, Optional if TYPE_CHECKING: # pragma: no cover - annotations only from PIL import Image @@ -34,6 +35,20 @@ def _validate_region(screen_region: List[int]) -> None: ) +def _grab_region(grabber: Any, screen_region: List[int]) -> Image.Image: + """Capture ``[left, top, right, bottom]`` wherever on the desktop it is. + + ``ImageGrab.grab(bbox=...)`` captures the primary monitor on Windows and + crops that, so a region on a monitor left of or above the primary one came + back black. There the region is cut from the whole virtual desktop instead. + """ + if sys.platform.startswith("win"): + from je_auto_control.utils.cv2_utils.region_capture import grab_windows_region + left, top, right, bottom = (int(value) for value in screen_region) + return grab_windows_region(left, top, right, bottom, grabber=grabber) + return grabber.grab(bbox=screen_region) + + def pil_screenshot(file_path: Optional[str] = None, screen_region: Optional[List[int]] = None) -> Image.Image: """ Take a screenshot through the platform's capture backend. @@ -47,15 +62,22 @@ def pil_screenshot(file_path: Optional[str] = None, screen_region: Optional[List :param file_path: (str | None) Path to save the screenshot. If None, do not save. 螢幕截圖的存檔路徑，若為 None 則不存檔 - :param screen_region: (list[int] | None) Region to capture [left, top, right, bottom]. - 擷取的螢幕區域 [左, 上, 右, 下]，若為 None 則擷取全螢幕 + :param screen_region: (list[int] | None) Region to capture [left, top, right, bottom], + in the coordinates the mouse takes. On Windows it may be on any + monitor (negative left of or above the primary one); the part of + it that is off the desktop is black, and a region entirely off + the desktop raises AutoControlScreenException. None captures the + primary screen. + 擷取的螢幕區域 [左, 上, 右, 下]（滑鼠座標）。Windows 上可在任何 + 螢幕；超出桌面的部分為黑色，完全在桌面外則丟 + AutoControlScreenException。None 擷取主螢幕 :return: PIL.Image.Image object 擷取到的影像物件 """ # 擷取螢幕畫面 Capture screen grabber = image_grabber() if screen_region is not None: _validate_region(screen_region) - image = grabber.grab(bbox=screen_region) + image = _grab_region(grabber, screen_region) else: image = grabber.grab() diff --git a/je_auto_control/utils/monitor_layout/logical_frame.py b/je_auto_control/utils/monitor_layout/logical_frame.py index 194717ca8..c356a1866 100644 --- a/je_auto_control/utils/monitor_layout/logical_frame.py +++ b/je_auto_control/utils/monitor_layout/logical_frame.py @@ -19,6 +19,22 @@ desktop starts at negative coordinates whenever a monitor sits left of or above the primary one. +``import je_auto_control`` now makes the process per-monitor DPI aware +(``windows/screen/win32_screen.py``), and then the two spaces are the same one: +``GetSystemMetrics`` reports physical pixels as well, nothing is rescaled, and a +scaled monitor is captured sharp. The rescale stays for a process whose +awareness was fixed before the import (an embedding host, a manifest). + +macOS has the requirement in a third form: ``screencapture`` returns a Retina +display at twice its size in points, the unit Quartz mouse events take, and +without a rectangle it captures the main display only. There the frame is built +by :mod:`je_auto_control.utils.monitor_layout.macos_frame`, one display at a +time, each scaled to points. + +A ``region`` is clipped to the captured frame and the origin returned is the +clipped one. Cropping past the frame pads with black, and a matcher handed that +padding can report a hit that is not on any screen. + Wayland has the same requirement without the DPI half: its capture spans the compositor's whole output layout, and that layout starts at a negative coordinate whenever an output sits left of or above the origin. There is no @@ -34,8 +50,11 @@ import sys from typing import Any, Callable, Optional, Sequence, Tuple +from je_auto_control.utils.exception.exceptions import AutoControlScreenException + Rect = Tuple[int, int, int, int] MetricsReader = Callable[[int], int] +DisplayReader = Callable[[], Sequence[Sequence[int]]] # GetSystemMetrics indices for the virtual desktop, in logical pixels. SM_XVIRTUALSCREEN = 76 @@ -111,22 +130,95 @@ def _resample(): return getattr(getattr(Image, "Resampling", Image), "LANCZOS") +def checked_region(region: Sequence[int]) -> Rect: + """``region`` as four ints ``(x, y, width, height)`` with a positive size. + + Raises ``AutoControlScreenException`` otherwise. A negative width reached + Pillow's ``crop`` and came back as a bare ``ValueError``, outside the + family every caller's containment catches. + """ + try: + left, top, width, height = (int(value) for value in region) + except (TypeError, ValueError) as error: + raise AutoControlScreenException( + f"region must be 4 ints (x, y, width, height); got {region!r}") from error + if width <= 0 or height <= 0: + raise AutoControlScreenException( + f"region must have positive width and height; got " + f"({left}, {top}, {width}, {height})") + return left, top, width, height + + +def intersect_rect(first: Sequence[int], second: Sequence[int]) -> Optional[Rect]: + """The overlap of two ``(x, y, width, height)`` rectangles, or ``None``.""" + left = max(first[0], second[0]) + top = max(first[1], second[1]) + right = min(first[0] + first[2], second[0] + second[2]) + bottom = min(first[1] + first[3], second[1] + second[3]) + if right <= left or bottom <= top: + return None + return left, top, right - left, bottom - top + + +def clip_region(region: Rect, frame: Rect) -> Rect: + """``region`` cut down to ``frame``; raise when none of it is on screen.""" + clipped = intersect_rect(region, frame) + if clipped is None: + raise AutoControlScreenException( + f"region {region} (x, y, width, height) is entirely off screen; " + f"the desktop is {frame}") + return clipped + + +def _is_macos() -> bool: + return sys.platform == "darwin" + + +def _is_pillow_grab(image_grab: Any) -> bool: + """Whether ``image_grab`` is Pillow's own ``ImageGrab`` module. + + The macOS path corrects what Pillow's ``screencapture`` call returns. A + grabber that is something else — a platform backend's, a test's — has + neither defect and keeps the generic path. + """ + return getattr(image_grab, "__name__", "") == "PIL.ImageGrab" + + def grab_logical(region: Optional[Sequence[int]] = None, *, all_screens: bool = True, grabber: Optional[Any] = None, - metrics: Optional[MetricsReader] = None) -> Tuple[Any, int, int]: + metrics: Optional[MetricsReader] = None, + displays: Optional[DisplayReader] = None) -> Tuple[Any, int, int]: """Capture the screen in mouse-coordinate space. :param region: ``(x, y, width, height)`` in mouse coordinates, or ``None`` - for everything. + for everything. It is clipped to the desktop; a region with no area, + or none of it on screen, raises ``AutoControlScreenException``. :param all_screens: include monitors beyond the primary one. :param grabber: ``ImageGrab``-shaped object, for tests. :param metrics: ``GetSystemMetrics``-shaped reader, for tests. + :param displays: macOS only — returns each display's ``(x, y, width, + height)`` in points; the default asks Quartz. Passing one selects the + macOS path on any platform, for tests. :return: ``(image, origin_x, origin_y)`` — add the origin to any hit found in - the image to get a coordinate the mouse can be sent to. + the image to get a coordinate the mouse can be sent to. With a + ``region`` the origin is the clipped region's corner, which differs + from the requested one when the region started off screen. """ + box = None if region is None else checked_region(region) image_grab = grabber or _load_image_grab() - if region is None and not all_screens: + if displays is not None or (_is_macos() and _is_pillow_grab(image_grab)): + from je_auto_control.utils.monitor_layout.macos_frame import grab_macos + frame = grab_macos(image_grab, box, all_screens=all_screens, displays=displays) + if frame is not None: + return frame + return _grab_virtual_desktop(image_grab, box, all_screens, metrics) + + +def _grab_virtual_desktop(image_grab: Any, box: Optional[Rect], all_screens: bool, + metrics: Optional[MetricsReader]) -> Tuple[Any, int, int]: + """The frame as Windows, X11 and Wayland build it: one capture, rescaled, cropped.""" + if box is None and not all_screens: # The primary-only grab is already in logical pixels and starts at (0, 0). return image_grab.grab(), 0, 0 @@ -135,12 +227,13 @@ def grab_logical(region: Optional[Sequence[int]] = None, *, origin_x, origin_y = (rect[0], rect[1]) if rect else _backend_frame_origin() if rect and needs_rescale((image.width, image.height), (rect[2], rect[3])): image = image.resize((rect[2], rect[3]), _resample()) - if region is None: + if box is None: return image, origin_x, origin_y # Crop on the rescaled frame, never through ImageGrab's bbox: that crop # happens in physical pixels and would cut the wrong place on a scaled screen. - left, top, width, height = (int(value) for value in region) + left, top, width, height = clip_region( + box, (origin_x, origin_y, image.width, image.height)) image = image.crop((left - origin_x, top - origin_y, left - origin_x + width, top - origin_y + height)) return image, left, top diff --git a/je_auto_control/utils/monitor_layout/macos_frame.py b/je_auto_control/utils/monitor_layout/macos_frame.py new file mode 100644 index 000000000..55bd8fadc --- /dev/null +++ b/je_auto_control/utils/monitor_layout/macos_frame.py @@ -0,0 +1,142 @@ +"""Build the macOS capture frame in points, across every display. + +``ImageGrab.grab()`` runs ``screencapture -x`` on macOS, which has two +properties the rest of the package cannot work with: + +* it captures the **main display only** (Pillow's ``all_screens`` is ignored on + darwin), so a target on a second display is never found; +* a Retina display comes back at **twice its size in points**, and points are + what Quartz mouse events take, so a hit read off the image lands at twice + its distance from the corner. + +``screencapture -R x,y,w,h`` takes a rectangle in global display coordinates — +points, negative left of or above the main display — and Pillow's +``scale_down=True`` resizes the result to the rectangle's size in points. So +the frame is built one display at a time: ask Quartz for each display's bounds +in points, capture exactly that rectangle scaled to points, and paste it where +the display sits. A region is cut the same way, piece by piece per display, +which also covers a region that spans a 1x and a 2x display. + +Nothing here imports Quartz at module level, and both the display list and the +grabber are injectable, so the arithmetic is tested with fakes on any platform. +It has not been run on Retina hardware. + +Imports no ``PySide6``. +""" +from typing import Any, Callable, List, Optional, Sequence, Tuple + +from je_auto_control.utils.monitor_layout.logical_frame import ( + clip_region, intersect_rect, +) + +Rect = Tuple[int, int, int, int] +DisplayReader = Callable[[], Sequence[Sequence[int]]] +_MAX_DISPLAYS = 32 + + +def quartz_display_bounds() -> List[Rect]: + """Every active display as ``(x, y, width, height)`` in points. + + Global display coordinates: the main display's top-left corner is + ``(0, 0)``, and a display left of or above it has a negative origin. These + are the coordinates ``CGEventCreateMouseEvent`` and ``screencapture -R`` + take. + """ + import Quartz + + error, display_ids, count = Quartz.CGGetActiveDisplayList(_MAX_DISPLAYS, None, None) + if error: + return [] + bounds: List[Rect] = [] + for display_id in list(display_ids)[:int(count)]: + rect = Quartz.CGDisplayBounds(display_id) + bounds.append((int(rect.origin.x), int(rect.origin.y), + int(rect.size.width), int(rect.size.height))) + return bounds + + +def _read_displays(displays: Optional[DisplayReader]) -> List[Rect]: + """The displays with a real area, or ``[]`` when Quartz is not installed.""" + try: + rows = (displays or quartz_display_bounds)() + except ImportError: + return [] + found: List[Rect] = [] + for row in rows: + x, y, width, height = (int(value) for value in row) + if width > 0 and height > 0: + found.append((x, y, width, height)) + return found + + +def display_union(displays: Sequence[Rect]) -> Rect: + """The bounding box of ``displays`` as ``(x, y, width, height)``.""" + left = min(rect[0] for rect in displays) + top = min(rect[1] for rect in displays) + right = max(rect[0] + rect[2] for rect in displays) + bottom = max(rect[1] + rect[3] for rect in displays) + return left, top, right - left, bottom - top + + +def main_display(displays: Sequence[Rect]) -> Rect: + """The display at the global origin, which is the main one; else the first.""" + for rect in displays: + if rect[0] == 0 and rect[1] == 0: + return rect + return displays[0] + + +def _grab_points(image_grab: Any, rect: Rect) -> Any: + """Capture ``rect`` (points) as an image exactly ``rect`` wide and tall.""" + left, top, width, height = rect + image = image_grab.grab(bbox=(left, top, left + width, top + height), + scale_down=True) + if image.size != (width, height): + # A grabber that ignored scale_down handed back device pixels. + from je_auto_control.utils.monitor_layout.logical_frame import _resample + image = image.resize((width, height), _resample()) + return image + + +def _stitch(image_grab: Any, box: Rect, displays: Sequence[Rect]) -> Any: + """``box`` (points) assembled from the part of it on each display.""" + pieces = [piece for piece in (intersect_rect(box, rect) for rect in displays) + if piece is not None] + if len(pieces) == 1 and pieces[0] == box: + return _grab_points(image_grab, box) + from PIL import Image + canvas = Image.new("RGB", (box[2], box[3])) + for piece in pieces: + canvas.paste(_grab_points(image_grab, piece).convert("RGB"), + (piece[0] - box[0], piece[1] - box[1])) + return canvas + + +def grab_macos(image_grab: Any, region: Optional[Rect] = None, *, + all_screens: bool = True, + displays: Optional[DisplayReader] = None, + ) -> Optional[Tuple[Any, int, int]]: + """Capture in points: ``(image, origin_x, origin_y)``, or ``None``. + + ``None`` means the display list is unavailable (pyobjc's Quartz is not + installed, or it reported no display) and the caller should capture the + way it does elsewhere. + + :param image_grab: ``ImageGrab``-shaped object. + :param region: ``(x, y, width, height)`` in points, already validated; it + is clipped to the displays and raises ``AutoControlScreenException`` + when none of it is on one. + :param all_screens: with no ``region``, every display or only the main one. + :param displays: returns each display's bounds in points; the default asks + Quartz. + """ + found = _read_displays(displays) + if not found: + return None + if region is not None: + box = clip_region(region, display_union(found)) + elif all_screens: + box = display_union(found) + else: + box = main_display(found) + return _stitch(image_grab, box, found), box[0], box[1] diff --git a/je_auto_control/utils/set_of_marks/set_of_marks.py b/je_auto_control/utils/set_of_marks/set_of_marks.py index 46994b3f4..13438e066 100644 --- a/je_auto_control/utils/set_of_marks/set_of_marks.py +++ b/je_auto_control/utils/set_of_marks/set_of_marks.py @@ -14,7 +14,7 @@ """ import io from pathlib import Path -from typing import Any, Dict, List, Optional +from typing import Any, Dict, List, Optional, Sequence from je_auto_control.utils.accessibility.element import element_box @@ -73,11 +73,14 @@ def resolve_mark(marks: List[Dict[str, Any]], return None -def _draw_marks(image: Any, marks: List[Dict[str, Any]]) -> Any: +def _draw_marks(image: Any, marks: List[Dict[str, Any]], + origin: Sequence[int] = (0, 0)) -> Any: + """Draw ``marks`` (screen coordinates) on an image whose corner is ``origin``.""" from PIL import ImageDraw draw = ImageDraw.Draw(image) for mark in marks: left, top, width, height = mark["bbox"] + left, top = left - int(origin[0]), top - int(origin[1]) draw.rectangle([left, top, left + width, top + height], outline=_OUTLINE, width=2) label = str(mark["id"]) @@ -88,11 +91,17 @@ def _draw_marks(image: Any, marks: List[Dict[str, Any]]) -> Any: def render_marks(image_bytes: bytes, - marks: List[Dict[str, Any]]) -> bytes: - """Draw numbered boxes for ``marks`` on a PNG; return annotated PNG bytes.""" + marks: List[Dict[str, Any]], + origin: Sequence[int] = (0, 0)) -> bytes: + """Draw numbered boxes for ``marks`` on a PNG; return annotated PNG bytes. + + ``origin`` is the screen coordinate of the image's top-left pixel: marks + are in screen coordinates, and a capture of the whole desktop starts at a + negative one whenever a monitor sits left of or above the primary. + """ from PIL import Image image = Image.open(io.BytesIO(image_bytes)).convert("RGB") - _draw_marks(image, marks) + _draw_marks(image, marks, origin) out = io.BytesIO() image.save(out, format="PNG") return out.getvalue() @@ -108,7 +117,10 @@ def mark_screen(app_name: Optional[str] = None, """Number the live accessibility elements; optionally render an overlay. Stores the marks for a later :func:`mark_click`. When ``render_path`` is - given, a screenshot is captured, annotated, and saved there. + given, every monitor is captured, annotated, and saved there; the result + then carries ``image_origin``, the screen coordinate of the image's + top-left pixel (``mark bbox - image_origin`` is where a mark is drawn). + Marks themselves stay in screen coordinates. """ from je_auto_control.utils.accessibility.accessibility_api import ( list_accessibility_elements) @@ -117,11 +129,15 @@ def mark_screen(app_name: Optional[str] = None, _last_marks.extend(marks) result: Dict[str, Any] = {"marks": marks} if render_path: - from je_auto_control.utils.cv2_utils.screenshot import pil_screenshot - image = _draw_marks(pil_screenshot().convert("RGB"), marks) + # The whole desktop, not pil_screenshot(): that is the primary monitor + # only, so an element on any other one was drawn off the image. + from je_auto_control.utils.monitor_layout.logical_frame import grab_logical + frame, origin_x, origin_y = grab_logical(None) + image = _draw_marks(frame.convert("RGB"), marks, (origin_x, origin_y)) target = Path(render_path) image.save(str(target), format="PNG") result["image_path"] = str(target.resolve()) + result["image_origin"] = [int(origin_x), int(origin_y)] return result diff --git a/je_auto_control/windows/screen/win32_screen.py b/je_auto_control/windows/screen/win32_screen.py index 52d36e552..db6f7806c 100644 --- a/je_auto_control/windows/screen/win32_screen.py +++ b/je_auto_control/windows/screen/win32_screen.py @@ -1,5 +1,5 @@ import sys -from typing import Tuple +from typing import Any, Tuple from je_auto_control.utils.exception.exception_tags import windows_import_error_message from je_auto_control.utils.exception.exceptions import AutoControlException @@ -40,14 +40,90 @@ _gdi32.GetPixel.argtypes = [wintypes.HDC, ctypes.c_int, ctypes.c_int] _gdi32.GetPixel.restype = wintypes.COLORREF -# 確保 DPI 感知，避免座標偏移。**這是行程層級的副作用**，而它發生在 import 時： -# 一旦設定就無法還原，之後所有 Win32 座標查詢都會拿到實體像素。這正是本模組被 -# import 的理由（螢幕尺寸與取色都必須是實體座標），但呼叫端要知道它會影響整個 -# 行程——擷取與滑鼠座標的換算請走 `utils/monitor_layout`。 -# -# Process-wide and irreversible, and it happens at import time; conversions -# between physical and logical coordinates belong to ``utils/monitor_layout``. -_user32.SetProcessDPIAware() +_DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2 = -4 +_AWARENESS_NAMES = {0: "unaware", 1: "system", 2: "per_monitor"} +_dpi_awareness_requested = False + + +def _request_dpi_awareness(user32: Any) -> None: + """Ask for per-monitor-v2 awareness, then system awareness; never raise.""" + try: + setter = user32.SetProcessDpiAwarenessContext + setter.argtypes = [ctypes.c_void_p] + setter.restype = wintypes.BOOL + if setter(ctypes.c_void_p(_DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2)): + return + except (AttributeError, OSError): + # Windows before 10 1703 exports no such function. + pass + try: + user32.SetProcessDPIAware() + except (AttributeError, OSError): + pass + + +def dpi_awareness(user32: Any = None) -> str: + """ + 這個行程實際的 DPI 感知：``unaware``／``system``／``per_monitor``／``unknown`` + The awareness this process really has: ``unaware`` / ``system`` / + ``per_monitor`` / ``unknown`` + + 問系統而不是記住自己要求過什麼：行程的感知只能設定一次，host 程式、manifest + 或先建立的 Qt 可能早就決定了。 + Asked of the system rather than remembered: awareness can be set once per + process, and a host application, a manifest or an earlier Qt may have + decided it already. + """ + library = user32 or _user32 + try: + get_context = library.GetThreadDpiAwarenessContext + get_context.argtypes = [] + get_context.restype = ctypes.c_void_p + get_awareness = library.GetAwarenessFromDpiAwarenessContext + get_awareness.argtypes = [ctypes.c_void_p] + get_awareness.restype = ctypes.c_int + return _AWARENESS_NAMES.get(int(get_awareness(get_context())), "unknown") + except (AttributeError, OSError): + return "unknown" + + +def enable_dpi_awareness(user32: Any = None) -> str: + """ + 讓行程成為 per-monitor v2 DPI 感知（做不到就退回系統感知）；回傳實際結果 + Make the process per-monitor-v2 DPI aware, falling back to system + awareness; return what it ended up with + + **行程層級、無法還原，而且發生在 import 時**（本模組底下呼叫一次）。 + per-monitor 之後，每個螢幕的 Win32 座標、滑鼠座標與擷取到的像素都是該螢幕的 + 實體像素。先前用的 `SetProcessDPIAware()` 是**系統**感知：只有 DPI 與主螢幕 + 相同的螢幕是實體像素，其他螢幕被 Windows 虛擬化——座標被縮放、截圖是縮過的 + 模糊影像。 + + 只要求一次：第二次呼叫不再碰 Win32，只回報現況。感知已被別人設定時（要求會 + 被拒絕）不丟例外，照樣回報現況；那種行程裡擷取與滑鼠座標的換算仍由 + `utils/monitor_layout` 負責。 + + Process-wide, irreversible, and it happens at import time (this module + calls it once below). Once per-monitor, Win32 coordinates, mouse + coordinates and captured pixels are each monitor's physical pixels. The + ``SetProcessDPIAware()`` used before is *system* awareness: only monitors + at the primary monitor's DPI were physical, and Windows virtualised the + rest — scaled coordinates and a blurred, resized capture. + + Requested once: a second call does not touch Win32 and only reports. A + process whose awareness someone else fixed first refuses the request; that + is not an error, and ``utils/monitor_layout`` still converts between + capture and mouse coordinates there. + """ + global _dpi_awareness_requested + library = user32 or _user32 + if not _dpi_awareness_requested: + _dpi_awareness_requested = True + _request_dpi_awareness(library) + return dpi_awareness(library) + + +enable_dpi_awareness() _CLR_INVALID = 0xFFFFFFFF diff --git a/test/unit_test/headless/test_platform_coordinate_contract.py b/test/unit_test/headless/test_platform_coordinate_contract.py new file mode 100644 index 000000000..4170edf39 --- /dev/null +++ b/test/unit_test/headless/test_platform_coordinate_contract.py @@ -0,0 +1,388 @@ +"""One pixel of a capture is one coordinate for the mouse, on every monitor. + +No real screen is read here: ``ImageGrab``, ``GetSystemMetrics``, ``user32`` and +Quartz are all fakes. + +* Windows asked for *system* DPI awareness, so a monitor at a different scale + from the primary one was virtualised. The process now asks for per-monitor-v2 + first and falls back. +* ``pil_screenshot(screen_region=...)`` — and with it ``screenshot``, + ``AC_screenshot``, ``keyword_screenshot`` and ``capture_window`` — captured + the primary monitor and cropped that, so a region on a monitor left of or + above it was black. +* ``mark_screen`` drew its marks on a primary-monitor screenshot. +* ``grab_logical`` cropped a region that ran off the desktop without clipping + it, so the padding could hold a "hit"; a negative size raised ``ValueError``. +* On macOS ``grab_logical`` returned Retina pixels and the main display only. +""" +import ctypes +import sys +import types + +import pytest +from PIL import Image + +from je_auto_control.utils.cv2_utils import region_capture, screenshot +from je_auto_control.utils.exception.exceptions import AutoControlScreenException +from je_auto_control.utils.monitor_layout import grab_logical, logical_frame, macos_frame +from je_auto_control.utils.set_of_marks import set_of_marks + +_WINDOWS = sys.platform in ("win32", "cygwin", "msys") +windows_only = pytest.mark.skipif(not _WINDOWS, reason="the Win32 modules import on Windows only") + +_WHITE, _RED, _BLUE, _BLACK = (255, 255, 255), (255, 0, 0), (0, 0, 255), (0, 0, 0) +_LEFT_MONITOR = [-1920, 0, -1720, 100] + + +# --- Windows: DPI awareness -------------------------------------------------- + +def _user32(calls, per_monitor=1, system=1, awareness=2): + """A ``user32`` recording which awareness was asked for. + + Plain functions, not methods: the module sets ``argtypes`` on what it + calls, as it has to on the real thing. + """ + def set_context(context): + # A handle is pointer-width and unsigned; -4 is how the SDK spells it. + calls.append(("context", ctypes.c_ssize_t(context.value).value)) + return per_monitor + + def set_aware(): + calls.append(("system",)) + return system + + return types.SimpleNamespace( + SetProcessDpiAwarenessContext=set_context, SetProcessDPIAware=set_aware, + GetThreadDpiAwarenessContext=lambda: 1, + GetAwarenessFromDpiAwarenessContext=lambda context: awareness) + + +@pytest.fixture +def win32_screen(monkeypatch): + from je_auto_control.windows.screen import win32_screen as module + monkeypatch.setattr(module, "_dpi_awareness_requested", False) + return module + + +@windows_only +def test_per_monitor_v2_is_asked_for_first(win32_screen): + calls = [] + assert win32_screen.enable_dpi_awareness(_user32(calls)) == "per_monitor" + assert calls == [("context", -4)], "no fallback once per-monitor was granted" + + +@windows_only +def test_a_refused_request_falls_back_to_system_awareness(win32_screen): + """Refused when something set the awareness first; that must not raise.""" + calls = [] + answer = win32_screen.enable_dpi_awareness(_user32(calls, per_monitor=0, awareness=1)) + assert calls == [("context", -4), ("system",)] + assert answer == "system" + + +@windows_only +def test_an_older_windows_without_the_function_falls_back(win32_screen): + calls = [] + fake = _user32(calls, awareness=1) + del fake.SetProcessDpiAwarenessContext + assert win32_screen.enable_dpi_awareness(fake) == "system" + assert calls == [("system",)] + + +@windows_only +def test_awareness_is_requested_once_per_process(win32_screen): + calls = [] + fake = _user32(calls) + win32_screen.enable_dpi_awareness(fake) + win32_screen.enable_dpi_awareness(fake) + assert calls == [("context", -4)] + + +@windows_only +def test_a_system_that_can_do_neither_is_not_an_error(win32_screen): + def refuse(*_args): + raise OSError("no such entry point") + + fake = types.SimpleNamespace(SetProcessDpiAwarenessContext=refuse, SetProcessDPIAware=refuse) + assert win32_screen.enable_dpi_awareness(fake) == "unknown" + + +@windows_only +def test_importing_the_package_left_the_process_dpi_aware(): + """Read-only: whatever set it, this process is not unaware after the import.""" + from je_auto_control.windows.screen import win32_screen as module + assert module.dpi_awareness() in ("system", "per_monitor") + + +# --- Windows: a region on another monitor ------------------------------------ + +class _Desktop: + """``ImageGrab`` as on Windows: a bbox crops the primary monitor, ``all_screens`` spans both. + + The primary monitor is at (0, 0); a second one is to its left and holds a + red block at (-1900, 10) to (-1800, 60). + """ + + def __init__(self, block=_RED): + self.image = Image.new("RGB", (3840, 1080), _WHITE) + if block: + self.image.paste(block, (20, 10, 120, 60)) + + def grab(self, bbox=None, all_screens=False, **_kwargs): + image, (x0, y0) = (self.image, (-1920, 0)) if all_screens \ + else (self.image.crop((1920, 0, 3840, 1080)), (0, 0)) + if bbox: + left, top, right, bottom = bbox + image = image.crop((left - x0, top - y0, right - x0, bottom - y0)) + return image.copy() + + +def _metrics(index): + return {76: -1920, 77: 0, 78: 3840, 79: 1080}[index] + + +@pytest.fixture +def desktop(monkeypatch): + """Windows with a monitor left of the primary one.""" + grabber = _Desktop() + monkeypatch.setattr(sys, "platform", "win32") + monkeypatch.setattr(logical_frame, "_system_metrics", _metrics) + monkeypatch.setattr(logical_frame, "_load_image_grab", lambda: grabber) + monkeypatch.setattr(screenshot, "image_grabber", lambda: grabber) + return grabber + + +def test_negative_secondary_monitor_capture(desktop): + image = screenshot.pil_screenshot(screen_region=_LEFT_MONITOR) + assert image.size == (200, 100) + assert image.getpixel((50, 30)) == _RED, "the block at (-1870, 30)" + captured_secondary_is_black = image.getpixel((50, 30)) == _BLACK + assert captured_secondary_is_black is False + + +def test_screenshot_and_its_keyword_see_the_other_monitor(desktop, tmp_path): + pytest.importorskip("cv2") + from je_auto_control.utils.pytest_plugin.keywords import keyword_screenshot + from je_auto_control.wrapper.auto_control_screen import screenshot as wrapper_screenshot + frame = wrapper_screenshot(screen_region=_LEFT_MONITOR) + assert frame.shape[:2] == (100, 200) + assert tuple(int(value) for value in frame[30, 50]) == (0, 0, 255), "red, as BGR" + target = tmp_path / "keyword.png" + keyword_screenshot(str(target), region=_LEFT_MONITOR) + with Image.open(target) as saved: + assert saved.convert("RGB").getpixel((50, 30)) == _RED + + +def test_a_window_on_the_other_monitor_is_captured(desktop, tmp_path): + pytest.importorskip("cv2") + from je_auto_control.utils.window_capture import capture_window + target = tmp_path / "window.png" + assert capture_window("Editor", target, geometry=lambda title: (-1920, 0, 200, 100)) == str(target) + with Image.open(target) as saved: + assert saved.size == (200, 100) + assert saved.convert("RGB").getpixel((50, 30)) == _RED + + +def test_a_region_running_off_the_desktop_keeps_its_size_and_corner(desktop): + """Still ``right - left`` wide, so a point in it is ``left`` plus its x.""" + image = screenshot.pil_screenshot(screen_region=[-2020, 0, -1720, 100]) + assert image.size == (300, 100) + assert image.getpixel((50, 30)) == _BLACK, "left of every monitor" + assert image.getpixel((150, 30)) == _RED, "the block, 100 px further in" + assert region_capture.grab_screen_region([-2020, 0, -1720, 100]).size == (300, 100) + + +def test_a_region_off_every_monitor_is_refused(desktop): + with pytest.raises(AutoControlScreenException, match="off screen"): + screenshot.pil_screenshot(screen_region=[5000, 5000, 5100, 5100]) + + +# --- grab_logical: clipping -------------------------------------------------- + +def test_a_located_region_is_clipped_to_the_desktop(desktop): + """The origin handed back is the clipped one, so a hit maps to a real pixel.""" + image, origin_x, origin_y = grab_logical((-2020, -50, 300, 150), grabber=desktop, + metrics=_metrics) + assert (image.size, origin_x, origin_y) == ((200, 100), -1920, 0) + assert image.getpixel((50, 30)) == _RED + + +@pytest.mark.parametrize("region", [(0, 0, -10, 10), (0, 0, 10, 0), (0, 0, 10), "nope"]) +def test_empty_region_rejected(desktop, monkeypatch, region): + """Before any capture, and as a framework error rather than a ValueError.""" + captured = [] + monkeypatch.setattr(desktop, "grab", lambda **kwargs: captured.append(kwargs)) + with pytest.raises(AutoControlScreenException): + grab_logical(region, grabber=desktop, metrics=_metrics) + assert captured == [] + + +def test_a_region_inside_the_desktop_is_untouched(desktop): + image, origin_x, origin_y = grab_logical((-1900, 10, 100, 50), grabber=desktop, + metrics=_metrics) + assert (image.size, origin_x, origin_y) == ((100, 50), -1900, 10) + + +# --- set-of-marks ------------------------------------------------------------ + +def test_marks_include_virtual_origin(monkeypatch, tmp_path): + from je_auto_control.utils.accessibility import accessibility_api + grabber = _Desktop(block=None) + monkeypatch.setattr(logical_frame, "_system_metrics", _metrics) + monkeypatch.setattr(logical_frame, "_load_image_grab", lambda: grabber) + elements =[{"bbox": [-1900, 10, 100, 50], "role": "button", "text": "left monitor"}, + {"bbox": [300, 400, 100, 50], "role": "button", "text": "primary"}] + monkeypatch.setattr(accessibility_api, "list_accessibility_elements", + lambda app_name=None: elements) + target = tmp_path / "marks.png" + result = set_of_marks.mark_screen(render_path=str(target)) + + marked_points = [mark["center"] for mark in result["marks"]] + expected_global_points = [[-1850, 35], [350, 425]] + assert marked_points == expected_global_points, "marks stay in screen coordinates" + assert result["image_origin"] == [-1920, 0] + with Image.open(target) as saved: + picture = saved.convert("RGB") + assert picture.size == (3840, 1080), "every monitor, not the primary one" + assert picture.getpixel((20 + 100, 10 + 25)) == _RED, "the left monitor's box edge" + assert picture.getpixel((1920 + 300 + 100, 400 + 25)) == _RED, "the primary's box edge" + assert picture.getpixel((300 + 100, 400 + 25)) == _WHITE, "not drawn 1920 px to the left" + + +def test_render_marks_takes_the_images_origin(): + import io + blank = io.BytesIO() + Image.new("RGB", (400, 200), _WHITE).save(blank, format="PNG") + marks = set_of_marks.mark_elements([{"bbox": [-150, 20, 100, 50]}]) + with Image.open(io.BytesIO(set_of_marks.render_marks(blank.getvalue(), marks, + origin=(-200, 0)))) as picture: + assert picture.convert("RGB").getpixel((50 + 100, 20 + 25)) == _RED + + +# --- macOS ------------------------------------------------------------------- + +_MAIN = (0, 0, 1440, 900) # Retina: 2880 x 1800 device pixels +_LEFT = (-1920, 0, 1920, 1080) # a 1x display to its left +_DISPLAYS = [_MAIN, _LEFT] + + +class _ScreenCapture: + """``ImageGrab`` as on macOS: ``screencapture -R`` in points, Retina at 2x. + + The desktop, in points, is white with a blue block at (100, 50)-(140, 70) on + the Retina main display and a red one at (-1900, 10)-(-1800, 60) on the 1x + display. A capture comes back in device pixels unless ``scale_down`` is set, + and a capture with no rectangle is the main display only. + """ + + def __init__(self): + self.world = Image.new("RGB", (3360, 1080), _BLACK) + self.world.paste(_WHITE, (0, 0, 1920, 1080)) + self.world.paste(_WHITE, (1920, 0, 3360, 900)) + self.world.paste(_BLUE, (1920 + 100, 50, 1920 + 140, 70)) + self.world.paste(_RED, (20, 10, 120, 60)) + self.rectangles = [] + + def grab(self, bbox=None, all_screens=False, scale_down=False, **_kwargs): + left, top, right, bottom = bbox or (0, 0, 1440, 900) + self.rectangles.append(bbox) + points = self.world.crop((left + 1920, top, right + 1920, bottom)) + scale = 1 if scale_down or left < 0 else 2 + return points.resize((points.width * scale, points.height * scale)) + + +def _displays(): + return list(_DISPLAYS) + + +def test_retina_maps_to_points(): + """A target found in the frame is where the mouse has to go, not twice as far.""" + capture = _ScreenCapture() + image, origin_x, origin_y = grab_logical(None, grabber=capture, displays=_displays) + assert (image.size, origin_x, origin_y) == ((3360, 1080), -1920, 0) + found = next((x, y) for y in range(40, 80) for x in range(1920, 2100) + if image.getpixel((x, y)) == _BLUE) + logical_click = (found[0] + origin_x, found[1] + origin_y) + assert logical_click == (100, 50) + + +def test_the_second_display_is_in_the_macos_frame(): + image, origin_x, _origin_y = grab_logical(None, grabber=_ScreenCapture(), displays=_displays) + assert image.getpixel((-1870 - origin_x, 30)) == _RED + assert image.getpixel((3000, 1000)) == _BLACK, "below the shorter display: no screen there" + + +def test_a_macos_region_is_captured_in_points(): + capture = _ScreenCapture() + image, origin_x, origin_y = grab_logical((90, 40, 100, 60), grabber=capture, displays=_displays) + assert (image.size, origin_x, origin_y) == ((100, 60), 90, 40) + assert image.getpixel((10, 10)) == _BLUE and image.getpixel((5, 5)) == _WHITE + assert capture.rectangles == [(90, 40, 190, 100)] + + +def test_a_macos_region_across_two_displays_is_stitched(): + capture = _ScreenCapture() + image, origin_x, _origin_y = grab_logical((-1900, 0, 2100, 100), grabber=capture, + displays=_displays) + assert (image.size, origin_x) == ((2100, 100), -1900) + assert image.getpixel((30, 30)) == _RED, "(-1870, 30) on the 1x display" + assert image.getpixel((1900 + 110, 60)) == _BLUE, "(110, 60) on the Retina display" + assert sorted(capture.rectangles) == [(-1900, 0, 0, 100), (0, 0, 200, 100)] + + +def test_a_macos_region_is_clipped_to_the_displays(): + image, origin_x, origin_y = grab_logical((-2000, -30, 200, 100), grabber=_ScreenCapture(), + displays=_displays) + assert (image.size, origin_x, origin_y) == ((120, 70), -1920, 0) + with pytest.raises(AutoControlScreenException, match="off screen"): + grab_logical((9000, 0, 10, 10), grabber=_ScreenCapture(), displays=_displays) + + +def test_the_macos_primary_only_frame_is_in_points_too(): + image, origin_x, origin_y = grab_logical(None, all_screens=False, grabber=_ScreenCapture(), + displays=_displays) + assert (image.size, origin_x, origin_y) == ((1440, 900), 0, 0) + assert image.getpixel((110, 60)) == _BLUE + + +def test_without_quartz_the_generic_capture_is_used(): + """No pyobjc: the frame is whatever the grabber returns, as before.""" + def no_quartz(): + raise ImportError("No module named 'Quartz'") + + capture = _ScreenCapture() + assert macos_frame.grab_macos(capture, None, displays=no_quartz) is None + image, origin_x, origin_y = grab_logical(None, grabber=capture, displays=no_quartz, + metrics=lambda index: 0) + assert (image.size, origin_x, origin_y) == ((2880, 1800), 0, 0) + + +def test_only_pillows_own_grabber_takes_the_macos_path(monkeypatch): + """A backend's or a test's grabber is not ``screencapture`` and is left alone.""" + from PIL import ImageGrab + monkeypatch.setattr(logical_frame, "_is_macos", lambda: True) + assert logical_frame._is_pillow_grab(ImageGrab) is True + capture = _ScreenCapture() + grab_logical(None, grabber=capture, metrics=lambda index: 0) + assert capture.rectangles == [None], "one plain capture, no per-display rectangles" + + +def test_display_bounds_are_read_from_quartz(monkeypatch): + def bounds(display_id): + x, y, width, height = {1: _MAIN, 2: _LEFT}[display_id] + return types.SimpleNamespace(origin=types.SimpleNamespace(x=float(x), y=float(y)), + size=types.SimpleNamespace(width=float(width), + height=float(height))) + + quartz = types.SimpleNamespace( + CGGetActiveDisplayList=lambda limit, displays, count: (0, (1, 2), 2), + CGDisplayBounds=bounds) + monkeypatch.setitem(sys.modules, "Quartz", quartz) + assert macos_frame.quartz_display_bounds() == _DISPLAYS + quartz.CGGetActiveDisplayList = lambda limit, displays, count: (1001, (), 0) + assert macos_frame.quartz_display_bounds() == [] + + +def test_the_main_display_is_the_one_at_the_origin(): + assert macos_frame.main_display([_LEFT, _MAIN]) == _MAIN + assert macos_frame.display_union(_DISPLAYS) == (-1920, 0, 3360, 1080) From 64f0e38cb62d103be53227731f97fba7c3fc1c4c Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:01:53 +0800 Subject: [PATCH 15/26] Read templates at any path and in grayscale, keep the left OCR box, floor negative centres A template under a non-ASCII folder was unreadable through cv2.imread, a 2-D or PIL L template raised cv2.error that no caller catches, a target starting at the end of a long OCR box was dropped with that box, and int() cut the centre of a hit left of the primary monitor toward zero. --- .../utils/cv2_utils/template_detection.py | 60 ++++++-- je_auto_control/utils/ocr/text_span.py | 22 ++- je_auto_control/wrapper/auto_control_image.py | 10 +- .../test_image_ocr_coordinate_contract.py | 141 ++++++++++++++++++ 4 files changed, 215 insertions(+), 18 deletions(-) create mode 100644 test/unit_test/headless/test_image_ocr_coordinate_contract.py diff --git a/je_auto_control/utils/cv2_utils/template_detection.py b/je_auto_control/utils/cv2_utils/template_detection.py index 97432c0bc..a0d8afb96 100644 --- a/je_auto_control/utils/cv2_utils/template_detection.py +++ b/je_auto_control/utils/cv2_utils/template_detection.py @@ -9,6 +9,7 @@ import os from typing import Any, List, Optional, Sequence, Tuple +from je_auto_control.utils.cv2_utils.image_file import read_image from je_auto_control.utils.cv2_utils.optional import require_cv2, require_je_open_cv from je_auto_control.utils.exception.exceptions import ImageNotFoundException from je_auto_control.utils.monitor_layout.logical_frame import grab_logical @@ -38,22 +39,61 @@ def _prepare(image: Any, detect_threshold: float, f"detect_threshold must be between 0 and 1, got {detect_threshold!r}") if isinstance(image, (str, os.PathLike)) and not os.path.isfile(image): raise ImageNotFoundException(f"template image not found: {image}") - open_cv = require_je_open_cv() + # Still the door that names the missing wheel on Windows arm64. + require_je_open_cv() cv2 = require_cv2() + template = _gray_template(cv2, image) grab_image, origin_x, origin_y = grab_logical(screen_region, all_screens=all_screens) - frame, template = open_cv.image_translate(grab_image, os.fspath(image) - if isinstance(image, os.PathLike) else image) - if template is None: - # cv2.imread returns None for an unreadable file; that used to surface - # as "'NoneType' object has no attribute 'shape'". - raise ImageNotFoundException(f"cannot read template image: {image}") - if template.shape[0] > frame.shape[0] or template.shape[1] > frame.shape[1]: - raise ImageNotFoundException("template is larger than the searched area") - scores = cv2.matchTemplate(frame, template, cv2.TM_CCOEFF_NORMED) + try: + frame = _to_gray(cv2, grab_image) + if template.shape[0] > frame.shape[0] or template.shape[1] > frame.shape[1]: + raise ImageNotFoundException("template is larger than the searched area") + scores = cv2.matchTemplate(frame, template, cv2.TM_CCOEFF_NORMED) + except cv2.error as error: + # An input OpenCV cannot match (an empty frame, a pixel type it does + # not take) is a search that found nothing, in the type callers catch. + raise ImageNotFoundException(f"cannot match template image: {error}") from error effective = min(threshold, 1.0 - _SCORE_EPSILON) return (frame, scores), template, effective, (origin_x, origin_y) +def _to_gray(cv2: Any, image: Any) -> Any: + """``image`` (PIL image or array) as a 2-D ``uint8`` grayscale array. + + A 2-D array or a PIL ``"L"`` image is grayscale already and is used as it + is: ``cvtColor`` raised ``cv2.error`` on it, which no caller's ``except`` + list names. + """ + import numpy as np + array = np.asarray(image) + if array.ndim == 3 and array.shape[2] == 1: + array = array[:, :, 0] + if array.ndim == 3: + array = cv2.cvtColor(array, cv2.COLOR_RGB2GRAY) + if array.ndim != 2: + raise ImageNotFoundException( + f"template detection needs a 2-D or 3-D image, got shape {array.shape}") + if array.dtype == np.bool_: + # PIL mode "1": True / False, which OpenCV has no pixel type for. + array = array.astype(np.uint8) * 255 + return array + + +def _gray_template(cv2: Any, image: Any) -> Any: + """Load ``image`` (path, PIL image or array) as a grayscale template. + + A path is read through :func:`image_file.read_image`, not ``cv2.imread``: + on Windows ``imread`` returns ``None`` for a path with non-ASCII + characters, so a template in ``測試\\t.png`` was reported unreadable. + """ + try: + if isinstance(image, (str, os.PathLike)): + return read_image(os.fspath(image), cv2.IMREAD_GRAYSCALE) + return _to_gray(cv2, image) + except (ValueError, TypeError, cv2.error) as error: + raise ImageNotFoundException(f"cannot read template image: {image!r}") from error + + def _draw(frame: Any, boxes: Sequence[Sequence[int]]) -> Any: cv2 = require_cv2() for x1, y1, x2, y2 in boxes: diff --git a/je_auto_control/utils/ocr/text_span.py b/je_auto_control/utils/ocr/text_span.py index e90249956..7a6ac022a 100644 --- a/je_auto_control/utils/ocr/text_span.py +++ b/je_auto_control/utils/ocr/text_span.py @@ -95,13 +95,27 @@ def _next_span(line: Sequence[Any], index: int, needle: str, if needle in accumulated: return _shrink_left(line, index, end, needle, case_sensitive), end if len(accumulated) > len(needle) + MAX_OVERSHOOT: - # This start can no longer produce a shortest match; drop the - # leftmost box and keep scanning instead of restarting the line. - index += 1 - accumulated = _joined(line, index, end, case_sensitive) + index, accumulated = _drop_left(line, index, end, needle, case_sensitive) return None +def _drop_left(line: Sequence[Any], index: int, end: int, needle: str, + case_sensitive: bool) -> Tuple[int, str]: + """Drop leading boxes that can no longer start a match; return the rest. + + A match that starts in box ``index`` and is not complete at ``end`` has to + take in every box after it, so the box can go only once those alone are as + long as the target. Dropping it for the run's total length threw away a + long sentence box that *ended* with the target's first word: ``Save As`` + after ``... then choose Save`` was never found. + """ + rest = _joined(line, index + 1, end, case_sensitive) + while index < end and len(rest) >= len(needle): + index += 1 + rest = _joined(line, index + 1, end, case_sensitive) + return index, _joined(line, index, end, case_sensitive) + + def find_spans(boxes: Sequence[Any], target: str, case_sensitive: bool = False, tolerance: float = LINE_TOLERANCE) -> List[List[Any]]: diff --git a/je_auto_control/wrapper/auto_control_image.py b/je_auto_control/wrapper/auto_control_image.py index 612fc2bd5..4efb0a2d8 100644 --- a/je_auto_control/wrapper/auto_control_image.py +++ b/je_auto_control/wrapper/auto_control_image.py @@ -45,8 +45,10 @@ def locate_image_center(image, detect_threshold: float = 1.0, image_data_array = template_detection.find_image(image, detect_threshold, draw_image) if image_data_array[0]: x1, y1, x2, y2 = image_data_array[1] - center_x = int((x1 + x2) / 2) - center_y = int((y1 + y2) / 2) + # Floor, not int(): truncation rounds toward zero, so on a monitor + # left of or above the primary the centre came out one pixel off. + center_x = int((x1 + x2) // 2) + center_y = int((y1 + y2) // 2) record_action_to_list("locate_image_center", {"image": image, "threshold": detect_threshold}) return center_x, center_y raise ImageNotFoundException(f"{cant_find_image_error_message} / {image}") @@ -70,8 +72,8 @@ def locate_and_click(image, mouse_keycode: Union[int, str], image_data_array = template_detection.find_image(image, detect_threshold, draw_image) if image_data_array[0]: x1, y1, x2, y2 = image_data_array[1] - center_x = int((x1 + x2) / 2) - center_y = int((y1 + y2) / 2) + center_x = int((x1 + x2) // 2) + center_y = int((y1 + y2) // 2) set_mouse_position(center_x, center_y) click_mouse(mouse_keycode) record_action_to_list("locate_and_click", {"image": image, "threshold": detect_threshold}) diff --git a/test/unit_test/headless/test_image_ocr_coordinate_contract.py b/test/unit_test/headless/test_image_ocr_coordinate_contract.py new file mode 100644 index 000000000..fce82b836 --- /dev/null +++ b/test/unit_test/headless/test_image_ocr_coordinate_contract.py @@ -0,0 +1,141 @@ +"""Template reading, OCR span matching and centre arithmetic, on synthetic data. + +From the 2026-09-24 audit: a template under a non-ASCII folder could not be +read (``cv2.imread`` goes through the C locale on Windows), a grayscale +template -- a 2-D array or a PIL ``"L"`` image -- raised ``cv2.error``, which +no caller catches, a target that began at the end of a long OCR box was never +found, and the centre of a hit left of or above the primary monitor was one +pixel off because ``int()`` cuts toward zero. + +The screen is a synthetic frame; nothing is captured or clicked. +""" +import types + +import numpy as np +import pytest +from PIL import Image + +pytest.importorskip("cv2") +pytest.importorskip("je_open_cv") + +from je_auto_control.utils.cv2_utils import template_detection # noqa: E402 +from je_auto_control.utils.cv2_utils.image_file import write_image # noqa: E402 +from je_auto_control.utils.exception.exceptions import ImageNotFoundException # noqa: E402 +from je_auto_control.utils.ocr import text_span # noqa: E402 +from je_auto_control.wrapper import auto_control_image # noqa: E402 + +_ORIGIN = (1000, 500) +_CROP = (130, 80, 170, 120) +_CENTRE = (1150, 600) + + +@pytest.fixture +def screen(monkeypatch): + """A smooth blob centred at (150, 100) of a 300x200 frame, on screen at _ORIGIN.""" + yy, xx = np.mgrid[0:200, 0:300] + blob = (255 * np.exp(-(((xx - 150) ** 2) / 400 + ((yy - 100) ** 2) / 300))).astype(np.uint8) + frame = Image.fromarray(np.stack([blob] * 3, axis=-1)) + monkeypatch.setattr(template_detection, "grab_logical", + lambda region, all_screens=True: (frame.copy(), *_ORIGIN)) + return frame + + +def test_a_template_under_a_non_ascii_folder_is_read(screen, tmp_path): + folder = tmp_path / "測試" + folder.mkdir() + path = folder / "樣板.png" + write_image(path, np.asarray(screen.crop(_CROP))) + assert auto_control_image.locate_image_center(str(path)) == _CENTRE + assert auto_control_image.locate_image_center(path) == _CENTRE + + +def test_a_pil_grayscale_template_is_matched(screen): + template = screen.crop(_CROP).convert("L") + assert auto_control_image.locate_image_center(template) == _CENTRE + + +def test_a_two_dimensional_array_template_is_matched(screen): + template = np.asarray(screen.crop(_CROP).convert("L")) + assert template.ndim == 2 + assert auto_control_image.locate_image_center(template) == _CENTRE + assert auto_control_image.locate_all_image(template, 0.95) == [[1130, 580, 1170, 620]] + + +def test_a_single_channel_three_dimensional_template_is_matched(screen): + template = np.asarray(screen.crop(_CROP).convert("L"))[:, :, None] + assert auto_control_image.locate_image_center(template) == _CENTRE + + +def test_a_colour_template_still_matches(screen): + assert auto_control_image.locate_image_center(screen.crop(_CROP)) == _CENTRE + assert auto_control_image.locate_image_center(np.asarray(screen.crop(_CROP))) == _CENTRE + + +@pytest.mark.parametrize("template", [ + np.zeros((5, 5, 2), dtype=np.uint8), # no such colour layout + np.zeros((5, 5), dtype=np.float64), # a pixel type matchTemplate refuses + np.zeros((2, 2, 2, 2), dtype=np.uint8), +]) +def test_a_template_opencv_cannot_use_is_the_typed_miss(screen, template): + with pytest.raises(ImageNotFoundException): + auto_control_image.locate_image_center(template) + + +def test_an_undecodable_file_is_the_typed_miss(screen, tmp_path): + path = tmp_path / "broken.png" + path.write_bytes(b"not an image") + with pytest.raises(ImageNotFoundException, match="broken.png"): + auto_control_image.locate_image_center(str(path)) + + +# --- centre arithmetic ------------------------------------------------------ + +@pytest.mark.parametrize("box, centre", [ + ([-5, -5, 0, 0], (-3, -3)), # pixels -5..-1: the middle one is -3 + ([-4, -9, -1, -2], (-3, -6)), + ([0, 0, 5, 5], (2, 2)), # the mirror image, unchanged + ([10, 20, 30, 40], (20, 30)), +]) +def test_the_centre_is_floored_on_both_sides_of_the_origin(monkeypatch, box, centre): + monkeypatch.setattr(template_detection, "find_image", lambda *a, **k: [True, box]) + assert auto_control_image.locate_image_center("any") == centre + + +def test_locate_and_click_uses_the_same_centre(monkeypatch): + moved, clicked = [], [] + monkeypatch.setattr(template_detection, "find_image", + lambda *a, **k: [True, [-5, -5, 0, 0]]) + monkeypatch.setattr(auto_control_image, "set_mouse_position", + lambda x, y: moved.append((x, y))) + monkeypatch.setattr(auto_control_image, "click_mouse", clicked.append) + assert auto_control_image.locate_and_click("any", "mouse_left") == (-3, -3) + assert moved == [(-3, -3)] and clicked == ["mouse_left"] + + +# --- OCR span matching ------------------------------------------------------ + +def _box(text, x): + return types.SimpleNamespace(text=text, x=x, y=10, width=40, height=12) + + +_LONG = "Open the File menu at the top of the window and then choose" + + +def test_a_target_starting_at_the_end_of_a_long_box_is_found(): + assert len(text_span.normalize(_LONG + "Save")) > len("saveas") + text_span.MAX_OVERSHOOT + boxes = [_box(_LONG + " Save", 0), _box("As", 500)] + spans = text_span.find_spans(boxes, "Save As") + assert [[box.text for box in span] for span in spans] == [[_LONG + " Save", "As"]] + + +def test_a_target_after_a_long_run_of_short_boxes_is_still_minimal(): + words = (_LONG * 2).split() + boxes = [_box(word, 50 * index) for index, word in enumerate([*words, "Save", "As", "now"])] + spans = text_span.find_spans(boxes, "Save As") + assert [[box.text for box in span] for span in spans] == [["Save", "As"]] + + +def test_a_long_box_is_dropped_once_it_cannot_start_the_target(): + boxes = [_box(_LONG, 0), _box("Export", 500), _box("Save", 600), _box("As", 700)] + spans = text_span.find_spans(boxes, "Save As") + assert [[box.text for box in span] for span in spans] == [["Save", "As"]] From 5690d293962c32641ee293adeff69388d9682339 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:01:54 +0800 Subject: [PATCH 16/26] Type capitals and Shift as written, one Enter per line break, and scroll up for a positive count everywhere write() typed capitals in lower case on Windows and X11 and is_shift did nothing outside macOS; CR LF pressed Enter twice; mouse_scroll's default direction made a positive count go down on X11 and up elsewhere; a NaN scroll point was clamped instead of refused and fractional coordinates were cut toward zero. Unicode typing sent line breaks and Tab as code points, a dead-key Shift level was labelled with the unshifted character, ISO keys had no label, the layout prototypes were set on the shared user32, and an unnamed clipboard format was called 'None'. --- docs/source/API/wrapper/keyboard.rst | 11 +- docs/source/API/wrapper/mouse.rst | 23 +- docs/source/Eng/doc/keyboard/keyboard_doc.rst | 13 +- docs/source/Eng/doc/mouse/mouse_doc.rst | 15 +- docs/source/Zh/doc/keyboard/keyboard_doc.rst | 11 +- docs/source/Zh/doc/mouse/mouse_doc.rst | 13 +- je_auto_control/gui/_auto_click_tab.py | 2 +- .../clipboard_formats/clipboard_formats.py | 9 +- .../utils/input_macro/input_macro.py | 4 +- .../utils/keyboard_layout/keyboard_layout.py | 64 +++- .../utils/mcp_server/fake_backend.py | 2 +- .../utils/mcp_server/tools/_handlers_input.py | 2 +- .../utils/text_unicode/text_unicode.py | 32 +- .../wrapper/auto_control_keyboard.py | 189 ++++++++--- je_auto_control/wrapper/auto_control_mouse.py | 33 +- .../headless/test_input_wrapper_contract.py | 321 ++++++++++++++++++ .../headless/test_platform_backend_binding.py | 2 +- .../headless/test_r3_platform_scroll_guard.py | 2 +- test/unit_test/headless/test_wrapper_audit.py | 2 +- .../headless/test_wrapper_seam_contract.py | 8 +- 20 files changed, 641 insertions(+), 117 deletions(-) create mode 100644 test/unit_test/headless/test_input_wrapper_contract.py diff --git a/docs/source/API/wrapper/keyboard.rst b/docs/source/API/wrapper/keyboard.rst index c0ce8f0b1..a410753db 100644 --- a/docs/source/API/wrapper/keyboard.rst +++ b/docs/source/API/wrapper/keyboard.rst @@ -61,7 +61,9 @@ press_keyboard_key :param keycode: Key name or key code to press. :type keycode: int or str - :param bool is_shift: Whether to press Shift simultaneously. + :param bool is_shift: Press the key with Shift held. On Windows and X11 + Shift goes down before the key and comes back up as soon as the key is + down, so a lone press never leaves Shift stuck. :param bool skip_record: If ``True``, this action will not be recorded. :returns: The key code that was pressed. :rtype: str @@ -77,7 +79,8 @@ release_keyboard_key :param keycode: Key name or key code to release. :type keycode: int or str - :param bool is_shift: Whether Shift was pressed. + :param bool is_shift: Whether Shift was pressed. Used by the macOS backend + only; on Windows and X11 the Shift of a press is already released. :param bool skip_record: If ``True``, this action will not be recorded. :returns: The key code that was released. :rtype: str @@ -93,7 +96,9 @@ type_keyboard :param keycode: Key name or key code to type. :type keycode: int or str - :param bool is_shift: Whether to press Shift simultaneously. + :param bool is_shift: Hold Shift around the key (pressed first, released + last, also when the key fails). Works on every platform; it used to be + ignored on Windows and X11. :param bool skip_record: If ``True``, this action will not be recorded. :returns: The key code that was typed. :rtype: str diff --git a/docs/source/API/wrapper/mouse.rst b/docs/source/API/wrapper/mouse.rst index a1de4c4a1..b212f69e3 100644 --- a/docs/source/API/wrapper/mouse.rst +++ b/docs/source/API/wrapper/mouse.rst @@ -121,14 +121,21 @@ click_mouse mouse_scroll ============ -.. function:: mouse_scroll(scroll_value, x=None, y=None, scroll_direction="scroll_down") - - Scrolls the mouse wheel. - - :param int scroll_value: Number of scroll units. - :param int x: X position (default: current position). - :param int y: Y position (default: current position). - :param str scroll_direction: Scroll direction (Linux only). One of: +.. function:: mouse_scroll(scroll_value, x=None, y=None, scroll_direction="scroll_up") + + Scrolls the mouse wheel. A positive ``scroll_value`` scrolls up and a + negative one down, on every platform. + + :param int scroll_value: Number of scroll notches; negative reverses the direction. + :param int x: X position (default: current position). A fractional value is + rounded to the nearest pixel; a value that is not a finite number raises + ``AutoControlMouseException`` before the cursor moves. + :param int y: Y position (default: current position). Same rules as ``x``. + :param str scroll_direction: The direction a *positive* ``scroll_value`` + takes. Read on X11 and Wayland only (Windows and macOS have one wheel + axis and use the sign alone). One of: ``"scroll_up"``, ``"scroll_down"``, ``"scroll_left"``, ``"scroll_right"``. + The default was ``"scroll_down"``, which made ``mouse_scroll(3)`` go down + on X11 / Wayland and up elsewhere; pass it explicitly to keep that. :returns: Tuple of ``(scroll_value, direction)``. :rtype: tuple diff --git a/docs/source/Eng/doc/keyboard/keyboard_doc.rst b/docs/source/Eng/doc/keyboard/keyboard_doc.rst index ea0468b7b..9680ccec6 100644 --- a/docs/source/Eng/doc/keyboard/keyboard_doc.rst +++ b/docs/source/Eng/doc/keyboard/keyboard_doc.rst @@ -85,13 +85,20 @@ Type a sequence of characters one by one: write("Hello World") +A capital letter is typed as a capital: ``write`` holds Shift around it on +Windows and X11 (it used to arrive in lower case). A Windows line ending +(CR LF) is one line break and presses Enter once, not twice. ``is_shift=True`` +holds Shift for every key, and does so on every platform; the same is true of +``type_keyboard``, ``press_keyboard_key`` and ``hotkey``, where it used to be +ignored outside macOS. + ``write`` logs the text, records it in the test record and returns it. For a password or token use ``write_secret`` (``AC_write_secret`` with ``secret``): the log gets the length only, the record a masked value, it returns nothing, and an error never names a character. It types every character as a Unicode key -event, so the text arrives exactly (``write`` types capitals as lower case on -Windows); a backend without Unicode typing (only Windows has it) refuses before -typing anything. +event, so the text arrives exactly whatever the layout and Caps Lock say (line +breaks, Tab and Backspace are pressed as keys); a backend without Unicode typing +(only Windows has it) refuses before typing anything. .. code-block:: python diff --git a/docs/source/Eng/doc/mouse/mouse_doc.rst b/docs/source/Eng/doc/mouse/mouse_doc.rst index 6880c342a..4bd4142e0 100644 --- a/docs/source/Eng/doc/mouse/mouse_doc.rst +++ b/docs/source/Eng/doc/mouse/mouse_doc.rst @@ -81,10 +81,19 @@ Scroll the mouse wheel: from je_auto_control import mouse_scroll - # Scroll down by 5 units + # Scroll up by 5 notches; a negative value scrolls down mouse_scroll(scroll_value=5) + mouse_scroll(scroll_value=-5) + +A positive value scrolls up and a negative one down on every platform. Pass +``x`` / ``y`` to scroll at a point: a fractional coordinate is rounded to the +nearest pixel, and one that is not a finite number raises +``AutoControlMouseException`` before the cursor moves. .. note:: - On Linux, you can specify the scroll direction using the ``scroll_direction`` parameter: - ``"scroll_up"``, ``"scroll_down"``, ``"scroll_left"``, ``"scroll_right"``. + On X11 and Wayland ``scroll_direction`` names the direction a positive value + takes: ``"scroll_up"`` (the default), ``"scroll_down"``, ``"scroll_left"``, + ``"scroll_right"``. The default used to be ``"scroll_down"``, so + ``mouse_scroll(5)`` scrolled down there and up on Windows and macOS; pass + ``scroll_direction="scroll_down"`` to keep the old meaning. diff --git a/docs/source/Zh/doc/keyboard/keyboard_doc.rst b/docs/source/Zh/doc/keyboard/keyboard_doc.rst index 971401dae..bf37173b3 100644 --- a/docs/source/Zh/doc/keyboard/keyboard_doc.rst +++ b/docs/source/Zh/doc/keyboard/keyboard_doc.rst @@ -83,12 +83,17 @@ Windows 的按鍵表也收常見的別名（``ctrl``、``alt``、``enter``、``e write("Hello World") +大寫字母會打成大寫：``write`` 在 Windows 與 X11 會在那個鍵外面按住 Shift（以前會 +打成小寫）。Windows 換行（CR LF）是一個換行，只按一次 Enter，不是兩次。 +``is_shift=True`` 會在每個鍵外面按住 Shift，而且每個平台都有效；``type_keyboard``、 +``press_keyboard_key`` 與 ``hotkey`` 也一樣，以前在 macOS 以外會被忽略。 + ``write`` 會把文字寫進 log、記進測試紀錄並回傳。密碼或 token 請用 ``write_secret``\ （命令是 ``AC_write_secret``，參數 ``secret``）：log 只記長度、 紀錄只留遮蔽值、不回傳任何東西，錯誤訊息也不會帶出任何字元。它把每個字元都以 -Unicode 按鍵事件送出，所以送到的文字一字不差（``write`` 在 Windows 上會把大寫打 -成小寫）；鍵盤後端不支援 Unicode 輸入時（目前只有 Windows 支援），在打任何字之前 -就拒絕。 +Unicode 按鍵事件送出，所以不論鍵盤配置與 Caps Lock 狀態，送到的文字一字不差 +（換行、Tab 與 Backspace 以按鍵送出）；鍵盤後端不支援 Unicode 輸入時（目前只有 +Windows 支援），在打任何字之前就拒絕。 .. code-block:: python diff --git a/docs/source/Zh/doc/mouse/mouse_doc.rst b/docs/source/Zh/doc/mouse/mouse_doc.rst index af5530a80..bf8456bf0 100644 --- a/docs/source/Zh/doc/mouse/mouse_doc.rst +++ b/docs/source/Zh/doc/mouse/mouse_doc.rst @@ -79,10 +79,17 @@ Windows 與 X11 依兩次點擊的時間差與位移判定雙擊，所以 ``inte from je_auto_control import mouse_scroll - # 向下捲動 5 個單位 + # 向上捲動 5 格；負值向下 mouse_scroll(scroll_value=5) + mouse_scroll(scroll_value=-5) + +每個平台都一樣：正值往上、負值往下。要在指定位置捲動就傳 ``x``／``y``：帶小數的 +座標會四捨五入到最近的像素，不是有限數值的座標會在游標移動之前丟出 +``AutoControlMouseException``。 .. note:: - 在 Linux 上，可以使用 ``scroll_direction`` 參數指定捲動方向： - ``"scroll_up"``、``"scroll_down"``、``"scroll_left"``、``"scroll_right"``。 + 在 X11 與 Wayland 上，``scroll_direction`` 指定「正值」的方向： + ``"scroll_up"``\ （預設）、``"scroll_down"``、``"scroll_left"``、``"scroll_right"``。 + 預設值原本是 ``"scroll_down"``，所以 ``mouse_scroll(5)`` 在那裡往下、在 Windows 與 + macOS 往上；要維持舊行為請明確傳 ``scroll_direction="scroll_down"``。 diff --git a/je_auto_control/gui/_auto_click_tab.py b/je_auto_control/gui/_auto_click_tab.py index 99397ea6b..1a848693b 100644 --- a/je_auto_control/gui/_auto_click_tab.py +++ b/je_auto_control/gui/_auto_click_tab.py @@ -285,7 +285,7 @@ def _send_write(self): def _send_scroll(self): try: val = int(self.scroll_value_input.text() or "3") - direction = self.scroll_dir_combo.currentText() if self.scroll_dir_combo else "scroll_down" + direction = self.scroll_dir_combo.currentText() if self.scroll_dir_combo else "scroll_up" mouse_scroll(val, scroll_direction=direction) except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: QMessageBox.warning(self, "Error", str(error)) diff --git a/je_auto_control/utils/clipboard_formats/clipboard_formats.py b/je_auto_control/utils/clipboard_formats/clipboard_formats.py index b25e3148c..35d08bac2 100644 --- a/je_auto_control/utils/clipboard_formats/clipboard_formats.py +++ b/je_auto_control/utils/clipboard_formats/clipboard_formats.py @@ -39,11 +39,16 @@ def _coerce(item: _Format) -> Tuple[int, str]: - """Normalise a format descriptor (int / ``{id,name}`` / ``(id,name)``).""" + """Normalise a format descriptor (int / ``{id,name}`` / ``(id,name)``). + + A missing or ``None`` name is ``""`` in every form: the pair form used to + give the string ``"None"``, so the same unnamed format compared unequal + across the two forms and ``diff_formats`` reported a change. + """ if isinstance(item, dict): return int(item.get("id", 0)), str(item.get("name") or "") if isinstance(item, (tuple, list)): - return int(item[0]), str(item[1] if len(item) > 1 else "") + return int(item[0]), str((item[1] if len(item) > 1 else "") or "") return int(item), "" diff --git a/je_auto_control/utils/input_macro/input_macro.py b/je_auto_control/utils/input_macro/input_macro.py index 1a8f63d4f..5d52847d1 100644 --- a/je_auto_control/utils/input_macro/input_macro.py +++ b/je_auto_control/utils/input_macro/input_macro.py @@ -52,8 +52,8 @@ def _sink_scroll(event: Dict[str, Any]) -> None: # direction on every backend, X11 and Wayland included. They used to # discard it and always scroll ``scroll_direction``, so a macro # recorded on Windows replayed backwards there, silently. The direction - # is named too: X11 and Wayland default to ``scroll_down`` for a positive - # value, the recorders' "up". + # is named too, so a positive value is the recorders' "up" whatever the + # default is (it was ``scroll_down`` on X11 and Wayland). mouse_scroll(int(event.get("value", event.get("delta", 1))), scroll_direction="scroll_up") diff --git a/je_auto_control/utils/keyboard_layout/keyboard_layout.py b/je_auto_control/utils/keyboard_layout/keyboard_layout.py index bdfd7f25c..a9a27e71d 100644 --- a/je_auto_control/utils/keyboard_layout/keyboard_layout.py +++ b/je_auto_control/utils/keyboard_layout/keyboard_layout.py @@ -20,7 +20,7 @@ mapping off Windows. Imports no ``PySide6``. """ import sys -from typing import Dict, Optional, Tuple +from typing import Any, Callable, Dict, Optional, Tuple from je_auto_control.utils.logging.logging_instance import autocontrol_logger @@ -43,7 +43,30 @@ _VK_SHIFT = 0x10 _VK_SPACE = 0x20 _MAPVK_VK_TO_VSC = 0 -_LAYOUT_CACHE: Dict[int, Dict[int, Tuple[str, str]]] = {} +#: Keys the US table has no row for but other layouts print from: ABNT C1/C2 +#: (Brazilian), OEM_8 (UK and others), OEM_AX, and OEM_102 -- the extra key +#: beside left Shift on ISO boards (``<`` on German, French and Nordic ones). +#: Without them those keys never had a label on any layout. +_EXTRA_CANDIDATE_VK: Tuple[int, ...] = (0xC1, 0xC2, 0xDF, 0xE1, 0xE2) + +#: ``{vk: (unshifted, shifted)}``; the shifted half is ``None`` for a key whose +#: Shift level prints no single character (a dead key, typically). +CharTable = Dict[int, Tuple[str, Optional[str]]] +_LAYOUT_CACHE: Dict[int, CharTable] = {} + + +def _user32() -> Any: + """A ``user32`` handle of this module's own. + + ``ctypes.windll.user32`` is one object for the whole process, so a + prototype set on it is set for everybody: after ``ToUnicodeEx.argtypes`` + was declared there with a ``c_char`` array, another caller passing the + usual ``c_ubyte`` array got ``ArgumentError``. A separate ``WinDLL`` + carries its own prototypes. + """ + import ctypes + # getattr: the name exists on Windows only, which is where this is called. + return getattr(ctypes, "WinDLL")("user32") def foreground_keyboard_layout() -> Optional[int]: @@ -51,8 +74,7 @@ def foreground_keyboard_layout() -> Optional[int]: if not sys.platform.startswith("win"): return None try: - import ctypes - user32 = ctypes.windll.user32 + user32 = _user32() window = user32.GetForegroundWindow() thread_id = user32.GetWindowThreadProcessId(window, None) if window else 0 return int(user32.GetKeyboardLayout(thread_id)) @@ -61,8 +83,12 @@ def foreground_keyboard_layout() -> Optional[int]: return None -def _translator(user32, layout: int): - """Return ``translate(vk, shifted) -> str`` for one layout.""" +def _translator(user32: Any, layout: int) -> Callable[[int, bool], str]: + """Return ``translate(vk, shifted) -> str`` for one layout. + + ``user32`` must be a private handle (:func:`_user32`): the prototypes + below are set on whatever is passed in. + """ import ctypes from ctypes import wintypes user32.ToUnicodeEx.argtypes = [ @@ -90,22 +116,27 @@ def _translate(vk: int, shifted: bool) -> str: return _translate -def _build_table(translate) -> Dict[int, Tuple[str, str]]: - """Translate every candidate key, keeping only the printable results.""" - table: Dict[int, Tuple[str, str]] = {} - for vk in US_PRINTABLE_VK: +def _build_table(translate: Callable[[int, bool], str]) -> CharTable: + """Translate every candidate key, keeping only the printable results. + + A key whose Shift level is a dead key (Shift+6 on US-International) or + prints nothing gets ``None`` for that half. It used to repeat the + unshifted character, so Shift+6 was labelled ``6``. + """ + table: CharTable = {} + for vk in (*US_PRINTABLE_VK, *_EXTRA_CANDIDATE_VK): plain, shifted = translate(vk, False), translate(vk, True) if len(plain) == 1 and plain.isprintable(): usable = len(shifted) == 1 and shifted.isprintable() - table[vk] = (plain, shifted if usable else plain) + table[vk] = (plain, shifted if usable else None) translate(_VK_SPACE, False) # flush any dead-key state left behind return table -def layout_char_table(layout: Optional[int] = None - ) -> Dict[int, Tuple[str, str]]: +def layout_char_table(layout: Optional[int] = None) -> CharTable: """``{vk: (unshifted, shifted)}`` for ``layout`` (default: the foreground one). + ``shifted`` is ``None`` where Shift plus the key prints no character. Empty off Windows or when the OS will not answer, so callers can fall back to :data:`US_PRINTABLE_VK`. """ @@ -116,8 +147,7 @@ def layout_char_table(layout: Optional[int] = None if layout in _LAYOUT_CACHE: return _LAYOUT_CACHE[layout] try: - import ctypes - table = _build_table(_translator(ctypes.windll.user32, layout)) + table = _build_table(_translator(_user32(), layout)) except (OSError, AttributeError, ValueError) as error: autocontrol_logger.info("layout table build failed: %r", error) return {} @@ -125,7 +155,7 @@ def layout_char_table(layout: Optional[int] = None return table -def char_table(layout: Optional[int] = None) -> Dict[int, Tuple[str, str]]: +def char_table(layout: Optional[int] = None) -> CharTable: """The layout's table, or the US table when the layout cannot be read. Not merged: a key missing from the layout's table (a dead key such as @@ -135,7 +165,7 @@ def char_table(layout: Optional[int] = None) -> Dict[int, Tuple[str, str]]: def vk_to_char(vk: int, shifted: bool = False, - table: Optional[Dict[int, Tuple[str, str]]] = None + table: Optional[CharTable] = None ) -> Optional[str]: """The character this key produces, or ``None`` if it produces none.""" pair = (char_table() if table is None else table).get(int(vk)) diff --git a/je_auto_control/utils/mcp_server/fake_backend.py b/je_auto_control/utils/mcp_server/fake_backend.py index b016a0c46..fbbae80aa 100644 --- a/je_auto_control/utils/mcp_server/fake_backend.py +++ b/je_auto_control/utils/mcp_server/fake_backend.py @@ -83,7 +83,7 @@ def _fake_release_mouse(mouse_keycode: Any, x: Any = None, def _fake_mouse_scroll(scroll_value: int, x: Any = None, y: Any = None, - scroll_direction: str = "scroll_down" + scroll_direction: str = "scroll_up" ) -> Tuple[int, str]: with _STATE_LOCK: _STATE.mouse_actions.append( diff --git a/je_auto_control/utils/mcp_server/tools/_handlers_input.py b/je_auto_control/utils/mcp_server/tools/_handlers_input.py index b9ccada8e..380c614b1 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers_input.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers_input.py @@ -38,7 +38,7 @@ def get_mouse_position() -> List[int]: def mouse_scroll(scroll_value: int, x: Optional[int] = None, y: Optional[int] = None, - scroll_direction: str = "scroll_down") -> List[Any]: + scroll_direction: str = "scroll_up") -> List[Any]: from je_auto_control.wrapper.auto_control_mouse import mouse_scroll as _scroll value, direction = _scroll(int(scroll_value), x, y, scroll_direction) return [int(value), str(direction)] diff --git a/je_auto_control/utils/text_unicode/text_unicode.py b/je_auto_control/utils/text_unicode/text_unicode.py index a3287dfe6..f19a40e55 100644 --- a/je_auto_control/utils/text_unicode/text_unicode.py +++ b/je_auto_control/utils/text_unicode/text_unicode.py @@ -25,6 +25,13 @@ Sink = Callable[[Dict[str, Any]], None] +#: Whitespace that means a *key*, not a character, and the key it means. Sent +#: as a Unicode code point these are dropped by most applications (a line feed +#: especially), so multi-line text arrived as one run-on line with no error. +#: ``write`` uses the same table (``WRITE_CONTROL_KEYS``). +CONTROL_KEYS: Dict[str, str] = {"\n": "return", "\r": "return", "\t": "tab", + "\b": "back"} + def unicode_code_units(text: str) -> List[int]: """Return the UTF-16 code units of ``text`` (surrogate pairs for > U+FFFF).""" @@ -49,11 +56,21 @@ def plan_paste(text: str, *, modifier: str = "ctrl") -> List[Dict[str, Any]]: def plan_unicode_keys(text: str) -> List[Dict[str, Any]]: """Return the op-plan to enter ``text`` as character-carrying key events. - One op per UTF-16 code unit, so a character above U+FFFF becomes the two - surrogates the platform layer has to send separately. + One ``unicode_unit`` op per UTF-16 code unit, so a character above U+FFFF + becomes the two surrogates the platform layer has to send separately. A + line break, Tab or Backspace is a ``key`` op naming the key to press + (:data:`CONTROL_KEYS`) instead of code point 10 / 13 / 9 / 8, which + applications drop; CR LF is one line break, as in ``write``. """ - return [{"op": "unicode_unit", "unit": unit} - for unit in unicode_code_units(text)] + plan: List[Dict[str, Any]] = [] + for char in (text or "").replace("\r\n", "\n"): + key = CONTROL_KEYS.get(char) + if key is not None: + plan.append({"op": "key", "key": key}) + else: + plan.extend({"op": "unicode_unit", "unit": unit} + for unit in unicode_code_units(char)) + return plan def unicode_keys_supported() -> bool: @@ -74,6 +91,9 @@ def _default_sink(event: Dict[str, Any]) -> None: elif op == "hotkey": from je_auto_control.wrapper.auto_control_keyboard import hotkey hotkey(list(event["keys"])) + elif op == "key": + from je_auto_control.wrapper.auto_control_keyboard import type_keyboard + type_keyboard(event["key"]) elif op == "unicode_unit": from je_auto_control.wrapper.platform_wrapper import keyboard # 平台縫沒有承諾這個成員——只有 Windows 有——所以照 `unicode_keys_supported` @@ -113,7 +133,9 @@ def type_unicode_keys(text: str, *, Requires a backend exposing ``type_unicode_unit`` (Windows today); callers that need a guaranteed route on every platform should use - :func:`type_unicode_text`. + :func:`type_unicode_text`. Line breaks, Tab and Backspace are pressed as + keys (see :func:`plan_unicode_keys`), so a custom ``sink`` receives + ``{"op": "key", "key": ...}`` for them. """ plan = plan_unicode_keys(text) return _dispatch(plan, text, sink, "keys") diff --git a/je_auto_control/wrapper/auto_control_keyboard.py b/je_auto_control/wrapper/auto_control_keyboard.py index 48cb50feb..2276d6301 100644 --- a/je_auto_control/wrapper/auto_control_keyboard.py +++ b/je_auto_control/wrapper/auto_control_keyboard.py @@ -11,7 +11,8 @@ import re import sys import warnings -from typing import Optional, Union, Tuple +from contextlib import contextmanager +from typing import Iterator, Optional, Union, Tuple from je_auto_control.utils.exception.exception_tags import ( keyboard_press_key_error_message, keyboard_release_key_error_message, @@ -25,7 +26,7 @@ from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.platform_id import is_windows, is_x11_unix from je_auto_control.utils.test_record.record_test_class import record_action_to_list -from je_auto_control.utils.text_unicode.text_unicode import unicode_code_units +from je_auto_control.utils.text_unicode.text_unicode import CONTROL_KEYS, unicode_code_units from je_auto_control.wrapper.platform_wrapper import ( keyboard, keyboard_check, keyboard_key_aliases, keyboard_keys_table, ) @@ -80,14 +81,52 @@ def _resolve_keycode(keycode: Union[int, str]) -> int: return keycode +def _backend_shift(is_shift: bool) -> bool: + """The ``is_shift`` to hand a key event whose Shift is already held. + + Only the macOS backend takes the flag (it posts the Shift event itself); + everywhere else Shift is a key this module holds, see ``_shift_held``. + """ + return is_shift and sys.platform == "darwin" + + +@contextmanager +def _shift_held(is_shift: bool) -> Iterator[None]: + """按住 Shift 直到區塊結束（Windows／X11）；macOS 由後端自己處理。 + Hold Shift for the block on Windows and X11; a no-op without ``is_shift`` + and on macOS, whose backend posts the Shift event from the flag. + + ``is_shift`` used to reach the macOS backend only, so on Windows and X11 + ``type_keyboard("a", is_shift=True)`` typed ``a`` while documenting Shift. + The release is in ``finally`` and never raises (``_release_still_held``): + a Shift left down changes every keystroke and click that follows. + """ + if not is_shift or sys.platform == "darwin": + yield + return + still_held: list = [] + try: + press_keyboard_key("shift", skip_record=True) + still_held.append("shift") + yield + finally: + _release_still_held(still_held, False) + + def press_keyboard_key(keycode: Union[int, str], is_shift: bool = False, skip_record: bool = False) -> Optional[str]: """ 按下指定鍵 Press a keyboard key + With ``is_shift`` the key goes down while Shift is held, on every platform. + On Windows and X11 Shift is pressed before the key and released right after + the key is down (the character is decided at key-down), so a lone press + never leaves Shift stuck; ``type_keyboard``, ``hotkey`` and ``write`` hold + it for their whole sequence instead. + :param keycode: 鍵盤代碼或字串 Keycode or string - :param is_shift: 是否同時按下 Shift + :param is_shift: 是否同時按下 Shift Hold Shift while the key goes down :param skip_record: 是否跳過紀錄 :return: keycode 字串 """ @@ -100,7 +139,8 @@ def press_keyboard_key(keycode: Union[int, str], is_shift: bool = False, if sys.platform == "darwin": keyboard.press_key(keycode, is_shift=is_shift) elif is_windows() or is_x11_unix(): - keyboard.press_key(keycode) + with _shift_held(is_shift): + keyboard.press_key(keycode) else: raise AutoControlKeyboardException( f"press_keyboard_key: no backend for {sys.platform!r}") @@ -121,6 +161,10 @@ def release_keyboard_key(keycode: Union[int, str], is_shift: bool = False, """ 放開指定鍵 Release a keyboard key + + ``is_shift`` reaches the macOS backend only: on Windows and X11 the Shift + of a press is already up by the time the key is released (see + ``press_keyboard_key``), so there is nothing left to release here. """ autocontrol_logger.info(f"release_keyboard_key, keycode={keycode}, is_shift={is_shift}, skip_record={skip_record}") try: @@ -185,11 +229,13 @@ def type_keyboard(keycode: Union[int, str], is_shift: bool = False, """ autocontrol_logger.info(f"type_keyboard, keycode={keycode}, is_shift={is_shift}, skip_record={skip_record}") still_held: list = [] + backend_shift = _backend_shift(is_shift) try: - press_keyboard_key(keycode, is_shift, skip_record=True) - still_held.append(keycode) - release_keyboard_key(keycode, is_shift, skip_record=True) - still_held.clear() + with _shift_held(is_shift): + press_keyboard_key(keycode, backend_shift, skip_record=True) + still_held.append(keycode) + release_keyboard_key(keycode, backend_shift, skip_record=True) + still_held.clear() if not skip_record: record_action_to_list("type_keyboard", {"keycode": keycode, "is_shift": is_shift}) @@ -207,7 +253,7 @@ def type_keyboard(keycode: Union[int, str], is_shift: bool = False, # TypeError, ValueError)` 名單的任何一項底下（實測確認）。也就是說最可能 # 發生的失敗（鍵名不在對照表裡、平台不支援、後端出錯）根本走不到那個 # `except`。`finally` 是唯一每條離開路徑都會跑到的地方。 - _release_still_held(still_held, is_shift) + _release_still_held(still_held, backend_shift) def check_key_is_press(keycode: Union[int, str]) -> Optional[bool]: """ @@ -239,8 +285,32 @@ def check_key_is_press(keycode: Union[int, str]) -> Optional[bool]: # Whitespace that means a *key*, not a character. Sent as a Unicode code point # these are silently dropped by most applications — a newline especially, which # turns a multi-line `write` into one run-on line with nothing reported. -WRITE_CONTROL_KEYS = {"\n": "return", "\r": "return", "\t": "tab", - "\b": "back"} +# One table for ``write`` and ``text_unicode``'s plans, so the two agree. +WRITE_CONTROL_KEYS = CONTROL_KEYS + +#: The key a character shares with its unshifted partner on a US layout. Only a +#: hint: Shift is added when the table really puts both on one key (see +#: ``_needs_shift``), so a layout that separates them is left alone. +_SHIFT_PARTNERS = { + "!": "1", "@": "2", "#": "3", "$": "4", "%": "5", "^": "6", "&": "7", + "*": "8", "(": "9", ")": "0", "_": "-", "+": "=", "{": "[", "}": "]", + "|": "\\", ":": ";", '"': "'", "<": ",", ">": ".", "?": "/", "~": "`", +} + + +def _needs_shift(single_char: str, key: int) -> bool: + """Whether typing ``single_char`` by its table key also needs Shift. + + The tables list a capital under the same code as its lower-case letter + (Windows: one virtual key; X11: one keycode), so pressing it alone typed + ``hi`` for ``"Hi"``. X11 lists shifted punctuation the same way. macOS is + left as it was: its backend is not part of this change. + """ + if sys.platform == "darwin": + return False + lowered = single_char.lower() + partner = lowered if lowered != single_char else _SHIFT_PARTNERS.get(single_char) + return partner is not None and keyboard_keys_table.get(partner) == key def _write_char_via_unicode(single_char: str) -> bool: @@ -259,6 +329,25 @@ def _write_char_via_unicode(single_char: str) -> bool: return True +def _write_one_char(single_char: str, is_shift: bool) -> None: + """Type one character of ``write``'s string, or raise if nothing can.""" + key = keyboard_keys_table.get(single_char) + control_key = WRITE_CONTROL_KEYS.get(single_char) + if control_key is not None and control_key in keyboard_keys_table: + # Before the table lookup: a newline must press Enter, not type + # U+000A and not fall through to the space fallback below. + type_keyboard(control_key, is_shift, skip_record=True) + elif key is not None: + type_keyboard(key, is_shift or _needs_shift(single_char, key), skip_record=True) + elif _write_char_via_unicode(single_char): + pass + elif single_char.isspace(): + type_keyboard("space", is_shift, skip_record=True) + else: + autocontrol_logger.error(f"write failed: {keyboard_write_cant_find_error_message}, char={single_char}") + raise AutoControlKeyboardException(keyboard_write_cant_find_error_message) + + def write(write_string: str, is_shift: bool = False) -> Optional[str]: """ 模擬輸入整個字串 @@ -270,34 +359,24 @@ def write(write_string: str, is_shift: bool = False) -> Optional[str]: backend supports them, and only raise where it does not — otherwise a single comma fails the whole string. + A capital letter is typed as a capital: the table gives it the key of its + lower-case letter, so Shift is held around it on Windows and X11 (it used + to come out lower case). A Windows line ending (CR LF) is one line break + and presses Enter once. ``is_shift`` holds Shift for every key typed from + the table. + :param write_string: 要輸入的字串 String to type - :param is_shift: 是否同時按下 Shift - :return: 輸入的字串 + :param is_shift: 是否同時按下 Shift Hold Shift for every key + :return: 輸入的字串 ``write_string``, unchanged """ autocontrol_logger.info(f"write, write_string={write_string}, is_shift={is_shift}") try: - record_write_chars = [] - for single_char in write_string: - key = keyboard_keys_table.get(single_char) - control_key = WRITE_CONTROL_KEYS.get(single_char) - if control_key is not None and control_key in keyboard_keys_table: - # Before the table lookup: a newline must press Enter, not type - # U+000A and not fall through to the space fallback below. - type_keyboard(control_key, is_shift, skip_record=True) - elif key is not None: - type_keyboard(key, is_shift, skip_record=True) - elif _write_char_via_unicode(single_char): - pass - elif single_char.isspace(): - type_keyboard("space", is_shift, skip_record=True) - else: - autocontrol_logger.error(f"write failed: {keyboard_write_cant_find_error_message}, char={single_char}") - raise AutoControlKeyboardException(keyboard_write_cant_find_error_message) - record_write_chars.append(single_char) - - result = "".join(record_write_chars) + # A Windows line ending is one Enter, not two: text read from a file + # used to gain an empty line after every line. + for single_char in write_string.replace("\r\n", "\n"): + _write_one_char(single_char, is_shift) record_action_to_list("write", {"write_string": write_string, "is_shift": is_shift}) - return result + return write_string except (OSError, RuntimeError, AttributeError, TypeError, ValueError) as error: record_action_to_list("write", {"write_string": write_string}, repr(error)) @@ -318,8 +397,9 @@ def write_secret(secret: str) -> None: ``write`` logs the text it types, records it in the test record and returns it, so a password typed through it lands in the log and in every run record. Here the log gets the length only and the record a masked argument. Every character goes - through Unicode key events, which type the exact character: the virtual-key path - of ``write`` types a capital letter as lower case on Windows. A backend without + through Unicode key events, which type the exact character whatever the layout + and Caps Lock say; a line break, Tab or Backspace is pressed as its key, since + applications drop those when they arrive as Unicode characters. A backend without Unicode typing (Windows has it) raises before typing anything, rather than risk a wrong character. A failure never names the character. @@ -332,8 +412,12 @@ def write_secret(secret: str) -> None: "write_secret: this platform's keyboard backend cannot type Unicode text exactly") autocontrol_logger.info(f"write_secret, {len(secret)} characters") try: - for single_char in secret: - _write_char_via_unicode(single_char) + for single_char in secret.replace("\r\n", "\n"): + control_key = WRITE_CONTROL_KEYS.get(single_char) + if control_key is not None and control_key in keyboard_keys_table: + type_keyboard(control_key, skip_record=True) + else: + _write_char_via_unicode(single_char) except (OSError, RuntimeError, AttributeError, TypeError, ValueError) as error: # The cause is dropped on purpose: its text could carry part of the secret. autocontrol_logger.error(f"write_secret failed: {type(error).__name__}") @@ -353,21 +437,26 @@ def hotkey(key_code_list: list, is_shift: bool = False) -> Tuple[str, str]: autocontrol_logger.info(f"hotkey, key_code_list={key_code_list}, is_shift={is_shift}") # 已經按下去、還沒放開的鍵，**依按下的順序**。放開時倒著走。 still_held: list = [] + backend_shift = _backend_shift(is_shift) try: press_list = [] release_list = [] - for key in key_code_list: - press_list.append(press_keyboard_key(key, is_shift, skip_record=True)) - # 按成功了才記——`press_keyboard_key` 丟例外時那個鍵並沒有被按下去， - # 記進來的話收尾會去放開一個從來沒按下的鍵。 - still_held.append(key) - - for key in reversed(key_code_list): - release_list.append(release_keyboard_key(key, is_shift, skip_record=True)) - # 放開的順序與 `still_held` 的堆疊順序一致（都是反序），所以 `pop()` - # 拿到的必定就是剛放開的那一個——同一個鍵重複出現在清單裡也對。 - still_held.pop() + # Shift 在整組按鍵外面按住：`is_shift` 以前在 Windows／X11 完全沒作用。 + # Shift is held around the whole chord; outside macOS ``is_shift`` used + # to do nothing at all. + with _shift_held(is_shift): + for key in key_code_list: + press_list.append(press_keyboard_key(key, backend_shift, skip_record=True)) + # 按成功了才記——`press_keyboard_key` 丟例外時那個鍵並沒有被按下去， + # 記進來的話收尾會去放開一個從來沒按下的鍵。 + still_held.append(key) + + for key in reversed(key_code_list): + release_list.append(release_keyboard_key(key, backend_shift, skip_record=True)) + # 放開的順序與 `still_held` 的堆疊順序一致（都是反序），所以 `pop()` + # 拿到的必定就是剛放開的那一個——同一個鍵重複出現在清單裡也對。 + still_held.pop() press_str = ",".join(filter(None, press_list)) release_str = ",".join(filter(None, release_list)) @@ -385,7 +474,7 @@ def hotkey(key_code_list: list, is_shift: bool = False) -> Tuple[str, str]: # 每一次點選與按鍵都變成別的意思，而畫面上沒有任何跡象。 # 為什麼是 `finally` 不是 `except`：見 `type_keyboard` 的同名說明 # （`AutoControlKeyboardException` 不在那份 except 名單的任何一項底下）。 - _release_still_held(still_held, is_shift) + _release_still_held(still_held, backend_shift) def send_key_event_to_window(window_title: str, keycode: Union[int, str]) -> None: """ diff --git a/je_auto_control/wrapper/auto_control_mouse.py b/je_auto_control/wrapper/auto_control_mouse.py index f6f23e15a..613688343 100644 --- a/je_auto_control/wrapper/auto_control_mouse.py +++ b/je_auto_control/wrapper/auto_control_mouse.py @@ -30,6 +30,7 @@ """ import ctypes import math +import numbers import sys import time import warnings @@ -126,10 +127,15 @@ def _coordinate(value: object, axis: str) -> int: A non-numeric or infinite value raised ValueError / OverflowError, which callers catching AutoControlException missed; a value past int32 was truncated by ctypes and moved the cursor somewhere else while this - reported the requested point. + reported the requested point. A fractional value goes to the nearest + pixel: ``int()`` alone cut toward zero, so ``-0.6`` landed on 0 and + ``10.9`` on 10. """ try: - number = int(value) # type: ignore[call-overload] + if isinstance(value, numbers.Real) and not isinstance(value, numbers.Integral): + number = int(round(value)) + else: + number = int(value) # type: ignore[call-overload] except (TypeError, ValueError, OverflowError) as error: raise AutoControlMouseException(f"{axis} must be a number, got {value!r}") from error if not _INT32_MIN <= number <= _INT32_MAX: @@ -340,7 +346,15 @@ def _scroll_to(x: Optional[int], y: Optional[int]) -> None: Query the cursor only when a coordinate is missing: when both are supplied the current position is never needed, so backends that cannot report it (e.g. Wayland) must not be forced to raise. + + 座標先驗證再夾限：NaN 在 ``min``／``max`` 裡會被悄悄換成桌面邊緣， + 游標就移到那裡才滾。 + Validated before the clamp: ``min`` / ``max`` quietly turn a NaN into the + desktop edge, so the cursor went there and the scroll happened anyway, + where ``set_mouse_position`` refuses the same value. """ + x = None if x is None else _coordinate(x, "x") + y = None if y is None else _coordinate(y, "y") left, top, width, height = _scroll_bounds() # 兩個座標都給定時不會被讀到，見下面的三元運算。 # Never read when both coordinates were supplied. @@ -403,17 +417,20 @@ def _resolve_scroll_axis(scroll_direction: str) -> int: def mouse_scroll(scroll_value: int, x: Optional[int] = None, y: Optional[int] = None, - scroll_direction: str = "scroll_down" + scroll_direction: str = "scroll_up" ) -> Tuple[int, Union[int, str]]: """ 模擬滑鼠滾輪操作 Simulate mouse scroll - 每個平台的規則相同：``scroll_value`` 為負就反向，絕對值是滾動格數。 - The sign of ``scroll_value`` reverses the direction on every platform, so a - call written on one works on the others. X11 and Wayland used to discard it - and always scroll ``scroll_direction``, which meant portable code scrolled - the opposite way there with no error and no warning. + 每個平台的規則相同：正值往上、負值往下，絕對值是滾動格數。 + A positive ``scroll_value`` scrolls up and a negative one down on every + platform, so a call written on one works on the others. Two things used + to break that on X11 and Wayland: the sign was discarded, and then the + default ``scroll_direction`` was ``"scroll_down"``, so ``mouse_scroll(3)`` + went down there and up on Windows and macOS. The default is now + ``"scroll_up"``; pass ``scroll_direction="scroll_down"`` to keep the old + X11 / Wayland meaning of a positive count. :param scroll_value: 滾動數值，負數代表反向 Scroll value; negative reverses :param x: X 座標，指定時會先將游標移到該處 X position; the cursor moves here first diff --git a/test/unit_test/headless/test_input_wrapper_contract.py b/test/unit_test/headless/test_input_wrapper_contract.py new file mode 100644 index 000000000..3aafa84ea --- /dev/null +++ b/test/unit_test/headless/test_input_wrapper_contract.py @@ -0,0 +1,321 @@ +"""What the keyboard / mouse wrappers type, scroll and click, pinned on fakes. + +The 2026-09-24 audit reproduced each of these against a recording backend: +``write("Hi")`` typed ``hi`` on Windows and X11 (a capital shares its key with +the lower-case letter and nothing held Shift), ``is_shift`` did nothing outside +macOS, CR LF pressed Enter twice, ``mouse_scroll(3)`` went down on X11 and up +everywhere else, a NaN scroll point was clamped to the desktop edge instead of +refused, fractional coordinates were cut toward zero, Unicode typing sent line +breaks and Tab as code points, a dead-key Shift level was labelled with the +unshifted character, and an unnamed clipboard format was called ``"None"``. + +Nothing here reaches the real desktop: every backend is a recorder. +""" +import ctypes +import importlib +import inspect +import sys +import types + +import pytest + +from je_auto_control.utils.exception.exceptions import ( + AutoControlKeyboardException, AutoControlMouseException, +) +from je_auto_control.utils.keyboard_layout import keyboard_layout as kl +from je_auto_control.utils.text_unicode import text_unicode +from je_auto_control.wrapper import auto_control_keyboard as kb +from je_auto_control.wrapper import auto_control_mouse as ms + +# The package re-exports a function of the same name, which hides the module. +cf = importlib.import_module("je_auto_control.utils.clipboard_formats.clipboard_formats") + +SHIFT, RETURN, TAB = 16, 13, 9 +_TABLE = { + "shift": SHIFT, "return": RETURN, "tab": TAB, "control": 17, "space": 32, + "back": 8, + "a": 65, "A": 65, "h": 72, "H": 72, "i": 73, "I": 73, "s": 83, "S": 83, + "b": 66, "B": 66, +} + + +class _Keys: + """A keyboard backend that records key-down / key-up instead of typing.""" + + def __init__(self, fail_on=None, unicode=False): + self.events = [] + self._fail_on = fail_on + if unicode: + self.type_unicode_unit = lambda unit: self.events.append(("unit", unit)) + + def press_key(self, keycode): + if keycode == self._fail_on: + raise OSError("SendInput refused the key") + self.events.append(("down", keycode)) + + def release_key(self, keycode): + self.events.append(("up", keycode)) + + +def _tap(code): + return [("down", code), ("up", code)] + + +def _shifted(code): + return [("down", SHIFT), *_tap(code), ("up", SHIFT)] + + +def _use(monkeypatch, backend, platform="win32", table=None): + monkeypatch.setattr(sys, "platform", platform) + monkeypatch.setattr(kb, "keyboard", backend) + monkeypatch.setattr(kb, "keyboard_keys_table", dict(table or _TABLE)) + return backend + + +@pytest.fixture(params=["win32", "linux"]) +def keys(request, monkeypatch): + """The recording backend, once as Windows and once as X11.""" + return _use(monkeypatch, _Keys(), request.param) + + +# --- case and Shift --------------------------------------------------------- + +def test_write_holds_shift_for_a_capital_letter(keys): + assert kb.write("Hi") == "Hi" + assert keys.events == _shifted(72) + _tap(73) + + +def test_write_leaves_lower_case_and_digits_alone(keys): + kb.write("hi") + assert keys.events == _tap(72) + _tap(73) + + +def test_write_shifts_punctuation_only_when_it_shares_its_key(monkeypatch): + """X11 lists ``!`` under the key of ``1``; ``<`` may have a key of its own.""" + table = {"shift": SHIFT, "1": 10, "!": 10, ",": 59, "<": 94} + backend = _use(monkeypatch, _Keys(), "linux", table) + kb.write("1!<") + assert backend.events == _tap(10) + _shifted(10) + _tap(94) + + +def test_is_shift_holds_shift_for_one_key(keys): + kb.type_keyboard("a", is_shift=True) + assert keys.events == _shifted(65) + + +def test_is_shift_holds_shift_for_every_key_of_write(keys): + kb.write("ab", is_shift=True) + assert keys.events == _shifted(65) + _shifted(66) + + +def test_is_shift_wraps_the_whole_hotkey(keys): + kb.hotkey(["control", "s"], is_shift=True) + assert keys.events == [("down", SHIFT), ("down", 17), ("down", 83), + ("up", 83), ("up", 17), ("up", SHIFT)] + + +def test_a_lone_press_with_is_shift_never_leaves_shift_down(keys): + kb.press_keyboard_key("a", is_shift=True) + assert keys.events == [("down", SHIFT), ("down", 65), ("up", SHIFT)] + keys.events.clear() + kb.release_keyboard_key("a", is_shift=True) + assert keys.events == [("up", 65)] + + +def test_shift_is_released_when_the_key_fails(monkeypatch): + backend = _use(monkeypatch, _Keys(fail_on=65)) + with pytest.raises(AutoControlKeyboardException): + kb.write("A") + assert backend.events == [("down", SHIFT), ("up", SHIFT)] + + +def test_macos_still_gets_the_flag_and_no_extra_shift_key(monkeypatch): + events = [] + backend = types.SimpleNamespace( + press_key=lambda code, is_shift: events.append(("down", code, is_shift)), + release_key=lambda code, is_shift: events.append(("up", code, is_shift))) + _use(monkeypatch, backend, "darwin") + kb.type_keyboard("a", is_shift=True) + kb.write("A") + assert events == [("down", 65, True), ("up", 65, True), + ("down", 65, False), ("up", 65, False)] + + +# --- line endings ----------------------------------------------------------- + +@pytest.mark.parametrize("text", ["a\r\nb", "a\nb", "a\rb"]) +def test_one_line_break_is_one_enter(keys, text): + assert kb.write(text) == text + assert keys.events == _tap(65) + _tap(RETURN) + _tap(66) + + +def test_two_line_breaks_are_still_two(keys): + kb.write("\r\n\r\n") + assert keys.events == _tap(RETURN) * 2 + + +def test_write_secret_presses_line_breaks_as_keys(monkeypatch): + backend = _use(monkeypatch, _Keys(unicode=True)) + kb.write_secret("a\r\nb\t") + assert backend.events == [("unit", 97), *_tap(RETURN), ("unit", 98), *_tap(TAB)] + + +# --- Unicode typing --------------------------------------------------------- + +def test_unicode_plan_presses_control_whitespace_as_keys(): + assert text_unicode.plan_unicode_keys("a\nb\tc") == [ + {"op": "unicode_unit", "unit": 97}, {"op": "key", "key": "return"}, + {"op": "unicode_unit", "unit": 98}, {"op": "key", "key": "tab"}, + {"op": "unicode_unit", "unit": 99}] + assert text_unicode.plan_unicode_keys("\r\n") == [{"op": "key", "key": "return"}] + assert kb.WRITE_CONTROL_KEYS is text_unicode.CONTROL_KEYS + + +def test_the_default_sink_types_a_key_op(monkeypatch): + typed = [] + monkeypatch.setattr(kb, "type_keyboard", lambda key, *a, **k: typed.append(key)) + text_unicode._default_sink({"op": "key", "key": "return"}) + assert typed == ["return"] + + +def test_type_unicode_keys_reports_what_it_dispatched(): + events = [] + result = text_unicode.type_unicode_keys("a\n", sink=events.append) + assert [event["op"] for event in events] == ["unicode_unit", "key"] + assert result["ops"] == 2 and result["method"] == "keys" + + +# --- mouse ------------------------------------------------------------------ + +class _Mouse: + """A mouse backend that records moves and scrolls.""" + + def __init__(self): + self.moves = [] + self.scrolls = [] + + def set_position(self, x, y): + self.moves.append((x, y)) + + def position(self): + return 5, 5 + + def scroll(self, *args): + self.scrolls.append(args) + + +@pytest.fixture +def mouse(monkeypatch): + backend = _Mouse() + monkeypatch.setattr(ms, "mouse", backend) + monkeypatch.setattr(ms, "_scroll_bounds", lambda: (-1920, 0, 3840, 1080)) + return backend + + +def test_coordinates_are_rounded_not_cut_toward_zero(mouse): + assert ms.set_mouse_position(-0.6, 10.9) == (-1, 11) + assert mouse.moves == [(-1, 11)] + assert ms.set_mouse_position(7, "12") == (7, 12) + + +@pytest.mark.parametrize("point", [ + {"x": float("nan"), "y": 100}, {"x": 100, "y": float("nan")}, + {"x": float("inf"), "y": 100}, {"x": "left", "y": 100}, +]) +def test_a_scroll_point_that_is_not_a_number_moves_nothing(mouse, point): + with pytest.raises(AutoControlMouseException): + ms.mouse_scroll(3, **point) + assert mouse.moves == [] and mouse.scrolls == [] + + +def test_a_scroll_point_off_the_desktop_is_still_clamped(mouse, monkeypatch): + monkeypatch.setattr(sys, "platform", "win32") + ms.mouse_scroll(3, x=99999, y=100.4) + assert mouse.moves == [(1919, 100)] and mouse.scrolls == [(3,)] + + +def test_a_positive_scroll_goes_up_by_default_on_x11(mouse, monkeypatch): + up, down = 4, 5 + monkeypatch.setattr(sys, "platform", "linux") + monkeypatch.setattr(ms, "special_mouse_keys_table", + {"scroll_up": up, "scroll_down": down}) + assert ms.mouse_scroll(3) == (3, up) + assert ms.mouse_scroll(3, scroll_direction="scroll_down") == (3, down) + assert mouse.scrolls == [(3, up), (3, down)] + + +def test_every_scroll_entry_point_shares_the_default(): + from je_auto_control.utils.mcp_server import fake_backend + from je_auto_control.utils.mcp_server.tools import _handlers_input + for function in (ms.mouse_scroll, _handlers_input.mouse_scroll, + fake_backend._fake_mouse_scroll): + default = inspect.signature(function).parameters["scroll_direction"].default + assert default == "scroll_up", function + + +@pytest.mark.skipif(sys.platform != "win32", reason="the Windows key table") +@pytest.mark.parametrize("name", ["plus", "minus", "comma", "period", "oem_2"]) +def test_the_windows_table_has_the_punctuation_keys(name): + from je_auto_control.wrapper.platform_wrapper import keyboard_keys_table + assert isinstance(keyboard_keys_table[name], int) + + +# --- keyboard layout -------------------------------------------------------- + +def test_a_dead_shift_level_is_none_and_iso_keys_get_a_label(): + """Shift+6 on US-International is a dead key: no character, so ``None``.""" + chars = {(0x36, False): "6", (0x36, True): "", # dead ^ + (0xE2, False): "<", (0xE2, True): ">", # OEM_102 + (0xDF, False): "`", (0xDF, True): "¬", # OEM_8 (UK) + (0x41, False): "a", (0x41, True): "A"} + table = kl._build_table(lambda vk, shifted: chars.get((vk, shifted), "")) + assert table == {0x36: ("6", None), 0xE2: ("<", ">"), + 0xDF: ("`", "¬"), 0x41: ("a", "A")} + assert kl.vk_to_char(0x36, True, table) is None + assert kl.vk_to_char(0x36, False, table) == "6" + + +class _FakeUser32: + """``ToUnicodeEx`` for a layout whose Shift+6 is a dead key.""" + + def __init__(self): + self.ToUnicodeEx = self._to_unicode + self.MapVirtualKeyExW = lambda vk, kind, layout: vk + + @staticmethod + def _to_unicode(vk, _scan, state, buffer, _size, _flags, _layout): + shifted = state[0x10] != b"\x00" + if vk == 0x36 and shifted: + return -1 + if 0x30 <= vk <= 0x39 and not shifted: + buffer.value = chr(vk) + return 1 + return 0 + + +def test_the_layout_table_is_built_on_a_private_handle(monkeypatch): + fake = _FakeUser32() + monkeypatch.setattr(sys, "platform", "win32") + monkeypatch.setattr(kl, "_user32", lambda: fake) + monkeypatch.setattr(kl, "_LAYOUT_CACHE", {}) + table = kl.layout_char_table(0x04090409) + assert table[0x36] == ("6", None) and table[0x31] == ("1", None) + + +@pytest.mark.skipif(sys.platform != "win32", reason="ctypes.windll is Windows-only") +def test_the_shared_user32_keeps_its_prototypes(): + """Another caller's ``c_ubyte`` key-state array must still be accepted.""" + shared = ctypes.windll.user32 + assert kl._user32() is not shared + before = shared.ToUnicodeEx.argtypes + kl._translator(kl._user32(), 0x04090409) + assert shared.ToUnicodeEx.argtypes == before + + +# --- clipboard formats ------------------------------------------------------ + +def test_an_unnamed_format_is_the_same_in_every_form(): + for item in ((49161, None), [49161, None], {"id": 49161, "name": None}, 49161): + assert cf._coerce(item) == (49161, "") + diff = cf.diff_formats([(13, None)], [{"id": 13, "name": None}]) + assert diff == {"added": [], "removed": [], "changed": False} diff --git a/test/unit_test/headless/test_platform_backend_binding.py b/test/unit_test/headless/test_platform_backend_binding.py index ee49e8e5f..708b86b14 100644 --- a/test/unit_test/headless/test_platform_backend_binding.py +++ b/test/unit_test/headless/test_platform_backend_binding.py @@ -119,7 +119,7 @@ def _scroll_env(monkeypatch, platform, *, cursor=(7, 7)): # No virtual desktop reported: the clamp falls back to screen_size. monkeypatch.setattr(auto_control_mouse, "logical_virtual_rect", lambda: None) monkeypatch.setattr(auto_control_mouse, "special_mouse_keys_table", - {"scroll_down": 5}) + {"scroll_up": 4, "scroll_down": 5}) monkeypatch.setattr(auto_control_mouse, "get_mouse_position", lambda: cursor) monkeypatch.setattr(sys, "platform", platform) diff --git a/test/unit_test/headless/test_r3_platform_scroll_guard.py b/test/unit_test/headless/test_r3_platform_scroll_guard.py index 455f57fac..a8667bcc5 100644 --- a/test/unit_test/headless/test_r3_platform_scroll_guard.py +++ b/test/unit_test/headless/test_r3_platform_scroll_guard.py @@ -40,7 +40,7 @@ def _install_common(monkeypatch, get_position): monkeypatch.setattr(acm, "screen_size", lambda: (1920, 1080)) monkeypatch.setattr(acm, "mouse", fake_mouse) monkeypatch.setattr(acm, "special_mouse_keys_table", - {"scroll_down": 5}, raising=False) + {"scroll_up": 4, "scroll_down": 5}, raising=False) monkeypatch.setattr(sys, "platform", "linux") def _set_pos(x, y): diff --git a/test/unit_test/headless/test_wrapper_audit.py b/test/unit_test/headless/test_wrapper_audit.py index 5196be13b..1ba59681f 100644 --- a/test/unit_test/headless/test_wrapper_audit.py +++ b/test/unit_test/headless/test_wrapper_audit.py @@ -74,7 +74,7 @@ def mouse_moves(monkeypatch): scroll=lambda *args: moves.append(("scroll",) + args)) monkeypatch.setattr(auto_control_mouse, "mouse", backend) monkeypatch.setattr(auto_control_mouse, "record_action_to_list", lambda *a, **k: None) - monkeypatch.setattr(auto_control_mouse, "special_mouse_keys_table", {"scroll_down": 5}) + monkeypatch.setattr(auto_control_mouse, "special_mouse_keys_table", {"scroll_up": 4, "scroll_down": 5}) return moves diff --git a/test/unit_test/headless/test_wrapper_seam_contract.py b/test/unit_test/headless/test_wrapper_seam_contract.py index 358fea0bb..4ae13064b 100644 --- a/test/unit_test/headless/test_wrapper_seam_contract.py +++ b/test/unit_test/headless/test_wrapper_seam_contract.py @@ -169,12 +169,12 @@ def test_scrolling_survives_an_unreportable_cursor(mouse_env, monkeypatch): """ monkeypatch.setattr(sys, "platform", "linux") monkeypatch.setattr(auto_control_mouse, "special_mouse_keys_table", - {"scroll_down": 5}) + {"scroll_up": 4, "scroll_down": 5}) monkeypatch.setattr(auto_control_mouse, "get_mouse_position", lambda: None) auto_control_mouse.mouse_scroll(3, x=100) - assert mouse_env == [("scroll", 3, 5)] + assert mouse_env == [("scroll", 3, 4)] def test_scrolling_reads_no_axis_table_where_there_is_none(mouse_env, @@ -183,7 +183,7 @@ def test_scrolling_reads_no_axis_table_where_there_is_none(mouse_env, monkeypatch.setattr(sys, "platform", "win32") monkeypatch.setattr(auto_control_mouse, "special_mouse_keys_table", None) - assert auto_control_mouse.mouse_scroll(-2) == (-2, "scroll_down") + assert auto_control_mouse.mouse_scroll(-2) == (-2, "scroll_up") assert mouse_env == [("scroll", -2)] @@ -201,7 +201,7 @@ def test_a_single_axis_wheel_is_scrolled_on_every_name_that_has_one( monkeypatch.setattr(sys, "platform", platform) monkeypatch.setattr(auto_control_mouse, "special_mouse_keys_table", None) - assert auto_control_mouse.mouse_scroll(4) == (4, "scroll_down") + assert auto_control_mouse.mouse_scroll(4) == (4, "scroll_up") assert mouse_env == [("scroll", 4)] From 4511bfd874d908f34b5151bb3eae54cb9442dafc Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:02:41 +0800 Subject: [PATCH 17/26] Type the cross-platform names mypy 2.4 could not, and check with 2.4 in CI mypy 2.4 resolves the 'sys.platform not in [...]' guards against the target, so on a foreign target the rest of those modules is unreachable and their unannotated names have no type for importers. The contract passes on 2.3 and 2.4 for all three targets. --- .github/workflows/quality.yml | 2 +- je_auto_control/osx/core/utils/osx_vk.py | 12 +++++++++--- je_auto_control/osx/record/osx_record.py | 2 +- .../windows/core/utils/win32_ctype_input.py | 5 ++++- je_auto_control/windows/record/win32_record.py | 2 +- 5 files changed, 16 insertions(+), 7 deletions(-) diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml index c75423f23..b994e1950 100644 --- a/.github/workflows/quality.yml +++ b/.github/workflows/quality.yml @@ -199,7 +199,7 @@ jobs: with: python-version: "3.12" - run: pip install -e . # NOSONAR githubactions:S8541,githubactions:S8544 # reason: installs the checked-out project itself, there is no upstream version to lock and the build must run - - run: "pip install --only-binary :all: mypy==2.3.0" + - run: "pip install --only-binary :all: mypy==2.4.0" # Deliberately NOT installing the optional extras: the contract forces # every non-base third-party module to `Any` so the result cannot depend # on what is installed, and installing them here would only hide a diff --git a/je_auto_control/osx/core/utils/osx_vk.py b/je_auto_control/osx/core/utils/osx_vk.py index 116f0a6ed..275e9fa43 100644 --- a/je_auto_control/osx/core/utils/osx_vk.py +++ b/je_auto_control/osx/core/utils/osx_vk.py @@ -3,9 +3,6 @@ from je_auto_control.utils.exception.exception_tags import osx_import_error_message from je_auto_control.utils.exception.exceptions import AutoControlException -if sys.platform not in ["darwin"]: - raise AutoControlException(osx_import_error_message) - # osx keyboard virtual keycode osx_key_a = osx_key_A = 0x00 @@ -111,3 +108,12 @@ osx_mouse_left = "Left" osx_mouse_middle = "Middle" osx_mouse_right = "Right" + +# The platform check sits below the tables on purpose. mypy checks this module +# for every target; with the check on top, everything under it is unreachable +# on a non-darwin target and the names above get no type, which mypy 2.4 +# reports in each importer ("Cannot determine type"). The tables are plain +# integers, so defining them first costs nothing, and importing this module +# anywhere but macOS still fails. +if sys.platform not in ["darwin"]: + raise AutoControlException(osx_import_error_message) diff --git a/je_auto_control/osx/record/osx_record.py b/je_auto_control/osx/record/osx_record.py index 6c16441a6..3a93fb340 100644 --- a/je_auto_control/osx/record/osx_record.py +++ b/je_auto_control/osx/record/osx_record.py @@ -38,4 +38,4 @@ def new_hook(self) -> OSXInputTap: # === 全域 Recorder 實例 Global Recorder Instance === -osx_recorder = OSXRecorder() +osx_recorder: OSXRecorder = OSXRecorder() diff --git a/je_auto_control/windows/core/utils/win32_ctype_input.py b/je_auto_control/windows/core/utils/win32_ctype_input.py index 29472d23a..eb05804da 100644 --- a/je_auto_control/windows/core/utils/win32_ctype_input.py +++ b/je_auto_control/windows/core/utils/win32_ctype_input.py @@ -8,9 +8,12 @@ import ctypes from ctypes import wintypes +from typing import Any from je_auto_control.windows.core.utils.win32_vk import WIN32_EventF_UNICODE, WIN32_VkToVSC -user32 = ctypes.WinDLL('user32', use_last_error=True) # type: ignore[attr-defined] # reason: win32-only ctypes +# Annotated: on a non-Windows target mypy 2.4 treats this module as unreachable +# past the guard above and cannot infer a type for importers to use. +user32: Any = ctypes.WinDLL('user32', use_last_error=True) # type: ignore[attr-defined] # reason: win32-only ctypes Mouse: int = 0 Keyboard: int = 1 diff --git a/je_auto_control/windows/record/win32_record.py b/je_auto_control/windows/record/win32_record.py index ceb45e0d2..5bd7afdc6 100644 --- a/je_auto_control/windows/record/win32_record.py +++ b/je_auto_control/windows/record/win32_record.py @@ -38,4 +38,4 @@ def new_hook(self) -> Win32InputHook: # 全域錄製器實例 Global recorder instance -win32_recorder = Win32Recorder() +win32_recorder: Win32Recorder = Win32Recorder() From d898c4520a651b8b6bfdfba6245c4e2e401f9f02 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:02:51 +0800 Subject: [PATCH 18/26] Let an MCP server be confined to root directories and keep host-pushed files in the viewer's download directory An MCP tool's file argument reached any file the process could open, even in read-only mode: ac_load_dotenv returned any file as KEY=VALUE. ac_resolve_ref read any environment variable and any file. Both stay unrestricted unless an operator opts in, so existing deployments are unchanged: JE_AUTOCONTROL_MCP_PATH_ROOTS and JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT give the roots, JE_AUTOCONTROL_MCP_ENV_REF_ALLOW the env:// allowlist. The schema says which arguments are paths ("format": "path"), by meaning rather than by name, so ac_json_query's JSONPath is left alone. A remote desktop viewer wrote a host-pushed file wherever the host said. A viewer cannot vouch for the host it dialled, so dest_path is now relative to the viewer's download directory and anything leaving it fails the transfer. The host side keeps trusting its token holders. --- .../Eng/doc/mcp_server/mcp_server_doc.rst | 72 +++ .../Eng/doc/new_features/new_features_doc.rst | 45 +- .../Eng/doc/new_features/v85_features_doc.rst | 10 + .../Zh/doc/mcp_server/mcp_server_doc.rst | 60 ++ .../Zh/doc/new_features/new_features_doc.rst | 41 +- .../Zh/doc/new_features/v85_features_doc.rst | 7 + je_auto_control/__init__.py | 3 + .../gui/remote_desktop/viewer_panel.py | 3 +- .../utils/mcp_server/_argument_policy.py | 86 +++ .../utils/mcp_server/_client_requests.py | 24 +- je_auto_control/utils/mcp_server/server.py | 13 +- .../utils/mcp_server/tools/_factories.py | 277 ++++----- je_auto_control/utils/path_guard/__init__.py | 6 +- je_auto_control/utils/path_guard/policy.py | 113 ++++ .../utils/remote_desktop/__init__.py | 6 +- .../utils/remote_desktop/file_transfer.py | 95 ++- je_auto_control/utils/remote_desktop/host.py | 3 + .../utils/remote_desktop/viewer.py | 13 +- je_auto_control/utils/secret_ref/__init__.py | 7 +- .../utils/secret_ref/secret_ref.py | 82 ++- .../headless/test_file_boundary_policy.py | 546 ++++++++++++++++++ 21 files changed, 1329 insertions(+), 183 deletions(-) create mode 100644 je_auto_control/utils/mcp_server/_argument_policy.py create mode 100644 je_auto_control/utils/path_guard/policy.py create mode 100644 test/unit_test/headless/test_file_boundary_policy.py diff --git a/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst b/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst index bf0622cda..ee4fe398b 100644 --- a/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst +++ b/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst @@ -437,6 +437,78 @@ clipboard reads, history, ...) survive: } } +Confining file arguments to root directories +============================================ + +Read-only mode limits *which tools* exist, not *which files* they open: +``ac_load_dotenv``, ``ac_read_document`` or ``ac_extract_pdf_text`` will read +any file the server process can. To bound that, give the server roots. It is +off by default — a server with neither variable set behaves exactly as +before, read-only mode included. + +``JE_AUTOCONTROL_MCP_PATH_ROOTS`` + Directories separated by ``os.pathsep`` (``;`` on Windows, ``:`` + elsewhere). Setting it turns the check on. + +``JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT`` + ``1`` / ``true`` / ``yes`` / ``on`` to also accept the directories the + client reports through ``roots/list``, in addition to the variable + above. It turns the check on by itself too; until the client has + answered, every file argument is refused rather than let through. Only + use it with a client you trust to describe the workspace — over HTTP any + caller that can reach the server can report roots. + +.. code-block:: json + + { + "mcpServers": { + "autocontrol_safe": { + "command": "python", + "args": ["-m", "je_auto_control.utils.mcp_server"], + "env": {"JE_AUTOCONTROL_MCP_READONLY": "1", + "JE_AUTOCONTROL_MCP_PATH_ROOTS": "C:/work/project"} + } + } + } + +With roots in force, every tool argument whose schema says +``"format": "path"`` is resolved with ``os.path.realpath`` and must land +inside a root. ``..``, a symlink or junction leading out, another drive and a +UNC share are all judged by where they really lead; a path starting with +``~`` has to be inside the roots both expanded and taken literally. A +refusal is a tool execution error (``isError: true``, ``Invalid arguments +for : ...``), like any other bad argument. The tool then receives the +canonical absolute path that was checked, so a relative path is relative to +the server's working directory. + +The annotation follows meaning, not the property's name: ``ac_json_query``'s +``path`` is a JSONPath expression and is left alone. What it does **not** +reach: + +* ``ac_execute_actions`` and the other tools that run an action list — an + action can open any file, which is why they are not read-only tools. +* Arguments that are a path only sometimes: ``target`` of ``ac_open_path`` / + ``ac_plan_open`` / ``ac_file_association`` (path or URL or extension) and of + ``ac_act_in_view`` (template path or text), ``ac_handle_file_dialog``'s + ``path`` (keystrokes typed into another application), ``argv`` of + ``ac_launch_process`` / ``ac_shell``, and paths inside free-form objects + (``ac_run_suite`` ``spec``, ``ac_run_dag`` ``definition``, + ``ac_assert_all`` ``specs``). +* Tools registered by plugins, unless their schema carries the annotation. + +``ac_resolve_ref`` / ``ac_resolve_refs`` follow the same roots for +``file://`` references, and have a switch of their own for ``env://``: + +``JE_AUTOCONTROL_MCP_ENV_REF_ALLOW`` + Comma-separated variable names, ``fnmatch`` patterns allowed + (``APP_*,HOME``). When set, ``env://NAME`` resolves only for a matching + name and anything else is a tool execution error. Unset, every variable + is readable, as before — including the ones that hold API keys. + +Programmatically, the same policy is ``server.argument_policy`` +(:class:`ArgumentPolicy` holding a :class:`je_auto_control.PathPolicy` and +the allowlist); assign another to a server you build yourself. + Confirmation prompts (elicitation) ================================== diff --git a/docs/source/Eng/doc/new_features/new_features_doc.rst b/docs/source/Eng/doc/new_features/new_features_doc.rst index 08cd73f77..83dcf7367 100644 --- a/docs/source/Eng/doc/new_features/new_features_doc.rst +++ b/docs/source/Eng/doc/new_features/new_features_doc.rst @@ -878,9 +878,24 @@ Three new message types form one transfer: * ``FILE_END`` — JSON ``{transfer_id, status, error?}`` Transfers are bidirectional, chunked (256 KiB per chunk), and have -*no aggregate size limit* and *no path restriction* on the -destination — token holders are trusted users. Progress is reported -locally on both sides without an extra wire message:: +*no aggregate size limit*. The two directions treat ``dest_path`` +differently, because the trust runs one way: + +* **Viewer → host**: ``dest_path`` is a path on the host, used as given. + The viewer holds the token, and token holders are trusted users. +* **Host → viewer**: ``dest_path`` is a path *relative to the viewer's + download directory*. A viewer cannot vouch for the host it dialled, so + an absolute path, a drive or UNC path, a ``..`` component or a symlink + leading out of that directory fails the transfer (``on_complete`` gets + ``ok=False``) and nothing is written. + +The download directory is ``~/Downloads/AutoControl`` unless +``JE_AUTOCONTROL_REMOTE_DOWNLOAD_DIR`` names another; it is created when +the first file arrives. To choose it per viewer, pass +``FileReceiver(base_dir=...)`` to ``set_file_receiver``. A receiver built +without ``base_dir`` is unconfined — that is what the host uses. + +Progress is reported locally on both sides without an extra wire message:: from je_auto_control.utils.remote_desktop import ( FileReceiver, RemoteDesktopHost, RemoteDesktopViewer, send_file, @@ -890,11 +905,13 @@ locally on both sides without an extra wire message:: viewer.send_file("local.bin", "/tmp/uploaded.bin", on_progress=lambda tid, done, total: print(done, total)) - # Host pushes to all viewers (each viewer needs a FileReceiver) + # Host pushes to all viewers; each one stores it under its own + # download directory, here /from_host/local.bin viewer.set_file_receiver(FileReceiver( on_progress=..., on_complete=..., + base_dir="~/Downloads/AutoControl", )) - host.send_file_to_viewers("local.bin", "/tmp/from_host.bin") + host.send_file_to_viewers("local.bin", "from_host/local.bin") GUI: *Send file...* opens a file picker + destination-path prompt and runs the upload on a ``QThread`` with a ``QProgressBar`` bound to the @@ -903,11 +920,19 @@ dragEnter / drop of local files; each dropped file kicks off the same upload flow. .. warning:: - Path is unrestricted and there is no size cap. Anyone with the - token can write any file to any location, and can fill the disk. - Keep ``trusted token holders == trusted users`` in mind, or wrap - the headless API in your own restricted ``FileReceiver`` subclass - that vets the destination path. + On the host, the path is unrestricted and there is no size cap. + Anyone with the token can write any file to any location on the + host, and can fill the disk. Keep ``trusted token holders == trusted + users`` in mind, or give the host a confined receiver with + ``host.set_file_receiver(FileReceiver(base_dir=...))``. A viewer is + confined to its download directory, but a host can still fill that + disk. + +.. note:: + Before this change a host-pushed ``dest_path`` was an absolute path + on the viewer. Scripts that call + ``host.send_file_to_viewers(src, "/tmp/x.bin")`` now fail on the + viewer; send a relative path instead. Remote desktop — AnyDesk-style popout window diff --git a/docs/source/Eng/doc/new_features/v85_features_doc.rst b/docs/source/Eng/doc/new_features/v85_features_doc.rst index fbedfae32..36aefe55b 100644 --- a/docs/source/Eng/doc/new_features/v85_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v85_features_doc.rst @@ -47,3 +47,13 @@ Their results are recorded (the executor record, the MCP result), so they refuse ``secret://`` with ``SecretRefError``: secret values never enter those records. Reference ``${secrets.NAME}`` in the step that needs the value, or call ``resolve_ref`` from Python. + +``RefResolver`` takes two optional restrictions, both off unless given: +``env_allowlist`` (the only variable names ``env://`` may read, ``fnmatch`` +patterns allowed) and ``path_policy`` (a ``PathPolicy`` whose roots ``file://`` +must stay inside). ``RefResolver.check_all(obj)`` applies them to a structure +without reading anything. The MCP server builds both from +``JE_AUTOCONTROL_MCP_ENV_REF_ALLOW`` and ``JE_AUTOCONTROL_MCP_PATH_ROOTS`` +for ``ac_resolve_ref`` / ``ac_resolve_refs``; see the MCP server page. The +executor commands are not restricted — an action file can already read any +variable or file. diff --git a/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst b/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst index 9ca641b7f..8f71ec8a0 100644 --- a/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst +++ b/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst @@ -379,6 +379,66 @@ client 不用改,可以和 2026-07-28 的 client 並存。 } } +把檔案參數限制在根目錄內 +======================== + +唯讀模式限制的是「有哪些工具」，不是「工具能開哪些檔案」：``ac_load_dotenv``、 +``ac_read_document``、``ac_extract_pdf_text`` 仍能讀伺服器行程讀得到的任何檔案。要限制這一點， +請給伺服器根目錄。預設關閉——兩個變數都沒設的伺服器行為與以前完全相同，唯讀模式也一樣。 + +``JE_AUTOCONTROL_MCP_PATH_ROOTS`` + 以 ``os.pathsep`` 分隔的目錄（Windows 是 ``;``，其他平台是 ``:``）。設了就啟用檢查。 + +``JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT`` + 設為 ``1`` / ``true`` / ``yes`` / ``on`` 時，除了上面那個變數之外，也接受用戶端透過 + ``roots/list`` 回報的目錄。單獨設它也會啟用檢查；在用戶端回覆之前，所有檔案參數都會被拒絕， + 而不是先放行。只在你信任用戶端描述的工作區時使用——走 HTTP 時，任何連得到伺服器的呼叫端 + 都能回報根目錄。 + +.. code-block:: json + + { + "mcpServers": { + "autocontrol_safe": { + "command": "python", + "args": ["-m", "je_auto_control.utils.mcp_server"], + "env": {"JE_AUTOCONTROL_MCP_READONLY": "1", + "JE_AUTOCONTROL_MCP_PATH_ROOTS": "C:/work/project"} + } + } + } + +根目錄生效後，schema 標了 ``"format": "path"`` 的工具參數都會先經過 ``os.path.realpath``， +結果必須落在某個根目錄內。``..``、指向外面的 symlink 或 junction、其他磁碟機、UNC 分享， +都以實際指向的位置判斷；以 ``~`` 開頭的路徑，展開與不展開兩種讀法都必須在根目錄內。 +被拒絕時回的是工具執行錯誤（``isError: true``，``Invalid arguments for : ...``）， +和其他參數錯誤一樣。通過檢查後，工具收到的是檢查過的那個絕對路徑，所以相對路徑是相對於 +伺服器的工作目錄。 + +這個標記看的是語意，不是屬性名稱：``ac_json_query`` 的 ``path`` 是 JSONPath，不受影響。 +它**涵蓋不到**的地方： + +* ``ac_execute_actions`` 與其他執行動作清單的工具——動作可以開任何檔案，這也是它們不屬於 + 唯讀工具的原因。 +* 只有某些情況下才是路徑的參數：``ac_open_path`` / ``ac_plan_open`` / + ``ac_file_association`` 的 ``target``（路徑、URL 或副檔名）、``ac_act_in_view`` 的 + ``target``（樣板路徑或文字）、``ac_handle_file_dialog`` 的 ``path``（打進別的應用程式的 + 按鍵）、``ac_launch_process`` / ``ac_shell`` 的 ``argv``，以及自由格式物件裡的路徑 + （``ac_run_suite`` 的 ``spec``、``ac_run_dag`` 的 ``definition``、``ac_assert_all`` 的 + ``specs``）。 +* 外掛註冊的工具，除非它的 schema 也帶這個標記。 + +``ac_resolve_ref`` / ``ac_resolve_refs`` 的 ``file://`` 參照套用同一組根目錄；``env://`` +另有自己的開關： + +``JE_AUTOCONTROL_MCP_ENV_REF_ALLOW`` + 以逗號分隔的變數名稱，可用 ``fnmatch`` 樣式（``APP_*,HOME``）。設了之後，``env://NAME`` + 只有名稱符合時才會解析，其餘回工具執行錯誤。沒設時和以前一樣，任何變數都讀得到—— + 包括放 API 金鑰的那些。 + +在程式裡，同一份設定是 ``server.argument_policy``（:class:`ArgumentPolicy`，內含 +:class:`je_auto_control.PathPolicy` 與允許清單）；自己建立的伺服器可以指派另一個。 + 破壞性動作確認(Elicitation) ============================= diff --git a/docs/source/Zh/doc/new_features/new_features_doc.rst b/docs/source/Zh/doc/new_features/new_features_doc.rst index 0961cce4a..9010b1180 100644 --- a/docs/source/Zh/doc/new_features/new_features_doc.rst +++ b/docs/source/Zh/doc/new_features/new_features_doc.rst @@ -822,9 +822,23 @@ GUI：Viewer 分頁有 *把本機剪貼簿文字送到 Host* 按鈕；host 收 * ``FILE_CHUNK`` — 36-byte ASCII transfer id + 原始 payload * ``FILE_END`` — JSON ``{transfer_id, status, error?}`` -雙向、分塊（256 KiB / chunk）、**沒有總大小上限**、**沒有目的路徑 -限制**（拿到 token 就視為信任使用者）。進度由兩端各自本地計算，不 -需要額外的 wire 訊息:: +雙向、分塊（256 KiB / chunk）、**沒有總大小上限**。兩個方向對 +``dest_path`` 的處理不同，因為信任是單向的： + +* **Viewer → host**：``dest_path`` 是 host 上的路徑，照單全收。Viewer + 持有 token，拿到 token 就視為信任使用者。 +* **Host → viewer**：``dest_path`` 是\ **相對於 viewer 下載目錄**\ 的 + 路徑。Viewer 無法替它連上的 host 背書，所以絕對路徑、磁碟機或 UNC + 路徑、``..``、指向目錄外的 symlink 都會讓傳輸失敗（``on_complete`` + 收到 ``ok=False``），不會寫入任何東西。 + +下載目錄預設是 ``~/Downloads/AutoControl``，可用 +``JE_AUTOCONTROL_REMOTE_DOWNLOAD_DIR`` 指定別的；第一個檔案到達時才 +建立。要個別指定，請把 ``FileReceiver(base_dir=...)`` 交給 +``set_file_receiver``。沒有 ``base_dir`` 的 receiver 不受限制——host +用的就是這種。 + +進度由兩端各自本地計算，不需要額外的 wire 訊息:: from je_auto_control.utils.remote_desktop import ( FileReceiver, RemoteDesktopHost, RemoteDesktopViewer, send_file, @@ -834,11 +848,13 @@ GUI：Viewer 分頁有 *把本機剪貼簿文字送到 Host* 按鈕；host 收 viewer.send_file("local.bin", "/tmp/uploaded.bin", on_progress=lambda tid, done, total: print(done, total)) - # Host 下發到所有 viewer（viewer 需要設一個 FileReceiver 來收） + # Host 下發到所有 viewer；每個 viewer 存進自己的下載目錄， + # 這裡是 <下載目錄>/from_host/local.bin viewer.set_file_receiver(FileReceiver( on_progress=..., on_complete=..., + base_dir="~/Downloads/AutoControl", )) - host.send_file_to_viewers("local.bin", "/tmp/from_host.bin") + host.send_file_to_viewers("local.bin", "from_host/local.bin") GUI：*傳送檔案...* 按鈕開啟檔案選擇器 + 目的路徑提示，上傳跑在 ``QThread`` 上，底下 ``QProgressBar`` 綁到 sender 的 progress 事 @@ -846,10 +862,17 @@ GUI：*傳送檔案...* 按鈕開啟檔案選擇器 + 目的路徑提示，上 去就走同一個流程上傳。 .. warning:: - 路徑無限制、大小無上限。任何拿到 token 的人都能把任意檔案寫到 - 任意位置（覆蓋 ``C:\\Windows\\System32\\*.dll`` 都可能），也能 - 塞滿磁碟。Token 持有者必須等同信任使用者；要更嚴格的話請自行 - 繼承 ``FileReceiver`` 在 ``handle_begin`` 內驗證 dest_path。 + 在 host 端，路徑無限制、大小無上限。任何拿到 token 的人都能把任意 + 檔案寫到 host 的任意位置（覆蓋 ``C:\\Windows\\System32\\*.dll`` + 都可能），也能塞滿磁碟。Token 持有者必須等同信任使用者；要更嚴格 + 的話請用 ``host.set_file_receiver(FileReceiver(base_dir=...))`` 給 + host 一個受限的 receiver。Viewer 被限制在下載目錄內，但 host 仍然 + 可以把那顆磁碟塞滿。 + +.. note:: + 在這次變更之前，host 推來的 ``dest_path`` 是 viewer 上的絕對路徑。 + 呼叫 ``host.send_file_to_viewers(src, "/tmp/x.bin")`` 的腳本現在會 + 在 viewer 端失敗，請改送相對路徑。 遠端桌面 — AnyDesk 風格彈出視窗 diff --git a/docs/source/Zh/doc/new_features/v85_features_doc.rst b/docs/source/Zh/doc/new_features/v85_features_doc.rst index c9c547c6a..84588f972 100644 --- a/docs/source/Zh/doc/new_features/v85_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v85_features_doc.rst @@ -37,3 +37,10 @@ governance 憑證 broker。``resolve_refs_in`` 走訪巢狀 dict/list 並就地 **Security** 分類下的命令提供。它們的結果會被記錄（executor 紀錄、MCP 回應），所以遇到 ``secret://`` 會丟 ``SecretRefError``：祕密的值不進這些紀錄。需要值的步驟請引用 ``${secrets.NAME}``，或在 Python 裡呼叫 ``resolve_ref``。 + +``RefResolver`` 有兩個選用的限制，沒給就不生效：``env_allowlist``（``env://`` 只能讀的變數名稱， +可用 ``fnmatch`` 樣式）與 ``path_policy``（``PathPolicy``，``file://`` 必須留在它的根目錄內）。 +``RefResolver.check_all(obj)`` 只檢查一個結構、不讀取任何東西。MCP 伺服器依 +``JE_AUTOCONTROL_MCP_ENV_REF_ALLOW`` 與 ``JE_AUTOCONTROL_MCP_PATH_ROOTS`` 為 +``ac_resolve_ref`` / ``ac_resolve_refs`` 建立這兩個限制，詳見 MCP 伺服器頁面。executor 指令不受 +限制——動作檔本來就能讀任何變數或檔案。 diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 0117874ed..188846a9f 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -741,6 +741,8 @@ from je_auto_control.utils.config_schema import ( ConfigField, ConfigSchema, coerce, validate_config, ) +# Opt-in confinement of file paths to configured root directories +from je_auto_control.utils.path_guard import PathNotAllowedError, PathPolicy # URI-scheme secret/value reference resolver (env:// / file:// / secret://) from je_auto_control.utils.secret_ref import ( RefResolver, SecretRefError, is_ref, resolve_ref, resolve_refs_in, @@ -1674,6 +1676,7 @@ def start_autocontrol_gui(*args, **kwargs): "dotenv_values", "dump_dotenv", "load_dotenv", "parse_dotenv", "LayeredConfig", "SourceTrace", "deep_merge", "ConfigField", "ConfigSchema", "coerce", "validate_config", + "PathNotAllowedError", "PathPolicy", "RefResolver", "SecretRefError", "is_ref", "resolve_ref", "resolve_refs_in", "redact_config", "redact_secret_text", "EventEmitter", "post_cloudevent", "to_cloudevent", diff --git a/je_auto_control/gui/remote_desktop/viewer_panel.py b/je_auto_control/gui/remote_desktop/viewer_panel.py index 78b30ce33..c7a9e84a9 100644 --- a/je_auto_control/gui/remote_desktop/viewer_panel.py +++ b/je_auto_control/gui/remote_desktop/viewer_panel.py @@ -22,7 +22,7 @@ ) from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.remote_desktop import ( - FileReceiver, RemoteDesktopViewer, WebSocketDesktopViewer, + FileReceiver, RemoteDesktopViewer, WebSocketDesktopViewer, default_download_dir, ) from je_auto_control.utils.remote_desktop.audio import ( AudioPlayer, is_audio_backend_available, @@ -239,6 +239,7 @@ def _connect(self) -> None: self._file_complete_signal.emit( tid, bool(ok), err or "", dst, ), + base_dir=default_download_dir(), )) viewer.connect(timeout=5.0) # ValueError: a host such as "a..b" fails IDNA encoding with diff --git a/je_auto_control/utils/mcp_server/_argument_policy.py b/je_auto_control/utils/mcp_server/_argument_policy.py new file mode 100644 index 000000000..d6d29d44b --- /dev/null +++ b/je_auto_control/utils/mcp_server/_argument_policy.py @@ -0,0 +1,86 @@ +"""Opt-in limits on what a tool call's arguments may name on this machine. + +Two schema annotations drive it, so the rule lives next to the argument it +governs instead of in a list of tool names: + +* ``"format": "path"`` — the string is a file or directory the tool will open. + With roots configured it must resolve inside one of them, and the tool is + handed the canonical path that was checked, so ``~`` and a relative path + cannot mean one thing to the check and another to the handler. +* ``"format": "value-ref"`` — the value holds ``env://`` / ``file://`` + references (``ac_resolve_ref`` / ``ac_resolve_refs``). ``file://`` targets + follow the same roots; ``env://`` names follow the allowlist. + +Nothing is restricted until an operator configures it: see +:class:`~je_auto_control.utils.path_guard.policy.PathPolicy` for the roots and +:func:`~je_auto_control.utils.secret_ref.secret_ref.env_allowlist_from_env`. +""" +from typing import Any, Dict, Mapping, Optional, Tuple + +from je_auto_control.utils.path_guard.policy import PathPolicy +from je_auto_control.utils.secret_ref.secret_ref import ( + RefResolver, env_allowlist_from_env, +) + +PATH_FORMAT = "path" +VALUE_REF_FORMAT = "value-ref" + + +class ArgumentPolicy: + """The path roots and ``env://`` allowlist one MCP server enforces.""" + + def __init__(self, path_policy: Optional[PathPolicy] = None, + env_allowlist: Optional[Tuple[str, ...]] = None) -> None: + self.path_policy = path_policy if path_policy is not None else PathPolicy() + self.env_allowlist = env_allowlist + + @classmethod + def from_env(cls, environ: Optional[Mapping[str, str]] = None) -> "ArgumentPolicy": + """Build the policy the process environment describes (default: none).""" + return cls(PathPolicy.from_env(environ), env_allowlist_from_env(environ)) + + @property + def enabled(self) -> bool: + """Whether :meth:`apply` can refuse or rewrite anything.""" + return self.path_policy.enabled or self.env_allowlist is not None + + def apply(self, tool_name: str, schema: Dict[str, Any], + arguments: Dict[str, Any]) -> Dict[str, Any]: + """Return ``arguments`` with every path canonicalised, or raise. + + Raises :class:`~je_auto_control.utils.path_guard.PathNotAllowedError` + for a path outside the roots and + :class:`~je_auto_control.utils.secret_ref.SecretRefError` for a + refused reference — both :class:`AutoControlException`. + """ + if not self.enabled: + return arguments + return self._walk(schema, arguments, f"{tool_name} $") + + def _walk(self, schema: Any, value: Any, where: str) -> Any: + if not isinstance(schema, dict): + return value + marker = schema.get("format") + if marker == VALUE_REF_FORMAT: + RefResolver(env_allowlist=self.env_allowlist, + path_policy=self.path_policy).check_all(value) + return value + if marker == PATH_FORMAT and isinstance(value, str): + return self._confine(value, where) + if isinstance(value, dict): + declared = schema.get("properties") or {} + extra = schema.get("additionalProperties") + return {key: self._walk(declared.get(key, extra), item, f"{where}.{key}") + for key, item in value.items()} + if isinstance(value, list): + return [self._walk(schema.get("items"), item, f"{where}[{index}]") + for index, item in enumerate(value)] + return value + + def _confine(self, value: str, where: str) -> str: + if not value or not self.path_policy.enabled: + return value # an empty path is the handler's "not given" + return str(self.path_policy.validate(value, operation=where)) + + +__all__ = ["ArgumentPolicy", "PATH_FORMAT", "VALUE_REF_FORMAT"] diff --git a/je_auto_control/utils/mcp_server/_client_requests.py b/je_auto_control/utils/mcp_server/_client_requests.py index 27d989a7d..30f71467b 100644 --- a/je_auto_control/utils/mcp_server/_client_requests.py +++ b/je_auto_control/utils/mcp_server/_client_requests.py @@ -36,6 +36,17 @@ def needs_confirmation(tool: MCPTool) -> bool: return not annotations.read_only and bool(annotations.destructive) +def _root_paths(roots_list: List[Any]) -> List[str]: + """Local directories named by a ``roots/list`` reply; other entries are skipped.""" + paths: List[str] = [] + for entry in roots_list: + uri = entry.get("uri") if isinstance(entry, dict) else None + local_path = _file_uri_to_path(uri) if isinstance(uri, str) else None + if local_path: + paths.append(local_path) + return paths + + class ClientRequestMixin: """Outbound half of the MCP session, mixed into :class:`MCPServer`. @@ -51,6 +62,7 @@ class ClientRequestMixin: _writer: Optional[Callable[[str], None]] _client_capabilities: Dict[str, Any] _resources: Any + argument_policy: Any _outbound_lock: threading.Lock _pending_outbound: Dict[Any, Dict[str, Any]] _outbound_id_counter: "itertools.count[int]" @@ -113,12 +125,20 @@ def _refresh_roots_safely(self) -> None: autocontrol_logger.info("MCP roots refresh skipped: %r", error) def refresh_roots(self, timeout: float = 10.0) -> List[Dict[str, Any]]: - """Send ``roots/list`` to the client and apply the first root.""" + """Send ``roots/list`` to the client and apply what it reports. + + The first root becomes the resource provider's workspace. Every root + is offered to the tool-argument path policy, which keeps them only + when ``JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT`` opted in. + """ result = self._send_outbound_request( "roots/list", params={}, timeout=timeout, ) roots_list = (result or {}).get("roots") or [] - if not isinstance(roots_list, list) or not roots_list: + if not isinstance(roots_list, list): + roots_list = [] + self.argument_policy.path_policy.set_client_roots(_root_paths(roots_list)) + if not roots_list: return [] first_uri = roots_list[0].get("uri") if isinstance(roots_list[0], dict) else None diff --git a/je_auto_control/utils/mcp_server/server.py b/je_auto_control/utils/mcp_server/server.py index 15b510fbc..623970515 100644 --- a/je_auto_control/utils/mcp_server/server.py +++ b/je_auto_control/utils/mcp_server/server.py @@ -15,6 +15,7 @@ from typing import Any, Callable, Dict, List, Optional, TextIO from je_auto_control.utils.cli_output import utf8_stream +from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.mcp_server.audit import AuditLogger from je_auto_control.utils.mcp_server.context import ( @@ -36,6 +37,7 @@ from je_auto_control.utils.mcp_server.tools._validation import ( undeclared_arguments, validate_arguments, ) +from je_auto_control.utils.mcp_server._argument_policy import ArgumentPolicy from je_auto_control.utils.mcp_server._client_requests import ( ClientRequestMixin, ) @@ -81,6 +83,9 @@ def __init__(self, tools: Optional[List[MCPTool]] = None, else AuditLogger()) self._rate_limiter = rate_limiter self._log_bridge = log_bridge + #: Path roots and env:// allowlist for tool arguments; off unless the + #: environment configures them. Assign another to change it. + self.argument_policy = ArgumentPolicy.from_env() self._stop = threading.Event() self._initialized = False self._peer_era: Optional[str] = None # the stdio peer's; see _note_peer_era @@ -604,7 +609,9 @@ def _prepare_tool_call( Raises :class:`_MCPError` when the request is malformed, the tool is unknown or the rate limit is hit, and :class:`_InvalidToolArguments` - when the arguments fail the tool's schema. + when the arguments fail the tool's schema or name a path or value + reference the configured :attr:`argument_policy` refuses. The + arguments returned carry the canonical form of every checked path. """ name = params.get("name") arguments = params.get("arguments") or {} @@ -619,6 +626,10 @@ def _prepare_tool_call( or undeclared_arguments(tool.input_schema, arguments)) if violation is not None: raise _InvalidToolArguments(f"Invalid arguments for {name}: {violation}") + try: + arguments = self.argument_policy.apply(name, tool.input_schema, arguments) + except AutoControlException as error: + raise _InvalidToolArguments(f"Invalid arguments for {name}: {error}") from error if self._rate_limiter is not None and not self._rate_limiter.try_acquire(): raise _MCPError(-32000, f"Rate limit exceeded for tool {name!r}") self._maybe_confirm_destructive(name, tool, arguments) diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index 55629ec6c..fecbc25ae 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -136,7 +136,7 @@ def screen_tools() -> List[MCPTool]: "(0 = virtual desktop spanning all, 1+ = single " "screens — see ac_list_monitors)."), input_schema=schema({ - "file_path": {"type": "string"}, + "file_path": {"type": "string", "format": "path"}, "screen_region": {"type": "array", "items": {"type": "integer"}}, "monitor_index": {"type": "integer"}, @@ -170,7 +170,7 @@ def screen_tools() -> List[MCPTool]: "``timeout`` seconds. Cancellable: clients can " "send notifications/cancelled to abort."), input_schema=schema({ - "image_path": {"type": "string"}, + "image_path": {"type": "string", "format": "path"}, "timeout": {"type": "number"}, "poll": {"type": "number"}, "detect_threshold": {"type": "number"}, @@ -204,8 +204,8 @@ def screen_tools() -> List[MCPTool]: "equal; components smaller than min_box_pixels " "are ignored to filter antialias noise."), input_schema=schema({ - "image_path_a": {"type": "string"}, - "image_path_b": {"type": "string"}, + "image_path_a": {"type": "string", "format": "path"}, + "image_path_b": {"type": "string", "format": "path"}, "threshold": {"type": "integer"}, "min_box_pixels": {"type": "integer"}, }, required=["image_path_a", "image_path_b"]), @@ -222,7 +222,7 @@ def image_and_ocr_tools() -> List[MCPTool]: description=("Find a template image on screen and return its " "centre [x, y]. detect_threshold is 0.0–1.0."), input_schema=schema({ - "image_path": {"type": "string"}, + "image_path": {"type": "string", "format": "path"}, "detect_threshold": {"type": "number"}, }, required=["image_path"]), handler=h_screen.locate_image_center, @@ -232,7 +232,7 @@ def image_and_ocr_tools() -> List[MCPTool]: name="ac_locate_and_click", description="Find a template image and click its centre.", input_schema=schema({ - "image_path": {"type": "string"}, + "image_path": {"type": "string", "format": "path"}, "mouse_keycode": {"type": "string"}, "detect_threshold": {"type": "number"}, }, required=["image_path"]), @@ -510,7 +510,7 @@ def system_tools() -> List[MCPTool]: description=("Place a Pillow-readable image file on the " "clipboard. Windows-only today; macOS / Linux " "raise NotImplementedError."), - input_schema=schema({"image_path": {"type": "string"}}, + input_schema=schema({"image_path": {"type": "string", "format": "path"}}, required=["image_path"]), handler=h_system.set_clipboard_image, annotations=DESTRUCTIVE, @@ -530,7 +530,7 @@ def system_tools() -> List[MCPTool]: MCPTool( name="ac_execute_action_file", description="Load a JSON action file from disk and execute it.", - input_schema=schema({"file_path": {"type": "string"}}, + input_schema=schema({"file_path": {"type": "string", "format": "path"}}, required=["file_path"]), handler=h_runs.execute_action_file, annotations=DESTRUCTIVE, @@ -565,7 +565,7 @@ def screen_record_tools() -> List[MCPTool]: "Codec defaults to XVID (.avi); use MP4V for .mp4."), input_schema=schema({ "recorder_name": {"type": "string"}, - "file_path": {"type": "string"}, + "file_path": {"type": "string", "format": "path"}, "codec": {"type": "string"}, "frame_per_sec": {"type": "integer"}, "width": {"type": "integer"}, @@ -629,7 +629,7 @@ def recording_tools() -> List[MCPTool]: MCPTool( name="ac_read_action_file", description="Read a JSON action file from disk and return its parsed contents.", - input_schema=schema({"file_path": {"type": "string"}}, + input_schema=schema({"file_path": {"type": "string", "format": "path"}}, required=["file_path"]), handler=h_runs.read_action_file, annotations=READ_ONLY, @@ -638,7 +638,7 @@ def recording_tools() -> List[MCPTool]: name="ac_write_action_file", description="Persist an action list to a JSON file at file_path.", input_schema=schema({ - "file_path": {"type": "string"}, + "file_path": {"type": "string", "format": "path"}, "actions": {"type": "array"}, }, required=["file_path", "actions"]), handler=h_runs.write_action_file, @@ -935,7 +935,7 @@ def chatops_tools() -> List[MCPTool]: input_schema=schema({ "message": {"type": "string"}, "context": {"type": "object"}, - "script_root": {"type": "string"}, + "script_root": {"type": "string", "format": "path"}, }, required=["message"]), handler=h_ops.chatops_dispatch, annotations=DESTRUCTIVE, @@ -1060,7 +1060,7 @@ def ios_tools() -> List[MCPTool]: name="ac_ios_screenshot", description="Save the device screen as a PNG to file_path.", input_schema=schema({ - "file_path": {"type": "string"}, + "file_path": {"type": "string", "format": "path"}, "url": {"type": "string"}, }, required=["file_path"]), handler=h_exec.ios_screenshot, @@ -1101,8 +1101,8 @@ def redaction_tools() -> List[MCPTool]: "unconditionally. Returns {output_path, boxes, " "detectors_used}."), input_schema=schema({ - "file_path": {"type": "string"}, - "output_path": {"type": "string"}, + "file_path": {"type": "string", "format": "path"}, + "output_path": {"type": "string", "format": "path"}, "policy": {"type": "string", "enum": ["off", "moderate", "strict"]}, "regions": {"type": "array", @@ -1227,7 +1227,7 @@ def webrunner_tools() -> List[MCPTool]: name="ac_web_screenshot", description="Convenience: save a screenshot of the active browser tab.", input_schema=schema({ - "file_path": {"type": "string"}, + "file_path": {"type": "string", "format": "path"}, }, required=["file_path"]), handler=h.web_screenshot, annotations=NON_DESTRUCTIVE, @@ -1697,7 +1697,7 @@ def ab_locator_tools() -> List[MCPTool]: "properties": { "kind": {"type": "string", "enum": ["image", "ocr", "vlm", "a11y"]}, - "template_path": {"type": "string"}, + "template_path": {"type": "string", "format": "path"}, "detect_threshold": {"type": "number"}, "text": {"type": "string"}, "min_confidence": {"type": "number"}, @@ -1762,8 +1762,8 @@ def failure_hook_tools() -> List[MCPTool]: "source": {"type": "string"}, "source_id": {"type": "string"}, "error_text": {"type": "string"}, - "script_path": {"type": "string"}, - "screenshot_path": {"type": "string"}, + "script_path": {"type": "string", "format": "path"}, + "screenshot_path": {"type": "string", "format": "path"}, "log_tail": {"type": "string"}, "metadata": {"type": "object"}, }, required=["source", "source_id"]), @@ -1832,7 +1832,7 @@ def smart_wait_tools() -> List[MCPTool]: "(spinner/toast/dialog vanished). 'detect_threshold', " "'timeout_s', 'poll_interval_s', 'gone_for_s'."), input_schema=schema({ - "image": {"type": "string"}, + "image": {"type": "string", "format": "path"}, "detect_threshold": {"type": "number"}, "timeout_s": {"type": "number"}, "poll_interval_s": {"type": "number"}, @@ -1937,7 +1937,7 @@ def smart_wait_tools() -> List[MCPTool]: "(i.e. a download finished writing). Returns a " "WaitOutcome (succeeded/reason/elapsed_s)."), input_schema=schema({ - "path": {"type": "string"}, + "path": {"type": "string", "format": "path"}, "timeout_s": {"type": "number"}, "poll_interval_s": {"type": "number"}, "stable_for_s": {"type": "number"}, @@ -2166,7 +2166,7 @@ def anchor_locator_tools() -> List[MCPTool]: "properties": { "kind": {"type": "string", "enum": ["image", "ocr", "vlm", "a11y"]}, - "template_path": {"type": "string"}, + "template_path": {"type": "string", "format": "path"}, "detect_threshold": {"type": "number"}, "text": {"type": "string"}, "min_confidence": {"type": "number"}, @@ -2243,7 +2243,7 @@ def self_healing_tools() -> List[MCPTool]: "attempt is appended to the self-healing audit log. " "Returns {found, coordinates, method, ...}."), input_schema=schema({ - "template_path": {"type": "string"}, + "template_path": {"type": "string", "format": "path"}, "description": {"type": "string"}, "detect_threshold": {"type": "number"}, "screen_region": {"type": "array", @@ -2261,7 +2261,7 @@ def self_healing_tools() -> List[MCPTool]: "description triggers the VLM fallback when the " "template fails."), input_schema=schema({ - "template_path": {"type": "string"}, + "template_path": {"type": "string", "format": "path"}, "description": {"type": "string"}, "mouse_keycode": {"type": "string"}, "detect_threshold": {"type": "number"}, @@ -2301,7 +2301,7 @@ def scheduler_tools() -> List[MCPTool]: "interval_seconds (run every N seconds) or " "cron_expression (5-field cron rule)."), input_schema=schema({ - "script_path": {"type": "string"}, + "script_path": {"type": "string", "format": "path"}, "interval_seconds": {"type": "number"}, "cron_expression": {"type": "string"}, "repeat": {"type": "boolean"}, @@ -2355,9 +2355,9 @@ def trigger_tools() -> List[MCPTool]: input_schema=schema({ "kind": {"type": "string", "enum": ["image", "window", "pixel", "file"]}, - "script_path": {"type": "string"}, + "script_path": {"type": "string", "format": "path"}, "repeat": {"type": "boolean"}, - "image_path": {"type": "string"}, + "image_path": {"type": "string", "format": "path"}, "threshold": {"type": "number"}, "title_substring": {"type": "string"}, "case_sensitive": {"type": "boolean"}, @@ -2366,7 +2366,7 @@ def trigger_tools() -> List[MCPTool]: "target_rgb": {"type": "array", "items": {"type": "integer"}}, "tolerance": {"type": "integer"}, - "watch_path": {"type": "string"}, + "watch_path": {"type": "string", "format": "path"}, }, required=["kind", "script_path"]), handler=h_sched.trigger_add, annotations=DESTRUCTIVE, @@ -2412,7 +2412,7 @@ def process_and_shell_tools() -> List[MCPTool]: "{pid, argv}. Optional working_directory."), input_schema=schema({ "argv": {"type": "array", "items": {"type": "string"}}, - "working_directory": {"type": "string"}, + "working_directory": {"type": "string", "format": "path"}, }, required=["argv"]), handler=h_system.launch_process, annotations=DESTRUCTIVE, @@ -2658,7 +2658,7 @@ def process_and_shell_tools() -> List[MCPTool]: def work_queue_tools() -> List[MCPTool]: - _Q = {"db": {"type": "string"}, "name": {"type": "string"}} + _Q = {"db": {"type": "string", "format": "path"}, "name": {"type": "string"}} return [ MCPTool( name="ac_queue_add", @@ -2733,7 +2733,7 @@ def synthetic_data_tools() -> List[MCPTool]: input_schema=schema({ "schema": {"type": "object"}, "count": {"type": "integer"}, - "path": {"type": "string"}, + "path": {"type": "string", "format": "path"}, "fmt": {"type": "string", "enum": ["json", "csv"]}, "seed": {"type": "integer"}, }, required=["schema"]), @@ -2752,7 +2752,7 @@ def mcp_registry_tools() -> List[MCPTool]: "'path' when given, else returns the manifest. " "include_tools embeds the live tool list."), input_schema=schema({ - "path": {"type": "string"}, + "path": {"type": "string", "format": "path"}, "include_tools": {"type": "boolean"}, }), handler=h_ops.mcp_manifest, @@ -2763,7 +2763,7 @@ def mcp_registry_tools() -> List[MCPTool]: def test_selection_tools() -> List[MCPTool]: _flows = {"flows": {"type": "array", "items": {"type": "string"}}, - "history_path": {"type": "string"}, + "history_path": {"type": "string", "format": "path"}, "window": {"type": "integer"}} return [ MCPTool( @@ -2791,7 +2791,7 @@ def test_selection_tools() -> List[MCPTool]: def element_repository_tools() -> List[MCPTool]: - _R = {"path": {"type": "string"}, "key": {"type": "string"}} + _R = {"path": {"type": "string", "format": "path"}, "key": {"type": "string"}} return [ MCPTool( name="ac_element_save", @@ -2830,7 +2830,7 @@ def element_repository_tools() -> List[MCPTool]: MCPTool( name="ac_element_list", description="List saved locator names in a repository file.", - input_schema=schema({"path": {"type": "string"}}, + input_schema=schema({"path": {"type": "string", "format": "path"}}, required=["path"]), handler=h.element_list, annotations=READ_ONLY, @@ -2856,7 +2856,7 @@ def flow_debugger_tools() -> List[MCPTool]: def skill_library_tools() -> List[MCPTool]: - _S = {"path": {"type": "string"}, "name": {"type": "string"}} + _S = {"path": {"type": "string", "format": "path"}, "name": {"type": "string"}} return [ MCPTool( name="ac_skill_save", @@ -2881,7 +2881,7 @@ def skill_library_tools() -> List[MCPTool]: MCPTool( name="ac_skill_list", description="List saved skill names in a library file.", - input_schema=schema({"path": {"type": "string"}}, + input_schema=schema({"path": {"type": "string", "format": "path"}}, required=["path"]), handler=h.skill_list, annotations=READ_ONLY, @@ -2897,7 +2897,7 @@ def skill_library_tools() -> List[MCPTool]: name="ac_skill_search", description=("Search skills by name/description/tags; returns " "matching names."), - input_schema=schema({"path": {"type": "string"}, + input_schema=schema({"path": {"type": "string", "format": "path"}, "query": {"type": "string"}}, required=["path", "query"]), handler=h.skill_search, @@ -2929,7 +2929,7 @@ def a2a_tools() -> List[MCPTool]: description=("Build an A2A (agent-to-agent) Agent Card describing " "AutoControl's skills. Writes to 'path' when given, " "else returns the card."), - input_schema=schema({"path": {"type": "string"}}), + input_schema=schema({"path": {"type": "string", "format": "path"}}), handler=h_ops.agent_card, annotations=SIDE_EFFECT_ONLY, ), @@ -2937,7 +2937,7 @@ def a2a_tools() -> List[MCPTool]: def office_tools() -> List[MCPTool]: - _P = {"path": {"type": "string"}} + _P = {"path": {"type": "string", "format": "path"}} return [ MCPTool( name="ac_read_workbook", @@ -3000,7 +3000,7 @@ def office_tools() -> List[MCPTool]: def agent_memory_tools() -> List[MCPTool]: - _D = {"db": {"type": "string"}} + _D = {"db": {"type": "string", "format": "path"}} return [ MCPTool( name="ac_memory_remember", @@ -3081,7 +3081,7 @@ def observer_tools() -> List[MCPTool]: "event": {"type": "string", "enum": ["appear", "vanish", "change"]}, "actions": {"type": "array"}, - "image": {"type": "string"}, + "image": {"type": "string", "format": "path"}, "threshold": {"type": "number"}, "text": {"type": "string"}, "x": {"type": "integer"}, "y": {"type": "integer"}, @@ -3137,7 +3137,7 @@ def sbom_tools() -> List[MCPTool]: "dependencies (supply-chain compliance). 'root' " "limits to a distribution's closure (empty = all " "installed). Writes to 'path' or returns the SBOM."), - input_schema=schema({"path": {"type": "string"}, + input_schema=schema({"path": {"type": "string", "format": "path"}, "root": {"type": "string"}}), handler=h_ops.generate_sbom, annotations=SIDE_EFFECT_ONLY, @@ -3156,7 +3156,7 @@ def sharding_tools() -> List[MCPTool]: input_schema=schema({ "flows": {"type": "array", "items": {"type": "string"}}, "shards": {"type": "integer"}, - "history_path": {"type": "string"}, + "history_path": {"type": "string", "format": "path"}, "window": {"type": "integer"}, }, required=["flows"]), handler=h_ops.shard_suite, @@ -3274,7 +3274,7 @@ def i18n_tools() -> List[MCPTool]: def checkpoint_tools() -> List[MCPTool]: - _R = {"run_id": {"type": "string"}, "db": {"type": "string"}} + _R = {"run_id": {"type": "string"}, "db": {"type": "string", "format": "path"}} return [ MCPTool( name="ac_run_resumable", @@ -3317,7 +3317,7 @@ def set_of_marks_tools() -> List[MCPTool]: "instead of pixels. Optionally render a numbered-box " "overlay screenshot to 'render_path'."), input_schema=schema({"app_name": {"type": "string"}, - "render_path": {"type": "string"}}), + "render_path": {"type": "string", "format": "path"}}), handler=h_loc.mark_screen, annotations=SIDE_EFFECT_ONLY, ), @@ -3517,7 +3517,7 @@ def process_doc_tools() -> List[MCPTool]: input_schema=schema({ "actions": {"type": "array"}, "title": {"type": "string"}, - "path": {"type": "string"}}, + "path": {"type": "string", "format": "path"}}, required=["actions"]), handler=h.generate_sop, annotations=SIDE_EFFECT_ONLY, @@ -3581,7 +3581,7 @@ def feature_match_tools() -> List[MCPTool]: "points), center, inliers, matches, score}}. 'min_inliers' " "is the confidence floor; 'ratio' the match cutoff."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, "max_features": {"type": "integer"}, "ratio": {"type": "number"}, @@ -3724,8 +3724,8 @@ def preprocess_tools() -> List[MCPTool]: "deskew/contrast (default grayscale,upscale,binarize); " "'scale' for upscale. Returns {path, width, height}."), input_schema=schema({ - "output_path": {"type": "string"}, - "source": {"type": "string"}, + "output_path": {"type": "string", "format": "path"}, + "source": {"type": "string", "format": "path"}, "steps": {"type": "array", "items": {"type": "string"}}, "scale": {"type": "number"}, "region": {"type": "array", "items": {"type": "integer"}}, @@ -3777,7 +3777,7 @@ def actionability_tools() -> List[MCPTool]: "waited_s}. 'timeout_s', 'stable_for_s', 'min_score', " "'region'."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "timeout_s": {"type": "number"}, "stable_for_s": {"type": "number"}, "min_score": {"type": "number"}, @@ -4017,7 +4017,7 @@ def clipboard_files_tools() -> List[MCPTool]: "file copy (Windows). 'paths' is a list of absolute paths. " "Returns {set, count}."), input_schema=schema({ - "paths": {"type": "array", "items": {"type": "string"}}}, + "paths": {"type": "array", "items": {"type": "string", "format": "path"}}}, required=["paths"]), handler=h_exec.set_clipboard_files, annotations=SIDE_EFFECT_ONLY, @@ -4107,7 +4107,7 @@ def clipboard_files_tools() -> List[MCPTool]: "'point' without sending it (pure dry-run). Returns " "{message, paths, point, wide, blob_size}."), input_schema=schema({ - "paths": {"type": "array", "items": {"type": "string"}}, + "paths": {"type": "array", "items": {"type": "string", "format": "path"}}, "point": {"type": "array", "items": {"type": "integer"}}}, required=["paths"]), handler=h_exec.plan_file_drop, @@ -4120,7 +4120,7 @@ def clipboard_files_tools() -> List[MCPTool]: "client-area drop coordinate. Returns {dropped, count}."), input_schema=schema({ "hwnd": {"type": "integer"}, - "paths": {"type": "array", "items": {"type": "string"}}, + "paths": {"type": "array", "items": {"type": "string", "format": "path"}}, "point": {"type": "array", "items": {"type": "integer"}}}, required=["hwnd", "paths"]), handler=h_exec.drop_files, @@ -4138,7 +4138,7 @@ def img_histogram_tools() -> List[MCPTool]: "hsv/rgb/gray, 'bins' per channel. Returns {bins, space, " "histogram}. A scale/illumination-robust view fingerprint."), input_schema=schema({ - "source": {"type": "string"}, + "source": {"type": "string", "format": "path"}, "bins": {"type": "integer"}, "space": {"type": "string"}, "region": {"type": "array", "items": {"type": "integer"}}}, @@ -4153,8 +4153,8 @@ def img_histogram_tools() -> List[MCPTool]: "'method' correlation/chisqr/intersection/bhattacharyya, " "'threshold', 'space'. Returns {changed, score}."), input_schema=schema({ - "reference": {"type": "string"}, - "current": {"type": "string"}, + "reference": {"type": "string", "format": "path"}, + "current": {"type": "string", "format": "path"}, "method": {"type": "string"}, "threshold": {"type": "number"}, "space": {"type": "string"}, @@ -4170,7 +4170,7 @@ def img_histogram_tools() -> List[MCPTool]: "variance — low=blurry), contrast (grayscale stddev), " "brightness (mean 0-255)}."), input_schema=schema({ - "source": {"type": "string"}, + "source": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}}), handler=h_exec.image_quality, annotations=READ_ONLY, @@ -4182,7 +4182,7 @@ def img_histogram_tools() -> List[MCPTool]: "blurry / low_contrast / too_dark / too_bright. Tune with " "'min_sharpness' / 'min_contrast'."), input_schema=schema({ - "source": {"type": "string"}, + "source": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, "min_sharpness": {"type": "number"}, "min_contrast": {"type": "number"}}), @@ -4196,8 +4196,8 @@ def img_histogram_tools() -> List[MCPTool]: "across 'scales'. Returns {found, result:{scale, " "scale_percent, score, center, margin, candidates}}."), input_schema=schema({ - "template": {"type": "string"}, - "haystack": {"type": "string"}, + "template": {"type": "string", "format": "path"}, + "haystack": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, "scales": {"type": "array", "items": {"type": "number"}}, "method": {"type": "string"}}, @@ -4212,8 +4212,8 @@ def img_histogram_tools() -> List[MCPTool]: "y, width, height, center}]} — the raw scores match_" "template discards."), input_schema=schema({ - "template": {"type": "string"}, - "haystack": {"type": "string"}, + "template": {"type": "string", "format": "path"}, + "haystack": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, "scales": {"type": "array", "items": {"type": "number"}}, "method": {"type": "string"}}, @@ -4228,7 +4228,7 @@ def img_histogram_tools() -> List[MCPTool]: "saliency — where to look with no template/text. Returns " "{regions:[{x,y,width,height,center,score}], count}."), input_schema=schema({ - "source": {"type": "string"}, + "source": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, "size": {"type": "integer"}, "threshold": {"type": "number"}, @@ -4242,7 +4242,7 @@ def img_histogram_tools() -> List[MCPTool]: "(default screen): {found, region:{x,y,width,height," "center,score}}. The first place to look."), input_schema=schema({ - "source": {"type": "string"}, + "source": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, "size": {"type": "integer"}, "threshold": {"type": "number"}, @@ -4348,7 +4348,7 @@ def img_histogram_tools() -> List[MCPTool]: "structure ('method' sobel/laplacian/zscore). " "'region' [x,y,w,h] clips the search. Returns {found, " "x, y, width, height, score}."), - input_schema=schema({"template": {"type": "string"}, + input_schema=schema({"template": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, "method": {"type": "string"}, @@ -4376,10 +4376,10 @@ def img_histogram_tools() -> List[MCPTool]: "a 'reference' image and the current screen (or " "'current' image). Returns {changes:[{box, score, " "changed}]}."), - input_schema=schema({"reference": {"type": "string"}, + input_schema=schema({"reference": {"type": "string", "format": "path"}, "boxes": {"type": "array", "items": {"type": "array"}}, - "current": {"type": "string"}, + "current": {"type": "string", "format": "path"}, "threshold": {"type": "number"}, "region": {"type": "array", "items": {"type": "integer"}}}, @@ -4402,7 +4402,7 @@ def img_histogram_tools() -> List[MCPTool]: description=("Classify the widget in a 'box' [x,y,w,h] of a " "'source' image from its pixel shape. Returns {type, " "features}."), - input_schema=schema({"source": {"type": "string"}, + input_schema=schema({"source": {"type": "string", "format": "path"}, "box": {"type": "array", "items": {"type": "integer"}}}, required=["source", "box"]), @@ -4446,8 +4446,8 @@ def motion_regions_tools() -> List[MCPTool]: "Returns {count, regions}. For spinners / animations / " "picking a quiet area. 'threshold'/'min_area'/'blur'."), input_schema=schema({ - "before": {"type": "string"}, - "after": {"type": "string"}, + "before": {"type": "string", "format": "path"}, + "after": {"type": "string", "format": "path"}, "threshold": {"type": "integer"}, "min_area": {"type": "integer"}, "blur": {"type": "integer"}}, @@ -4461,8 +4461,8 @@ def motion_regions_tools() -> List[MCPTool]: "(default: screen). Returns {moved, activity} where " "activity is the fraction of pixels that changed."), input_schema=schema({ - "before": {"type": "string"}, - "after": {"type": "string"}, + "before": {"type": "string", "format": "path"}, + "after": {"type": "string", "format": "path"}, "threshold": {"type": "integer"}, "min_area": {"type": "integer"}}, required=["before"]), @@ -4533,8 +4533,8 @@ def perceptual_diff_tools() -> List[MCPTool]: "total_pixels, diff_ratio, regions}; pass 'max_diff_ratio' " "to raise when exceeded. 'threshold' 0..1 sensitivity."), input_schema=schema({ - "actual": {"type": "string"}, - "expected": {"type": "string"}, + "actual": {"type": "string", "format": "path"}, + "expected": {"type": "string", "format": "path"}, "threshold": {"type": "number"}, "include_aa": {"type": "boolean"}, "max_diff_ratio": {"type": "number"}}, @@ -4883,7 +4883,7 @@ def barcode_tools() -> List[MCPTool]: "{count, barcodes:[{text, type, points}]}. QR codes have " "their own tool."), input_schema=schema({ - "source": {"type": "string"}, + "source": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}}, required=[]), handler=h_exec.read_barcodes, @@ -4902,8 +4902,8 @@ def ssim_tools() -> List[MCPTool]: "a list of [x,y,w,h] boxes to exclude (clocks/cursors). " "Returns {score}. Perceptual, unlike pixel diff."), input_schema=schema({ - "reference": {"type": "string"}, - "current": {"type": "string"}, + "reference": {"type": "string", "format": "path"}, + "current": {"type": "string", "format": "path"}, "ignore": {"type": "array", "items": {"type": "array", "items": {"type": "integer"}}}, @@ -4920,8 +4920,8 @@ def ssim_tools() -> List[MCPTool]: "'ignore' [x,y,w,h] boxes suppressed. Returns " "{count, regions} (largest first)."), input_schema=schema({ - "reference": {"type": "string"}, - "current": {"type": "string"}, + "reference": {"type": "string", "format": "path"}, + "current": {"type": "string", "format": "path"}, "ignore": {"type": "array", "items": {"type": "array", "items": {"type": "integer"}}}, @@ -4944,7 +4944,7 @@ def visual_match_tools() -> List[MCPTool]: "score,scale,center}}. 'scales' [..] for DPI/zoom, " "'min_score', 'region', 'method'."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "scales": {"type": "array", "items": {"type": "number"}}, "region": {"type": "array", "items": {"type": "integer"}}, @@ -4959,7 +4959,7 @@ def visual_match_tools() -> List[MCPTool]: ">= 'min_score', overlaps removed by NMS. " "Returns {count, matches}."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "max_results": {"type": "integer"}, "nms_iou": {"type": "number"}, @@ -4976,8 +4976,8 @@ def visual_match_tools() -> List[MCPTool]: "over a transparent or varying background. Returns " "{found, match}. 'min_score', 'region'."), input_schema=schema({ - "template": {"type": "string"}, - "mask": {"type": "string"}, + "template": {"type": "string", "format": "path"}, + "mask": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "region": {"type": "array", "items": {"type": "integer"}}}, required=["template"]), @@ -4989,8 +4989,8 @@ def visual_match_tools() -> List[MCPTool]: description=("Find EVERY masked match of 'template' >= 'min_score', " "overlaps removed by NMS. Returns {count, matches}."), input_schema=schema({ - "template": {"type": "string"}, - "mask": {"type": "string"}, + "template": {"type": "string", "format": "path"}, + "mask": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "max_results": {"type": "integer"}, "nms_iou": {"type": "number"}, @@ -5012,7 +5012,7 @@ def rotated_match_tools() -> List[MCPTool]: "scale,angle,center}}. Use when a control is skewed / a " "rotated icon / a dial. 'min_score', 'region', 'method'."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "scales": {"type": "array", "items": {"type": "number"}}, "angles": {"type": "array", "items": {"type": "number"}}, @@ -5028,7 +5028,7 @@ def rotated_match_tools() -> List[MCPTool]: ">= 'min_score' over the angle x scale sweep, overlaps " "removed by NMS. Returns {count, matches}."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "scales": {"type": "array", "items": {"type": "number"}}, "angles": {"type": "array", "items": {"type": "number"}}, @@ -5048,7 +5048,7 @@ def rotated_match_tools() -> List[MCPTool]: "button) - do NOT blindly click. 'ambiguous_ratio' (default " "0.9), 'min_score', 'scales', 'region', 'method'."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "scales": {"type": "array", "items": {"type": "number"}}, "ambiguous_ratio": {"type": "number"}, @@ -5066,7 +5066,7 @@ def rotated_match_tools() -> List[MCPTool]: "separability near 0 = unimodal (no clear match) - do NOT " "trust the threshold. 'region', 'method'."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, "method": {"type": "string"}}, required=["template"]), @@ -5080,7 +5080,7 @@ def rotated_match_tools() -> List[MCPTool]: "tune. 'floor' (default 0.5) clamps the threshold so a noisy " "surface can't match junk. Returns {count, matches}."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "floor": {"type": "number"}, "max_results": {"type": "integer"}, "region": {"type": "array", "items": {"type": "integer"}}, @@ -5097,7 +5097,7 @@ def rotated_match_tools() -> List[MCPTool]: "ORB can't key on. Returns {found, match}. 'min_score', " "'scales', 'region'."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "scales": {"type": "array", "items": {"type": "number"}}, "region": {"type": "array", "items": {"type": "integer"}}}, @@ -5111,7 +5111,7 @@ def rotated_match_tools() -> List[MCPTool]: "'min_score', overlaps removed by NMS. Returns " "{count, matches}."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "max_results": {"type": "integer"}, "nms_iou": {"type": "number"}, @@ -5128,7 +5128,7 @@ def rotated_match_tools() -> List[MCPTool]: "cx/cy are float for drag / slider / high-DPI accuracy. " "'min_score', 'region', 'method'."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "min_score": {"type": "number"}, "region": {"type": "array", "items": {"type": "integer"}}, "method": {"type": "string"}}, @@ -5145,7 +5145,7 @@ def rotated_match_tools() -> List[MCPTool]: ">= 'min_votes' references agree within 'agree_px'. Cuts false " "positives on themed / animated UI. 'min_score', 'region'."), input_schema=schema({ - "templates": {"type": "array", "items": {"type": "string"}}, + "templates": {"type": "array", "items": {"type": "string", "format": "path"}}, "min_score": {"type": "number"}, "agree_px": {"type": "number"}, "min_votes": {"type": "integer"}, @@ -5177,7 +5177,7 @@ def rotated_match_tools() -> List[MCPTool]: "targets), 'min_score', 'scales', 'region'. For solid colour " "blobs use find_color_region instead."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "channels": {"type": "array", "items": {"type": "string"}}, "min_score": {"type": "number"}, "scales": {"type": "array", "items": {"type": "number"}}, @@ -5192,7 +5192,7 @@ def rotated_match_tools() -> List[MCPTool]: "'min_score', overlaps removed by NMS. Returns " "{count, matches}."), input_schema=schema({ - "template": {"type": "string"}, + "template": {"type": "string", "format": "path"}, "channels": {"type": "array", "items": {"type": "string"}}, "min_score": {"type": "number"}, "max_results": {"type": "integer"}, @@ -5210,7 +5210,7 @@ def rotated_match_tools() -> List[MCPTool]: "(min_ssim >= 'settle_threshold') - match only when stable to " "avoid mid-animation hits."), input_schema=schema({ - "frames": {"type": "array", "items": {"type": "string"}}, + "frames": {"type": "array", "items": {"type": "string", "format": "path"}}, "settle_threshold": {"type": "number"}}, required=["frames"]), handler=h_exec.region_stability, @@ -5224,8 +5224,8 @@ def rotated_match_tools() -> List[MCPTool]: "'agree_px' - a steady match, not one lucky frame. " "'min_score'."), input_schema=schema({ - "template": {"type": "string"}, - "frames": {"type": "array", "items": {"type": "string"}}, + "template": {"type": "string", "format": "path"}, + "frames": {"type": "array", "items": {"type": "string", "format": "path"}}, "min_score": {"type": "number"}, "agree_px": {"type": "number"}}, required=["template", "frames"]), @@ -5638,9 +5638,9 @@ def plugin_sdk_tools() -> List[MCPTool]: def governance_tools() -> List[MCPTool]: _AD = {"action": {"type": "string"}, "requester": {"type": "string"}, - "db": {"type": "string"}} + "db": {"type": "string", "format": "path"}} _TA = {"token": {"type": "string"}, "approver": {"type": "string"}, - "db": {"type": "string"}} + "db": {"type": "string", "format": "path"}} return [ MCPTool( name="ac_approval_request", @@ -5675,7 +5675,7 @@ def governance_tools() -> List[MCPTool]: "rejected) and an 'approved' boolean to gate an " "action on."), input_schema=schema({"token": {"type": "string"}, - "db": {"type": "string"}}, ["token"]), + "db": {"type": "string", "format": "path"}}, ["token"]), handler=h_ops.approval_status, annotations=READ_ONLY, ), @@ -5757,7 +5757,7 @@ def egress_tools() -> List[MCPTool]: def approval_testing_tools() -> List[MCPTool]: _ND = {"name": {"type": "string"}, - "approvals_dir": {"type": "string"}, + "approvals_dir": {"type": "string", "format": "path"}, "extension": {"type": "string"}} return [ MCPTool( @@ -5785,7 +5785,7 @@ def approval_testing_tools() -> List[MCPTool]: name="ac_pending_artifacts", description=("List artifact names with a received file awaiting " "approval under 'approvals_dir'. Returns {pending}."), - input_schema=schema({"approvals_dir": {"type": "string"}}), + input_schema=schema({"approvals_dir": {"type": "string", "format": "path"}}), handler=h_ops.pending_artifacts, annotations=READ_ONLY, ), @@ -5829,7 +5829,7 @@ def compliance_tools() -> List[MCPTool]: input_schema=schema( {"evidence": {"type": "object"}, "frameworks": {"type": "array", "items": {"type": "string"}}, - "path": {"type": "string"}, + "path": {"type": "string", "format": "path"}, "fmt": {"type": "string", "enum": ["json", "html"]}}, ["evidence"]), handler=h_ops.compliance_report, @@ -5896,8 +5896,9 @@ def video_report_tools() -> List[MCPTool]: "burned in. Writes 'output' (mp4/avi). Returns " "{output, steps, fps, frame_count}."), input_schema=schema( - {"steps": {"type": "array", "items": {"type": "object"}}, - "output": {"type": "string"}, + {"steps": {"type": "array", + "items": {"type": "object", "properties": {"image": {"format": "path"}}}}, + "output": {"type": "string", "format": "path"}, "fps": {"type": "integer"}, "seconds_per_step": {"type": "number"}}, ["steps", "output"]), @@ -5956,7 +5957,7 @@ def artifact_store_tools() -> List[MCPTool]: "S3-compatible store. Optional 'key' (defaults to the " "file name). Returns {key}."), input_schema=schema( - {"local_path": {"type": "string"}, "key": {"type": "string"}}, + {"local_path": {"type": "string", "format": "path"}, "key": {"type": "string"}}, ["local_path"]), handler=h_ops.s3_upload, annotations=DESTRUCTIVE, @@ -5967,7 +5968,7 @@ def artifact_store_tools() -> List[MCPTool]: "'local_path'. Returns {path}."), input_schema=schema( {"key": {"type": "string"}, - "local_path": {"type": "string"}}, ["key", "local_path"]), + "local_path": {"type": "string", "format": "path"}}, ["key", "local_path"]), handler=h_ops.s3_download, annotations=SIDE_EFFECT_ONLY, ), @@ -5998,7 +5999,7 @@ def image_dedup_tools() -> List[MCPTool]: "comparison. 'algo' is 'average' (default) or " "'dhash'. Returns {hash} (hex)."), input_schema=schema( - {"path": {"type": "string"}, + {"path": {"type": "string", "format": "path"}, "algo": {"type": "string", "enum": ["average", "dhash"]}}, ["path"]), handler=h_loc.image_hash, @@ -6010,7 +6011,7 @@ def image_dedup_tools() -> List[MCPTool]: "keeping the first of each cluster (images within " "'max_distance' bits are dropped). Returns {unique}."), input_schema=schema( - {"paths": {"type": "array", "items": {"type": "string"}}, + {"paths": {"type": "array", "items": {"type": "string", "format": "path"}}, "max_distance": {"type": "integer"}}, ["paths"]), handler=h_loc.dedupe_images, annotations=READ_ONLY, @@ -6223,7 +6224,7 @@ def process_mining_tools() -> List[MCPTool]: def asset_tools() -> List[MCPTool]: - _ENV = {"environment": {"type": "string"}, "db": {"type": "string"}} + _ENV = {"environment": {"type": "string"}, "db": {"type": "string", "format": "path"}} return [ MCPTool( name="ac_set_asset", @@ -7119,7 +7120,7 @@ def dotenv_tools() -> List[MCPTool]: description=("Load a .env file at 'path' into a fresh {values} dict. " "'override' is accepted for symmetry (fresh dict)."), input_schema=schema( - {"path": {"type": "string"}, "override": {"type": "boolean"}}, + {"path": {"type": "string", "format": "path"}, "override": {"type": "boolean"}}, ["path"]), handler=h_exec.load_dotenv, annotations=READ_ONLY, @@ -7219,7 +7220,7 @@ def secret_ref_tools() -> List[MCPTool]: name="ac_resolve_ref", description=("Resolve a value reference 'ref' (env://VAR, " "file://path, or secret://name) to {value}."), - input_schema=schema({"ref": {"type": "string"}}, ["ref"]), + input_schema=schema({"ref": {"type": "string", "format": "value-ref"}}, ["ref"]), handler=h_exec.resolve_ref, annotations=READ_ONLY, ), @@ -7227,7 +7228,7 @@ def secret_ref_tools() -> List[MCPTool]: name="ac_resolve_refs", description=("Recursively resolve every env:// / file:// / secret:// " "reference inside 'obj'. Returns {resolved}."), - input_schema=schema({"obj": {"type": "object"}}, ["obj"]), + input_schema=schema({"obj": {"type": "object", "format": "value-ref"}}, ["obj"]), handler=h_exec.resolve_refs, annotations=READ_ONLY, ), @@ -7523,7 +7524,7 @@ def provenance_tools() -> List[MCPTool]: "list of file 'paths' (sha256 subjects). Returns " "{statement}."), input_schema=schema( - {"paths": {"type": "array"}, "builder_id": {"type": "string"}}, + {"paths": {"type": "array", "items": {"format": "path"}}, "builder_id": {"type": "string"}}, ["paths"]), handler=h_ops.build_provenance, annotations=READ_ONLY, @@ -7533,7 +7534,8 @@ def provenance_tools() -> List[MCPTool]: description=("Re-hash 'files' (name->path) against a provenance " "'statement'. Returns {ok, mismatches}."), input_schema=schema( - {"statement": {"type": "object"}, "files": {"type": "object"}}, + {"statement": {"type": "object"}, + "files": {"type": "object", "additionalProperties": {"format": "path"}}}, ["statement", "files"]), handler=h_ops.verify_provenance, annotations=READ_ONLY, @@ -7759,7 +7761,7 @@ def decision_table_tools() -> List[MCPTool]: def locator_repair_tools() -> List[MCPTool]: - _DB = {"db": {"type": "string"}} + _DB = {"db": {"type": "string", "format": "path"}} return [ MCPTool( name="ac_repair_record", @@ -7845,7 +7847,7 @@ def sarif_tools() -> List[MCPTool]: "Returns {sarif, path?}."), input_schema=schema( {"findings": {"type": "array", "items": {"type": "object"}}, - "path": {"type": "string"}, + "path": {"type": "string", "format": "path"}, "tool_name": {"type": "string"}}, ["findings"]), handler=h_ops.export_sarif, annotations=SIDE_EFFECT_ONLY, @@ -7950,7 +7952,7 @@ def hotkey_tools() -> List[MCPTool]: "to begin listening."), input_schema=schema({ "combo": {"type": "string"}, - "script_path": {"type": "string"}, + "script_path": {"type": "string", "format": "path"}, "binding_id": {"type": "string"}, }, required=["combo", "script_path"]), handler=h_sched.hotkey_bind, @@ -8323,7 +8325,7 @@ def assertion_tools() -> List[MCPTool]: description=("Assert a template image is (or is not) visible on " "screen at the given match threshold."), input_schema=schema({ - "template_path": {"type": "string"}, + "template_path": {"type": "string", "format": "path"}, "threshold": {"type": "number"}, "present": {"type": "boolean"}, "raise_on_fail": {"type": "boolean"}, @@ -8401,7 +8403,7 @@ def assertion_tools() -> List[MCPTool]: "or a minimum byte size (min_size). Set exists=false " "to assert the file is absent."), input_schema=schema({ - "path": {"type": "string"}, + "path": {"type": "string", "format": "path"}, "exists": {"type": "boolean"}, "contains": {"type": "string"}, "sha256": {"type": "string"}, @@ -8486,7 +8488,7 @@ def data_source_tools() -> List[MCPTool]: "with the AC_for_each_row flow-control command to " "drive a script once per row."), input_schema=schema({ - "source": {"type": "object"}, + "source": {"type": "object", "properties": {"path": {"format": "path"}}}, "limit": {"type": "integer"}, }, required=["source"]), handler=hq.load_data, @@ -8503,7 +8505,7 @@ def pdf_tools() -> List[MCPTool]: "pages), a 1-based page number, or a list of them. " "Requires the optional pypdf package."), input_schema=schema({ - "path": {"type": "string"}, + "path": {"type": "string", "format": "path"}, "pages": {"type": ["integer", "array", "null"]}, }, required=["path"]), handler=hq.extract_pdf_text, @@ -8517,7 +8519,7 @@ def pdf_tools() -> List[MCPTool]: "case-insensitive match. Raises on failure unless " "raise_on_fail is false."), input_schema=schema({ - "path": {"type": "string"}, + "path": {"type": "string", "format": "path"}, "text": {"type": "string"}, "present": {"type": "boolean"}, "page": {"type": "integer"}, @@ -8541,7 +8543,8 @@ def email_tools() -> List[MCPTool]: "use_tls?, use_ssl?, timeout?}; TLS is on by default. " "Sends mail (irreversible side effect)."), input_schema=schema({ - "message": {"type": "object"}, + "message": {"type": "object", + "properties": {"attachments": {"items": {"format": "path"}}}}, "smtp": {"type": "object"}, }, required=["message", "smtp"]), handler=hq.send_email, @@ -8561,7 +8564,7 @@ def sql_tools() -> List[MCPTool]: "'params' (?/:name placeholders) — never interpolate. " "A single read-only statement only."), input_schema=schema({ - "database": {"type": "string"}, + "database": {"type": "string", "format": "path"}, "query": {"type": "string"}, "params": {"type": ["array", "object"]}, "fetch": {"type": "string", "enum": ["all", "one", "scalar"]}, @@ -8577,7 +8580,7 @@ def sql_tools() -> List[MCPTool]: "Bind values via 'params'. Raises on failure unless " "raise_on_fail is false."), input_schema=schema({ - "database": {"type": "string"}, + "database": {"type": "string", "format": "path"}, "query": {"type": "string"}, "params": {"type": ["array", "object"]}, "op": {"type": "string"}, @@ -8626,7 +8629,7 @@ def visual_regression_tools() -> List[MCPTool]: "screen (or a [x, y, w, h] region) for later visual " "regression checks."), input_schema=schema({ - "path": {"type": "string"}, + "path": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, }, required=["path"]), handler=hq.take_golden, @@ -8641,11 +8644,11 @@ def visual_regression_tools() -> List[MCPTool]: "and passes unless create_if_missing=false. Pass " "diff_path to save a highlighted diff on mismatch."), input_schema=schema({ - "golden_path": {"type": "string"}, + "golden_path": {"type": "string", "format": "path"}, "region": {"type": "array", "items": {"type": "integer"}}, "tolerance": {"type": "number"}, "per_pixel_threshold": {"type": "integer"}, - "diff_path": {"type": "string"}, + "diff_path": {"type": "string", "format": "path"}, "create_if_missing": {"type": "boolean"}, "raise_on_fail": {"type": "boolean"}, }, required=["golden_path"]), @@ -8683,13 +8686,13 @@ def codegen_tools() -> List[MCPTool]: "execute_action). Pass 'output' to also write the file. " "Returns the generated source code."), input_schema=schema({ - "source": {"type": ["array", "string"], + "source": {"type": ["array", "string"], "format": "path", "description": "Action list, or path to a JSON action file."}, "target": {"type": "string", "enum": ["pytest", "python", "robot"]}, "style": {"type": "string", "enum": ["calls", "actions"]}, "name": {"type": "string"}, - "output": {"type": "string"}, + "output": {"type": "string", "format": "path"}, }, required=["source"]), handler=hq.generate_code, annotations=SIDE_EFFECT_ONLY, @@ -8823,8 +8826,8 @@ def suite_tools() -> List[MCPTool]: "spec": {"type": "object"}, "tags": {"type": "array", "items": {"type": "string"}}, "respect_quarantine": {"type": "boolean"}, - "junit_path": {"type": "string"}, - "allure_dir": {"type": "string"}, + "junit_path": {"type": "string", "format": "path"}, + "allure_dir": {"type": "string", "format": "path"}, }, required=["spec"]), handler=hq.run_suite, annotations=DESTRUCTIVE, @@ -8977,7 +8980,7 @@ def media_assert_tools() -> List[MCPTool]: "segment of a recorded video and assert motion " "(expect_motion=true) or a static segment."), input_schema=schema({ - "video_path": {"type": "string"}, + "video_path": {"type": "string", "format": "path"}, "start_s": {"type": "number"}, "end_s": {"type": "number"}, "threshold": {"type": "number"}, diff --git a/je_auto_control/utils/path_guard/__init__.py b/je_auto_control/utils/path_guard/__init__.py index 6f65b514a..5d4185c67 100644 --- a/je_auto_control/utils/path_guard/__init__.py +++ b/je_auto_control/utils/path_guard/__init__.py @@ -3,8 +3,12 @@ ALLOWED_ROOTS_ENV, PathNotAllowedError, default_allowed_roots, validate_path, ) +from je_auto_control.utils.path_guard.policy import ( + MCP_CLIENT_ROOTS_ENV, MCP_PATH_ROOTS_ENV, PathPolicy, +) __all__ = [ - "ALLOWED_ROOTS_ENV", "PathNotAllowedError", "default_allowed_roots", + "ALLOWED_ROOTS_ENV", "MCP_CLIENT_ROOTS_ENV", "MCP_PATH_ROOTS_ENV", + "PathNotAllowedError", "PathPolicy", "default_allowed_roots", "validate_path", ] diff --git a/je_auto_control/utils/path_guard/policy.py b/je_auto_control/utils/path_guard/policy.py new file mode 100644 index 000000000..1a3ba5f81 --- /dev/null +++ b/je_auto_control/utils/path_guard/policy.py @@ -0,0 +1,113 @@ +"""Opt-in confinement of file paths to a set of root directories. + +:func:`~je_auto_control.utils.path_guard.path_guard.validate_path` bounds the +paths a CLI takes from ``argv`` and always has roots. A :class:`PathPolicy` is +the long-lived counterpart for a server: it is **off** until it is given roots, +its roots can change while it runs (an MCP client's ``roots/list``), and every +check names the operation that asked, so a refusal reads as "which argument of +which tool". + +A path is canonicalised with ``os.path.realpath`` before it is compared, so +``..``, a symlink that points out of a root, another drive and a UNC share are +all judged by where they really lead. + +Headless module: imports no PySide6. +""" +from __future__ import annotations + +import os +import threading +from pathlib import Path +from typing import Iterable, Mapping, Optional, Tuple + +from je_auto_control.utils.path_guard.path_guard import ( + PathNotAllowedError, _canonical, _is_within, +) + +#: ``os.pathsep``-separated directories MCP tool file arguments must stay in. +MCP_PATH_ROOTS_ENV = "JE_AUTOCONTROL_MCP_PATH_ROOTS" +#: Truthy to also accept the roots the MCP client reports through ``roots/list``. +MCP_CLIENT_ROOTS_ENV = "JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT" + +_TRUTHY = frozenset({"1", "true", "yes", "on"}) + + +class PathPolicy: + """Roots a path must resolve into; inactive until it has some. + + ``roots`` are fixed for the life of the policy. With ``use_client_roots`` + the directories passed to :meth:`set_client_roots` count as well, and the + policy is active even before the first of them arrives — a path is then + refused rather than let through while the roots are still unknown. + """ + + def __init__(self, roots: Iterable[os.PathLike | str] = (), *, + use_client_roots: bool = False) -> None: + self._static: Tuple[Path, ...] = tuple(_canonical(root) for root in roots) + self._use_client_roots = bool(use_client_roots) + self._client: Tuple[Path, ...] = () + self._lock = threading.Lock() + + @classmethod + def from_env(cls, environ: Optional[Mapping[str, str]] = None) -> "PathPolicy": + """Build the policy the MCP environment variables describe. + + Neither variable set gives an inactive policy, which is the default. + """ + source = os.environ if environ is None else environ + roots = [entry.strip() + for entry in source.get(MCP_PATH_ROOTS_ENV, "").split(os.pathsep) + if entry.strip()] + from_client = source.get(MCP_CLIENT_ROOTS_ENV, "").strip().lower() in _TRUTHY + return cls(roots, use_client_roots=from_client) + + @property + def enabled(self) -> bool: + """Whether :meth:`validate` restricts anything.""" + return bool(self._static) or self._use_client_roots + + @property + def use_client_roots(self) -> bool: + """Whether roots reported by the client are honoured.""" + return self._use_client_roots + + def set_client_roots(self, roots: Iterable[os.PathLike | str]) -> None: + """Replace the client-reported roots; ignored unless they are honoured.""" + if not self._use_client_roots: + return + resolved = tuple(_canonical(root) for root in roots) + with self._lock: + self._client = resolved + + def roots(self) -> Tuple[Path, ...]: + """Every root currently in force, configured ones first.""" + with self._lock: + return self._static + self._client + + def validate(self, path: os.PathLike | str, *, operation: str) -> Path: + """Return ``path`` canonicalised, or raise :class:`PathNotAllowedError`. + + ``operation`` says what wanted the path and is quoted in the refusal. + An inactive policy canonicalises and accepts. A path starting with + ``~`` must be inside the roots both expanded and taken literally, + because callers differ on whether they expand it. + """ + text = os.fspath(path) + if not text or "\x00" in text: + raise PathNotAllowedError(f"{operation}: invalid path {text!r}") + candidate = _canonical(text) + if not self.enabled: + return candidate + roots = self.roots() + readings = [candidate] + if text.startswith("~"): + readings.append(Path(os.path.realpath(text))) + for reading in readings: + if not any(_is_within(reading, root) for root in roots): + raise PathNotAllowedError( + f"{operation}: {text!r} resolves to {reading}, outside the allowed " + f"roots ({', '.join(str(root) for root in roots) or 'none known yet'})") + return candidate + + +__all__ = ["MCP_CLIENT_ROOTS_ENV", "MCP_PATH_ROOTS_ENV", "PathPolicy"] diff --git a/je_auto_control/utils/remote_desktop/__init__.py b/je_auto_control/utils/remote_desktop/__init__.py index f700ef903..b4d024350 100644 --- a/je_auto_control/utils/remote_desktop/__init__.py +++ b/je_auto_control/utils/remote_desktop/__init__.py @@ -17,7 +17,8 @@ ClipboardSyncError, ) from je_auto_control.utils.remote_desktop.file_transfer import ( - FileReceiver, FileSendResult, FileTransferError, send_file, + FileReceiver, FileSendResult, FileTransferError, confine_destination, + default_download_dir, send_file, ) from je_auto_control.utils.remote_desktop.host import RemoteDesktopHost from je_auto_control.utils.remote_desktop.host_access import ( @@ -183,7 +184,8 @@ def is_webrtc_available() -> bool: "AudioBackendError", "AudioCapture", "AudioPlayer", "is_audio_backend_available", "ClipboardSyncError", - "FileReceiver", "FileSendResult", "FileTransferError", "send_file", + "FileReceiver", "FileSendResult", "FileTransferError", "confine_destination", + "default_download_dir", "send_file", "PresenceError", "PresenceListener", "PresenceRegistry", "ROLE_CONTROLLER", "ROLE_OBSERVER", "ViewerPresence", "default_presence_registry", diff --git a/je_auto_control/utils/remote_desktop/file_transfer.py b/je_auto_control/utils/remote_desktop/file_transfer.py index ab159a929..aad7644f1 100644 --- a/je_auto_control/utils/remote_desktop/file_transfer.py +++ b/je_auto_control/utils/remote_desktop/file_transfer.py @@ -16,6 +16,14 @@ this should keep ``trusted token holders == trusted users`` in mind, and treat the dropbox / destination filesystem accordingly. +That trust runs one way. A host writes where an authenticated viewer says, +because the viewer holds the token. A viewer has no such assurance about the +host it connected to, so a viewer's receiver is built with ``base_dir``: +``dest_path`` is then a path *relative to that directory*, and an absolute +path, a drive or UNC path, a ``..`` component or a symlink leading out of it +fails the transfer. :func:`default_download_dir` is the directory the viewers +use unless told otherwise. + The receiver writes to a ``.part`` file beside the destination and renames it into place only when ``FILE_END`` reports success and exactly the announced number of bytes arrived, so a failed transfer never truncates an @@ -23,12 +31,13 @@ """ import json import os +import re import threading from collections import OrderedDict import uuid from dataclasses import dataclass -from pathlib import Path -from typing import Any, Callable, Dict, Optional, Tuple +from pathlib import Path, PurePosixPath, PureWindowsPath +from typing import Any, Callable, Dict, List, Optional, Tuple, Union from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.logging.logging_instance import autocontrol_logger @@ -41,6 +50,12 @@ f"transfer_id must be a {TRANSFER_ID_LEN}-char UUID string" ) +#: Overrides where a viewer stores the files its host pushes. +DOWNLOAD_DIR_ENV = "JE_AUTOCONTROL_REMOTE_DOWNLOAD_DIR" + +_SEPARATORS = re.compile(r"[\\/]+") + +PathLike = Union[str, "os.PathLike[str]"] ProgressCallback = Callable[[str, int, int], None] CompleteCallback = Callable[[str, bool, Optional[str], str], None] @@ -49,6 +64,53 @@ class FileTransferError(AutoControlException, RuntimeError): """Raised when a file-transfer payload is malformed.""" +def default_download_dir() -> Path: + """Where a viewer keeps host-pushed files: the env override, else ``~/Downloads/AutoControl``. + + Not created here; the receiver makes it when the first file arrives. + """ + override = os.environ.get(DOWNLOAD_DIR_ENV, "").strip() + if override: + return Path(os.path.expanduser(override)) + return Path(os.path.expanduser("~")) / "Downloads" / "AutoControl" + + +def confine_destination(base_dir: PathLike, dest_path: str) -> Path: + """Return where relative ``dest_path`` lands under ``base_dir``, or raise. + + Raises :class:`FileTransferError` for an absolute, drive or UNC path, a + ``..`` component, and anything whose real location (symlinks resolved) is + outside ``base_dir``. Both separators are honoured whatever the local + platform, since the path was written on another machine. + """ + parts = _relative_parts(dest_path) + base = Path(os.path.realpath(os.path.expanduser(os.fspath(base_dir)))) + target = Path(os.path.realpath(base.joinpath(*parts))) + if base not in target.parents: + raise FileTransferError(f"dest_path leaves the download directory: {dest_path!r}") + return target + + +def _relative_parts(dest_path: str) -> List[str]: + """Split a relative ``dest_path`` into components, refusing every way out.""" + if "\x00" in dest_path: + raise FileTransferError("dest_path contains a NUL byte") + if PureWindowsPath(dest_path).anchor or PurePosixPath(dest_path).is_absolute(): + raise FileTransferError( + f"dest_path must be relative to the download directory: {dest_path!r}") + parts = [part for part in _SEPARATORS.split(dest_path) if part not in ("", ".")] + if not parts: + raise FileTransferError("dest_path names no file") + if any(_climbs_out(part) for part in parts): + raise FileTransferError(f"dest_path leaves the download directory: {dest_path!r}") + return parts + + +def _climbs_out(part: str) -> bool: + """A parent reference, or on Windows a colon: a drive or an alternate data stream.""" + return part == ".." or (os.name == "nt" and ":" in part) + + def new_transfer_id() -> str: """Return a fresh 36-character ASCII transfer ID.""" return str(uuid.uuid4()) @@ -151,12 +213,20 @@ def _discard(part: Path) -> None: class FileReceiver: - """Demultiplex incoming FILE_* messages into one or more file writes.""" + """Demultiplex incoming FILE_* messages into one or more file writes. + + With ``base_dir`` every ``dest_path`` is relative to that directory and a + transfer that would land outside it fails (see + :func:`confine_destination`). Without it the sender's path is used as + given, which is right only when the sender is trusted — the host side. + """ def __init__(self, on_progress: Optional[ProgressCallback] = None, - on_complete: Optional[CompleteCallback] = None) -> None: + on_complete: Optional[CompleteCallback] = None, + base_dir: Optional[PathLike] = None) -> None: self._on_progress = on_progress self._on_complete = on_complete + self._base_dir = base_dir self._active: Dict[str, _Incoming] = {} # Transfers aborted before FILE_BEGIN registered them. A viewer that # disconnected while its begin was opening the part file had the @@ -180,9 +250,11 @@ def handle_begin(self, payload: bytes) -> None: if cancelled: self._fire_complete(transfer_id, False, "cancelled before it began", str(dest_path)) return - path = Path(os.path.expanduser(dest_path)) - if not path.name: # ".", "/" or "C:\\": with_name raised ValueError past the handler - self._fire_complete(transfer_id, False, "dest_path names no file", str(path)) + try: + path = self._destination(dest_path) + # ValueError: a NUL in the name reaches realpath before any open(). + except (FileTransferError, ValueError) as error: + self._fire_complete(transfer_id, False, str(error), str(dest_path)) return part = path.with_name(f".{path.name}.{transfer_id[:8]}.part") try: @@ -202,6 +274,15 @@ def handle_begin(self, payload: bytes) -> None: if self._on_progress is not None: self._on_progress(transfer_id, 0, total_size) + def _destination(self, dest_path: str) -> Path: + """Where ``dest_path`` is written, confined to ``base_dir`` when there is one.""" + if self._base_dir is not None: + return confine_destination(self._base_dir, dest_path) + path = Path(os.path.expanduser(dest_path)) + if not path.name: # ".", "/" or "C:\\": with_name raised ValueError past the handler + raise FileTransferError("dest_path names no file") + return path + def _register(self, incoming: _Incoming) -> bool: """Make ``incoming`` active, unless it was aborted while its file opened.""" with self._lock: diff --git a/je_auto_control/utils/remote_desktop/host.py b/je_auto_control/utils/remote_desktop/host.py index dd6180538..387e70131 100644 --- a/je_auto_control/utils/remote_desktop/host.py +++ b/je_auto_control/utils/remote_desktop/host.py @@ -398,6 +398,9 @@ def send_file_to_viewers(self, source_path: str, dest_path: str, on_progress=None) -> int: """Stream ``source_path`` to every authenticated viewer. + ``dest_path`` is relative to each viewer's download directory: a + viewer refuses an absolute path or one that leaves that directory. + Returns the number of viewers the transfer was attempted on. Each viewer gets its own ``transfer_id`` so progress callbacks can be demultiplexed in the GUI. diff --git a/je_auto_control/utils/remote_desktop/viewer.py b/je_auto_control/utils/remote_desktop/viewer.py index 1ea65f445..407c9f6e7 100644 --- a/je_auto_control/utils/remote_desktop/viewer.py +++ b/je_auto_control/utils/remote_desktop/viewer.py @@ -13,7 +13,7 @@ ClipboardSyncError, decode as decode_clipboard, encode_image, encode_text, ) from je_auto_control.utils.remote_desktop.file_transfer import ( - FileReceiver, FileTransferError, send_file, + FileReceiver, FileTransferError, default_download_dir, send_file, ) from je_auto_control.utils.remote_desktop.host_id import validate_host_id from je_auto_control.utils.remote_desktop.protocol import ( @@ -293,12 +293,19 @@ def send_clipboard_image(self, png_bytes: bytes) -> None: self._channel.send_typed(MessageType.CLIPBOARD, encode_image(png_bytes)) def set_file_receiver(self, receiver: FileReceiver) -> None: - """Replace the default ``FileReceiver`` used for incoming files.""" + """Replace the default ``FileReceiver`` used for incoming files. + + The default keeps what the host pushes inside + :func:`~je_auto_control.utils.remote_desktop.file_transfer.default_download_dir`; + pass ``FileReceiver(base_dir=...)`` to choose the directory. + """ self._file_receiver = receiver def _ensure_file_receiver(self) -> FileReceiver: if self._file_receiver is None: - self._file_receiver = FileReceiver() + # Confined: the host names the destination, and a viewer cannot + # vouch for the host it dialled. + self._file_receiver = FileReceiver(base_dir=default_download_dir()) return self._file_receiver def send_file(self, source_path: str, dest_path: str, diff --git a/je_auto_control/utils/secret_ref/__init__.py b/je_auto_control/utils/secret_ref/__init__.py index 224c4a2de..a63e18f78 100644 --- a/je_auto_control/utils/secret_ref/__init__.py +++ b/je_auto_control/utils/secret_ref/__init__.py @@ -1,9 +1,10 @@ """URI-scheme value reference resolution for AutoControl.""" from je_auto_control.utils.secret_ref.secret_ref import ( - RefResolver, SecretRefError, is_ref, refuse_secret_refs, resolve_ref, resolve_refs_in, + MCP_ENV_REF_ALLOW_ENV, RefResolver, SecretRefError, env_allowlist_from_env, is_ref, + refuse_secret_refs, resolve_ref, resolve_refs_in, ) __all__ = [ - "RefResolver", "SecretRefError", "is_ref", "refuse_secret_refs", "resolve_ref", - "resolve_refs_in", + "MCP_ENV_REF_ALLOW_ENV", "RefResolver", "SecretRefError", "env_allowlist_from_env", + "is_ref", "refuse_secret_refs", "resolve_ref", "resolve_refs_in", ] diff --git a/je_auto_control/utils/secret_ref/secret_ref.py b/je_auto_control/utils/secret_ref/secret_ref.py index d52be9fcb..3594cff55 100644 --- a/je_auto_control/utils/secret_ref/secret_ref.py +++ b/je_auto_control/utils/secret_ref/secret_ref.py @@ -9,11 +9,21 @@ Pure standard library (``os`` / ``re``); imports no ``PySide6``. The env reader, secret resolver, and base directory are injectable, so resolution is safe and deterministic in CI. + +Two optional restrictions, both off unless given: ``env_allowlist`` names the +only environment variables ``env://`` may read (``fnmatch`` patterns such as +``APP_*``), and ``path_policy`` confines ``file://`` to that policy's roots. +The MCP server builds both from its environment, because a model that can ask +for ``env://ANTHROPIC_API_KEY`` or ``file:///etc/passwd`` gets the value back. """ +import fnmatch import os import re from pathlib import Path -from typing import Any, Callable, Mapping, Optional +from typing import TYPE_CHECKING, Any, Callable, Iterable, Mapping, Optional, Tuple + +if TYPE_CHECKING: + from je_auto_control.utils.path_guard.policy import PathPolicy from je_auto_control.utils.exception.exceptions import AutoControlException @@ -22,6 +32,10 @@ EnvReader = Mapping[str, str] SecretResolver = Callable[[str], Optional[str]] +#: Comma-separated ``fnmatch`` patterns: the only variables ``env://`` may read +#: through the MCP server's ``ac_resolve_ref`` / ``ac_resolve_refs``. +MCP_ENV_REF_ALLOW_ENV = "JE_AUTOCONTROL_MCP_ENV_REF_ALLOW" + _DRIVE_URL_PATH = re.compile(r"^/[A-Za-z]:[/\\]") @@ -62,6 +76,18 @@ def refuse_secret_refs(obj: Any) -> None: "reference ${secrets.NAME} in the step that needs the value") +def env_allowlist_from_env( + environ: Optional[Mapping[str, str]] = None) -> Optional[Tuple[str, ...]]: + """Read ``JE_AUTOCONTROL_MCP_ENV_REF_ALLOW``; ``None`` (unset) means no restriction. + + A value that names nothing, such as a lone comma, allows no variable at all. + """ + raw = (os.environ if environ is None else environ).get(MCP_ENV_REF_ALLOW_ENV) + if raw is None or not raw.strip(): + return None + return tuple(entry.strip() for entry in raw.split(",") if entry.strip()) + + def _default_secret(name: str) -> str: from je_auto_control.utils.governance import default_broker token = default_broker.lease(name, ttl=1.0) @@ -79,10 +105,40 @@ class RefResolver: def __init__(self, *, env: Optional[EnvReader] = None, secret_resolver: Optional[SecretResolver] = None, - base_dir: Optional[str] = None) -> None: + base_dir: Optional[str] = None, + env_allowlist: Optional[Iterable[str]] = None, + path_policy: Optional["PathPolicy"] = None) -> None: self._env = env self._secret_resolver = secret_resolver self._base_dir = base_dir + self._env_allowlist = None if env_allowlist is None else tuple(env_allowlist) + self._path_policy = path_policy + + def check_all(self, obj: Any) -> None: + """Raise ``SecretRefError`` for any reference in ``obj`` the restrictions refuse. + + Nothing is read: this is the gate a server runs on a request before it + hands the structure to a resolver. + """ + if isinstance(obj, dict): + for value in obj.values(): + self.check_all(value) + elif isinstance(obj, list): + for item in obj: + self.check_all(item) + elif isinstance(obj, str): + match = _REF_RE.match(obj) + if match is not None and match.group(1) == "env": + self._check_env(match.group(2)) + elif match is not None and match.group(1) == "file": + self._file_target(match.group(2)) + + def _check_env(self, name: str) -> None: + allowed = self._env_allowlist + if allowed is None: + return + if not any(fnmatch.fnmatchcase(name, pattern) for pattern in allowed): + raise SecretRefError(f"env var {name!r} is not on the env:// allowlist") def resolve(self, ref: str) -> str: """Resolve a single reference string to its value.""" @@ -109,12 +165,22 @@ def resolve_all(self, obj: Any) -> Any: return obj def _resolve_env(self, name: str) -> str: + self._check_env(name) source = self._env if self._env is not None else os.environ if name not in source: raise SecretRefError(f"env var {name!r} is not set") return source[name] def _resolve_file(self, path: str) -> str: + resolved = self._file_target(path) + try: + return Path(resolved).read_text(encoding="utf-8") + # ValueError: an embedded NUL, or a file that is not UTF-8. + except (OSError, ValueError) as error: + raise SecretRefError(f"cannot read {path!r}: {error}") from error + + def _file_target(self, path: str) -> str: + """The real path a ``file://`` target names, refused if it is out of bounds.""" if _DRIVE_URL_PATH.match(path): path = path[1:] # file:///C:/x names C:/x, not the drive-relative /C:/x if "\0" in path: @@ -128,11 +194,13 @@ def _resolve_file(self, path: str) -> str: resolved = os.path.realpath(os.path.join(base, path)) if not _is_within(base, resolved): raise SecretRefError(f"path escapes base dir: {path!r}") - try: - return Path(resolved).read_text(encoding="utf-8") - # ValueError: an embedded NUL, or a file that is not UTF-8. - except (OSError, ValueError) as error: - raise SecretRefError(f"cannot read {path!r}: {error}") from error + if self._path_policy is not None and self._path_policy.enabled: + from je_auto_control.utils.path_guard.path_guard import PathNotAllowedError + try: + self._path_policy.validate(resolved, operation="file:// reference") + except PathNotAllowedError as error: + raise SecretRefError(str(error)) from error + return resolved def _resolve_secret(self, name: str) -> str: resolver = self._secret_resolver diff --git a/test/unit_test/headless/test_file_boundary_policy.py b/test/unit_test/headless/test_file_boundary_policy.py new file mode 100644 index 000000000..3d3332b10 --- /dev/null +++ b/test/unit_test/headless/test_file_boundary_policy.py @@ -0,0 +1,546 @@ +"""File boundaries: MCP tool paths, value references and viewer downloads. + +Three places took a path from the other side of a trust boundary and used it +as given. An MCP tool's file argument reached any file on the machine, even in +read-only mode (``ac_load_dotenv`` returned any file as KEY=VALUE). +``ac_resolve_ref`` read any environment variable and any file. And a remote +desktop viewer wrote a host-pushed file wherever the host said. + +The first two are opt-in — a server nobody configured must behave as before — +so each has a test for the unconfigured case as well as the confined one. +""" +import json +import os +import sys +from pathlib import Path +from typing import Any, Dict, Iterator, List, Optional, Tuple + +import pytest + +from je_auto_control.utils.mcp_server._argument_policy import ArgumentPolicy +from je_auto_control.utils.mcp_server.server import MCPServer +from je_auto_control.utils.mcp_server.tools import MCPTool, build_default_tool_registry +from je_auto_control.utils.path_guard import PathNotAllowedError, PathPolicy +from je_auto_control.utils.path_guard.policy import ( + MCP_CLIENT_ROOTS_ENV, MCP_PATH_ROOTS_ENV, +) +from je_auto_control.utils.remote_desktop.file_transfer import ( + DOWNLOAD_DIR_ENV, FileReceiver, FileTransferError, confine_destination, + default_download_dir, encode_begin, encode_chunk, encode_end, new_transfer_id, +) +from je_auto_control.utils.secret_ref import ( + MCP_ENV_REF_ALLOW_ENV, RefResolver, SecretRefError, env_allowlist_from_env, +) + +_POLICY_ENV = (MCP_PATH_ROOTS_ENV, MCP_CLIENT_ROOTS_ENV, MCP_ENV_REF_ALLOW_ENV) + + +@pytest.fixture(autouse=True) +def _no_ambient_policy(monkeypatch): + """A developer's own environment must not configure the servers under test.""" + for name in _POLICY_ENV + (DOWNLOAD_DIR_ENV,): + monkeypatch.delenv(name, raising=False) + + +@pytest.fixture(scope="module") +def registry() -> Dict[str, MCPTool]: + return {tool.name: tool + for tool in build_default_tool_registry(read_only=False, aliases=False)} + + +def _path_pointers(schema: Dict[str, Any], prefix: str = "") -> Iterator[str]: + """Every place in ``schema`` annotated ``"format": "path"``.""" + if schema.get("format") == "path": + yield prefix + for key, child in (schema.get("properties") or {}).items(): + yield from _path_pointers(child, f"{prefix}.{key}" if prefix else key) + for key in ("items", "additionalProperties"): + if isinstance(schema.get(key), dict): + yield from _path_pointers(schema[key], f"{prefix}[]") + + +def _call(server: MCPServer, name: str, arguments: Dict[str, Any]) -> Dict[str, Any]: + line = server.handle_line(json.dumps({ + "jsonrpc": "2.0", "id": 1, "method": "tools/call", + "params": {"name": name, "arguments": arguments}, + })) + return json.loads(line)["result"] + + +def _text(result: Dict[str, Any]) -> str: + return "".join(block.get("text", "") for block in result["content"]) + + +def _server(names: List[str], *, read_only: bool = False) -> MCPServer: + tools = [tool for tool in build_default_tool_registry(read_only=read_only, aliases=False) + if tool.name in names] + assert {tool.name for tool in tools} == set(names) + return MCPServer(tools=tools) + + +def _symlink(link: Path, target: Path) -> None: + """Link ``link`` to ``target``; a directory falls back to an NTFS junction.""" + try: + link.symlink_to(target, target_is_directory=target.is_dir()) + except (OSError, NotImplementedError): + if sys.platform != "win32" or not target.is_dir(): + pytest.skip("this account cannot create symlinks") + import _winapi + _winapi.CreateJunction(str(target), str(link)) + + +# --- which arguments are paths -------------------------------------------- + +@pytest.mark.parametrize("name, pointer", [ + ("ac_load_dotenv", "path"), ("ac_read_document", "path"), + ("ac_extract_pdf_text", "path"), ("ac_sql_query", "database"), + ("ac_read_action_file", "file_path"), ("ac_screenshot", "file_path"), + ("ac_queue_stats", "db"), ("ac_match_template", "template"), + ("ac_ssim_compare", "reference"), ("ac_anchor_locate", "anchor.template_path"), + ("ac_set_clipboard_files", "paths[]"), ("ac_build_provenance", "paths[]"), + ("ac_verify_provenance", "files[]"), ("ac_load_data", "source.path"), + ("ac_write_step_video", "steps[].image"), ("ac_send_email", "message.attachments[]"), + ("ac_generate_code", "source"), +]) +def test_file_arguments_are_marked_as_paths(registry, name, pointer): + assert pointer in set(_path_pointers(registry[name].input_schema)) + + +@pytest.mark.parametrize("name, argument", [ + ("ac_json_query", "path"), # a JSONPath expression + ("ac_handle_file_dialog", "path"), # keystrokes typed into another app's dialog + ("ac_queue_complete", "output"), # the work item's result + ("ac_queue_add", "reference"), # a dedupe key + ("ac_cua_command", "source"), # anthropic / openai / canonical + ("ac_generate_sbom", "root"), # a distribution name + ("ac_gettext_translate", "po"), # the .po text itself + ("ac_perceptual_diff", "min_area"), +]) +def test_semantic_path_fields_only(registry, name, argument): + """The annotation follows meaning: a property merely called ``path`` is left alone.""" + assert argument not in set(_path_pointers(registry[name].input_schema)) + + +def test_aliases_carry_the_same_annotations(): + tools = {tool.name: tool for tool in build_default_tool_registry(aliases=True)} + assert "file_path" in set(_path_pointers(tools["screenshot"].input_schema)) + + +# --- PathPolicy ----------------------------------------------------------- + +def test_policy_without_roots_restricts_nothing(tmp_path): + policy = PathPolicy() + assert not policy.enabled + assert policy.validate(tmp_path / "x", operation="t") == Path(os.path.realpath(tmp_path / "x")) + + +def test_policy_accepts_inside_and_rejects_outside(tmp_path): + root = tmp_path / "root" + root.mkdir() + policy = PathPolicy([root]) + assert policy.validate(root / "a" / "b.txt", operation="t") == \ + Path(os.path.realpath(root / "a" / "b.txt")) + assert policy.validate(root, operation="t") == Path(os.path.realpath(root)) + with pytest.raises(PathNotAllowedError, match="tool x"): + policy.validate(tmp_path / "other.txt", operation="tool x") + + +@pytest.mark.parametrize("escape", ["..", "sub/../../outside.txt", "./../x"]) +def test_policy_rejects_dot_dot_escape(tmp_path, escape): + root = tmp_path / "root" + root.mkdir() + with pytest.raises(PathNotAllowedError): + PathPolicy([root]).validate(str(root / escape), operation="t") + + +def test_symlink_escape_is_rejected(tmp_path): + root = tmp_path / "root" + outside = tmp_path / "outside" + root.mkdir() + outside.mkdir() + (outside / "secret.txt").write_text("s", encoding="utf-8") + _symlink(root / "link", outside) + policy = PathPolicy([root]) + with pytest.raises(PathNotAllowedError): + policy.validate(root / "link" / "secret.txt", operation="t") + _symlink(root / "file_link", outside / "secret.txt") + with pytest.raises(PathNotAllowedError): + policy.validate(root / "file_link", operation="t") + + +@pytest.mark.skipif(sys.platform != "win32", reason="drive and UNC paths are Windows forms") +@pytest.mark.parametrize("path", [ + r"\\server\share\x.txt", r"\\?\UNC\server\share\x.txt", r"\\.\NUL", "NUL", + "Z:\\x.txt", "Z:x.txt", +]) +def test_drive_and_unc_escapes_are_rejected(tmp_path, monkeypatch, path): + root = tmp_path / "root" + root.mkdir() + monkeypatch.chdir(root) + with pytest.raises(PathNotAllowedError): + PathPolicy([root]).validate(path, operation="t") + + +def test_tilde_must_be_inside_both_expanded_and_literal(tmp_path, monkeypatch): + home = tmp_path / "home" + cwd = tmp_path / "cwd" + home.mkdir() + cwd.mkdir() + monkeypatch.setenv("HOME", str(home)) + monkeypatch.setenv("USERPROFILE", str(home)) + monkeypatch.chdir(cwd) + with pytest.raises(PathNotAllowedError): # a handler that expands it leaves the root + PathPolicy([cwd]).validate("~/x.txt", operation="t") + with pytest.raises(PathNotAllowedError): # a handler that does not leaves it too + PathPolicy([home]).validate("~/x.txt", operation="t") + assert PathPolicy([home, cwd]).validate("~/x.txt", operation="t") == \ + Path(os.path.realpath(home / "x.txt")) + + +@pytest.mark.parametrize("bad", ["", "a\x00b"]) +def test_policy_rejects_malformed_paths(tmp_path, bad): + with pytest.raises(PathNotAllowedError): + PathPolicy([tmp_path]).validate(bad, operation="t") + + +def test_policy_from_env(tmp_path): + first, second = tmp_path / "a", tmp_path / "b" + assert not PathPolicy.from_env({}).enabled + policy = PathPolicy.from_env({MCP_PATH_ROOTS_ENV: f"{first}{os.pathsep} {second} {os.pathsep}"}) + assert policy.roots() == (Path(os.path.realpath(first)), Path(os.path.realpath(second))) + assert not policy.use_client_roots + assert PathPolicy.from_env({MCP_CLIENT_ROOTS_ENV: "1"}).use_client_roots + assert not PathPolicy.from_env({MCP_CLIENT_ROOTS_ENV: "0"}).enabled + + +def test_client_roots_need_the_opt_in_and_fail_closed_until_known(tmp_path): + ignored = PathPolicy() + ignored.set_client_roots([tmp_path]) + assert ignored.roots() == () and not ignored.enabled + + policy = PathPolicy(use_client_roots=True) + assert policy.enabled + with pytest.raises(PathNotAllowedError, match="none known yet"): + policy.validate(tmp_path / "x", operation="t") + policy.set_client_roots([tmp_path]) + assert policy.validate(tmp_path / "x", operation="t") + policy.set_client_roots([]) + with pytest.raises(PathNotAllowedError): + policy.validate(tmp_path / "x", operation="t") + + +# --- the walk over a tool's arguments -------------------------------------- + +_SCHEMA = {"type": "object", "properties": { + "path": {"type": "string", "format": "path"}, + "expr": {"type": "string"}, + "many": {"type": "array", "items": {"format": "path"}}, + "spec": {"type": "object", "properties": {"path": {"format": "path"}}}, + "files": {"type": "object", "additionalProperties": {"format": "path"}}, + "source": {"type": ["array", "string"], "format": "path"}, +}} + + +def test_walk_canonicalises_every_marked_path_and_nothing_else(tmp_path): + policy = ArgumentPolicy(PathPolicy([tmp_path])) + inside = str(tmp_path / "sub" / ".." / "a.txt") + real = os.path.realpath(tmp_path / "a.txt") + out = policy.apply("tool", _SCHEMA, { + "path": inside, "expr": "../../$.a", "many": [inside], "spec": {"path": inside, "kind": "csv"}, + "files": {"x": inside}, "source": [["AC_noop", {"path": "/elsewhere"}]], + }) + assert out == {"path": real, "expr": "../../$.a", "many": [real], + "spec": {"path": real, "kind": "csv"}, "files": {"x": real}, + "source": [["AC_noop", {"path": "/elsewhere"}]]} + + +@pytest.mark.parametrize("arguments, where", [ + ({"path": "OUT"}, "tool $.path"), ({"many": ["OUT"]}, r"tool \$\.many\[0\]"), + ({"spec": {"path": "OUT"}}, "tool $.spec.path"), ({"files": {"x": "OUT"}}, "tool $.files.x"), + ({"source": "OUT"}, "tool $.source"), +]) +def test_walk_rejects_an_outside_path_wherever_it_sits(tmp_path, arguments, where): + root = tmp_path / "root" + root.mkdir() + outside = str(tmp_path / "outside.txt") + text = json.dumps(arguments).replace("OUT", outside.replace("\\", "\\\\")) + with pytest.raises(PathNotAllowedError, match=where.replace("$", r"\$") if "\\" not in where + else where): + ArgumentPolicy(PathPolicy([root])).apply("tool", _SCHEMA, json.loads(text)) + + +def test_walk_leaves_an_empty_path_to_the_handler(tmp_path): + arguments = {"path": ""} + assert ArgumentPolicy(PathPolicy([tmp_path])).apply("tool", _SCHEMA, arguments) == arguments + + +def test_unconfigured_walk_returns_the_arguments_untouched(): + arguments = {"path": "../x"} + assert ArgumentPolicy().apply("tool", _SCHEMA, arguments) is arguments + + +# --- through the server ---------------------------------------------------- + +def _dotenv(directory: Path) -> Path: + directory.mkdir(parents=True, exist_ok=True) + target = directory / "app.env" + target.write_text("TOKEN=abc\n", encoding="utf-8") + return target + + +def test_unconfigured_server_reads_anywhere_even_read_only(tmp_path): + """The default is unchanged: no roots, no restriction, read-only included.""" + target = _dotenv(tmp_path / "elsewhere") + server = _server(["ac_load_dotenv"], read_only=True) + assert not server.argument_policy.enabled + result = _call(server, "ac_load_dotenv", {"path": str(target)}) + assert result["isError"] is False and "abc" in _text(result) + + +def test_configured_roots_reject_a_path_outside_them(tmp_path, monkeypatch): + root = tmp_path / "root" + inside = _dotenv(root) + outside = _dotenv(tmp_path / "elsewhere") + monkeypatch.setenv(MCP_PATH_ROOTS_ENV, str(root)) + server = _server(["ac_load_dotenv"], read_only=True) + + escaped = _call(server, "ac_load_dotenv", {"path": str(outside)}) + assert escaped["isError"] is True + assert "Invalid arguments for ac_load_dotenv" in _text(escaped) + assert "abc" not in _text(escaped) + dotted = _call(server, "ac_load_dotenv", {"path": str(root / ".." / "elsewhere" / "app.env")}) + assert dotted["isError"] is True + + allowed = _call(server, "ac_load_dotenv", {"path": str(inside)}) + assert allowed["isError"] is False and "abc" in _text(allowed) + + +def test_configured_roots_leave_a_json_path_alone(tmp_path, monkeypatch): + monkeypatch.setenv(MCP_PATH_ROOTS_ENV, str(tmp_path)) + server = _server(["ac_json_query"]) + result = _call(server, "ac_json_query", {"data": {"a": {"b": 7}}, "path": "$.a.b"}) + assert result["isError"] is False and "7" in _text(result) + + +def test_server_rejects_a_symlink_out_of_the_roots(tmp_path, monkeypatch): + root = tmp_path / "root" + root.mkdir() + _dotenv(tmp_path / "elsewhere") + _symlink(root / "link", tmp_path / "elsewhere") + monkeypatch.setenv(MCP_PATH_ROOTS_ENV, str(root)) + result = _call(_server(["ac_load_dotenv"]), "ac_load_dotenv", + {"path": str(root / "link" / "app.env")}) + assert result["isError"] is True and "abc" not in _text(result) + + +def test_roots_list_feeds_the_policy_only_when_opted_in(tmp_path, monkeypatch): + workspace = tmp_path / "ws" + inside = _dotenv(workspace) + outside = _dotenv(tmp_path / "elsewhere") + uri = "file:///" + str(workspace).replace("\\", "/").lstrip("/") + reply = {"roots": [{"uri": uri}, {"uri": "https://example.invalid/x"}, "junk"]} + + plain = _server(["ac_load_dotenv"]) + monkeypatch.setattr(plain, "_send_outbound_request", lambda *args, **kwargs: reply) + plain.refresh_roots() + assert _call(plain, "ac_load_dotenv", {"path": str(outside)})["isError"] is False + + monkeypatch.setenv(MCP_CLIENT_ROOTS_ENV, "true") + server = _server(["ac_load_dotenv"]) + assert _call(server, "ac_load_dotenv", {"path": str(inside)})["isError"] is True + monkeypatch.setattr(server, "_send_outbound_request", lambda *args, **kwargs: reply) + server.refresh_roots() + assert server.argument_policy.path_policy.roots() == (Path(os.path.realpath(workspace)),) + assert _call(server, "ac_load_dotenv", {"path": str(inside)})["isError"] is False + assert _call(server, "ac_load_dotenv", {"path": str(outside)})["isError"] is True + + +def test_env_roots_and_client_roots_add_up(tmp_path, monkeypatch): + configured, reported = tmp_path / "configured", tmp_path / "reported" + monkeypatch.setenv(MCP_PATH_ROOTS_ENV, str(configured)) + monkeypatch.setenv(MCP_CLIENT_ROOTS_ENV, "1") + policy = ArgumentPolicy.from_env().path_policy + policy.set_client_roots([reported]) + assert policy.validate(configured / "a", operation="t") + assert policy.validate(reported / "a", operation="t") + + +# --- value references ------------------------------------------------------ + +def test_env_ref_allowlist(): + env = {"APP_URL": "u", "APP_KEY": "k", "ANTHROPIC_API_KEY": "sk"} + assert RefResolver(env=env).resolve("env://ANTHROPIC_API_KEY") == "sk" + limited = RefResolver(env=env, env_allowlist=["APP_*", "HOME"]) + assert limited.resolve("env://APP_URL") == "u" + with pytest.raises(SecretRefError, match="allowlist"): + limited.resolve("env://ANTHROPIC_API_KEY") + with pytest.raises(SecretRefError, match="allowlist"): + limited.check_all({"a": [{"b": "env://ANTHROPIC_API_KEY"}]}) + limited.check_all({"a": ["env://APP_KEY", "plain", 3, "secret://x"]}) + with pytest.raises(SecretRefError): + RefResolver(env=env, env_allowlist=[]).resolve("env://APP_URL") + + +@pytest.mark.parametrize("raw, expected", [ + (None, None), ("", None), (" ", None), ("A, B_*", ("A", "B_*")), (",", ()), +]) +def test_env_allowlist_from_env(raw: Optional[str], expected: Optional[Tuple[str, ...]]): + environ = {} if raw is None else {MCP_ENV_REF_ALLOW_ENV: raw} + assert env_allowlist_from_env(environ) == expected + + +def test_file_ref_follows_the_path_policy(tmp_path): + root = tmp_path / "root" + root.mkdir() + (root / "ok.txt").write_text("fine", encoding="utf-8") + (tmp_path / "secret.txt").write_text("no", encoding="utf-8") + resolver = RefResolver(path_policy=PathPolicy([root])) + assert resolver.resolve("file://" + str(root / "ok.txt").replace("\\", "/")) == "fine" + outside = "file://" + str(tmp_path / "secret.txt").replace("\\", "/") + with pytest.raises(SecretRefError, match="outside the allowed roots"): + resolver.resolve(outside) + with pytest.raises(SecretRefError): + resolver.check_all([outside]) + assert RefResolver(path_policy=PathPolicy()).resolve(outside) == "no" + + +def test_unconfigured_server_resolves_any_env_ref(monkeypatch): + monkeypatch.setenv("AC_TEST_BOUNDARY_VALUE", "visible") + result = _call(_server(["ac_resolve_ref"]), "ac_resolve_ref", + {"ref": "env://AC_TEST_BOUNDARY_VALUE"}) + assert result["isError"] is False and "visible" in _text(result) + + +def test_server_applies_the_env_allowlist_to_both_ref_tools(monkeypatch): + monkeypatch.setenv("AC_TEST_BOUNDARY_VALUE", "visible") + monkeypatch.setenv("AC_TEST_BOUNDARY_SECRET", "hidden") + monkeypatch.setenv(MCP_ENV_REF_ALLOW_ENV, "AC_TEST_BOUNDARY_VALUE") + server = _server(["ac_resolve_ref", "ac_resolve_refs"]) + allowed = _call(server, "ac_resolve_ref", {"ref": "env://AC_TEST_BOUNDARY_VALUE"}) + assert allowed["isError"] is False and "visible" in _text(allowed) + for name, arguments in [ + ("ac_resolve_ref", {"ref": "env://AC_TEST_BOUNDARY_SECRET"}), + ("ac_resolve_refs", {"obj": {"k": ["env://AC_TEST_BOUNDARY_SECRET"]}}), + ]: + refused = _call(server, name, arguments) + assert refused["isError"] is True + assert "allowlist" in _text(refused) and "hidden" not in _text(refused) + + +def test_server_applies_the_roots_to_file_refs(tmp_path, monkeypatch): + root = tmp_path / "root" + root.mkdir() + (root / "ok.txt").write_text("fine", encoding="utf-8") + (tmp_path / "secret.txt").write_text("leak", encoding="utf-8") + monkeypatch.setenv(MCP_PATH_ROOTS_ENV, str(root)) + server = _server(["ac_resolve_ref", "ac_resolve_refs"]) + inside = "file://" + str(root / "ok.txt").replace("\\", "/") + outside = "file://" + str(tmp_path / "secret.txt").replace("\\", "/") + assert "fine" in _text(_call(server, "ac_resolve_ref", {"ref": inside})) + for name, arguments in [("ac_resolve_ref", {"ref": outside}), + ("ac_resolve_refs", {"obj": {"k": outside}})]: + refused = _call(server, name, arguments) + assert refused["isError"] is True and "leak" not in _text(refused) + + +# --- viewer downloads ------------------------------------------------------ + +def _receive(receiver: FileReceiver, dest_path: str, data: bytes = b"payload") -> None: + transfer_id = new_transfer_id() + receiver.handle_begin(encode_begin(transfer_id, dest_path, len(data))) + receiver.handle_chunk(encode_chunk(transfer_id, data)) + receiver.handle_end(encode_end(transfer_id)) + + +def _confined(base: Path) -> Tuple[FileReceiver, List[Tuple[bool, Optional[str], str]]]: + outcomes: List[Tuple[bool, Optional[str], str]] = [] + receiver = FileReceiver( + on_complete=lambda _tid, ok, error, dest: outcomes.append((ok, error, dest)), + base_dir=base) + return receiver, outcomes + + +def test_viewer_file_stays_in_download_root(tmp_path): + download_root = tmp_path / "downloads" + receiver, outcomes = _confined(download_root) + _receive(receiver, "reports/2026\\q3/out.bin") + received_path = Path(outcomes[-1][2]) + assert outcomes[-1][:2] == (True, None) + assert received_path.is_relative_to(Path(os.path.realpath(download_root))) + assert received_path == Path(os.path.realpath(download_root / "reports" / "2026" / "q3" / "out.bin")) + assert received_path.read_bytes() == b"payload" + assert not list(download_root.rglob("*.part")) + + +_ESCAPES = ["/tmp/from_host.bin", "/etc/cron.d/x", "C:\\Windows\\x.dll", "C:x.dll", "c:/x", + "\\\\server\\share\\x", "//server/share/x", "\\x.bin", "../x.bin", "a/../../x.bin", + "a\\..\\..\\x.bin", "..", ".", "", "a/..", "./"] + + +@pytest.mark.parametrize("dest_path", [path for path in _ESCAPES if path]) +def test_viewer_refuses_a_destination_outside_the_download_root(tmp_path, dest_path): + download_root = tmp_path / "downloads" + receiver, outcomes = _confined(download_root) + _receive(receiver, dest_path) + assert len(outcomes) == 1 and outcomes[0][0] is False + assert outcomes[0][2] == dest_path + written = [path for path in tmp_path.rglob("*") if path.is_file()] + assert written == [] + + +@pytest.mark.parametrize("dest_path", _ESCAPES) +def test_confine_destination_raises_for_every_escape(tmp_path, dest_path): + with pytest.raises(FileTransferError): + confine_destination(tmp_path, dest_path) + + +def test_viewer_refuses_a_symlink_out_of_the_download_root(tmp_path): + download_root = tmp_path / "downloads" + outside = tmp_path / "outside" + download_root.mkdir() + outside.mkdir() + (outside / "victim.txt").write_text("original", encoding="utf-8") + _symlink(download_root / "dir_link", outside) + receiver, outcomes = _confined(download_root) + _receive(receiver, "dir_link/new.bin") + _receive(receiver, "dir_link/victim.txt") + assert [outcome[0] for outcome in outcomes] == [False, False] + assert (outside / "victim.txt").read_text(encoding="utf-8") == "original" + assert sorted(path.name for path in outside.iterdir()) == ["victim.txt"] + _symlink(download_root / "file_link", outside / "victim.txt") + _receive(receiver, "file_link") + assert outcomes[-1][0] is False + assert (outside / "victim.txt").read_text(encoding="utf-8") == "original" + + +def test_receiver_without_base_dir_keeps_the_host_side_behaviour(tmp_path): + """The host trusts its authenticated viewers: an absolute path is honoured.""" + outcomes: List[Tuple[bool, Optional[str], str]] = [] + receiver = FileReceiver(on_complete=lambda _t, ok, err, dst: outcomes.append((ok, err, dst))) + target = tmp_path / "anywhere" / "a.bin" + _receive(receiver, str(target)) + assert outcomes == [(True, None, str(target))] + assert target.read_bytes() == b"payload" + + +def test_default_download_dir(tmp_path, monkeypatch): + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setenv("USERPROFILE", str(tmp_path)) + assert default_download_dir() == tmp_path / "Downloads" / "AutoControl" + monkeypatch.setenv(DOWNLOAD_DIR_ENV, str(tmp_path / "inbox")) + assert default_download_dir() == tmp_path / "inbox" + + +def test_viewer_default_receiver_is_confined_and_host_default_is_not(tmp_path, monkeypatch): + from je_auto_control.utils.remote_desktop import RemoteDesktopHost, RemoteDesktopViewer + monkeypatch.setenv(DOWNLOAD_DIR_ENV, str(tmp_path / "inbox")) + viewer = RemoteDesktopViewer(host="127.0.0.1", port=1, token="t") + receiver = viewer._ensure_file_receiver() + _receive(receiver, str(tmp_path / "absolute.bin")) + assert not (tmp_path / "absolute.bin").exists() + _receive(receiver, "kept.bin") + assert (tmp_path / "inbox" / "kept.bin").read_bytes() == b"payload" + + host = RemoteDesktopHost(token="t", bind="127.0.0.1", port=0) + _receive(host._ensure_file_receiver(), str(tmp_path / "host_side.bin")) + assert (tmp_path / "host_side.bin").read_bytes() == b"payload" From 82fb01fcc041f1c6fa234288ca46426a64f5785a Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:03:43 +0800 Subject: [PATCH 19/26] Mark the tab registry's import as table-driven for the static analyser --- architecture_explore.md | 8 ++++---- je_auto_control/gui/tab_registry.py | 3 ++- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/architecture_explore.md b/architecture_explore.md index e466e94c7..53d2b9873 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -20,7 +20,7 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | | Python 模組總數（含周邊子專案） | 1,068 | -| 程式碼總行數 | 158,285 | +| 程式碼總行數 | 158,286 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 778 | | 套件門面 `__all__` 公開名稱數 | 1,244 | @@ -883,7 +883,7 @@ GUI 是**選用 extra**（`pip install je_auto_control[gui]`，PySide6 + qt-mate | `gui/__init__.py` | 25 | `start_autocontrol_gui()`：**唯一**會延遲匯入 PySide6 的地方，維持頂層套件 Qt-free。 | | `main_window.py` | 379 | `QMainWindow`：選單列（File／Actions／View／…）、左側導覽面板 dock（`Ctrl+K` 搜尋、`Ctrl+B` 收合）、即時語言切換、字級預設、深色／淺色主題（`theme.py` 的 token，不再用 qt-material）。分頁分為 core／editing／detection／automation／system 五類。 | | `main_widget.py` | 368 | 工作區：擁有 `QTabWidget`，依 `tab_registry.TAB_SPECS` 註冊 48 個分頁，並暴露 show/hide/activate/list API 給選單列與導覽面板。只有預設開啟的三個分頁與自己的 mixin 表單在啟動時建立，其餘第一次開啟才匯入模組、建立 widget。核心分頁在 `_own_tab_builders` 宣告 `(label_key, handler)` 動作對。 | -| `tab_registry.py` | 123 | 分頁表：每個分頁一筆 `TabSpec`（鍵、標題鍵、分類、模組與類別名），`TabEntry` 在第一次存取 `widget` 時才呼叫 factory。不匯入 Qt。 | +| `tab_registry.py` | 124 | 分頁表：每個分頁一筆 `TabSpec`（鍵、標題鍵、分類、模組與類別名），`TabEntry` 在第一次存取 `widget` 時才呼叫 factory。不匯入 Qt。 | | `navigation.py` | 200 | `NavigationPanel`：搜尋框 + 依分類的功能樹，列出每個已註冊分頁（開啟中的以粗體標示），只回報被選的鍵，開啟分頁仍由視窗負責。 | | `theme.py` | 178 | 設計 token（`ThemeTokens`：顏色、圓角、間距、字族）、深色與淺色兩組值、由 token 產生的樣式表與對應的 `QPalette`；不載入圖檔或字型檔。 | | `_auto_click_tab.py` | 291 | 自動點擊分頁的 mixin 建構器。 | @@ -1079,7 +1079,7 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | 層／子系統 | 檔案數 | 行數 | | --- | ---: | ---: | -| `gui/` | 98 | 28,397 | +| `gui/` | 98 | 28,398 | | `utils/mcp_server/` | 35 | 18,898 | | `utils/remote_desktop/` | 56 | 13,014 | | `utils/executor/` | 8 | 9,606 | @@ -1100,5 +1100,5 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 852 | | 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 682 | 57,156 | -| **總計** | **1,062** | **158,220** | +| **總計** | **1,062** | **158,221** | diff --git a/je_auto_control/gui/tab_registry.py b/je_auto_control/gui/tab_registry.py index 424b18266..4562045d4 100644 --- a/je_auto_control/gui/tab_registry.py +++ b/je_auto_control/gui/tab_registry.py @@ -30,7 +30,8 @@ class TabSpec: def lazy_factory(module: str, class_name: str) -> WidgetFactory: """Return a factory that imports ``module`` and builds ``class_name``.""" def build() -> Any: - return getattr(import_module(module), class_name)() + # reason: both names come from the TAB_SPECS table below, never from user input. + return getattr(import_module(module), class_name)() # nosemgrep: python.lang.security.audit.non-literal-import.non-literal-import return build From fae33465f0ae95c5277056ec48e1fbaae8512032 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:03:56 +0800 Subject: [PATCH 20/26] Authorise REST routes and MCP tools by role when a user store is configured utils/rbac had users, roles and token checks that nothing consulted: both servers compared one shared token, so every caller could do everything and the audit trail could not say who had. RBAC is now opt-in through JE_AUTOCONTROL_RBAC_USERS or a user_store argument; without it the shared token behaves exactly as before. With it a token is one user's, each route, tool and privileged AC_* command needs a capability of that user's role, tools/list agrees with tools/call, and audit entries carry the user id. Signing an action file gets its own capability so that running actions is not approving them. --- .../Eng/doc/mcp_server/mcp_server_doc.rst | 26 ++ .../operations_layer/operations_layer_doc.rst | 84 ++++ .../Zh/doc/mcp_server/mcp_server_doc.rst | 21 + .../operations_layer/operations_layer_doc.rst | 74 +++ je_auto_control/__init__.py | 8 + .../utils/executor/action_executor.py | 2 + .../utils/executor/flow_control.py | 9 +- je_auto_control/utils/mcp_server/_authz.py | 108 +++++ je_auto_control/utils/mcp_server/audit.py | 22 +- .../utils/mcp_server/http_transport.py | 66 +-- je_auto_control/utils/mcp_server/server.py | 4 +- je_auto_control/utils/rbac/__init__.py | 26 +- je_auto_control/utils/rbac/authorization.py | 106 +++++ je_auto_control/utils/rbac/policy.py | 208 +++++++++ je_auto_control/utils/rbac/users.py | 92 ++-- je_auto_control/utils/rest_api/__main__.py | 14 +- je_auto_control/utils/rest_api/rest_auth.py | 75 ++- .../utils/rest_api/rest_openapi.py | 14 +- .../utils/rest_api/rest_registry.py | 6 +- je_auto_control/utils/rest_api/rest_server.py | 157 +++++-- .../headless/test_r3_net_socket_rest.py | 4 + .../headless/test_rbac_server_wiring.py | 427 ++++++++++++++++++ 22 files changed, 1430 insertions(+), 123 deletions(-) create mode 100644 je_auto_control/utils/mcp_server/_authz.py create mode 100644 je_auto_control/utils/rbac/authorization.py create mode 100644 je_auto_control/utils/rbac/policy.py create mode 100644 test/unit_test/headless/test_rbac_server_wiring.py diff --git a/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst b/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst index bf0622cda..b7f355494 100644 --- a/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst +++ b/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst @@ -289,6 +289,32 @@ box), start the same dispatcher behind HTTP: Bearer token can also come from ``JE_AUTOCONTROL_MCP_TOKEN``. +**Roles (opt-in RBAC).** Set ``JE_AUTOCONTROL_RBAC_USERS`` to a user store +file, or pass ``user_store=UserStore(path)`` to ``start_mcp_http_server`` / +``HttpMCPServer``, and the HTTP transport authenticates each request as one +user of that store instead of comparing a shared token (``auth_token`` and +``JE_AUTOCONTROL_MCP_TOKEN`` are then not accepted, and a bearer token is +always required). The store, its roles and how to create users are described +under *Roles* in the operations-layer REST API chapter; both servers can +share one file. Without a store nothing changes. The stdio transport has no +bearer token and is never subject to RBAC. + +- A tool needs ``read_screen`` when it is marked ``readOnlyHint`` and + ``drive_input`` otherwise, so a ``viewer`` gets exactly the read-only + tools and an ``operator`` the rest. ``ac_remote_host_start`` / ``_stop``, + ``ac_usb_acl_add`` / ``_remove`` / ``_set_default``, + ``ac_usb_passthrough_enable``, ``ac_egress_allow`` / ``_reset`` and + ``ac_load_plugins`` need ``manage_hosts`` (``admin``). +- ``tools/list`` returns only the tools the caller may call, and + ``tools/call`` on any other answers JSON-RPC error ``-32003`` + (``Forbidden: ...``, ``data.required_capability``) without running it. +- A tool that takes an action list (``ac_execute_actions`` and the like) is + refused the same way when the list contains a command the caller's role + does not grant, such as ``AC_sign_action_file`` for an operator. +- A token whose role the store does not define gets HTTP 403. +- Each audit line carries ``user_id`` and ``role``; a refused call is + recorded with ``"status": "denied"``. + Browser requests are refused unless they come from this machine: a request whose ``Origin`` header is not a loopback origin gets 403, and when the server is bound to loopback so does one whose ``Host`` header does not name diff --git a/docs/source/Eng/doc/operations_layer/operations_layer_doc.rst b/docs/source/Eng/doc/operations_layer/operations_layer_doc.rst index 45fea94da..7782d1108 100644 --- a/docs/source/Eng/doc/operations_layer/operations_layer_doc.rst +++ b/docs/source/Eng/doc/operations_layer/operations_layer_doc.rst @@ -134,6 +134,90 @@ CLI:: python -m je_auto_control.utils.rest_api --host 127.0.0.1 --port 9939 +Roles (opt-in RBAC) +------------------- + +By default every caller holds the one shared token and may use every +endpoint. Point the server at a user store and each caller gets a token of +their own and a role instead: + +- **Switching it on**: set ``JE_AUTOCONTROL_RBAC_USERS`` to the user store + file, pass ``--users `` to ``python -m je_auto_control.utils.rest_api``, + or pass ``user_store=UserStore(path)`` to ``RestApiServer`` / + ``start_rest_api_server``. Nothing else switches it on -- in particular a + ``~/.je_auto_control/users.json`` that merely exists does not. +- **With it on, the shared token is refused.** A bearer token must belong to + one user of the store; a store that is empty or unreadable admits nobody. +- **Without it, nothing changes**: the single shared token, no roles, no 403. + +Create the users from Python; the plain token is shown once and only its +hash is stored:: + + import je_auto_control as ac + + store = ac.UserStore("/etc/autocontrol/users.json") + token = store.add_user(user_id="alice", display_name="Alice", role="operator") + store.set_role("alice", "viewer") # roles: viewer / operator / admin + store.rotate_token("alice") + store.remove_user("alice") + +A running server re-reads the file when it changes, so a removed user, a +rotated token or a changed role applies to the next request. + +.. list-table:: + :header-rows: 1 + + * - Capability + - Roles + - Endpoints + * - ``read_screen`` + - viewer, operator, admin + - every ``GET`` except the two below, ``/metrics`` included + * - ``drive_input`` + - operator, admin + - ``POST /execute``, ``/execute_file``, ``/usb/loopback/open``, + ``/usb/remote/open`` + * - ``read_audit`` + - admin + - ``GET /audit/list``, ``/audit/verify`` + * - ``manage_hosts`` + - admin + - ``POST /config/export``, ``/config/import``, ``/usb/passthrough/enable``, + ``/usb/acl/add``, ``/usb/acl/remove``, ``/usb/acl/default``, and any + route that has not been given a capability + * - ``sign_actions`` + - admin + - the ``AC_sign_action_file`` command (see below) + +A role that lacks the capability gets ``403`` +``{"error": "forbidden", "required_capability": "...", "role": "..."}``. +The same value is published per operation as ``x-required-capability`` in +``/openapi.json``. + +Being allowed to ``POST /execute`` is being allowed to run actions, not every +command. Signing an action file (``sign_actions``), reading the audit log +(``read_audit``), and the commands that administer the host itself -- +``AC_admin_*``, starting or stopping the REST / MCP / remote-desktop / +webhook servers, ``AC_usb_acl_*``, ``AC_usb_passthrough_enable``, +``AC_config_import`` / ``AC_config_export``, ``AC_egress_allow`` / +``AC_egress_reset``, ``AC_load_plugins``, ``AC_add_package_to_executor``, +``AC_secret_init`` / ``set`` / ``remove`` / ``lock`` / ``unlock``, +``AC_audit_log_clear`` (all ``manage_hosts``) -- need their own capability +wherever they appear in the action list, however deeply nested. The request +is answered 403 with the ``command`` named before its first action runs, and +the executor refuses the same commands when they come from an action file. + +The roles are not a sandbox: an operator drives the real keyboard and can +launch programs, so whatever the desktop user could do by hand is within +reach. They protect the host's own privileged state. Work an operator defers +to another thread -- a scheduler job, a trigger, a hotkey binding -- runs +later without the caller's role attached. + +Every audit row of an RBAC request names the user: the ``viewer_id`` column +holds the user id and the detail reads +``POST /execute -> ok:200 user=alice role=operator`` (``forbidden:`` +for a refused one). + Endpoint surface ---------------- diff --git a/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst b/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst index 9ca641b7f..6a461427d 100644 --- a/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst +++ b/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst @@ -267,6 +267,27 @@ HTTP 傳輸(含 SSE / Auth / TLS) Bearer token 也可從 ``JE_AUTOCONTROL_MCP_TOKEN`` 環境變數讀取。 +**角色（選用的 RBAC）。** 把 ``JE_AUTOCONTROL_RBAC_USERS`` 設為使用者存放檔，或對 +``start_mcp_http_server``／``HttpMCPServer`` 傳 ``user_store=UserStore(path)``，HTTP +傳輸就會把每個請求驗證為存放檔中的某位使用者，而不是比對共用 token（此時 +``auth_token`` 與 ``JE_AUTOCONTROL_MCP_TOKEN`` 不再被接受，且一律需要 bearer token）。 +存放檔、角色與建立使用者的方式見維運層 REST API 章節的「角色」；兩個伺服器可以共用 +同一個檔案。沒有設定存放檔時行為完全不變。stdio 傳輸沒有 bearer token，不受 RBAC +約束。 + +- 工具標示 ``readOnlyHint`` 時需要 ``read_screen``，否則需要 ``drive_input``，所以 + ``viewer`` 拿到的正好是唯讀工具，``operator`` 再加上其餘工具。 + ``ac_remote_host_start``／``_stop``、``ac_usb_acl_add``／``_remove``／ + ``_set_default``、``ac_usb_passthrough_enable``、``ac_egress_allow``／``_reset`` 與 + ``ac_load_plugins`` 需要 ``manage_hosts``\ （``admin``）。 +- ``tools/list`` 只回呼叫者可以呼叫的工具；對其他工具 ``tools/call`` 會回 JSON-RPC + 錯誤 ``-32003``\ （``Forbidden: ...``、``data.required_capability``），且不會執行。 +- 接受動作清單的工具（``ac_execute_actions`` 等）在清單含有呼叫者角色沒有的指令時 + 同樣被拒絕，例如 operator 的 ``AC_sign_action_file``。 +- Token 的角色不在存放檔定義的角色之內時回 HTTP 403。 +- 每一行稽核紀錄都帶 ``user_id`` 與 ``role``；被拒絕的呼叫記為 + ``"status": "denied"``。 + 瀏覽器送來的請求只接受本機來源：``Origin`` 不是 loopback 的一律回 403；伺服器綁在 loopback 時，``Host`` 不是 loopback 名稱的也回 403（防 DNS rebinding）。非瀏覽器的 客戶端不送 ``Origin``，不受影響。要讓其他來源的瀏覽器客戶端連線，把完整來源列在 diff --git a/docs/source/Zh/doc/operations_layer/operations_layer_doc.rst b/docs/source/Zh/doc/operations_layer/operations_layer_doc.rst index 389175208..7fcd83e1c 100644 --- a/docs/source/Zh/doc/operations_layer/operations_layer_doc.rst +++ b/docs/source/Zh/doc/operations_layer/operations_layer_doc.rst @@ -125,6 +125,80 @@ CLI:: python -m je_auto_control.utils.rest_api --host 127.0.0.1 --port 9939 +角色（選用的 RBAC） +-------------------- + +預設情況下所有呼叫端共用同一個 token，可以使用所有端點。把伺服器指向一個使用者 +存放檔之後，每個呼叫端改為擁有自己的 token 與角色： + +- **啟用方式**：把 ``JE_AUTOCONTROL_RBAC_USERS`` 設為使用者存放檔路徑、對 + ``python -m je_auto_control.utils.rest_api`` 傳 ``--users <檔案>``，或對 + ``RestApiServer``／``start_rest_api_server`` 傳 ``user_store=UserStore(path)``。 + 除此之外不會自動啟用——單純存在 ``~/.je_auto_control/users.json`` 並不算。 +- **啟用後共用 token 一律被拒絕。** Bearer token 必須屬於存放檔中的某位使用者； + 存放檔是空的或讀不出來時，任何人都進不來。 +- **沒有啟用時完全不變**：單一共用 token、沒有角色、不會有 403。 + +使用者以 Python 建立；明文 token 只顯示一次，檔案裡只存雜湊:: + + import je_auto_control as ac + + store = ac.UserStore("/etc/autocontrol/users.json") + token = store.add_user(user_id="alice", display_name="Alice", role="operator") + store.set_role("alice", "viewer") # 角色：viewer / operator / admin + store.rotate_token("alice") + store.remove_user("alice") + +執行中的伺服器會在檔案變動時重新讀取，所以移除使用者、輪替 token 或改角色都在 +下一個請求生效。 + +.. list-table:: + :header-rows: 1 + + * - 能力 + - 角色 + - 端點 + * - ``read_screen`` + - viewer、operator、admin + - 除下列兩個之外的所有 ``GET``，包含 ``/metrics`` + * - ``drive_input`` + - operator、admin + - ``POST /execute``、``/execute_file``、``/usb/loopback/open``、 + ``/usb/remote/open`` + * - ``read_audit`` + - admin + - ``GET /audit/list``、``/audit/verify`` + * - ``manage_hosts`` + - admin + - ``POST /config/export``、``/config/import``、``/usb/passthrough/enable``、 + ``/usb/acl/add``、``/usb/acl/remove``、``/usb/acl/default``，以及任何 + 尚未指定能力的路由 + * - ``sign_actions`` + - admin + - ``AC_sign_action_file`` 指令（見下） + +角色沒有該能力時回 ``403`` +``{"error": "forbidden", "required_capability": "...", "role": "..."}``。 +同一個值也以 ``x-required-capability`` 列在 ``/openapi.json`` 的每個 operation 上。 + +可以 ``POST /execute`` 代表可以執行動作，不代表可以執行每一個指令。替動作檔簽章 +（``sign_actions``）、讀取稽核紀錄（``read_audit``），以及管理主機本身的指令—— +``AC_admin_*``、啟動或停止 REST／MCP／遠端桌面／webhook 伺服器、``AC_usb_acl_*``、 +``AC_usb_passthrough_enable``、``AC_config_import``／``AC_config_export``、 +``AC_egress_allow``／``AC_egress_reset``、``AC_load_plugins``、 +``AC_add_package_to_executor``、``AC_secret_init``／``set``／``remove``／``lock``／ +``unlock``、``AC_audit_log_clear``\ （皆為 ``manage_hosts``）——不論在動作清單裡 +巢狀多深，都需要各自的能力。請求會在第一個動作執行前就回 403 並指出 ``command``； +同樣的指令若來自動作檔，則由 executor 拒絕。 + +角色不是沙箱：operator 操作的是真實鍵盤，也能啟動程式，桌面使用者手動做得到的事 +它都做得到。角色保護的是主機自身的特權狀態。operator 交給其他執行緒延後執行的 +工作（排程工作、觸發器、熱鍵綁定）之後執行時不會帶著呼叫者的角色。 + +RBAC 請求的每一筆稽核紀錄都會記下使用者：``viewer_id`` 欄位存使用者 id，detail 為 +``POST /execute -> ok:200 user=alice role=operator``\ （被拒絕時為 +``forbidden:<能力>``）。 + 端點清單 -------- diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 0117874ed..36451cac0 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -1095,6 +1095,11 @@ from je_auto_control.utils.rest_api.rest_server import ( RestApiServer, start_rest_api_server, ) +# RBAC for the REST API and the MCP HTTP transport (opt-in) +from je_auto_control.utils.rbac import ( + AuthorizationContext, AuthorizationError, UserAuthError, UserRecord, + UserStore, authorization_scope, +) # Admin console (headless multi-host client) from je_auto_control.utils.admin import ( AdminConsoleClient, AdminHost, default_admin_console, @@ -1753,6 +1758,9 @@ def start_autocontrol_gui(*args, **kwargs): "register_plugin_commands", # REST API "RestApiServer", "start_rest_api_server", + # RBAC + "AuthorizationContext", "AuthorizationError", "UserAuthError", + "UserRecord", "UserStore", "authorization_scope", # Admin console "AdminConsoleClient", "AdminHost", "default_admin_console", # WebRTC inspector diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index b9a7bf614..21c85500d 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -68,6 +68,7 @@ from je_auto_control.utils.json.json_file import read_action_json from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.mcp_server.http_transport import start_mcp_http_server +from je_auto_control.utils.rbac.policy import authorize_command from je_auto_control.utils.mcp_server.server import start_mcp_stdio_server from je_auto_control.utils.package_manager.package_manager_class import package_manager from je_auto_control.utils.project.create_project_structure import create_project_dir @@ -8118,6 +8119,7 @@ def _execute_event(self, action: list) -> Any: Execute a single event """ name = action[0] + authorize_command(name) # RBAC; a no-op outside a server's user scope block_handler = self._block_commands.get(name) if block_handler is not None: args = action[1] if len(action) == 2 else {} diff --git a/je_auto_control/utils/executor/flow_control.py b/je_auto_control/utils/executor/flow_control.py index 9bc76e463..54e2a5fe5 100644 --- a/je_auto_control/utils/executor/flow_control.py +++ b/je_auto_control/utils/executor/flow_control.py @@ -22,6 +22,7 @@ exec_sql_to_var, exec_transform_var, ) from je_auto_control.utils.logging.logging_instance import autocontrol_logger +from je_auto_control.utils.rbac.authorization import authorization_scope, current_authorization from je_auto_control.wrapper.auto_control_image import locate_image_center from je_auto_control.wrapper.auto_control_screen import get_pixel from je_auto_control.utils.timeouts import deadline_after @@ -461,6 +462,9 @@ def __init__(self, executor: Any, branches: list) -> None: self._macros = dict(executor.macros) self._strict = getattr(executor_module._STRICT_BODIES, "value", False) self._macro_depth = getattr(_MACRO_DEPTH, "value", 0) + # The RBAC user is bound to the thread that accepted the request; a + # branch thread without it ran privileged commands unchecked. + self._caller = current_authorization() self.results: list = [None] * len(branches) self._failures: list = [0] * len(branches) self._errors: Dict[int, str] = {} @@ -489,8 +493,9 @@ def run_branch(self, index: int, branch: Any) -> None: _MACRO_DEPTH.value = self._macro_depth self._module.reset_recorded_failures() try: - self.results[index] = self._branch_executor().execute_action( - branch, raise_on_error=self._strict, _validated=True) + with authorization_scope(self._caller): + self.results[index] = self._branch_executor().execute_action( + branch, raise_on_error=self._strict, _validated=True) self._failures[index] = self._module.recorded_failures() except AutoControlAssertionException as error: self._assertions[index] = error diff --git a/je_auto_control/utils/mcp_server/_authz.py b/je_auto_control/utils/mcp_server/_authz.py new file mode 100644 index 000000000..2ff4c145a --- /dev/null +++ b/je_auto_control/utils/mcp_server/_authz.py @@ -0,0 +1,108 @@ +"""Bearer authentication and per-role tool authorisation for the MCP server. + +Two halves, one per layer. :func:`check_bearer` is the HTTP transport's: it +turns the ``Authorization`` header value into either a refusal or -- when a user +store is configured -- the caller's identity. :func:`visible_tools` and +:func:`authorize_tool_call` are the dispatcher's: ``tools/list`` shows only +what the caller may call, and ``tools/call`` refuses the rest, so the two +answers cannot disagree. + +Without a user store nothing here identifies anyone: the transport keeps +its optional shared token and the dispatcher offers every tool, as before. +""" +import hmac +from typing import Any, Dict, List, Optional, Tuple + +from je_auto_control.utils.mcp_server._protocol import _MCPError +from je_auto_control.utils.mcp_server.audit import AuditLogger +from je_auto_control.utils.mcp_server.tools import MCPTool +from je_auto_control.utils.rbac.authorization import ( + AuthorizationContext, current_authorization, resolve_token, +) +from je_auto_control.utils.rbac.policy import capability_for_tool, denied_command_in +from je_auto_control.utils.rbac.users import Capability, UserStore + +#: JSON-RPC error code of a call refused for the caller's role. +FORBIDDEN_CODE = -32003 + +#: ``(HTTP status, error text)`` of a refused request. +Refusal = Tuple[int, str] + + +def check_bearer(authorization: Optional[str], expected: Optional[str], + users: Optional[UserStore], + ) -> Tuple[Optional[AuthorizationContext], Optional[Refusal]]: + """``(caller, refusal)`` for a request's ``Authorization`` header. + + With ``users`` the token must be one user's and the caller is returned; + the shared ``expected`` token is not consulted, because a token that + passed for everyone could not be given a role. Without ``users`` the + caller is always ``None`` and ``expected`` is compared as before -- no + token configured means no check. + """ + if users is None and expected is None: + return None, None + # The scheme is case-insensitive (RFC 7235 2.1): "bearer tok" was + # refused here while the REST gate accepted it. + scheme, _, provided = (authorization or "").strip().partition(" ") + if scheme.lower() != "bearer": + return None, (401, "missing bearer token") + provided = provided.strip() + if users is None: + # Bytes: compare_digest raises TypeError on a non-ASCII str, and + # http.server decodes headers as latin-1, so a crafted token used to + # kill the request thread instead of being refused. + if hmac.compare_digest(provided.encode("utf-8"), str(expected).encode("utf-8")): + return None, None + return None, (401, "invalid bearer token") + caller = resolve_token(users, provided) if provided else None + if caller is None: + return None, (401, "invalid bearer token") + if not caller.allows(Capability.READ_SCREEN): + # A role the store does not define grants nothing; without this the + # ungated methods (resources, prompts) would still answer it. + return None, (403, f"role {caller.role!r} grants no access") + return caller, None + + +def _tool_capability(tool: MCPTool) -> str: + return capability_for_tool(tool.name, tool.annotations.read_only) + + +def visible_tools(tools: List[MCPTool]) -> List[MCPTool]: + """The tools the current caller may call; all of them outside an RBAC scope.""" + caller = current_authorization() + if caller is None: + return tools + return [tool for tool in tools if caller.allows(_tool_capability(tool))] + + +def authorize_tool_call(tool: MCPTool, arguments: Dict[str, Any], + audit: AuditLogger) -> None: + """Refuse, and record, a call the current caller's role does not grant. + + Two things can be missing: the capability of the tool itself, or that of + a privileged ``AC_*`` command inside an action list among its arguments + -- ``ac_execute_actions`` is open to an operator, signing a file through + it is not. + """ + caller = current_authorization() + if caller is None: + return + needed: Optional[str] = _tool_capability(tool) + reason = f"tool {tool.name!r}" + if caller.allows(str(needed)): + denied = denied_command_in(arguments, caller) + if denied is None: + return + reason, needed = f"command {denied[0]!r}", denied[1] + message = (f"Forbidden: {reason} needs the {needed!r} capability; " + f"role {caller.role!r} does not grant it") + audit.record(tool=tool.name, arguments=arguments, status="denied", + duration_seconds=0.0, error_text=message) + raise _MCPError(FORBIDDEN_CODE, message, {"required_capability": needed}) + + +__all__ = [ + "FORBIDDEN_CODE", "authorize_tool_call", "check_bearer", "visible_tools", +] diff --git a/je_auto_control/utils/mcp_server/audit.py b/je_auto_control/utils/mcp_server/audit.py index 3efa367dc..71b7eb172 100644 --- a/je_auto_control/utils/mcp_server/audit.py +++ b/je_auto_control/utils/mcp_server/audit.py @@ -5,6 +5,10 @@ duration. The default sink is ``$JE_AUTOCONTROL_MCP_AUDIT`` (or ``mcp_audit.jsonl`` next to the cwd) so deployments that need a forensic trail get it without code changes. + +When the call was made by an authenticated RBAC user the line also carries +``user_id`` and ``role``; a call refused for its role is recorded with +status ``denied``. """ import json import os @@ -14,6 +18,7 @@ from je_auto_control.utils.executor.action_redaction import SENSITIVE_ARGUMENT_NAMES, redact_actions from je_auto_control.utils.logging.logging_instance import autocontrol_logger +from je_auto_control.utils.rbac.authorization import current_authorization class AuditLogger: @@ -39,10 +44,19 @@ def enabled(self) -> bool: def record(self, *, tool: str, arguments: Dict[str, Any], status: str, duration_seconds: float, error_text: Optional[str] = None, - artifact_path: Optional[str] = None) -> None: - """Append one audit entry. No-ops when no path is configured.""" + artifact_path: Optional[str] = None, + user_id: Optional[str] = None) -> None: + """Append one audit entry. No-ops when no path is configured. + + ``user_id`` defaults to the RBAC user the calling thread is serving, + so the entry names who made the call without each call site passing + it along; with no such user the entry has no ``user_id`` at all. + """ if self._path is None: return + caller = current_authorization() + if user_id is None and caller is not None: + user_id = caller.user_id entry = { "ts": time.time(), "tool": tool, @@ -50,6 +64,10 @@ def record(self, *, tool: str, arguments: Dict[str, Any], "status": status, "duration_seconds": float(duration_seconds), } + if user_id is not None: + entry["user_id"] = user_id + if caller is not None and caller.user_id == user_id: + entry["role"] = caller.role if error_text is not None: entry["error"] = error_text if artifact_path is not None: diff --git a/je_auto_control/utils/mcp_server/http_transport.py b/je_auto_control/utils/mcp_server/http_transport.py index cdf3641bc..0b7e93b20 100644 --- a/je_auto_control/utils/mcp_server/http_transport.py +++ b/je_auto_control/utils/mcp_server/http_transport.py @@ -9,6 +9,10 @@ ``Content-Type: application/json``. The default bind is ``127.0.0.1`` to honour the project's least-privilege policy. +**RBAC.** With a user store -- ``user_store=`` or ``JE_AUTOCONTROL_RBAC_USERS`` +-- a bearer token must be one user's, the shared ``auth_token`` is not +accepted, and each request is dispatched as that user (:mod:`._authz`). + **Sessions.** ``initialize`` mints an ``Mcp-Session-Id`` and returns it as a response header; a client that echoes it back keeps one dispatcher scope across every connection it makes, and may open a standing server-to-client @@ -25,7 +29,6 @@ ``subscriptions/listen`` holds the response stream open for the change notifications it asked for, until the client closes it or the server stops. """ -import hmac import json import os import ssl @@ -38,6 +41,7 @@ bearer_challenge, log_safe, parse_content_length, wire_json_text, ) from je_auto_control.utils.logging.logging_instance import autocontrol_logger +from je_auto_control.utils.mcp_server._authz import check_bearer from je_auto_control.utils.mcp_server._http_stateless import ( PROTOCOL_VERSION_HEADER, is_stateless, read_message, stateless_refusal, status_for, unsupported_header_refusal, @@ -53,6 +57,10 @@ HttpSession, SESSION_HEADER, SessionRegistry, session_id_from_headers, ) from je_auto_control.utils.mcp_server.server import MCPServer +from je_auto_control.utils.rbac.authorization import ( + AuthorizationContext, authorization_scope, user_store_from_env, +) +from je_auto_control.utils.rbac.users import UserStore DEFAULT_PATH = "/mcp" _MAX_BODY = 1_000_000 @@ -108,6 +116,8 @@ class _MCPHttpHandler(BaseHTTPRequestHandler): # Set once this request's body has been read off the socket, so a later # error response knows there is nothing left to drain. _body_consumed = False + # The RBAC user this request authenticated as; None under the shared token. + _caller: Optional[AuthorizationContext] = None # socketserver applies this to the connection socket in setup(); it bounds # every read (headers *and* body) so a stalled request cannot pin a worker. timeout = _REQUEST_TIMEOUT @@ -132,10 +142,13 @@ def do_POST(self) -> None: # noqa: N802 # reason: stdlib API if refused is not None: self._send_raw_json(refused.body, status=refused.status) return - if is_stateless(self.headers, message): - self._serve_stateless(bridge, line, message) - return - self._serve_in_session(bridge, line) + # One scope around every way the line is dispatched: each of them + # runs it on this thread. + with authorization_scope(self._caller): + if is_stateless(self.headers, message): + self._serve_stateless(bridge, line, message) + return + self._serve_in_session(bridge, line) def _serve_in_session(self, bridge: MCPServer, line: str) -> None: """Serve a handshake-era request under its session, or its connection.""" @@ -303,30 +316,18 @@ def _caller_allowed(self) -> bool: if not self._origin_allowed(): self._send_json({"error": "origin not allowed"}, status=403) return False - expected: Optional[str] = self.server.auth_token # type: ignore[attr-defined] - if expected is None: + self._caller, refusal = check_bearer( + self.headers.get("Authorization"), self.server.auth_token, # type: ignore[attr-defined] + self.server.user_store) # type: ignore[attr-defined] + if refusal is None: return True - # The scheme is case-insensitive (RFC 7235 2.1): "bearer tok" was - # refused here while the REST gate accepted it. - scheme, _, provided = self.headers.get("Authorization", "").strip().partition(" ") + status, text = refusal # 401 with a challenge for a missing *and* a wrong token: the MCP # authorization spec requires both, and RFC 9110 the header. challenge = {"WWW-Authenticate": bearer_challenge( - "autocontrol-mcp", self.headers.get("Authorization"))} - if scheme.lower() != "bearer": - self._send_json({"error": "missing bearer token"}, status=401, - extra_headers=challenge) - return False - provided = provided.strip() - # Bytes: compare_digest raises TypeError on a non-ASCII str, and - # http.server decodes headers as latin-1, so a crafted token used to - # kill the request thread instead of being refused. - if not hmac.compare_digest(provided.encode("utf-8"), - expected.encode("utf-8")): - self._send_json({"error": "invalid bearer token"}, status=401, - extra_headers=challenge) - return False - return True + "autocontrol-mcp", self.headers.get("Authorization"))} if status == 401 else None + self._send_json({"error": text}, status=status, extra_headers=challenge) + return False def _origin_allowed(self) -> bool: """True unless a browser on another site, or a rebound name, sent this. @@ -574,10 +575,12 @@ class _MCPHttpServer(ThreadingHTTPServer): def __init__(self, server_address: Tuple[str, int], mcp: MCPServer, - auth_token: Optional[str] = None) -> None: + auth_token: Optional[str] = None, + user_store: Optional[UserStore] = None) -> None: super().__init__(server_address, _MCPHttpHandler) self.mcp = mcp self.auth_token = auth_token + self.user_store = user_store # Dropping a session releases the dispatcher state scoped to its id — # the same release a closing socket used to perform, moved to the # identity that actually owns that state. @@ -631,8 +634,11 @@ def __init__(self, mcp: Optional[MCPServer] = None, host: str = "127.0.0.1", port: int = 9940, auth_token: Optional[str] = None, ssl_context: Optional[ssl.SSLContext] = None, + user_store: Optional[UserStore] = None, ) -> None: + """``user_store`` switches RBAC on; ``None`` reads ``JE_AUTOCONTROL_RBAC_USERS``.""" self._mcp = mcp if mcp is not None else MCPServer() + self._users = user_store if user_store is not None else user_store_from_env() self._address: Tuple[str, int] = (host, port) self._auth_token = auth_token if auth_token is not None else ( os.environ.get("JE_AUTOCONTROL_MCP_TOKEN") or None @@ -661,6 +667,7 @@ def start(self) -> None: return self._server = _MCPHttpServer( self._address, self._mcp, auth_token=self._auth_token, + user_store=self._users, ) if self._ssl_context is not None: # Defer the handshake so it runs in get_request() under a timeout @@ -683,8 +690,8 @@ def start(self) -> None: ) self._thread.start() scheme = "https" if self._ssl_context is not None else "http" - autocontrol_logger.info("MCP %s listening on %s:%d", scheme, - *self._address) + autocontrol_logger.info("MCP %s listening on %s:%d (rbac=%s)", scheme, + *self._address, "on" if self._users is not None else "off") def stop(self, timeout: float = 2.0) -> None: if self._server is None: @@ -706,11 +713,12 @@ def start_mcp_http_server(host: str = "127.0.0.1", port: int = 9940, mcp: Optional[MCPServer] = None, auth_token: Optional[str] = None, ssl_context: Optional[ssl.SSLContext] = None, + user_store: Optional[UserStore] = None, ) -> HttpMCPServer: """Start and return an :class:`HttpMCPServer`; convenience wrapper.""" server = HttpMCPServer( mcp=mcp, host=host, port=port, - auth_token=auth_token, ssl_context=ssl_context, + auth_token=auth_token, ssl_context=ssl_context, user_store=user_store, ) server.start() return server diff --git a/je_auto_control/utils/mcp_server/server.py b/je_auto_control/utils/mcp_server/server.py index 15b510fbc..0099c8598 100644 --- a/je_auto_control/utils/mcp_server/server.py +++ b/je_auto_control/utils/mcp_server/server.py @@ -16,6 +16,7 @@ from je_auto_control.utils.cli_output import utf8_stream from je_auto_control.utils.logging.logging_instance import autocontrol_logger +from je_auto_control.utils.mcp_server._authz import authorize_tool_call, visible_tools from je_auto_control.utils.mcp_server.audit import AuditLogger from je_auto_control.utils.mcp_server.context import ( OperationCancelledError, ToolCallContext, @@ -522,7 +523,7 @@ def _handle_tools_list(self) -> Dict[str, Any]: # client as "-32603 dictionary changed size during iteration". with self._tools_lock: tools = list(self._tools.values()) - return {"tools": [tool.to_descriptor() for tool in tools]} + return {"tools": [tool.to_descriptor() for tool in visible_tools(tools)]} def _handle_resources_list(self) -> Dict[str, Any]: """List descriptors for every registered resource.""" @@ -615,6 +616,7 @@ def _prepare_tool_call( tool = self._tools.get(name) if tool is None: raise _MCPError(-32602, f"Unknown tool: {name}") + authorize_tool_call(tool, arguments, self._audit) # RBAC; a no-op without a user violation = (validate_arguments(tool.input_schema, arguments) or undeclared_arguments(tool.input_schema, arguments)) if violation is not None: diff --git a/je_auto_control/utils/rbac/__init__.py b/je_auto_control/utils/rbac/__init__.py index 9bc9be596..beb31a9d7 100644 --- a/je_auto_control/utils/rbac/__init__.py +++ b/je_auto_control/utils/rbac/__init__.py @@ -12,21 +12,37 @@ * Token authentication: ``authenticate(token)`` constant-time compares against every user's hashed token. -It is a building block only: the REST API and the MCP server do not consult -it yet, and the audit log has no ``user_id`` field -- both still use their -single shared token. +It is opt-in. The REST API and the MCP HTTP transport consult a store only +when one is configured -- ``JE_AUTOCONTROL_RBAC_USERS`` naming the file, or +a ``user_store=`` argument -- and otherwise keep their single shared token +exactly as before. With a store, each request is authenticated as one user +(:mod:`.authorization`), every REST route, MCP tool and privileged ``AC_*`` +command is checked against the capability it needs (:mod:`.policy`), and +the audit entries carry the ``user_id``. The store is intentionally tiny — no LDAP, no OAuth, no row-level permissions. Operators who need more should stand up a proper IdP in front of the REST endpoint; this is the "good-enough-for-small-team" baseline. """ +from je_auto_control.utils.rbac.authorization import ( + USERS_ENV, AuthorizationContext, AuthorizationError, authorization_scope, + current_authorization, resolve_token, user_store_from_env, +) +from je_auto_control.utils.rbac.policy import ( + authorize_command, capability_for_command, capability_for_route, + capability_for_tool, denied_command_in, +) from je_auto_control.utils.rbac.users import ( Capability, Role, UserAuthError, UserRecord, UserStore, can, default_user_store, role_capabilities, ) __all__ = [ - "Capability", "Role", "UserAuthError", "UserRecord", "UserStore", - "can", "default_user_store", "role_capabilities", + "AuthorizationContext", "AuthorizationError", "Capability", "Role", + "USERS_ENV", "UserAuthError", "UserRecord", "UserStore", + "authorization_scope", "authorize_command", "can", + "capability_for_command", "capability_for_route", "capability_for_tool", + "current_authorization", "default_user_store", "denied_command_in", + "resolve_token", "role_capabilities", "user_store_from_env", ] diff --git a/je_auto_control/utils/rbac/authorization.py b/je_auto_control/utils/rbac/authorization.py new file mode 100644 index 000000000..1370c53c4 --- /dev/null +++ b/je_auto_control/utils/rbac/authorization.py @@ -0,0 +1,106 @@ +"""Who is making this request: the authenticated user and the scope it runs in. + +A server that has a :class:`~je_auto_control.utils.rbac.users.UserStore` +configured turns a bearer token into an :class:`AuthorizationContext` with +:func:`resolve_token` and serves the request inside +:func:`authorization_scope`. Code further down -- the executor, the MCP +dispatcher, the audit writers -- asks :func:`current_authorization` who the +caller is instead of having the identity threaded through every signature. + +No scope means no RBAC: a deployment that never configured a user store, the +stdio MCP transport, a script calling the library directly. Nothing is +checked there, which is what keeps those callers working exactly as before. +""" +from __future__ import annotations + +import contextlib +import os +from contextvars import ContextVar +from dataclasses import dataclass +from pathlib import Path +from typing import Iterator, Optional + +from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.rbac.users import UserAuthError, UserStore, can + +#: Names the user store file. Setting it is what switches RBAC on for the +#: REST API and the MCP HTTP transport; unset or empty leaves both on their +#: shared token. +USERS_ENV = "JE_AUTOCONTROL_RBAC_USERS" + + +class AuthorizationError(AutoControlException, PermissionError): + """The authenticated user's role does not grant what was asked for.""" + + def __init__(self, message: str, capability: str = "") -> None: + super().__init__(message) + self.capability = capability + + +@dataclass(frozen=True) +class AuthorizationContext: + """One authenticated caller: who it is and the role it holds.""" + + user_id: str + role: str + + def allows(self, capability: str) -> bool: + """``True`` when this caller's role grants ``capability``.""" + return can(self.role, capability) + + +def user_store_from_env() -> Optional[UserStore]: + """The user store ``JE_AUTOCONTROL_RBAC_USERS`` names, or ``None`` when unset. + + Only an explicit path opts in. The default ``~/.je_auto_control/users.json`` + is never picked up on its own: a file left there by an experiment must not + silently retire the shared token of a running deployment. + """ + raw = os.environ.get(USERS_ENV, "").strip() + if not raw: + return None + return UserStore(Path(os.path.realpath(os.path.expanduser(raw)))) + + +def resolve_token(store: UserStore, token: str) -> Optional[AuthorizationContext]: + """The caller ``token`` belongs to, or ``None`` when it is nobody's. + + The store is re-read first when its file changed, so removing a user or + rotating a token takes effect on the next request. + """ + store.refresh() + try: + record = store.authenticate(token) + except UserAuthError: + return None + return AuthorizationContext(user_id=record.user_id, role=record.role) + + +_CURRENT: ContextVar[Optional[AuthorizationContext]] = ContextVar( + "je_auto_control_authorization", default=None) + + +def current_authorization() -> Optional[AuthorizationContext]: + """The caller this thread is serving, or ``None`` outside any RBAC scope.""" + return _CURRENT.get() + + +@contextlib.contextmanager +def authorization_scope(context: Optional[AuthorizationContext]) -> Iterator[None]: + """Serve the enclosed work as ``context``; ``None`` clears any outer scope. + + The scope belongs to the calling thread. Work handed to another thread -- + a scheduler job, a trigger, a hotkey -- runs outside it. + """ + token = _CURRENT.set(context) + try: + yield + finally: + _CURRENT.reset(token) + + +__all__ = [ + "AuthorizationContext", "AuthorizationError", "USERS_ENV", + "authorization_scope", "current_authorization", "resolve_token", + "user_store_from_env", +] diff --git a/je_auto_control/utils/rbac/policy.py b/je_auto_control/utils/rbac/policy.py new file mode 100644 index 000000000..b38466e56 --- /dev/null +++ b/je_auto_control/utils/rbac/policy.py @@ -0,0 +1,208 @@ +"""Which capability each REST route, MCP tool and ``AC_*`` command needs. + +Three surfaces, one rule each: + +* **REST routes** are listed one by one. A route missing from the list needs + ``manage_hosts``, so a route added without a decision is admin-only rather + than open. +* **MCP tools** follow their own annotation -- a tool marked read-only needs + ``read_screen``, any other ``drive_input`` -- except the few named here + that change how the host itself is exposed. +* **``AC_*`` commands** reach the executor through anything that runs an + action list (``POST /execute``, ``ac_execute_actions``, an action file). + Being allowed to run actions is ``drive_input``; the commands named here + need more, and are checked both where the list is submitted and again by + the executor, which is what covers a list read from a file. + +The roles are not a sandbox. An operator drives the real keyboard and can +launch processes, so anything the desktop user could do by hand is within +reach. What the table protects is the host's own privileged state: its +signing key, its user and host administration, the servers it runs, its USB +and egress policy, its secrets vault and its audit log. +""" +from __future__ import annotations + +from typing import Any, Dict, Iterator, Optional, Tuple + +from je_auto_control.utils.rbac.authorization import ( + AuthorizationContext, AuthorizationError, current_authorization, +) +from je_auto_control.utils.rbac.users import Capability + +_READ = Capability.READ_SCREEN +_DRIVE = Capability.DRIVE_INPUT +_HOSTS = Capability.MANAGE_HOSTS +_AUDIT = Capability.READ_AUDIT + +REST_ROUTE_CAPABILITIES: Dict[Tuple[str, str], str] = { + ("GET", "/metrics"): _READ, + ("GET", "/jobs"): _READ, + ("GET", "/history"): _READ, + ("GET", "/screenshot"): _READ, + ("GET", "/mouse_position"): _READ, + ("GET", "/screen_size"): _READ, + ("GET", "/windows"): _READ, + ("GET", "/sessions"): _READ, + ("GET", "/commands"): _READ, + ("GET", "/inspector/recent"): _READ, + ("GET", "/inspector/summary"): _READ, + ("GET", "/usb/devices"): _READ, + ("GET", "/usb/events"): _READ, + ("GET", "/usb/passthrough/status"): _READ, + ("GET", "/usb/acl"): _READ, + ("GET", "/usb/loopback/devices"): _READ, + ("GET", "/usb/remote/devices"): _READ, + ("GET", "/diagnose"): _READ, + ("GET", "/openapi.json"): _READ, + ("GET", "/audit/list"): _AUDIT, + ("GET", "/audit/verify"): _AUDIT, + ("POST", "/execute"): _DRIVE, + ("POST", "/execute_file"): _DRIVE, + ("POST", "/usb/loopback/open"): _DRIVE, + ("POST", "/usb/remote/open"): _DRIVE, + ("POST", "/config/export"): _HOSTS, + ("POST", "/config/import"): _HOSTS, + ("POST", "/usb/passthrough/enable"): _HOSTS, + ("POST", "/usb/acl/add"): _HOSTS, + ("POST", "/usb/acl/remove"): _HOSTS, + ("POST", "/usb/acl/default"): _HOSTS, +} + +#: Commands that need more than ``drive_input``. Every other command is +#: covered by the capability of the surface that submitted the action list. +COMMAND_CAPABILITIES: Dict[str, str] = { + "AC_sign_action_file": Capability.SIGN_ACTIONS, + "AC_audit_log_list": _AUDIT, + "AC_audit_log_verify": _AUDIT, + "AC_audit_log_clear": _HOSTS, + "AC_admin_add_host": _HOSTS, + "AC_admin_remove_host": _HOSTS, + "AC_admin_list_hosts": _HOSTS, + "AC_admin_poll": _HOSTS, + "AC_admin_broadcast_execute": _HOSTS, + "AC_rest_api_start": _HOSTS, + "AC_rest_api_stop": _HOSTS, + "AC_rest_api_status": _HOSTS, # its reply carries the shared token + "AC_start_mcp_server": _HOSTS, + "AC_start_mcp_http_server": _HOSTS, + "AC_start_remote_host": _HOSTS, + "AC_stop_remote_host": _HOSTS, + "AC_start_webrtc_host": _HOSTS, + "AC_stop_webrtc_host": _HOSTS, + "AC_start_ws_host": _HOSTS, + "AC_stop_ws_host": _HOSTS, + "AC_webhook_add": _HOSTS, + "AC_webhook_remove": _HOSTS, + "AC_webhook_start": _HOSTS, + "AC_webhook_stop": _HOSTS, + "AC_usb_acl_add": _HOSTS, + "AC_usb_acl_remove": _HOSTS, + "AC_usb_acl_set_default": _HOSTS, + "AC_usb_acl_import": _HOSTS, + "AC_usb_acl_export": _HOSTS, + "AC_usb_passthrough_enable": _HOSTS, + "AC_config_import": _HOSTS, + "AC_config_export": _HOSTS, + "AC_egress_allow": _HOSTS, + "AC_egress_reset": _HOSTS, + "AC_load_plugins": _HOSTS, + "AC_add_package_to_executor": _HOSTS, + "AC_add_package_to_callback_executor": _HOSTS, + "AC_secret_init": _HOSTS, + "AC_secret_set": _HOSTS, + "AC_secret_remove": _HOSTS, + "AC_secret_unlock": _HOSTS, + "AC_secret_lock": _HOSTS, +} + +#: MCP tools whose capability is not the one their read-only hint implies. +TOOL_CAPABILITIES: Dict[str, str] = { + "ac_remote_host_start": _HOSTS, + "ac_remote_host_stop": _HOSTS, + "ac_usb_acl_add": _HOSTS, + "ac_usb_acl_remove": _HOSTS, + "ac_usb_acl_set_default": _HOSTS, + "ac_usb_passthrough_enable": _HOSTS, + "ac_egress_allow": _HOSTS, + "ac_egress_reset": _HOSTS, + "ac_load_plugins": _HOSTS, +} + + +def capability_for_route(method: str, path: str) -> str: + """The capability ``method path`` needs; an unlisted route is admin-only.""" + return REST_ROUTE_CAPABILITIES.get((method.upper(), path), _HOSTS) + + +def capability_for_tool(name: str, read_only: bool) -> str: + """The capability the MCP tool ``name`` needs, given its read-only hint.""" + listed = TOOL_CAPABILITIES.get(name) + if listed is not None: + return listed + return _READ if read_only else _DRIVE + + +def capability_for_command(name: str) -> str: + """The capability the ``AC_*`` command ``name`` needs.""" + return COMMAND_CAPABILITIES.get(name, _DRIVE) + + +def authorize_command(name: Any) -> None: + """Refuse a privileged command the current caller's role does not grant. + + Called by the executor for every action. Outside an RBAC scope it does + nothing, and inside one it only looks at the commands listed in + :data:`COMMAND_CAPABILITIES` -- whether the caller may run actions at all + was decided by the surface that accepted the request. + """ + context = current_authorization() + if context is None or not isinstance(name, str): + return + needed = COMMAND_CAPABILITIES.get(name) + if needed is not None and not context.allows(needed): + raise AuthorizationError( + f"{name} needs the {needed!r} capability; user {context.user_id!r} " + f"has role {context.role!r}", capability=needed) + + +def denied_command_in(payload: Any, + context: AuthorizationContext) -> Optional[Tuple[str, str]]: + """``(command, capability)`` for the first privileged command ``context`` may not run. + + Walks ``payload`` -- a request body, a tool's arguments -- wherever an + action list could be nested in it, so a request is refused whole before + its first action runs rather than part-way through. A list that merely + starts with a command's name is treated as that command: refusing a + look-alike costs a retry, missing a real one is the failure. + """ + for name in _leading_names(payload, depth=0): + needed = COMMAND_CAPABILITIES.get(name) + if needed is not None and not context.allows(needed): + return name, needed + return None + + +_MAX_WALK_DEPTH = 200 + + +def _leading_names(node: Any, depth: int) -> Iterator[str]: + """Yield the first item of every list in ``node`` that is a string.""" + if depth > _MAX_WALK_DEPTH: + return + if isinstance(node, dict): + for value in node.values(): + yield from _leading_names(value, depth + 1) + return + if not isinstance(node, (list, tuple)): + return + if node and isinstance(node[0], str): + yield node[0] + for item in node: + yield from _leading_names(item, depth + 1) + + +__all__ = [ + "COMMAND_CAPABILITIES", "REST_ROUTE_CAPABILITIES", "TOOL_CAPABILITIES", + "authorize_command", "capability_for_command", "capability_for_route", + "capability_for_tool", "denied_command_in", +] diff --git a/je_auto_control/utils/rbac/users.py b/je_auto_control/utils/rbac/users.py index 7abf711a1..ca5b2aea0 100644 --- a/je_auto_control/utils/rbac/users.py +++ b/je_auto_control/utils/rbac/users.py @@ -9,7 +9,7 @@ import threading from dataclasses import asdict, dataclass, field, replace from pathlib import Path -from typing import Dict, List, Optional, Set +from typing import Dict, List, Optional, Set, Tuple from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.json_store.json_store import atomic_write_text @@ -34,11 +34,15 @@ class Capability: MANAGE_HOSTS = "manage_hosts" MANAGE_USERS = "manage_users" READ_AUDIT = "read_audit" + #: Signing an action file with this host's key. Separate from + #: ``DRIVE_INPUT`` so that being allowed to run actions is not being + #: allowed to approve the files that may be run. + SIGN_ACTIONS = "sign_actions" @classmethod def all(cls) -> List[str]: return [cls.READ_SCREEN, cls.DRIVE_INPUT, cls.MANAGE_HOSTS, - cls.MANAGE_USERS, cls.READ_AUDIT] + cls.MANAGE_USERS, cls.READ_AUDIT, cls.SIGN_ACTIONS] _ROLE_CAPABILITIES: Dict[str, Set[str]] = { @@ -104,6 +108,9 @@ def __init__(self, path: Optional[Path] = None) -> None: # Why the file on disk could not be read, if it could not. Saving # then would replace every user in it with what this instance holds. self._unreadable: Optional[str] = None + # What the file looked like when it was last read or written, so + # refresh() can tell that another process has changed it. + self._loaded_stamp: Optional[Tuple[int, int]] = None self._load() @property @@ -185,37 +192,65 @@ def get(self, user_id: str) -> Optional[UserRecord]: record = self._users.get(user_id) return None if record is None else _copy(record) + def refresh(self) -> bool: + """Re-read the file if it changed on disk; ``True`` when it was re-read. + + Users are managed from another process than the server that checks + their tokens. Without this a removed user, a rotated token or a + lowered role kept working until that server was restarted. + """ + stamp = self._stamp() + with self._lock: + if stamp == self._loaded_stamp: + return False + self._load() + return True + + def _stamp(self) -> Optional[Tuple[int, int]]: + """``(mtime_ns, size)`` of the file, or ``None`` when there is none.""" + try: + status = self._path.stat() + except OSError: + return None + return status.st_mtime_ns, status.st_size + def _load(self) -> None: + stamp = self._stamp() + users, problem = self._read() + with self._lock: + self._users = users + self._unreadable = problem + self._loaded_stamp = stamp + if problem is not None: + autocontrol_logger.error( + "user store %s unreadable (%s); no user can sign in and it will not be overwritten", + self._path, problem) + + def _read(self) -> Tuple[Dict[str, UserRecord], Optional[str]]: + """The users on disk, and why the file could not be read if it could not.""" if not self._path.exists(): - return + return {}, None try: body = json.loads(self._path.read_text(encoding="utf-8")) except (OSError, ValueError) as error: # ValueError: bad JSON or not UTF-8 - self._mark_unreadable(repr(error)) - return - users = body.get("users") if isinstance(body, dict) else None - if not isinstance(users, list): - self._mark_unreadable("no 'users' list") - return - with self._lock: - for entry in users: - if not isinstance(entry, dict): - continue - record = UserRecord( - user_id=str(entry.get("user_id", "")), - display_name=str(entry.get("display_name", "")), - role=str(entry.get("role", Role.VIEWER)), - token_hash=str(entry.get("token_hash", "")), - tags=_tags(entry.get("tags")), - ) - if record.user_id: - self._users[record.user_id] = record - - def _mark_unreadable(self, reason: str) -> None: - self._unreadable = reason - autocontrol_logger.error( - "user store %s unreadable (%s); no user can sign in and it will not be overwritten", - self._path, reason) + return {}, repr(error) + entries = body.get("users") if isinstance(body, dict) else None + if not isinstance(entries, list): + return {}, "no 'users' list" + users: Dict[str, UserRecord] = {} + for entry in entries: + if not isinstance(entry, dict): + continue + record = UserRecord( + user_id=str(entry.get("user_id", "")), + display_name=str(entry.get("display_name", "")), + role=str(entry.get("role", Role.VIEWER)), + token_hash=str(entry.get("token_hash", "")), + tags=_tags(entry.get("tags")), + ) + if record.user_id: + users[record.user_id] = record + return users, None def _commit_locked(self, users: Dict[str, UserRecord]) -> None: """Save ``users``, then make them the store's; on any failure nothing changes. @@ -237,6 +272,7 @@ def _commit_locked(self, users: Dict[str, UserRecord]) -> None: except OSError as error: raise UserAuthError(f"could not save user store {self._path}: {error}") from error self._users = users + self._loaded_stamp = self._stamp() def _copy(record: UserRecord) -> UserRecord: diff --git a/je_auto_control/utils/rest_api/__main__.py b/je_auto_control/utils/rest_api/__main__.py index 21f032535..4cc1c233f 100644 --- a/je_auto_control/utils/rest_api/__main__.py +++ b/je_auto_control/utils/rest_api/__main__.py @@ -9,8 +9,10 @@ import argparse import sys import time +from pathlib import Path from typing import Optional +from je_auto_control.utils.rbac.users import UserStore from je_auto_control.utils.rest_api.rest_server import RestApiServer @@ -27,6 +29,9 @@ def _build_arg_parser() -> argparse.ArgumentParser: help="bearer token (auto-generated if omitted)") parser.add_argument("--no-audit", action="store_true", help="disable audit-log writes") + parser.add_argument("--users", default=None, + help="user store file; switches RBAC on " + "(default: JE_AUTOCONTROL_RBAC_USERS)") return parser @@ -35,12 +40,17 @@ def main(argv: Optional[list] = None) -> int: server = RestApiServer( host=args.host, port=args.port, token=args.token, enable_audit=not args.no_audit, + user_store=UserStore(Path(args.users)) if args.users else None, ) server.start() host, port = server.address print(f"REST API listening at http://{host}:{port}") - print(f"Bearer token: {server.token}") - print("Send Authorization: Bearer on every non-/health call.") + if server.user_store is not None: + print(f"RBAC on: tokens are checked against {server.user_store.path}") + print("Send Authorization: Bearer on every non-/health call.") + else: + print(f"Bearer token: {server.token}") + print("Send Authorization: Bearer on every non-/health call.") print("Press Ctrl-C to stop.") try: while True: diff --git a/je_auto_control/utils/rest_api/rest_auth.py b/je_auto_control/utils/rest_api/rest_auth.py index 7f27a3cad..69110f32e 100644 --- a/je_auto_control/utils/rest_api/rest_auth.py +++ b/je_auto_control/utils/rest_api/rest_auth.py @@ -9,6 +9,10 @@ * Comparison uses :func:`secrets.compare_digest` to avoid timing leaks. * The token is generated once at server start and surfaced on the ``RestApiServer`` instance so the GUI / CLI can show it to the user. + * With a ``user_store`` (opt-in RBAC) the shared token is not accepted at + all: a bearer token must belong to one user of the store, and the gate + hands back who that is so the server can authorise the route. One token + that passed for everyone could not be told apart by role. Rate limit: * One token bucket per client IP, refilled at ``_REQUESTS_PER_MINUTE`` @@ -22,7 +26,10 @@ import threading import time from dataclasses import dataclass -from typing import Dict, Optional +from typing import Dict, Optional, Tuple + +from je_auto_control.utils.rbac.authorization import AuthorizationContext, resolve_token +from je_auto_control.utils.rbac.users import UserStore _DEFAULT_TOKEN_BYTES = 24 @@ -57,18 +64,28 @@ class _Bucket: failed_window_start: float = 0.0 +@dataclass(frozen=True) +class AuthResult: + """What the gate decided, and who the caller is when RBAC identified one.""" + + verdict: str + context: Optional[AuthorizationContext] = None + + class RestAuthGate: """Bearer-token check + per-IP token bucket. - ``check(...)`` is the only entry point handlers should call. - Returns one of ``"ok"``, ``"unauthorized"``, ``"rate_limited"``, - ``"locked_out"``. + ``authenticate(...)`` returns the verdict together with the caller's + identity; ``check(...)`` returns the verdict alone -- one of ``"ok"``, + ``"unauthorized"``, ``"rate_limited"``, ``"locked_out"``. """ def __init__(self, expected_token: str, *, requests_per_minute: float = _REQUESTS_PER_MINUTE, - burst: float = _BURST) -> None: + burst: float = _BURST, + user_store: Optional[UserStore] = None) -> None: self._token = expected_token + self._users = user_store self._rate_per_s = float(requests_per_minute) / 60.0 self._burst = float(burst) self._buckets: Dict[str, _Bucket] = {} @@ -78,7 +95,17 @@ def __init__(self, expected_token: str, def expected_token(self) -> str: return self._token + @property + def rbac_enabled(self) -> bool: + """``True`` when tokens are resolved through a user store.""" + return self._users is not None + def check(self, *, client_ip: str, header_value: Optional[str]) -> str: + """The verdict of :meth:`authenticate`, for callers that need no identity.""" + return self.authenticate(client_ip=client_ip, header_value=header_value).verdict + + def authenticate(self, *, client_ip: str, + header_value: Optional[str]) -> AuthResult: """Rate-limit, then authenticate; a valid token is never locked out. The lockout used to be checked first, keyed by IP alone -- and every @@ -87,14 +114,26 @@ def check(self, *, client_ip: str, header_value: Optional[str]) -> str: only answers wrong tokens; the per-IP rate limit still applies to all. """ if not self._consume_token(client_ip): - return "rate_limited" - if _matches_bearer(header_value, self._token): + return AuthResult("rate_limited") + accepted, context = self._identify(header_value) + if accepted: self._reset_failures(client_ip) - return "ok" + return AuthResult("ok", context) if self._is_locked_out(client_ip): - return "locked_out" + return AuthResult("locked_out") self._note_failure(client_ip) - return "unauthorized" + return AuthResult("unauthorized") + + def _identify(self, header_value: Optional[str], + ) -> Tuple[bool, Optional[AuthorizationContext]]: + """``(accepted, caller)``; the caller is ``None`` under the shared token.""" + provided = _bearer_token(header_value) + if provided is None: + return False, None + if self._users is None: + return constant_time_equal(provided, self._token), None + context = resolve_token(self._users, provided) + return context is not None, context def _consume_token(self, client_ip: str) -> bool: now = time.monotonic() @@ -143,15 +182,21 @@ def _reset_failures(self, client_ip: str) -> None: bucket.failed = 0 -def _matches_bearer(header_value: Optional[str], expected: str) -> bool: +def _bearer_token(header_value: Optional[str]) -> Optional[str]: + """The token of an ``Authorization: Bearer `` header, or ``None``.""" if not header_value: - return False + return None parts = header_value.strip().split(None, 1) if len(parts) != 2 or parts[0].lower() != "bearer": - return False - return constant_time_equal(parts[1], expected) + return None + return parts[1] + + +def _matches_bearer(header_value: Optional[str], expected: str) -> bool: + provided = _bearer_token(header_value) + return provided is not None and constant_time_equal(provided, expected) __all__ = [ - "RestAuthGate", "generate_token", "constant_time_equal", + "AuthResult", "RestAuthGate", "generate_token", "constant_time_equal", ] diff --git a/je_auto_control/utils/rest_api/rest_openapi.py b/je_auto_control/utils/rest_api/rest_openapi.py index a3b6bfa3e..e39854f1f 100644 --- a/je_auto_control/utils/rest_api/rest_openapi.py +++ b/je_auto_control/utils/rest_api/rest_openapi.py @@ -14,6 +14,8 @@ from typing import Any, Dict, List, Tuple +from je_auto_control.utils.rbac.policy import capability_for_route + _BEARER_SCHEME_NAME = "BearerAuth" _API_VERSION = "1.0.0" @@ -323,7 +325,10 @@ def build_openapi_spec(*, server_url: str = "http://127.0.0.1:9939", "AutoControl REST API. All non-public endpoints require " "an `Authorization: Bearer ` header. The bearer " "token is generated at server start and surfaced via the " - "REST API GUI tab or the CLI." + "REST API GUI tab or the CLI. When the server was started " + "with a user store (RBAC), the token is one user's instead, " + "and each operation needs the capability named in its " + "`x-required-capability`; a role without it gets 403." ), }, "servers": [{"url": server_url}], @@ -352,6 +357,8 @@ def _operation_object(method: str, path: str, } if meta.get("public"): op["security"] = [] # explicit empty array overrides global security + else: + op["x-required-capability"] = capability_for_route(method, path) if meta.get("params"): op["parameters"] = list(meta["params"]) if meta.get("request_body"): @@ -379,6 +386,11 @@ def _build_responses(meta: Dict[str, Any]) -> Dict[str, Any]: "description": "Missing or wrong bearer token.", "content": {_JSON_MEDIA_TYPE: {"schema": _error_schema()}}, } + responses["403"] = { + "description": ("RBAC only: the authenticated user's role does not " + "grant the capability this operation needs."), + "content": {_JSON_MEDIA_TYPE: {"schema": _error_schema()}}, + } responses["429"] = { "description": "Rate limited or locked out after repeated auth failures.", "content": {_JSON_MEDIA_TYPE: {"schema": _error_schema()}}, diff --git a/je_auto_control/utils/rest_api/rest_registry.py b/je_auto_control/utils/rest_api/rest_registry.py index 6afe8b09c..e466ef5e3 100644 --- a/je_auto_control/utils/rest_api/rest_registry.py +++ b/je_auto_control/utils/rest_api/rest_registry.py @@ -9,6 +9,7 @@ import threading from typing import Any, Dict, Optional +from je_auto_control.utils.rbac.users import UserStore from je_auto_control.utils.rest_api.rest_server import RestApiServer @@ -26,7 +27,8 @@ def server(self) -> Optional[RestApiServer]: def start(self, host: str = "127.0.0.1", port: int = 9939, *, token: Optional[str] = None, - enable_audit: bool = True) -> Dict[str, Any]: + enable_audit: bool = True, + user_store: Optional[UserStore] = None) -> Dict[str, Any]: """Stop any existing server, then start a fresh one with the config. The whole start lifecycle (stop existing → construct → bind → @@ -40,7 +42,7 @@ def start(self, host: str = "127.0.0.1", port: int = 9939, previous.stop(timeout=2.0) server = RestApiServer( host=host, port=int(port), token=token, - enable_audit=enable_audit, + enable_audit=enable_audit, user_store=user_store, ) server.start() self._server = server diff --git a/je_auto_control/utils/rest_api/rest_server.py b/je_auto_control/utils/rest_api/rest_server.py index 3a41de490..af3f2268b 100644 --- a/je_auto_control/utils/rest_api/rest_server.py +++ b/je_auto_control/utils/rest_api/rest_server.py @@ -22,6 +22,11 @@ bearer_challenge, log_safe, parse_content_length, wire_json_text, ) from je_auto_control.utils.logging.logging_instance import autocontrol_logger +from je_auto_control.utils.rbac.authorization import ( + AuthorizationContext, authorization_scope, user_store_from_env, +) +from je_auto_control.utils.rbac.policy import capability_for_route, denied_command_in +from je_auto_control.utils.rbac.users import UserStore from je_auto_control.utils.rest_api.rest_auth import RestAuthGate, generate_token from je_auto_control.utils.rest_api.rest_handlers import ( HandlerResult, RouteContext, @@ -150,17 +155,9 @@ def _serve_dashboard(self, path: str) -> None: def _serve_metrics(self) -> None: client_ip = self.client_address[0] if self.client_address else "?" - verdict = self._gate().check( - client_ip=client_ip, - header_value=self.headers.get("Authorization"), - ) - if verdict != "ok": - if verdict == "unauthorized": - self._metrics().record_failed_auth() - self._reject(verdict) - self._metrics().record_request( - "GET", "/metrics", _verdict_to_status(verdict), - ) + admitted, _caller = self._admit("GET", _PATH_METRICS, client_ip, None, + audit_refusal=False) + if not admitted: return body = self._metrics().render( audit_row_count=_count_audit_rows(getattr(self.server, "audit_log", None)), @@ -192,41 +189,98 @@ def _dispatch(self, method: str, routes: Dict[str, HandlerFn], self._answer_unrouted(parsed.path) return client_ip = self.client_address[0] if self.client_address else "?" + caller: Optional[AuthorizationContext] = None if parsed.path not in _PUBLIC_PATHS: - verdict = self._gate().check( - client_ip=client_ip, - header_value=self.headers.get("Authorization"), - ) - if verdict != "ok": - if verdict == "unauthorized": - self._metrics().record_failed_auth() - self._drain_unread_body(body) - self._reject(verdict) - self._audit(method, parsed.path, client_ip, verdict) - self._metrics().record_request( - method, parsed.path, _verdict_to_status(verdict), - ) + admitted, caller = self._admit(method, parsed.path, client_ip, body) + if not admitted: return if body is _BODY_PENDING: body = self._read_json_body() if body is _BODY_ERROR_SENT: return + if self._refuse_privileged_commands(method, parsed.path, client_ip, caller, body): + return ctx = RouteContext(query=parsed.query, body=body, client_ip=client_ip) try: - status, payload = handler(ctx) + # The scope is what lets the executor refuse a privileged command + # found in an action file, which nothing here has read. + with authorization_scope(caller): + status, payload = handler(ctx) except _HANDLER_ERRORS as error: autocontrol_logger.error( "rest-api %s %s handler raised: %r", method, parsed.path, error, ) self._send_json({"error": "handler crashed"}, status=500) - self._audit(method, parsed.path, client_ip, "error") + self._audit(method, parsed.path, client_ip, "error", caller) self._metrics().record_request(method, parsed.path, 500) return status = self._send_json(payload, status=status, default=str) if parsed.path not in _PUBLIC_PATHS: - self._audit(method, parsed.path, client_ip, f"ok:{status}") + self._audit(method, parsed.path, client_ip, f"ok:{status}", caller) self._metrics().record_request(method, parsed.path, status) + def _admit(self, method: str, path: str, client_ip: str, body: Any, + *, audit_refusal: bool = True, + ) -> Tuple[bool, Optional[AuthorizationContext]]: + """Authenticate, then authorise the route; ``False`` means a reply was sent. + + The caller is ``None`` under the shared token: there is no user to + hold a role, so there is nothing to authorise. + """ + result = self._gate().authenticate( + client_ip=client_ip, + header_value=self.headers.get("Authorization"), + ) + if result.verdict != "ok": + if result.verdict == "unauthorized": + self._metrics().record_failed_auth() + self._drain_unread_body(body) + self._reject(result.verdict) + if audit_refusal: + self._audit(method, path, client_ip, result.verdict) + self._metrics().record_request( + method, path, _verdict_to_status(result.verdict), + ) + return False, None + needed = capability_for_route(method, path) + if result.context is not None and not result.context.allows(needed): + self._drain_unread_body(body) + self._forbid(method, path, client_ip, result.context, needed) + return False, None + return True, result.context + + def _refuse_privileged_commands(self, method: str, path: str, client_ip: str, + caller: Optional[AuthorizationContext], + body: Any) -> bool: + """Answer 403 when the body names a command the caller's role lacks. + + Being allowed to ``POST /execute`` is being allowed to run actions, + not to run every command: signing a file or starting a server needs + more. The whole body is refused before its first action runs. + """ + if caller is None or body is None: + return False + denied = denied_command_in(body, caller) + if denied is None: + return False + self._forbid(method, path, client_ip, caller, denied[1], command=denied[0]) + return True + + def _forbid(self, method: str, path: str, client_ip: str, + caller: AuthorizationContext, needed: str, + command: Optional[str] = None) -> None: + """Answer 403, naming the capability the caller's role does not grant.""" + payload: Dict[str, Any] = { + "error": "forbidden", "required_capability": needed, "role": caller.role, + } + outcome = f"forbidden:{needed}" + if command is not None: + payload["command"] = command + outcome += f":{command}" + self._send_json(payload, status=403) + self._audit(method, path, client_ip, outcome, caller) + self._metrics().record_request(method, path, 403) + def _gate(self) -> RestAuthGate: return self.server.auth_gate # type: ignore[attr-defined] @@ -234,15 +288,25 @@ def _metrics(self) -> RestMetrics: return self.server.metrics # type: ignore[attr-defined] def _audit(self, method: str, path: str, client_ip: str, - outcome: str) -> None: + outcome: str, + caller: Optional[AuthorizationContext] = None) -> None: + """Record one request; with RBAC the row names the authenticated user. + + The user id goes in the row's ``viewer_id`` column -- the caller's + identity, as it is for a remote-desktop viewer -- and in the detail + text, so it shows wherever the row is read. + """ audit = getattr(self.server, "audit_log", None) if audit is None: return + if caller is None: + identity: Dict[str, Any] = {} + detail = f"{method} {path} -> {outcome}" + else: + identity = {"viewer_id": caller.user_id} + detail = f"{method} {path} -> {outcome} user={caller.user_id} role={caller.role}" try: - audit.log( - "rest_api", host_id=client_ip, - detail=f"{method} {path} -> {outcome}", - ) + audit.log("rest_api", host_id=client_ip, detail=detail, **identity) except (OSError, RuntimeError) as error: autocontrol_logger.warning("rest-api audit write failed: %r", error) @@ -410,12 +474,21 @@ class RestApiServer: def __init__(self, host: str = "127.0.0.1", port: int = 9939, *, token: Optional[str] = None, - enable_audit: bool = True) -> None: + enable_audit: bool = True, + user_store: Optional[UserStore] = None) -> None: + """``user_store`` switches RBAC on; ``None`` reads ``JE_AUTOCONTROL_RBAC_USERS``. + + With neither, every authenticated request holds the one shared + ``token`` and nothing is authorised per role -- the behaviour this + server has always had. With a store the shared token is refused and + each request is authorised as the user its own token belongs to. + """ self._address: Tuple[str, int] = (host, port) self._server: Optional[ThreadingHTTPServer] = None self._thread: Optional[threading.Thread] = None self._token = token if token else generate_token() - self._auth = RestAuthGate(expected_token=self._token) + self._users = user_store if user_store is not None else user_store_from_env() + self._auth = RestAuthGate(expected_token=self._token, user_store=self._users) self._audit_log = self._open_audit_log() if enable_audit else None self._metrics = RestMetrics() @@ -440,6 +513,11 @@ def address(self) -> Tuple[str, int]: def token(self) -> str: return self._token + @property + def user_store(self) -> Optional[UserStore]: + """The store RBAC resolves tokens through, or ``None`` when RBAC is off.""" + return self._users + @property def is_running(self) -> bool: return self._server is not None @@ -474,9 +552,10 @@ def start(self) -> None: ) self._thread.start() autocontrol_logger.info( - "REST API listening on %s:%d (audit=%s)", + "REST API listening on %s:%d (audit=%s, rbac=%s)", self._address[0], self._address[1], "on" if self._audit_log is not None else "off", + "on" if self._users is not None else "off", ) def stop(self, timeout: float = 2.0) -> None: @@ -493,10 +572,12 @@ def stop(self, timeout: float = 2.0) -> None: def start_rest_api_server(host: str = "127.0.0.1", port: int = 9939, *, token: Optional[str] = None, - enable_audit: bool = True) -> RestApiServer: + enable_audit: bool = True, + user_store: Optional[UserStore] = None) -> RestApiServer: """Construct, start, and return a ``RestApiServer``.""" server = RestApiServer( host=host, port=port, token=token, enable_audit=enable_audit, + user_store=user_store, ) server.start() return server @@ -522,11 +603,15 @@ def _main(argv: Optional[list] = None) -> int: parser.add_argument("--port", type=int, default=9939) parser.add_argument("--token", default=os.environ.get("AC_TOKEN")) parser.add_argument("--no-audit", action="store_true") + parser.add_argument("--users", default=None, + help="user store file; switches RBAC on (default: " + "JE_AUTOCONTROL_RBAC_USERS)") args = parser.parse_args(argv) server = start_rest_api_server( host=args.host, port=args.port, token=args.token, enable_audit=not args.no_audit, + user_store=UserStore(Path(args.users)) if args.users else None, ) stop_event = threading.Event() diff --git a/test/unit_test/headless/test_r3_net_socket_rest.py b/test/unit_test/headless/test_r3_net_socket_rest.py index b18be0480..3156a00b5 100644 --- a/test/unit_test/headless/test_r3_net_socket_rest.py +++ b/test/unit_test/headless/test_r3_net_socket_rest.py @@ -77,6 +77,10 @@ class _Gate: def check(self, **_kwargs) -> str: return "ok" + def authenticate(self, **_kwargs): + from je_auto_control.utils.rest_api.rest_auth import AuthResult + return AuthResult("ok") + class _Metrics: def __init__(self) -> None: diff --git a/test/unit_test/headless/test_rbac_server_wiring.py b/test/unit_test/headless/test_rbac_server_wiring.py new file mode 100644 index 000000000..da9d3c548 --- /dev/null +++ b/test/unit_test/headless/test_rbac_server_wiring.py @@ -0,0 +1,427 @@ +"""RBAC wired into the REST API, the MCP HTTP transport and the executor. + +Opt-in: a server with no user store keeps its single shared token. With a +store, a token is one user's, each route / tool / privileged command needs a +capability of that user's role, and the audit entries name the user. + +Nothing here touches the real mouse, keyboard or screen: REST handlers that +would are replaced by recorders, and the MCP tools are local fakes. +""" +import json +import urllib.error +import urllib.request + +import pytest + +from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.executor.action_executor import executor +from je_auto_control.utils.mcp_server.audit import AuditLogger +from je_auto_control.utils.mcp_server.http_transport import DEFAULT_PATH, HttpMCPServer +from je_auto_control.utils.mcp_server.prompts import StaticPromptProvider +from je_auto_control.utils.mcp_server.resources import ChainProvider +from je_auto_control.utils.mcp_server.server import MCPServer +from je_auto_control.utils.mcp_server.tools import MCPTool, build_default_tool_registry +from je_auto_control.utils.mcp_server.tools._base import DESTRUCTIVE, READ_ONLY, schema +from je_auto_control.utils.rbac import ( + USERS_ENV, AuthorizationContext, AuthorizationError, Capability, Role, + UserStore, authorization_scope, authorize_command, capability_for_route, + capability_for_tool, current_authorization, denied_command_in, +) +from je_auto_control.utils.rbac.policy import ( + COMMAND_CAPABILITIES, REST_ROUTE_CAPABILITIES, TOOL_CAPABILITIES, +) +from je_auto_control.utils.rest_api import rest_server +from je_auto_control.utils.rest_api.rest_auth import RestAuthGate +from je_auto_control.utils.rest_api.rest_server import RestApiServer + +_SCHEME = "http" # NOSONAR localhost-only ephemeral test server; TLS out of scope +_SIGN = "AC_sign_action_file" + + +@pytest.fixture(autouse=True) +def _no_ambient_rbac(monkeypatch): + """The developer's own environment must not switch RBAC on for these tests.""" + monkeypatch.delenv(USERS_ENV, raising=False) + monkeypatch.delenv("JE_AUTOCONTROL_MCP_TOKEN", raising=False) + + +@pytest.fixture() +def users(tmp_path): + """A store with one user per role; ``tokens`` maps the role to its token.""" + store = UserStore(tmp_path / "users.json") + store.tokens = { + role: store.add_user(user_id=f"{role}-user", display_name=role, role=role) + for role in Role.all() + } + return store + + +class _FakeAudit: + """Stands in for the tamper-evident audit log; keeps what it was given.""" + + def __init__(self): + self.rows = [] + + def log(self, event_type, **fields): + self.rows.append({"event_type": event_type, **fields}) + + def query(self, **_kwargs): + return list(self.rows) + + +@pytest.fixture() +def rest(monkeypatch): + """Start REST servers on ephemeral ports; ``/execute`` only records its body.""" + executed = [] + + def fake_execute(ctx): + executed.append(ctx.body) + return 200, {"result": "ran"} + + monkeypatch.setitem(rest_server._POST_ROUTES, "/execute", fake_execute) + started = [] + + def start(**kwargs): + server = RestApiServer(host="127.0.0.1", port=0, enable_audit=False, **kwargs) + server._audit_log = _FakeAudit() + server.start() + server.executed = executed + started.append(server) + return server + + yield start + for server in started: + server.stop(timeout=1.0) + + +def _call(server, method, path, token=None, body=None): + """``(status, decoded JSON)`` of one request; an error status is returned, not raised.""" + host, port = server.address + data = json.dumps(body).encode("utf-8") if body is not None else None + headers = {"Content-Type": "application/json"} if data else {} + if token is not None: + headers["Authorization"] = f"Bearer {token}" + request = urllib.request.Request( + f"{_SCHEME}://{host}:{port}{path}", data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(request, timeout=5) as response: # nosec B310 # reason: loopback test server + return response.status, json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as error: + return error.code, json.loads(error.read().decode("utf-8")) + + +# --- REST --------------------------------------------------------------- + +def test_unconfigured_rbac_keeps_shared_token(rest): + server = rest(token="shared-secret") + assert server.user_store is None + assert _call(server, "GET", "/commands", "shared-secret")[0] == 200 + status, _body = _call(server, "POST", "/execute", "shared-secret", {"actions": [[_SIGN]]}) + assert status == 200, "the shared token is not restricted by any role" + assert _call(server, "GET", "/commands", "wrong")[0] == 401 + assert "user=" not in server._audit_log.rows[0]["detail"] + assert "viewer_id" not in server._audit_log.rows[0] + + +def test_viewer_cannot_execute(rest, users): + server = rest(user_store=users) + viewer = users.tokens[Role.VIEWER] + assert _call(server, "GET", "/commands", viewer)[0] == 200 + status, body = _call(server, "POST", "/execute", viewer, + {"actions": [["AC_slugify", {"text": "x"}]]}) + assert status == 403 + assert body == {"error": "forbidden", "role": Role.VIEWER, + "required_capability": Capability.DRIVE_INPUT} + assert server.executed == [] + + +def test_operator_executes_but_cannot_sign(rest, users, tmp_path): + server = rest(user_store=users) + operator = users.tokens[Role.OPERATOR] + target = tmp_path / "actions.json" + target.write_text("[]", encoding="utf-8") + assert _call(server, "POST", "/execute", operator, + {"actions": [["AC_slugify", {"text": "x"}]]})[0] == 200 + assert len(server.executed) == 1 + nested = {"actions": [["AC_loop", {"times": 1, "body": [[_SIGN, {"path": str(target)}]]}]]} + status, body = _call(server, "POST", "/execute", operator, nested) + assert status == 403 + assert body["required_capability"] == Capability.SIGN_ACTIONS + assert body["command"] == _SIGN + assert len(server.executed) == 1, "the refused list must not reach the handler" + + +def test_admin_may_sign_and_shared_token_is_refused_under_rbac(rest, users): + server = rest(user_store=users, token="shared-secret") + admin = users.tokens[Role.ADMIN] + assert _call(server, "POST", "/execute", admin, {"actions": [[_SIGN]]})[0] == 200 + assert _call(server, "GET", "/commands", "shared-secret")[0] == 401 + assert _call(server, "GET", "/commands")[0] == 401 + + +@pytest.mark.parametrize("role,path,expected", [ + (Role.VIEWER, "/audit/list", 403), + (Role.OPERATOR, "/audit/list", 403), + (Role.OPERATOR, "/metrics", 200), + (Role.ADMIN, "/metrics", 200), +]) +def test_get_routes_follow_the_role(rest, users, role, path, expected): + server = rest(user_store=users) + assert _call_status(server, path, users.tokens[role]) == expected + + +def _call_status(server, path, token): + host, port = server.address + request = urllib.request.Request( + f"{_SCHEME}://{host}:{port}{path}", headers={"Authorization": f"Bearer {token}"}) + try: + with urllib.request.urlopen(request, timeout=5) as response: # nosec B310 # reason: loopback test server + return response.status + except urllib.error.HTTPError as error: + return error.code + + +def test_operator_cannot_sign_through_an_action_file(users, tmp_path): + """The file's content is never seen by the server: the executor refuses it.""" + target = tmp_path / "payload.json" + target.write_text("[]", encoding="utf-8") + script = tmp_path / "script.json" + script.write_text(json.dumps( + [["AC_parallel", {"branches": [[[_SIGN, {"path": str(target), "key": "k"}]]]}], + [_SIGN, {"path": str(target), "key": "k"}]]), encoding="utf-8") + operator = AuthorizationContext("operator-user", Role.OPERATOR) + with authorization_scope(operator): + executor.execute_files([str(script)]) + assert not (tmp_path / "payload.json.sig").exists() + with authorization_scope(AuthorizationContext("admin-user", Role.ADMIN)): + executor.execute_files([str(script)]) + assert (tmp_path / "payload.json.sig").exists() + + +def test_executor_check_is_typed_and_silent_outside_a_scope(): + authorize_command(_SIGN) # no scope: the library used directly is unrestricted + assert current_authorization() is None + with authorization_scope(AuthorizationContext("op", Role.OPERATOR)): + authorize_command("AC_click_mouse") + with pytest.raises(AuthorizationError) as raised: + authorize_command(_SIGN) + assert raised.value.capability == Capability.SIGN_ACTIONS + assert isinstance(raised.value, (AutoControlException, PermissionError)) + with authorization_scope(None): + authorize_command(_SIGN) + assert current_authorization() is None + + +def test_rest_audit_rows_name_the_user(rest, users): + server = rest(user_store=users) + _call(server, "GET", "/commands", users.tokens[Role.OPERATOR]) + _call(server, "POST", "/execute", users.tokens[Role.VIEWER], {"actions": []}) + allowed, refused = server._audit_log.rows + assert allowed["viewer_id"] == "operator-user" + assert "user=operator-user role=operator" in allowed["detail"] + assert refused["viewer_id"] == "viewer-user" + assert "forbidden:drive_input" in refused["detail"] + + +def test_user_changes_apply_without_a_restart(rest, users): + server = rest(user_store=users) + viewer = users.tokens[Role.VIEWER] + assert _call(server, "GET", "/commands", viewer)[0] == 200 + # Another process manages the users: a second store on the same file. + manager = UserStore(users.path) + manager.set_role("viewer-user", Role.OPERATOR) + assert _call(server, "POST", "/execute", viewer, {"actions": []})[0] == 200 + manager.remove_user("viewer-user") + assert _call(server, "GET", "/commands", viewer)[0] == 401 + + +def test_broken_or_empty_store_admits_nobody(rest, tmp_path): + path = tmp_path / "users.json" + path.write_text("{not json", encoding="utf-8") + server = rest(user_store=UserStore(path), token="shared-secret") + assert _call(server, "GET", "/commands", "shared-secret")[0] == 401 + + +def test_environment_variable_switches_rbac_on(monkeypatch, users): + assert RestApiServer(enable_audit=False).user_store is None + assert HttpMCPServer(mcp=_mcp([]))._users is None + monkeypatch.setenv(USERS_ENV, str(users.path)) + assert RestApiServer(enable_audit=False).user_store.path == users.path.resolve() + assert HttpMCPServer(mcp=_mcp([]))._users.path == users.path.resolve() + + +def test_gate_check_keeps_its_string_verdicts(users): + gate = RestAuthGate("shared", user_store=users) + header = f"Bearer {users.tokens[Role.VIEWER]}" + assert gate.rbac_enabled + assert gate.check(client_ip="1.1.1.1", header_value=header) == "ok" + assert gate.check(client_ip="1.1.1.1", header_value="Bearer shared") == "unauthorized" + result = gate.authenticate(client_ip="1.1.1.1", header_value=header) + assert result.context == AuthorizationContext("viewer-user", Role.VIEWER) + legacy = RestAuthGate("shared").authenticate(client_ip="1.1.1.1", header_value="Bearer shared") + assert (legacy.verdict, legacy.context) == ("ok", None) + + +# --- the policy tables -------------------------------------------------- + +def test_every_rest_route_has_a_decided_capability(): + routes = {("GET", path) for path in rest_server._GET_ROUTES} + routes |= {("POST", path) for path in rest_server._POST_ROUTES} + routes.add(("GET", "/metrics")) + routes -= {("GET", path) for path in rest_server._PUBLIC_PATHS} + assert routes == set(REST_ROUTE_CAPABILITIES) + assert capability_for_route("POST", "/not/listed") == Capability.MANAGE_HOSTS + + +def test_policy_tables_name_things_that_exist(): + assert set(COMMAND_CAPABILITIES) <= executor.known_commands() + assert set(TOOL_CAPABILITIES) <= {tool.name for tool in build_default_tool_registry()} + known = set(Capability.all()) + assert set(COMMAND_CAPABILITIES.values()) | set(TOOL_CAPABILITIES.values()) <= known + assert set(REST_ROUTE_CAPABILITIES.values()) <= known + + +def test_only_admin_holds_the_signing_capability(): + assert capability_for_tool("anything", read_only=True) == Capability.READ_SCREEN + assert capability_for_tool("anything", read_only=False) == Capability.DRIVE_INPUT + holders = [role for role in Role.all() + if AuthorizationContext("u", role).allows(Capability.SIGN_ACTIONS)] + assert holders == [Role.ADMIN] + + +def test_denied_command_is_found_wherever_a_list_nests(): + operator = AuthorizationContext("op", Role.OPERATOR) + assert denied_command_in({"actions": [["AC_click_mouse"]]}, operator) is None + deep = {"spec": {"steps": [{"then": [["AC_try", {"body": [[_SIGN, {}]]}]]}]}} + assert denied_command_in(deep, operator) == (_SIGN, Capability.SIGN_ACTIONS) + assert denied_command_in(deep, AuthorizationContext("root", Role.ADMIN)) is None + + +# --- MCP ---------------------------------------------------------------- + +def _mcp(tools, audit_path=None): + return MCPServer(tools=tools, resource_provider=ChainProvider([]), + prompt_provider=StaticPromptProvider([]), + audit_logger=AuditLogger(path=audit_path)) + + +@pytest.fixture() +def mcp_http(tmp_path): + """Start MCP HTTP servers over three fake tools; ``calls`` records invocations.""" + calls = [] + + def tool(name, annotations): + def handler(**arguments): + calls.append((name, arguments)) + return {"ran": name} + return MCPTool(name=name, description=name, annotations=annotations, handler=handler, + input_schema=schema({"actions": {"type": "array"}})) + + tools = [tool("peek", READ_ONLY), tool("poke", DESTRUCTIVE), + tool("ac_load_plugins", DESTRUCTIVE)] + audit_path = tmp_path / "mcp_audit.jsonl" + started = [] + + def start(**kwargs): + server = HttpMCPServer(mcp=_mcp(tools, str(audit_path)), host="127.0.0.1", port=0, + **kwargs) + server.start() + server.calls = calls + server.audit_path = audit_path + started.append(server) + return server + + yield start + for server in started: + server.stop(timeout=1.0) + + +def _rpc(server, method, params=None, token=None): + return _call(server, "POST", DEFAULT_PATH, token, + {"jsonrpc": "2.0", "id": 1, "method": method, "params": params or {}}) + + +def _tool_names(server, token): + _status, body = _rpc(server, "tools/list", token=token) + return [tool["name"] for tool in body["result"]["tools"]] + + +def _audit_entries(server): + return [json.loads(line) for line in + server.audit_path.read_text(encoding="utf-8").splitlines()] + + +def test_mcp_tool_list_matches_what_each_role_may_call(mcp_http, users): + server = mcp_http(user_store=users) + expected = {Role.VIEWER: ["peek"], Role.OPERATOR: ["peek", "poke"], + Role.ADMIN: ["peek", "poke", "ac_load_plugins"]} + for role, names in expected.items(): + token = users.tokens[role] + assert _tool_names(server, token) == names + for name in ("peek", "poke", "ac_load_plugins"): + _status, body = _rpc(server, "tools/call", {"name": name, "arguments": {}}, token) + assert ("result" in body) is (name in names), (role, name, body) + + +def test_mcp_viewer_cannot_execute(mcp_http, users): + server = mcp_http(user_store=users) + status, body = _rpc(server, "tools/call", {"name": "poke", "arguments": {}}, + users.tokens[Role.VIEWER]) + assert status == 200 + assert body["error"]["code"] == -32003 + assert body["error"]["data"] == {"required_capability": Capability.DRIVE_INPUT} + assert server.calls == [] + + +def test_mcp_operator_cannot_sign_through_an_action_list(mcp_http, users): + server = mcp_http(user_store=users) + arguments = {"actions": [["AC_if_var", {"then": [[_SIGN, {"path": "x"}]]}]]} + _status, body = _rpc(server, "tools/call", {"name": "poke", "arguments": arguments}, + users.tokens[Role.OPERATOR]) + assert body["error"]["data"] == {"required_capability": Capability.SIGN_ACTIONS} + assert server.calls == [] + _status, body = _rpc(server, "tools/call", {"name": "poke", "arguments": arguments}, + users.tokens[Role.ADMIN]) + assert "result" in body and len(server.calls) == 1 + + +def test_user_id_in_audit(mcp_http, users): + server = mcp_http(user_store=users) + _rpc(server, "tools/call", {"name": "poke", "arguments": {}}, users.tokens[Role.OPERATOR]) + _rpc(server, "tools/call", {"name": "poke", "arguments": {}}, users.tokens[Role.VIEWER]) + ran, refused = _audit_entries(server) + assert (ran["user_id"], ran["role"], ran["status"]) == ("operator-user", "operator", "ok") + assert (refused["user_id"], refused["status"]) == ("viewer-user", "denied") + assert "drive_input" in refused["error"] + + +def test_mcp_rbac_refuses_the_shared_token_and_unknown_roles(mcp_http, users): + users.add_user(user_id="odd", display_name="odd", role=Role.VIEWER, token="odd-token") + stored = json.loads(users.path.read_text(encoding="utf-8")) + stored["users"][-1]["role"] = "superuser" + users.path.write_text(json.dumps(stored), encoding="utf-8") + server = mcp_http(user_store=users, auth_token="shared-secret") + assert _rpc(server, "tools/list", token="shared-secret")[0] == 401 + assert _rpc(server, "tools/list")[0] == 401 + assert _rpc(server, "tools/list", token="odd-token")[0] == 403 + + +def test_mcp_unconfigured_rbac_keeps_shared_token(mcp_http): + server = mcp_http(auth_token="shared-secret") + assert _rpc(server, "tools/list")[0] == 401 + assert _tool_names(server, "shared-secret") == ["peek", "poke", "ac_load_plugins"] + _status, body = _rpc(server, "tools/call", {"name": "ac_load_plugins", "arguments": {}}, + "shared-secret") + assert "result" in body + assert "user_id" not in _audit_entries(server)[0] + assert _tool_names(mcp_http(), None) == ["peek", "poke", "ac_load_plugins"] + + +def test_default_registry_offers_a_viewer_only_read_only_tools(users): + registry = build_default_tool_registry(read_only=False) + with authorization_scope(AuthorizationContext("viewer-user", Role.VIEWER)): + listed = MCPServer(tools=registry).handle_line( + json.dumps({"jsonrpc": "2.0", "id": 1, "method": "tools/list"})) + names = {tool["name"] for tool in json.loads(listed)["result"]["tools"]} + assert names == {tool.name for tool in registry if tool.annotations.read_only} + assert "ac_execute_actions" not in names and "ac_click_mouse" not in names From 70cb44863414373d7d50635adb60017cbb66d9a7 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:04:11 +0800 Subject: [PATCH 21/26] Keep the analyser marker within the line limit --- architecture_explore.md | 8 ++++---- je_auto_control/gui/tab_registry.py | 3 ++- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/architecture_explore.md b/architecture_explore.md index 53d2b9873..cff317661 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -20,7 +20,7 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | | Python 模組總數（含周邊子專案） | 1,068 | -| 程式碼總行數 | 158,286 | +| 程式碼總行數 | 158,287 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 778 | | 套件門面 `__all__` 公開名稱數 | 1,244 | @@ -883,7 +883,7 @@ GUI 是**選用 extra**（`pip install je_auto_control[gui]`，PySide6 + qt-mate | `gui/__init__.py` | 25 | `start_autocontrol_gui()`：**唯一**會延遲匯入 PySide6 的地方，維持頂層套件 Qt-free。 | | `main_window.py` | 379 | `QMainWindow`：選單列（File／Actions／View／…）、左側導覽面板 dock（`Ctrl+K` 搜尋、`Ctrl+B` 收合）、即時語言切換、字級預設、深色／淺色主題（`theme.py` 的 token，不再用 qt-material）。分頁分為 core／editing／detection／automation／system 五類。 | | `main_widget.py` | 368 | 工作區：擁有 `QTabWidget`，依 `tab_registry.TAB_SPECS` 註冊 48 個分頁，並暴露 show/hide/activate/list API 給選單列與導覽面板。只有預設開啟的三個分頁與自己的 mixin 表單在啟動時建立，其餘第一次開啟才匯入模組、建立 widget。核心分頁在 `_own_tab_builders` 宣告 `(label_key, handler)` 動作對。 | -| `tab_registry.py` | 124 | 分頁表：每個分頁一筆 `TabSpec`（鍵、標題鍵、分類、模組與類別名），`TabEntry` 在第一次存取 `widget` 時才呼叫 factory。不匯入 Qt。 | +| `tab_registry.py` | 125 | 分頁表：每個分頁一筆 `TabSpec`（鍵、標題鍵、分類、模組與類別名），`TabEntry` 在第一次存取 `widget` 時才呼叫 factory。不匯入 Qt。 | | `navigation.py` | 200 | `NavigationPanel`：搜尋框 + 依分類的功能樹，列出每個已註冊分頁（開啟中的以粗體標示），只回報被選的鍵，開啟分頁仍由視窗負責。 | | `theme.py` | 178 | 設計 token（`ThemeTokens`：顏色、圓角、間距、字族）、深色與淺色兩組值、由 token 產生的樣式表與對應的 `QPalette`；不載入圖檔或字型檔。 | | `_auto_click_tab.py` | 291 | 自動點擊分頁的 mixin 建構器。 | @@ -1079,7 +1079,7 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | 層／子系統 | 檔案數 | 行數 | | --- | ---: | ---: | -| `gui/` | 98 | 28,398 | +| `gui/` | 98 | 28,399 | | `utils/mcp_server/` | 35 | 18,898 | | `utils/remote_desktop/` | 56 | 13,014 | | `utils/executor/` | 8 | 9,606 | @@ -1100,5 +1100,5 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 852 | | 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 682 | 57,156 | -| **總計** | **1,062** | **158,221** | +| **總計** | **1,062** | **158,222** | diff --git a/je_auto_control/gui/tab_registry.py b/je_auto_control/gui/tab_registry.py index 4562045d4..dd9633464 100644 --- a/je_auto_control/gui/tab_registry.py +++ b/je_auto_control/gui/tab_registry.py @@ -31,7 +31,8 @@ def lazy_factory(module: str, class_name: str) -> WidgetFactory: """Return a factory that imports ``module`` and builds ``class_name``.""" def build() -> Any: # reason: both names come from the TAB_SPECS table below, never from user input. - return getattr(import_module(module), class_name)() # nosemgrep: python.lang.security.audit.non-literal-import.non-literal-import + loaded = import_module(module) # nosemgrep: python.lang.security.audit.non-literal-import.non-literal-import + return getattr(loaded, class_name)() return build From 4a41551462f11bc4a7f915cc3939de1b0e79250c Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:11:52 +0800 Subject: [PATCH 22/26] Re-measure the module map after merging the sweep branches --- architecture_explore.md | 194 ++++++++++++++++++++-------------------- 1 file changed, 97 insertions(+), 97 deletions(-) diff --git a/architecture_explore.md b/architecture_explore.md index cff317661..561375631 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -19,8 +19,8 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | -| Python 模組總數（含周邊子專案） | 1,068 | -| 程式碼總行數 | 158,287 | +| Python 模組總數（含周邊子專案） | 1,074 | +| 程式碼總行數 | 160,861 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 778 | | 套件門面 `__all__` 公開名稱數 | 1,244 | @@ -154,9 +154,9 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | 模組 | 行數 | 職責 | | --- | ---: | --- | -| `je_auto_control/__init__.py` | 1,978 | **套件門面**。集中匯入並再匯出 1,251 個公開名稱，以功能區塊註解分段（callback／exception／executor／a11y／vision／clipboard…）。 | +| `je_auto_control/__init__.py` | 1,986 | **套件門面**。集中匯入並再匯出 1,251 個公開名稱，以功能區塊註解分段（callback／exception／executor／a11y／vision／clipboard…）。 | | `je_auto_control/__main__.py` | 98 | 舊版 argparse 進入點：`-e` 執行單檔、`-d` 執行整個目錄、`--execute_str` 執行 JSON 字串、`-c` 建立專案。 | -| `je_auto_control/cli.py` | 353 | **主 CLI**（`je_auto_control` console script）。子命令：`run`（含 `--var`／`--dry-run`）、`validate`／`lint`、`list-commands`、`fmt`、`record`、`codegen`、`failure-bundle`、`list-jobs`、`start-server`、`start-rest`、`version`。所有子命令延遲匯入，確保不碰 Qt。 | +| `je_auto_control/cli.py` | 372 | **主 CLI**（`je_auto_control` console script）。子命令：`run`（含 `--var`／`--dry-run`）、`validate`／`lint`、`list-commands`、`fmt`、`record`、`codegen`、`failure-bundle`、`list-jobs`、`start-server`、`start-rest`、`version`。所有子命令延遲匯入，確保不碰 Qt。 | | `je_auto_control/api/__init__.py` | 22 | 版本化整合進入點。 | | `je_auto_control/api/core.py` | 19 | **穩定無頭 API 門面**：只暴露 `execute_action`、`execute_action_with_vars`、`generate_code`、`run_diagnostics`、`create_failure_bundle`、`failure_bundle_on_error`、`FailureBundleOptions`。mypy 型別契約以此為起點，現已擴到整包（見「設定基線」）。 | | `je_auto_control/utils/cli_output.py` | 44 | 本套件命令列工具與 stdio 伺服器的標準串流：不論碼頁一律 UTF-8。 | @@ -172,45 +172,45 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | 模組 | 行數 | 職責 | | --- | ---: | --- | | `wrapper/platform_wrapper.py` | 119 | **Strategy 樞紐**。依 `sys.platform` 匯入唯一後端並匯出 `keyboard`、`keyboard_check`、`keyboard_keys_table`、`mouse`、`mouse_keys_table`、`special_mouse_keys_table`、`screen`、`recorder`；八個名稱都帶著 `backend_contract` 的型別出去，其中 `keyboard`／`mouse` 因為四個分支綁的是三種互不相容的形狀，先落在私有的 `_keyboard`／`_mouse`（`Any`）上再標合約；載入失敗直接拋 `AutoControlException`（fail fast）。 | -| `wrapper/backend_contract.py` | 238 | 平台縫的型別合約：`ScreenBackend`／`KeyboardCheckBackend`／`RecorderBackend` 三個跨平台 Protocol，加上 `keyboard`／`mouse` 各自的三份——`Win32*`（SendInput 與 Interception）、`Darwin*`（Quartz）、`X11Unix*`（XTest／uinput／Wayland／BSD），因為這兩個名稱的呼叫形狀真的因平台而異；`KeyboardBackend`／`MouseBackend` 依 `sys.platform` 別名到其中一組，所以呼叫端被檢查的是它真的會走到的簽章。四個 `_platform_*` 組裝模組各自標注自己綁的是什麼，少一個成員就在該後端自己的檔案裡紅掉，而不是在三層之上的呼叫點。 | +| `wrapper/backend_contract.py` | 249 | 平台縫的型別合約：`ScreenBackend`／`KeyboardCheckBackend`／`RecorderBackend` 三個跨平台 Protocol，加上 `keyboard`／`mouse` 各自的三份——`Win32*`（SendInput 與 Interception）、`Darwin*`（Quartz）、`X11Unix*`（XTest／uinput／Wayland／BSD），因為這兩個名稱的呼叫形狀真的因平台而異；`KeyboardBackend`／`MouseBackend` 依 `sys.platform` 別名到其中一組，所以呼叫端被檢查的是它真的會走到的簽章。四個 `_platform_*` 組裝模組各自標注自己綁的是什麼，少一個成員就在該後端自己的檔案裡紅掉，而不是在三層之上的呼叫點。 | | `wrapper/_platform_windows.py` | 366 | Windows 後端組裝：Win32 ctypes 模組 + 虛擬鍵表（含 OEM 鍵與 `keyboard_key_aliases` 別名表）+ 選用 Interception 驅動。 | | `wrapper/_platform_osx.py` | 162 | macOS 後端組裝（Quartz 事件 + osx 虛擬鍵表）。 | | `wrapper/_platform_linux.py` | 278 | X11 後端組裝（python-Xlib + 選用 uinput）。 | | `wrapper/_platform_wayland.py` | 63 | Wayland 後端組裝（libei／ydotool／grim）。 | -| `wrapper/auto_control_mouse.py` | 537 | 滑鼠 API：位置讀寫、按下／放開／點擊、捲動、座標前處理、送訊息給指定視窗。 | -| `wrapper/auto_control_keyboard.py` | 432 | 鍵盤 API：鍵表查詢、`keyboard_key_name` 反查標準鍵名（不回別名）、按下／放開／敲擊、`write` 字串、`hotkey` 組合鍵、按鍵狀態偵測。**`type_keyboard` 與 `hotkey` 的放開走 `finally`**（見下）。 | +| `wrapper/auto_control_mouse.py` | 575 | 滑鼠 API：位置讀寫、按下／放開／點擊、捲動、座標前處理、送訊息給指定視窗。 | +| `wrapper/auto_control_keyboard.py` | 521 | 鍵盤 API：鍵表查詢、`keyboard_key_name` 反查標準鍵名（不回別名）、按下／放開／敲擊、`write` 字串、`hotkey` 組合鍵、按鍵狀態偵測。**`type_keyboard` 與 `hotkey` 的放開走 `finally`**（見下）。 | | `wrapper/auto_control_screen.py` | 111 | 螢幕 API：`screen_size`、`screenshot`（可指定區域）、`get_pixel`。 | -| `wrapper/auto_control_image.py` | 83 | 影像 API：`locate_all_image`、`locate_image_center`、`locate_and_click`。 | +| `wrapper/auto_control_image.py` | 85 | 影像 API：`locate_all_image`、`locate_image_center`、`locate_and_click`。 | | `wrapper/auto_control_record.py` | 124 | 錄製 API：`record`／`stop_record`／`record_to_json`（支援 stop event 與逾時）。 | -| `wrapper/auto_control_window.py` | 287 | 視窗管理門面：列舉、尋找、聚焦、等待、關閉、顯示狀態、幾何、所屬行程 PID、依行程列舉／最小化視窗、不搶焦點的投遞式輸入（目前僅 Windows 實作）。 | -| `wrapper/window_backends/` | 1,004 | 視窗管理的平台縫（`base` / `windows_backend` / `x11_backend` / `macos_backend` / `null_backend`）。放在 `wrapper/` 而不是 `utils/`，因為它必須 import `windows/`、`linux_with_x11/`、`osx/`，而 `utils/` 在分層上在那三者之上。 | +| `wrapper/auto_control_window.py` | 329 | 視窗管理門面：列舉、尋找、聚焦、等待、關閉、顯示狀態、幾何、所屬行程 PID、依行程列舉／最小化視窗、不搶焦點的投遞式輸入（目前僅 Windows 實作）。 | +| `wrapper/window_backends/` | 1,100 | 視窗管理的平台縫（`base` / `windows_backend` / `x11_backend` / `macos_backend` / `null_backend`）。放在 `wrapper/` 而不是 `utils/`，因為它必須 import `windows/`、`linux_with_x11/`、`osx/`，而 `utils/` 在分層上在那三者之上。 | ### 5.3 平台後端 -#### Windows（`windows/`，23 檔／1,976 行） +#### Windows（`windows/`，23 檔／2,154 行） | 模組 | 行數 | 職責 | | --- | ---: | --- | -| `core/utils/win32_ctype_input.py` | 73 | `SendInput` 的 ctypes 結構定義與送出。 | +| `core/utils/win32_ctype_input.py` | 76 | `SendInput` 的 ctypes 結構定義與送出。 | | `core/utils/win32_vk.py` | 205 | Windows 虛擬鍵碼對照表。 | | `core/utils/win32_keypress_check.py` | 22 | `GetAsyncKeyState` 按鍵狀態查詢。 | | `mouse/win32_ctype_mouse_control.py` | 220 | 滑鼠事件產生（含多螢幕絕對座標換算）。 | | `keyboard/win32_ctype_keyboard_control.py` | 98 | 鍵盤事件產生。 | | `record/win32_input_hook.py` | 253 | 單一一組低階鍵鼠 hook（`WH_KEYBOARD_LL`／`WH_MOUSE_LL`）＋訊息迴圈，產生帶時間戳的事件時間軸；停止時以 `PostThreadMessageW(WM_QUIT)` 收掉執行緒，不會每錄一次就漏一條。 | | `record/win32_record.py` | 41 | 把 `win32_input_hook` 的時間軸轉成 action list（含按鍵放開、滾輪與間隔）；整形本體與 macOS 共用 `utils/input_macro/recorder_base.py`。 | -| `screen/win32_screen.py` | 95 | 螢幕尺寸與像素讀取。**每支 Win32 函式都明寫 argtypes/restype**（HDC 是指標寬度，走預設的 c_int 會截斷，錯誤會沉默地擴散到 GetPixel／ReleaseDC），並持有自己的 user32／gdi32 handle。import 時呼叫 `SetProcessDPIAware()`——**行程層級且不可還原**，實體↔邏輯座標換算請走 `utils/monitor_layout`。 | -| `window/windows_window_manage.py` | 374 | 視窗列舉／聚焦／關閉／最小化／幾何／所屬行程 PID／投遞式輸入（`auto_control_window` 的實作）。**每支 Win32 函式都明寫 argtypes/restype**，並持有自己的 user32 handle，避免把原型外溢到別的模組；hwnd 一律是 int。 | +| `screen/win32_screen.py` | 171 | 螢幕尺寸與像素讀取。**每支 Win32 函式都明寫 argtypes/restype**（HDC 是指標寬度，走預設的 c_int 會截斷，錯誤會沉默地擴散到 GetPixel／ReleaseDC），並持有自己的 user32／gdi32 handle。import 時呼叫 `SetProcessDPIAware()`——**行程層級且不可還原**，實體↔邏輯座標換算請走 `utils/monitor_layout`。 | +| `window/windows_window_manage.py` | 473 | 視窗列舉／聚焦／關閉／最小化／幾何／所屬行程 PID／投遞式輸入（`auto_control_window` 的實作）。**每支 Win32 函式都明寫 argtypes/restype**，並持有自己的 user32 handle，避免把原型外溢到別的模組；hwnd 一律是 int。 | | `message/window_message.py` | 97 | 直接對視窗送 `WM_*` 訊息（背景輸入）。 | | `interception/_dll.py` | 232 | `interception.dll` 的延遲 ctypes 載入與結構定義。 | | `interception/keyboard.py` | 70 | 經 Interception 驅動的鍵盤輸入（繞過部分反自動化偵測）。 | | `interception/mouse.py` | 160 | 經 Interception 驅動的滑鼠輸入。 | -#### macOS（`osx/`，17 檔／925 行） +#### macOS（`osx/`，17 檔／981 行） | 模組 | 行數 | 職責 | | --- | ---: | --- | -| `core/utils/osx_vk.py` | 113 | macOS 虛擬鍵碼表。 | -| `mouse/osx_mouse.py` | 143 | Quartz `CGEvent` 滑鼠事件。 | +| `core/utils/osx_vk.py` | 119 | macOS 虛擬鍵碼表。 | +| `mouse/osx_mouse.py` | 193 | Quartz `CGEvent` 滑鼠事件。 | | `keyboard/osx_keyboard.py` | 144 | Quartz 鍵盤事件。 | | `keyboard/osx_keyboard_check.py` | 24 | 按鍵狀態查詢。 | | `listener/osx_listener.py` | 261 | 專屬執行緒上的 listen-only `CGEventTap`＋自己的 `CFRunLoopRunInMode` 切片；不在 import 時建 `NSApplication`，也不用會卡住呼叫緒的 `AppHelper.runEventLoop()`。修飾鍵由 `flagsChanged` 的旗標還原成 press／release，座標取 `CGEventGetLocation`（左上原點，與重播送出的座標同一空間）。 | @@ -272,18 +272,18 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.1 執行引擎與腳本資產 -> 24 個套件、約 14,735 行。 +> 24 個套件、約 15,216 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | | `utils/action_lint/` | 429 | action 檔 linter 與 JSON Schema 產生器（CI 用 `python -m` 進入點） | -| `utils/action_signing/` | 380 | action 檔 HMAC-SHA256 簽章與 Fernet 加密，`execute_files` 會強制驗簽 | +| `utils/action_signing/` | 716 | action 檔 HMAC-SHA256 簽章與 Fernet 加密，`execute_files` 會強制驗簽 | | `utils/checkpoint/` | 129 | 流程檢查點與續跑，讓長 action list 具持久性 | | `utils/codegen/` | 294 | 由 action list 產生可執行的 pytest / python / robot 測試碼 | -| `utils/dag/` | 536 | 跨主機 DAG 編排器（圖模型 + runner） | +| `utils/dag/` | 559 | 跨主機 DAG 編排器（圖模型 + runner） | | `utils/decision_table/` | 112 | DMN 風格決策表：規則 + 命中策略，把分支外部化 | | `utils/deterministic/` | 116 | 決定性執行控制：固定亂數種子 + 凍結時鐘 | -| `utils/executor/` | 9,606 | **核心**。`Executor` 指令分派表（778 個 `AC_*`）、參數插值、乾跑、逐步 callback；`flow_control` 提供 34 個區塊指令（迴圈／分支／try／巨集／變數） | +| `utils/executor/` | 9,659 | **核心**。`Executor` 指令分派表（778 個 `AC_*`）、參數插值、乾跑、逐步 callback；`flow_control` 提供 34 個區塊指令（迴圈／分支／try／巨集／變數） | | `utils/flow_debugger/` | 166 | action list 的單步除錯器與追蹤器 | | `utils/input_macro/` | 462 | 定時輸入事件：錄製結果的整形（`timeline`／`InputRecorder`，Windows 與 macOS 共用）、重播與宣告式輸入序列 DSL | | `utils/json/` | 99 | action JSON 檔讀寫與正規化格式化（`fmt --check` 的後端） | @@ -294,7 +294,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/project/` | 183 | 專案腳手架：建立目錄結構與範本 action 檔 | | `utils/recording_edit/` | 165 | 不重錄的前提下裁切／過濾／縮放已錄製的 action list | | `utils/saga/` | 103 | Saga 協調器：失敗時以 LIFO 補償動作回滾 | -| `utils/script_vars/` | 218 | 執行期變數作用域與 `${var}` / `${secrets.*}` 插值 | +| `utils/script_vars/` | 287 | 執行期變數作用域與 `${var}` / `${secrets.*}` 插值 | | `utils/skill_library/` | 145 | 具名可重用 action 序列（skill）的持久化倉庫 | | `utils/state_machine/` | 268 | 宣告式有限狀態機驅動 action JSON | | `utils/stubs/` | 311 | 為 `AC_*` 指令面產生型別 stub | @@ -303,7 +303,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.2 框架基礎設施 -> 14 個套件、約 3,105 行。 +> 14 個套件、約 3,256 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -312,19 +312,19 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/critical_exit/` | 132 | 監看緊急停止鍵的守護執行緒，用於中止失控腳本 | | `utils/diagnostics/` | 330 | 跨子系統的「一切正常嗎」健檢，附 `python -m` 進入點 | | `utils/dbus_client/` | 714 | 只用標準函式庫的 D-Bus session bus 客戶端。原本在 `linux_wayland/` 為 portal 交握而寫，AT-SPI 無障礙後端成為第二個使用者後搬到這裡（`utils/` 在分層上在各 OS 套件之上） | -| `utils/exception/` | 213 | **例外階層根**。所有錯誤繼承 `AutoControlException`，加上集中式錯誤訊息字串（`exception_tags`） | +| `utils/exception/` | 223 | **例外階層根**。所有錯誤繼承 `AutoControlException`，加上集中式錯誤訊息字串（`exception_tags`） | | `utils/failure_bundle/` | 229 | 可攜、已遮蔽的失敗診斷 ZIP（截圖 + 診斷 + log 尾段） | | `utils/file_process/` | 40 | 目錄檔案列舉（`execute_dir` 的後端） | | `utils/logging/` | 168 | `autocontrol_logger` 單例 + 家目錄共用記錄檔 handler（`JE_AUTOCONTROL_LOG_FILE` 可改） | -| `utils/package_manager/` | 155 | 動態載入套件並把 executor 注入其中；載入前先過套件閘門（允許清單加上只能從 Python 呼叫的開關，工作區 X-12） | -| `utils/path_guard/` | 114 | 命令列傳入路徑的正規化與邊界檢查（防路徑穿越） | +| `utils/package_manager/` | 179 | 動態載入套件並把 executor 注入其中；載入前先過套件閘門（允許清單加上只能從 Python 呼叫的開關，工作區 X-12） | +| `utils/path_guard/` | 231 | 命令列傳入路徑的正規化與邊界檢查（防路徑穿越） | | `utils/platform_id/` | 62 | 作業系統家族的單一判定點。`sys.platform` 原本在一百多處跟字面清單比對，而那些清單都沒有 BSD；`is_x11_unix()` 問的是「這是不是 X11 unix」，這才是守衛一直想問的問題 | | `utils/shell_process/` | 279 | `ShellManager`：以 argv list 執行外部命令（禁用 `shell=True`） | | `utils/start_exe/` | 36 | 啟動另一個執行檔行程 | ### 5.4.3 排程、觸發與背景監看 -> 11 個套件、約 4,300 行。 +> 11 個套件、約 4,302 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -336,13 +336,13 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/scheduler/` | 506 | 間隔式與 cron 式的 action JSON 排程器 | | `utils/session_guard/` | 62 | 驅動輸入前先偵測工作階段是否已鎖定／非互動 | | `utils/triggers/` | 1,383 | 事件驅動觸發引擎：影像／視窗／像素／檔案／webhook／IMAP 郵件 | -| `utils/voice/` | 97 | 語音指令路由：把辨識到的語句對應到 `AC_*` action list | +| `utils/voice/` | 99 | 語音指令路由：把辨識到的語句對應到 `AC_*` action list | | `utils/watchdog/` | 195 | 背景彈窗／中斷看門狗，供無人值守自動化 | | `utils/watcher/` | 90 | 無頭輪詢原語：滑鼠位置、像素顏色、log tail | ### 5.4.4 輸入模擬與動作品質 -> 22 個套件、約 2,919 行。 +> 22 個套件、約 2,971 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -364,14 +364,14 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/step_repair/` | 136 | 失敗／無效動作的修復策略（自我修正迴圈） | | `utils/table_grid_fill/` | 187 | 以 OCR 文字填滿格線表格，取得可定址的表格 | | `utils/input_reach/` | 111 | 送出去的輸入到不到得了：桌面鎖定查詢（免費）＋ 實際送一個 F13 確認沒有被過濾（有副作用，只給診斷用） | -| `utils/keyboard_layout/` | 152 | 向系統問「這個鍵盤配置下每個鍵印出什麼字」（`ToUnicodeEx`），問不到退回 US 對照表 | -| `utils/text_unicode/` | 151 | 輸入任意 Unicode（emoji／CJK／重音字）：優先送字元按鍵事件，不支援時退回剪貼簿貼上 | +| `utils/keyboard_layout/` | 182 | 向系統問「這個鍵盤配置下每個鍵印出什麼字」（`ToUnicodeEx`），問不到退回 US 對照表 | +| `utils/text_unicode/` | 173 | 輸入任意 Unicode（emoji／CJK／重音字）：優先送字元按鍵事件，不支援時退回剪貼簿貼上 | | `utils/tween_drag/` | 182 | 沿曲線的緩動插值拖曳；`_drag_through` 是它與 `drag_path` 共用的按下／移動／放開（`step_delay_s`、`settle_s` 節奏，失敗時在 `finally` 裡於最後到達點放開） | | `utils/verify_field/` | 118 | 打字後讀回欄位，確認內容確實落地 | ### 5.4.5 影像辨識與畫面分析 -> 37 個套件、約 6,064 行。 +> 37 個套件、約 6,397 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -381,7 +381,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/color_region/` | 107 | 以顏色定位畫面區域（遮罩 + 連通元件） | | `utils/color_stats/` | 103 | 區域顏色統計：平均色與主色 | | `utils/coordinate_space/` | 93 | 模型網格座標與實體像素之間的座標空間對映 | -| `utils/cv2_utils/` | 847 | OpenCV 基礎層：擷取後端選擇（`screen_grabber`，Pillow／mss 或平台後端）、截圖、`[left, top, right, bottom]` 區域擷取（`region_capture`，Windows 經 `grab_logical` 涵蓋所有螢幕）、樣板比對（走 `grab_logical`，涵蓋所有螢幕）、螢幕錄影、影片錄製（兩者都經 `frame_clock` 依 fps 配速）、連通元件、影像堆疊的取用口（`optional`，Windows arm64 沒有 wheel 時語意報錯）、非 ASCII 路徑也讀寫得到的影像檔存取（`image_file`） | +| `utils/cv2_utils/` | 929 | OpenCV 基礎層：擷取後端選擇（`screen_grabber`，Pillow／mss 或平台後端）、截圖、`[left, top, right, bottom]` 區域擷取（`region_capture`，Windows 經 `grab_logical` 涵蓋所有螢幕）、樣板比對（走 `grab_logical`，涵蓋所有螢幕）、螢幕錄影、影片錄製（兩者都經 `frame_clock` 依 fps 配速）、連通元件、影像堆疊的取用口（`optional`，Windows arm64 沒有 wheel 時語意報錯）、非 ASCII 路徑也讀寫得到的影像檔存取（`image_file`） | | `utils/edge_lines/` | 131 | 以 Hough 轉換偵測線條／格線／分隔線 | | `utils/edge_match/` | 115 | 邊緣形狀（Chamfer／距離轉換）樣板比對 | | `utils/feature_match/` | 143 | ORB 特徵比對：在旋轉／縮放／主題變更下定位樣板 | @@ -395,7 +395,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/match_ensemble/` | 67 | 多樣板共識比對（多張參考圖投票到同一位置） | | `utils/match_stability/` | 70 | 比對前的靜止閘門與跨影格的比對持續性 | | `utils/match_trust/` | 154 | 樣板比對可信度評分（次峰比 + peak-to-sidelobe） | -| `utils/monitor_layout/` | 320 | 多螢幕／虛擬桌面幾何（在哪個螢幕、位置、重映射）＋ `logical_frame` 以滑鼠座標空間擷取畫面 | +| `utils/monitor_layout/` | 555 | 多螢幕／虛擬桌面幾何（在哪個螢幕、位置、重映射）＋ `logical_frame` 以滑鼠座標空間擷取畫面 | | `utils/motion_regions/` | 78 | 兩影格間的局部變化／活動偵測（absdiff） | | `utils/perceptual_diff/` | 202 | 感知式（YIQ）影像差異，抑制反鋸齒邊緣誤報 | | `utils/preprocess/` | 276 | OCR／比對前的影像前處理（灰階、二值化、去傾斜…） | @@ -404,7 +404,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/saliency/` | 114 | 頻譜殘差視覺顯著性：顯著圖與排序後的顯著區域 | | `utils/scale_detect/` | 105 | 偵測樣板實際渲染的顯示縮放／視覺 DPI | | `utils/screen_grid/` | 146 | 供 VLM 接地用的粗粒度標號網格（點 ↔ 格對映） | -| `utils/set_of_marks/` | 153 | Set-of-Marks 疊圖：為畫面元素編號供 VLM 指認 | +| `utils/set_of_marks/` | 169 | Set-of-Marks 疊圖：為畫面元素編號供 VLM 指認 | | `utils/shape_locator/` | 108 | 以邊緣／輪廓偵測定位元件（矩形／形狀，免樣板） | | `utils/ssim/` | 162 | 結構相似度比較：感知分數 + 變化區域 | | `utils/subpixel_match/` | 109 | 以二次曲面擬合做次像素級比對精修 | @@ -415,7 +415,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.6 OCR 與文字理解 -> 19 個套件、約 3,803 行。 +> 19 個套件、約 3,817 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -428,7 +428,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/guardrail/` | 117 | 針對畫面／OCR 文字的啟發式 prompt-injection 防護 | | `utils/heading_segment/` | 71 | 判定 OCR 行是標題或內文，建出文件大綱 | | `utils/near_dup/` | 108 | 近似重複文字偵測（SimHash／MinHash） | -| `utils/ocr/` | 1,367 | OCR 引擎門面 + 三個後端（Tesseract／EasyOCR／PaddleOCR）、版面結構化與跨詞比對（`text_span`）、Tesseract 的尋找／設定／健康檢查（`tesseract_setup`） | +| `utils/ocr/` | 1,381 | OCR 引擎門面 + 三個後端（Tesseract／EasyOCR／PaddleOCR）、版面結構化與跨詞比對（`text_span`）、Tesseract 的尋找／設定／健康檢查（`tesseract_setup`） | | `utils/pii_text/` | 141 | 自由文字中的 PII 偵測與遮蔽（email／電話／SSN／卡號／IP／IBAN） | | `utils/readability/` | 140 | 可讀性評分（Flesch、Flesch-Kincaid、Gunning Fog、SMOG、ARI） | | `utils/reading_flow/` | 165 | 以遞迴 XY-cut 推導欄位感知的閱讀順序 | @@ -494,12 +494,12 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.9 AI / Agent / LLM -> 13 個套件、約 23,246 行。 +> 13 個套件、約 23,540 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | | `utils/a2a/` | 92 | A2A（agent-to-agent）agent card 產生 | -| `utils/agent/` | 1,975 | 閉環 Computer-Use Agent 主迴圈 + Anthropic／OpenAI／Computer-Use 三後端 | +| `utils/agent/` | 2,142 | 閉環 Computer-Use Agent 主迴圈 + Anthropic／OpenAI／Computer-Use 三後端 | | `utils/agent_memory/` | 166 | agent 的持久化情節記憶（goal → trajectory → outcome） | | `utils/agent_replay/` | 67 | 可攜的 agent 軌跡追蹤（記錄 observation→action 並重播） | | `utils/agent_trace/` | 172 | agent 可觀測性：OpenTelemetry GenAI 慣例的 LLM span | @@ -507,32 +507,32 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/cua_action/` | 204 | 標準化 computer-use 動作結構（Anthropic／OpenAI → `AC_*`） | | `utils/llm/` | 365 | 自然語言 → action list 規劃器 + Anthropic／null 後端 | | `utils/mcp_registry/` | 97 | MCP registry `server.json` 資訊清單產生（可被發現） | -| `utils/mcp_server/` | 18,898 | **無頭 MCP 伺服器**（16K LOC，預設註冊 680 個工具＝661 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | +| `utils/mcp_server/` | 19,025 | **無頭 MCP 伺服器**（16K LOC，預設註冊 680 個工具＝661 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | | `utils/tool_use_schema/` | 195 | 把 `AC_*` 指令匯出成 Claude／OpenAI 的 tool-use schema | | `utils/trajectory_eval/` | 132 | agent 軌跡評估：依評分規準為一次執行打分 | | `utils/vision/` | 538 | VLM 元素定位器（依描述找元素）+ Anthropic／OpenAI／null 後端 | ### 5.4.10 遠端桌面與 USB -> 6 個套件、約 19,487 行。 +> 6 個套件、約 20,018 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | | `utils/admin/` | 418 | 多主機管理主控台：平行輪詢 N 個 AutoControl REST 端點 | | `utils/config_sync/` | 332 | 透過訊令伺服器做跨機器設定同步 | | `utils/device_matrix/` | 143 | 行動裝置矩陣：同一 action list 於多台裝置平行執行 | -| `utils/remote_desktop/` | 13,014 | **遠端桌面子系統**（56 檔／11.7K LOC）：TCP／WebSocket／WebRTC 三條傳輸路徑、主機與檢視端、訊令伺服器、TURN／中繼、多檢視者、錄影、信任清單、TOTP、稽核鏈 | -| `utils/usb/` | 4,572 | 跨平台 USB 列舉／熱插拔／裝置直通（WinUSB、IOKit、libusb 後端 + ACL + WebRTC DataChannel 通道） | +| `utils/remote_desktop/` | 13,301 | **遠端桌面子系統**（56 檔／11.7K LOC）：TCP／WebSocket／WebRTC 三條傳輸路徑、主機與檢視端、訊令伺服器、TURN／中繼、多檢視者、錄影、信任清單、TOTP、稽核鏈 | +| `utils/usb/` | 4,816 | 跨平台 USB 列舉／熱插拔／裝置直通（WinUSB、IOKit、libusb 後端 + ACL + WebRTC DataChannel 通道） | | `utils/usbip/` | 1,008 | USB/IP 線路協定主機端（協定封包、TCP 伺服器、libusb URB 後端） | ### 5.4.11 伺服器、網路協定與外部整合 -> 24 個套件、約 7,035 行。 +> 24 個套件、約 6,947 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | | `utils/acme_v2/` | 617 | 完整 ACME v2 用戶端（RFC 8555），不依賴 certbot | -| `utils/chatops/` | 667 | Chat-ops bot：接收 Slack／Discord／webhook 的 slash 指令並路由到動作 | +| `utils/chatops/` | 669 | Chat-ops bot：接收 Slack／Discord／webhook 的 slash 指令並路由到動作 | | `utils/cookie_jar/` | 191 | RFC 6265 cookie jar | | `utils/email_send/` | 118 | SMTP 寄信（email 觸發器的發送端搭檔） | | `utils/events/` | 106 | 對外 CloudEvents 發送（執行生命週期事件） | @@ -548,9 +548,9 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/notify_channels/` | 105 | 對外聊天／webhook 通知（Slack／Discord／Teams／raw） | | `utils/otp/` | 37 | TOTP 一次性密碼產生（自動化 2FA 登入） | | `utils/outbox/` | 107 | 交易式 outbox，保證至少一次的事件投遞 | -| `utils/pytest_plugin/` | 380 | pytest 外掛 + BDD step library（`pytest11` entry point） | -| `utils/rest_api/` | 1,881 | 純標準庫 REST 前端：路由、Bearer 驗證、限流、Prometheus 指標、OpenAPI 3.1 產生 | -| `utils/socket_server/` | 160 | 執行 action JSON 的執行緒式 TCP 指令伺服器（預設綁 127.0.0.1） | +| `utils/pytest_plugin/` | 283 | pytest 外掛 + BDD step library（`pytest11` entry point） | +| `utils/rest_api/` | 1,885 | 純標準庫 REST 前端：路由、Bearer 驗證、限流、Prometheus 指標、OpenAPI 3.1 產生 | +| `utils/socket_server/` | 163 | 執行 action JSON 的執行緒式 TCP 指令伺服器（預設綁 127.0.0.1） | | `utils/sse_client/` | 142 | Server-Sent Events 用戶端解析 | | `utils/tls_acme/` | 473 | TLS 自動化：HTTP-01 挑戰伺服器、金鑰／CSR、自動續期 | | `utils/url_canon/` | 194 | RFC 3986 URL 正規化與查詢字串工具 | @@ -558,7 +558,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.12 報表、可觀測性與測試治理 -> 34 個套件、約 7,716 行。 +> 34 個套件、約 7,721 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -583,7 +583,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/profiler/` | 451 | 逐動作效能剖析器 + 資源剖析器 | | `utils/quarantine/` | 204 | 易碎測試隔離區，讓套件執行器跳過已知不穩定案例 | | `utils/run_diff/` | 143 | 兩次執行軌跡的差異（LCS 對齊：新增／移除／狀態翻轉／退化） | -| `utils/run_history/` | 449 | 執行歷史儲存與產出物管理 | +| `utils/run_history/` | 454 | 執行歷史儲存與產出物管理 | | `utils/sarif/` | 187 | 以 SARIF 2.1.0 匯出發現項，供 GitHub／Azure code scanning | | `utils/slo/` | 115 | SLO 評估：SLI、錯誤預算與多視窗燃燒率告警 | | `utils/smoothing/` | 67 | 數列移動平均平滑 | @@ -630,7 +630,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.14 安全、機密與合規 -> 13 個套件、約 2,964 行。 +> 13 個套件、約 3,035 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -642,8 +642,8 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/rbac/` | 302 | 角色型存取控制：使用者、角色與權杖驗證（尚未接到 REST／MCP） | | `utils/redaction/` | 508 | 截圖遮蔽層：規則偵測 + 政策 + 協調器（上傳 VLM 前先遮） | | `utils/sbom/` | 170 | SBOM（CycloneDX）產生 | -| `utils/secret_ref/` | 169 | URI scheme 形式的值參照解析 | -| `utils/secrets/` | 360 | 加密機密儲存庫，供 `${secrets.NAME}` 解析 | +| `utils/secret_ref/` | 238 | URI scheme 形式的值參照解析 | +| `utils/secrets/` | 362 | 加密機密儲存庫，供 `${secrets.NAME}` 解析 | | `utils/secrets_scan/` | 138 | 掃描 action JSON／資料中應入庫卻硬編碼的機密 | | `utils/vex/` | 178 | OpenVEX 陳述撰寫與漏洞分類處置 | | `utils/vuln_scan/` | 276 | 以 OSV 比對 SBOM 元件的漏洞（純標準庫） | @@ -671,13 +671,13 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.16 系統、視窗與剪貼簿 -> 16 個套件、約 2,616 行。 +> 16 個套件、約 2,697 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | | `utils/clipboard/` | 465 | 跨平台無頭剪貼簿存取（文字 + 影像）＋ `win32_clipboard_api.py`：**所有剪貼簿格式共用的 Win32 原型與 open/alloc/lock 流程**（`open_clipboard()` 會等過短暫被別的行程佔住的剪貼簿——Win32 一次只允許一個行程開啟，別人正在複製就必然失敗）（`argtypes` 只宣告一半曾讓四支 writer 在 64 位元上必然丟 `OverflowError`，見 CHANGELOG）。`set_clipboard_image` 同時接受 PNG 位元組與檔案路徑——先前這個名字在本子套件裡有**兩份不同簽章的實作**（`clipboard.py` 吃 bytes、`clipboard_image.py` 吃路徑），匯錯來源只會在執行期才炸，已合併成一支 | | `utils/clipboard_files/` | 118 | 剪貼簿檔案清單（CF_HDROP）：純 DROPFILES 封裝 + Win32 存取 | -| `utils/clipboard_formats/` | 151 | 檢視與分類剪貼簿可用格式（純分類／差異 + Win32 列舉） | +| `utils/clipboard_formats/` | 156 | 檢視與分類剪貼簿可用格式（純分類／差異 + Win32 列舉） | | `utils/clipboard_history/` | 114 | 剪貼簿歷史：環形緩衝 + 背景輪詢器 | | `utils/clipboard_rich_formats/` | 328 | 豐富剪貼簿格式 — RTF 與 CSV/TSV 編解碼 + Windows 存取 | | `utils/file_assoc/` | 98 | 解析哪個應用程式被註冊來開啟某副檔名 | @@ -687,32 +687,32 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/shell_open/` | 99 | 以預設應用開啟檔案，或以預設瀏覽器開啟 URL | | `utils/system_volume/` | 212 | 讀取與控制系統主音量與靜音狀態 | | `utils/trash/` | 93 | 把檔案移到系統資源回收筒（可復原刪除） | -| `utils/window_capture/` | 304 | 逐視窗截圖、視窗版面儲存／還原、貼齊與排列 | +| `utils/window_capture/` | 379 | 逐視窗截圖、視窗版面儲存／還原、貼齊與排列 | | `utils/window_geometry/` | 87 | 視窗客戶區幾何（外框內縮、client→screen 對映） | | `utils/window_layout/` | 137 | 視窗拼貼／版面規劃器（左右半、四象限、網格、層疊） | -| `utils/window_zorder/` | 76 | 視窗 z 序控制（最上層／移到最前／送到最後） | +| `utils/window_zorder/` | 77 | 視窗 z 序控制（最上層／移到最前／送到最後） | ### 5.4.17 大型子系統的檔案級剖析 上表以子套件為單位；以下把行數最大的幾個子系統展開到檔案層。 -#### `utils/executor/`（9,606 行）— 執行核心 +#### `utils/executor/`（9,659 行）— 執行核心 | 檔案 | 行數 | 職責 | | --- | ---: | --- | -| `action_executor.py` | 8,387 | `Executor` 類別與 `event_dict` 分派表（778 個指令），另含數百個把 utils 能力接成指令的 adapter 函式；全域單例 `executor` 與 `add_command_to_executor()` 擴充點。 | -| `flow_control.py` | 644 | 真正的流程控制：`AC_loop`／`AC_for_each`／`AC_while_*`／`AC_if_*`／`AC_try`／`AC_retry`／`AC_parallel`／`AC_define_macro`／`AC_call_macro`／變數指令（`AC_set_var`／`AC_get_var`／`AC_inc_var`）。`LoopBreak`／`LoopContinue` 以例外實作。34 個區塊指令的分派表 `BLOCK_COMMANDS` 也在這裡，含下一列匯入的資料來源指令。 | +| `action_executor.py` | 8,433 | `Executor` 類別與 `event_dict` 分派表（778 個指令），另含數百個把 utils 能力接成指令的 adapter 函式；全域單例 `executor` 與 `add_command_to_executor()` 擴充點。 | +| `flow_control.py` | 651 | 真正的流程控制：`AC_loop`／`AC_for_each`／`AC_while_*`／`AC_if_*`／`AC_try`／`AC_retry`／`AC_parallel`／`AC_define_macro`／`AC_call_macro`／變數指令（`AC_set_var`／`AC_get_var`／`AC_inc_var`）。`LoopBreak`／`LoopContinue` 以例外實作。34 個區塊指令的分派表 `BLOCK_COMMANDS` 也在這裡，含下一列匯入的資料來源指令。 | | `flow_data_commands.py` | 272 | `AC_*_to_var` 資料來源與轉換指令：shell、時鐘、亂數、PDF、TOTP、SQL、檔案、HTTP、OCR，加上 `AC_assert_var`／`AC_assert_db`／`AC_assert_duration`／`AC_transform_var`。都不執行巢狀 action list，所以沒有迴圈／分支語意。 | | `action_schema.py` | 159 | action list 的結構驗證：形狀、參數型別、未知指令拒絕。單一走訪同時支援兩種消費方式：`validate_actions()` 遇到第一個問題就拋、`unknown_command_names()` 收齊全部不認得的名字（REST `/execute` 用它回 400）。 | | `action_redaction.py` | 86 | 記錄與紀錄鍵用的遮蔽：`AC_secret_*` 的參數（金庫通行碼、機密值）在寫進 log、當成結果紀錄的鍵之前換成 `***`，巢狀在區塊指令裡的也一樣。 | | `mouse_aliases.py` | 39 | 單鍵點擊別名（`AC_click_left` 等），executor 與 callback executor 共用。 | | `flags.py` | 19 | 旗標讀取 `as_bool`：依拼法（`"true"`／`"yes"`／`"on"`／`"1"`）而非 Python 真值判斷，所以 JSON、CLI、MCP 傳來的 `"false"` 是關；所有 adapter 與流程指令共用。 | -#### `utils/mcp_server/`（18,898 行，680 個工具）— 最大子系統 +#### `utils/mcp_server/`（19,025 行，680 個工具）— 最大子系統 | 檔案 | 行數 | 職責 | | --- | ---: | --- | -| `tools/_factories.py` | 9,060 | 工具工廠：每個函式回傳一個領域的 `MCPTool` 清單（把 `AC_*` 能力包成 MCP 工具）。 | +| `tools/_factories.py` | 9,063 | 工具工廠：每個函式回傳一個領域的 `MCPTool` 清單（把 `AC_*` 能力包成 MCP 工具）。 | | `tools/_handlers.py` | 547 | 把 MCP 工具呼叫橋接到 AutoControl 無頭 API 的 adapter；主題模組拆完之後這裡留的是資料／文字／HTTP 那一類與 WebRunner 橋接。 | | `tools/_handlers_qa.py` | 419 | 同一種 adapter，QA 主題：斷言 DSL、資料驅動、SQL／PDF／郵件／HTTP 步驟、codegen、視覺回歸、狀態機、flaky 偵測與隔離、suite runner、無障礙稽核、裝置矩陣、媒體斷言。從 `_handlers.py` 依主題拆出的第一塊（750 行上限）；兩者互不引用。 | | `tools/_handlers_input.py` | 221 | 同一種 adapter，輸入主題：滑鼠、鍵盤、虛擬手把（ViGEm）。 | @@ -724,11 +724,11 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `tools/_handlers_executor_bridge.py` | 1,429 | 252 個純委派（中位數 3 行，最長的 16 行全是參數簽章）：每個都是 `from action_executor import _x` 再 `return _x(...)`，沒有分支邏輯。超過 750 行,理由記在 `Progress.md` 的豁免表（再切只能照 MCP 工廠領域分,會把同一種委派散進十幾個沒有語意邊界的檔）。 | | `tools/_handlers_locators.py` | 436 | 同一種 adapter，定位主題：無障礙樹、智慧等待、自我修復、螢幕觀察、座標空間、視覺與 OCR、影像去重、元件倉庫、A/B 定位。 | | `tools/_handlers_operations.py` | 629 | 同一種 adapter，營運主題：agent 與其記憶／追蹤、治理與合規、成本與遙測、失敗掛鉤、看門狗、速率限制、檢查點、核可、產物與資產、測試選擇與分片、佇列與 saga。 | -| `server.py` | 721 | JSON-RPC 2.0 over stdio 的最小 MCP 伺服器：連線範圍狀態、行內／併發分派、工具與 resource／prompt 處理器；握手時代的方法表（`_run_method`），兩個協定時代的逐請求分派在 `_stateless.py`。 | +| `server.py` | 732 | JSON-RPC 2.0 over stdio 的最小 MCP 伺服器：連線範圍狀態、行內／併發分派、工具與 resource／prompt 處理器；握手時代的方法表（`_run_method`），兩個協定時代的逐請求分派在 `_stateless.py`。 | | `http_transport.py` | 719 | MCP 的 HTTP 傳輸；宣告 2026-07-28 的請求走 `_http_stateless.py` 的標頭規則，不發 session。 | | `_http_stateless.py` | 185 | MCP 2026-07-28 在 Streamable HTTP 上的規則：`MCP-Protocol-Version`／`Mcp-Method`／`Mcp-Name` 必須與 body 相符（`=?base64?…?=` 先解碼），不符是 400＋`HeaderMismatch`；版本與中繼資料錯誤 400、未知方法 404。純函式，由 `http_transport.py` 回覆。 | | `http_sessions.py` | 247 | MCP 的 HTTP 傳輸用的 session 身分:`Mcp-Session-Id` 註冊表,以及每個 session 那條常駐的 server→client SSE 串流。 | -| `_client_requests.py` | 254 | 伺服器主動送出的請求：`roots/list`／`elicitation/create`／`sampling/createMessage`,對應表與回應路由,以及破壞性工具的確認交握。只屬於握手時代：無狀態請求裡送出會丟例外。 | +| `_client_requests.py` | 274 | 伺服器主動送出的請求：`roots/list`／`elicitation/create`／`sampling/createMessage`,對應表與回應路由,以及破壞性工具的確認交握。只屬於握手時代：無狀態請求裡送出會丟例外。 | | `_stateless.py` | 267 | MCP 2026-07-28 無狀態版本，與以 `initialize` 握手的版本並存：逐請求的 `_meta`（版本、client 能力、`logLevel`）、`server/discover`、結果的 `resultType`／`serverInfo`／快取提示、`-32020`～`-32022` 錯誤碼，以及兩個時代逐請求分派的 mixin。 | | `_input_required.py` | 160 | 多輪往返請求（MRTR）：`input_required` 結果，與 HMAC 簽章、會過期、只兌換一次的 `requestState`；破壞性工具確認在無狀態請求裡的形式。 | | `_subscriptions.py` | 256 | 變更通知：握手時代的 `resources/subscribe`／`unsubscribe` 與未經訂閱的 `resources/updated`、`tools/list_changed`（不送給無狀態的對端）；2026-07-28 的 `subscriptions/listen`：確認、以訂閱 id 標記的通知、取消與伺服器結束時的完成回覆。 | @@ -737,7 +737,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `prompts.py` | 220 | MCP prompt 目錄。 | | `fake_backend.py` | 187 | CI／無頭測試用的記憶體內假後端。 | | `plugin_watcher.py` | 168 | 檔案變更時熱重載外掛工具的背景 watcher。 | -| `tools/_base.py` | 146 | 工具註冊表的共用型別與輔助。 | +| `tools/_base.py` | 153 | 工具註冊表的共用型別與輔助。 | | `tools/_validation.py` | 122 | MCP 工具用到的 JSON Schema 子集驗證器。 | | `tools/plugin_tools.py` | 89 | 把外掛載入的 `AC_*` callable 包成 `MCPTool`。 | | `log_bridge.py` | 118 | 把 Python logging 記錄橋接成 MCP `notifications/message`；2026-07-28 的請求只收到自己設了 `logLevel` 時產生的記錄。 | @@ -746,7 +746,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `rate_limit.py` | 48 | 工具呼叫的 token bucket 限流。 | | `__main__.py` | 92 | `je_auto_control_mcp` console script 進入點。 | -#### `utils/remote_desktop/`（13,014 行／56 檔） +#### `utils/remote_desktop/`（13,301 行／56 檔） 三條傳輸路徑並存：**TCP**（JPEG 影格）、**WebSocket**（同協定換傳輸）、**WebRTC**（aiortc 視訊 + DataChannel）。 @@ -754,18 +754,18 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | --- | ---: | --- | | `webrtc_host.py` | 722 | WebRTC 主機：串流螢幕視訊並接受檢視端輸入;session 生命週期、DataChannel 接線、檔案收發。 | | `webrtc_viewer.py` | 677 | WebRTC 檢視端：接收視訊並送出輸入。 | -| `host.py` | 673 | TCP 主機：接受迴圈、TLS 包裝、連線／認證握手、音訊與剪貼簿廣播、檔案推送、單次 token。 | -| `viewer.py` | 634 | TCP 檢視端。 | +| `host.py` | 676 | TCP 主機：接受迴圈、TLS 包裝、連線／認證握手、音訊與剪貼簿廣播、檔案推送、單次 token。 | +| `viewer.py` | 641 | TCP 檢視端。 | | `host_service.py` | 562 | 無頭 WebRTC 主機執行器 + 多平台服務安裝器。 | | `host_client.py` | 453 | TCP 主機的每連線處理器：一個檢視端一個實例,擁有它的認證交換、sender／audio／receiver 三條執行緒,以及入站訊息的路由表。 | -| `registry.py` | 370 | `AC_remote_*` 指令使用的行程級單例。 | +| `registry.py` | 564 | `AC_remote_*` 指令使用的行程級單例。 | | `webrtc_transport.py` | 421 | 共用 WebRTC 管線：asyncio 橋接執行緒、螢幕視訊軌、設定。 | | `multi_viewer.py` | 357 | 每個連入檢視端各跑一個 `WebRTCDesktopHost` 的協調器。 | | `signaling_server.py` | 427 | 獨立的 WebRTC SDP 交換 rendezvous 服務。 | | `audit_log.py` | 355 | SQLite 雜湊鏈稽核記錄。 | | `host_capture.py` | 323 | TCP 主機的影格與游標產生：螢幕列舉、監視器索引轉擷取區域、預設 JPEG／游標 provider,以及 `FrameProductionMixin`（游標輪詢、擷取迴圈、上線編碼）。 | | `ws_protocol.py` | 318 | 最小 RFC 6455 WebSocket 框架與握手。 | -| `file_transfer.py` | 371 | 分塊檔案傳輸。 | +| `file_transfer.py` | 452 | 分塊檔案傳輸。 | | `relay.py` | 315 | NAT 穿透失敗時的 TCP 中繼。 | | `fingerprint.py` | 246 | TOFU 主機指紋驗證。 | | `turn_config.py` | 249 | coturn 設定產生器。 | @@ -797,19 +797,19 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `permissions.py` / `clipboard_sync.py` / `wake_on_lan.py` / `session_actions.py` / `auth.py` | 64 / 74 / 56 / 40 / 28 | 逐 session 權限、剪貼簿同步、WOL、SAS 注入與螢幕遮蔽、HMAC 挑戰回應。 | | `ws_host.py` / `ws_viewer.py` / `jpeg_recorder.py` | 40 / 29 / 146 | WebSocket 傳輸變體與 TCP 路徑錄影。 | -#### `utils/usb/`（4,572 行）與 `utils/usbip/`（1,008 行） +#### `utils/usb/`（4,816 行）與 `utils/usbip/`（1,008 行） | 檔案 | 行數 | 職責 | | --- | ---: | --- | -| `usb/passthrough/session.py` | 642 | 逐 peer 的 USB 直通 session。 | -| `usb/passthrough/viewer_client.py` | 600 | 檢視端的直通協定用戶端。 | +| `usb/passthrough/session.py` | 690 | 逐 peer 的 USB 直通 session。 | +| `usb/passthrough/viewer_client.py` | 748 | 檢視端的直通協定用戶端。 | | `usb/passthrough/backend.py` | 488 | 後端 ABC + libusb 實作。 | | `usb/passthrough/winusb_backend.py` | 488 | Windows WinUSB 後端（ctypes）。 | | `usb/passthrough/acl.py` | 495 | 逐裝置 ACL。 | | `usb/passthrough/iokit_backend.py` | 221 | macOS IOKit 後端。 | | `usb/passthrough/webrtc_channel.py` | 180 | 把直通協定橋到 WebRTC `usb` DataChannel。 | | `usb/passthrough/loopback.py` | 159 | 行程內 loopback 傳輸（測試用）。 | -| `usb/passthrough/protocol.py` | 133 | 線路框格式。 | +| `usb/passthrough/protocol.py` | 177 | 線路框格式。 | | `usb/passthrough/descriptor.py` | 134 | USB 標準裝置描述元解析。 | | `usb/passthrough/key_provider.py` | 125 | ACL 的可插拔 HMAC 金鑰來源。 | | `usb/passthrough/commands.py` | 150 | 無頭直通指令（單一真實來源）。 | @@ -820,12 +820,12 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `usbip/libusb_backend.py` | 224 | 以 PyUSB／libusb 執行 URB 的正式後端。 | | `usbip/backend.py` | 87 | 可插拔 URB 執行後端。 | -#### `utils/rest_api/`（1,881 行） +#### `utils/rest_api/`（1,885 行） | 檔案 | 行數 | 職責 | | --- | ---: | --- | | `rest_server.py` | 549 | HTTP 前端主體。 | -| `rest_handlers.py` | 524 | 端點實作。 | +| `rest_handlers.py` | 528 | 端點實作。 | | `rest_openapi.py` | 431 | 走訪路由表產生 OpenAPI 3.1 規格。 | | `rest_auth.py` | 157 | Bearer token 驗證 + 逐 client 限流閘門。 | | `rest_metrics.py` | 75 | Prometheus 曝露端點。 | @@ -894,10 +894,10 @@ GUI 是**選用 extra**（`pip install je_auto_control[gui]`，PySide6 + qt-mate | `_report_tab.py` | 88 | 報表分頁 mixin。 | | `_i18n_helpers.py` | 66 | 需要即時語言切換的分頁共用的翻譯註冊 mixin。 | | `_validators.py` | 29 | `int_validator()`／`double_validator()`：以 C locale 驗證的數字輸入框 validator，接受的正是 `int()`／`float()` 讀得懂的寫法（預設 locale 在法文、德文下只收小數逗號）。所有數字 `QLineEdit` 都用它。 | -| `_screen_geometry.py` | 52 | Qt 邏輯座標與截圖用的原生像素互轉：`native_region()`、`screen_at_native()`、`logical_point()`（每個螢幕的左上角在兩者相同，螢幕內依 device pixel ratio 縮放）。區域選取與主機端標註覆蓋層都用它。 | +| `_screen_geometry.py` | 55 | Qt 邏輯座標與截圖用的原生像素互轉：`native_region()`、`screen_at_native()`、`logical_point()`（每個螢幕的左上角在兩者相同，螢幕內依 device pixel ratio 縮放）。區域選取與主機端標註覆蓋層都用它。 | | `_daemon_thread.py` | 79 | `DaemonThread`：`QThread` 的替代品，保留遠端桌面 worker 用到的介面（`start`／`run`／`isRunning`／`wait`／`requestInterruption`／`started`／`finished`），但 `run()` 跑在 daemon `threading.Thread` 上，刪除物件或程式結束都不會銷毀執行中的執行緒。 | | `_worker_thread.py` | 216 | `start_worker()`：在 daemon `threading.Thread` 上執行 `QObject` worker 的 `run()`（沒有 `QThread` 可被銷毀），並經由分頁擁有的中繼物件回報結果（回呼一律在 GUI 執行緒；worker 沒處理的例外也送到 `on_fail`）；worker 留在模組登錄表直到 GUI 執行緒看到它結束，回傳 `WorkerHandle`（`isRunning()`）；程式結束時先呼叫 worker 的 `request_stop()`，最多等 10 秒，仍在跑的隨行程結束。 | -| `language_wrapper/` | 5,063 | 四語系字典（英／日／簡中／繁中）+ `multi_language_wrapper` 執行期切換器與監聽註冊表。 | +| `language_wrapper/` | 5,067 | 四語系字典（英／日／簡中／繁中）+ `multi_language_wrapper` 執行期切換器與監聽註冊表。 | | `selector/` | 216 | 拖曳選取螢幕區域的半透明全螢幕覆蓋層與樣板裁切工具（互動式，但都有對應的程式化 API）。 | > **分頁指令一律走 Actions 選單**：分頁本身只放輸入、表格與結果檢視，指令由視窗層選單暴露。 @@ -957,7 +957,7 @@ GUI 是**選用 extra**（`pip install je_auto_control[gui]`，PySide6 + qt-mate | diagnostics | `diagnostics_tab.py` | 91 | 執行子系統檢查並顯示結果。 | | report | `_report_tab.py` | 81 | 產生 HTML／JSON／XML 報表。 | -#### 遠端桌面 GUI（`gui/remote_desktop/`，19 檔／6,646 行） +#### 遠端桌面 GUI（`gui/remote_desktop/`，19 檔／6,735 行） | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -965,14 +965,14 @@ GUI 是**選用 extra**（`pip install je_auto_control[gui]`，PySide6 + qt-mate | `webrtc_dialogs.py` | 519 | WebRTC GUI 用的自訂對話框與清單元件（待審檢視者、信任清單、通訊錄、遠端檔案表、稽核記錄、LAN 瀏覽）。 | | `advanced_group.py` | 92 | 兩個 WebRTC 面板共用的 Advanced STUN／TURN（含選用硬體編碼器）群組，含它寫回面板的 Protocol。 | | `trusted_group.py` | 70 | WebRTC host 面板的信任 viewer 清單群組（移除／清空／匯入／匯出），含它寫回面板的 Protocol。 | -| `connection_screen.py` | 704 | Quick Connect —— AnyDesk 風格單畫面入口。 | -| `viewer_panel.py` | 521 | 「控制另一台機器」子分頁。 | +| `connection_screen.py` | 729 | Quick Connect —— AnyDesk 風格單畫面入口。 | +| `viewer_panel.py` | 543 | 「控制另一台機器」子分頁。 | | `webrtc_known_hosts.py` | 346 | TOFU 釘選庫瀏覽器：`KnownHostsDialog` 與帶外釘選用的小表單。由 `webrtc_dialogs` 再匯出。 | -| `host_panel.py` | 353 | 「分享這台機器」子分頁。 | +| `host_panel.py` | 371 | 「分享這台機器」子分頁。 | | `frame_display.py` | 228 | 繪製 JPEG 影格並發出遠端輸入事件的元件。 | | `webrtc_workers.py` | 237 | 訊令流程的背景 worker（`DaemonThread`，長輪詢比面板或程式活得久也不會中止行程）。 | | `tab.py` | 165 | 外層容器分頁。 | -| `_helpers.py` | 249 | 面板共用輔助：翻譯、Qt→AC 鍵滑鼠對應、TLS context、狀態徽章、指紋與時間格式化。 | +| `_helpers.py` | 273 | 面板共用輔助：翻譯、Qt→AC 鍵滑鼠對應、TLS context、狀態徽章、指紋與時間格式化。 | | `remote_screen_window.py` | 140 | 檢視端的彈出視窗。 | | `tray_icon.py` | 106 | WebRTC 主機的系統匣圖示。 | | `annotation_overlay.py` | 174 | 主機端標註的透明最上層覆蓋。 | @@ -1079,26 +1079,26 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | 層／子系統 | 檔案數 | 行數 | | --- | ---: | ---: | -| `gui/` | 98 | 28,399 | -| `utils/mcp_server/` | 35 | 18,898 | -| `utils/remote_desktop/` | 56 | 13,014 | -| `utils/executor/` | 8 | 9,606 | -| `utils/usb/` | 17 | 4,572 | -| `je_auto_control/`（頂層 3 檔） | 3 | 2,429 | +| `gui/` | 98 | 28,507 | +| `utils/mcp_server/` | 36 | 19,025 | +| `utils/remote_desktop/` | 56 | 13,301 | +| `utils/executor/` | 8 | 9,659 | +| `utils/usb/` | 17 | 4,816 | +| `je_auto_control/`（頂層 3 檔） | 3 | 2,456 | | `utils/accessibility/` | 14 | 3,143 | -| `wrapper/` | 19 | 3,805 | -| `windows/` | 23 | 1,976 | -| `utils/rest_api/` | 8 | 1,881 | -| `utils/agent/` | 9 | 1,975 | +| `wrapper/` | 19 | 4,083 | +| `windows/` | 23 | 2,154 | +| `utils/rest_api/` | 8 | 1,885 | +| `utils/agent/` | 9 | 2,142 | | `linux_with_x11/` | 19 | 1,281 | | `linux_wayland/` | 17 | 2,921 | | `utils/triggers/` | 4 | 1,383 | -| `utils/ocr/` | 10 | 1,367 | +| `utils/ocr/` | 10 | 1,381 | | `utils/usbip/` | 5 | 1,008 | | `utils/assertion/` | 3 | 887 | -| `osx/` | 17 | 925 | +| `osx/` | 17 | 981 | | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 852 | -| 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 682 | 57,156 | -| **總計** | **1,062** | **158,222** | +| 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 687 | 58,187 | +| **總計** | **1,068** | **160,796** | From 9414cb8c72442353cd43cfa387e35fbe9a8e944f Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:23:03 +0800 Subject: [PATCH 23/26] Record the Progress sweep and confirm the foreground only where the backend can focus_window's new check compared the platform's foreground id with the id it had focused; on macOS and X11 a mismatch does not mean the request was refused, so the check is limited to the Windows backend. Progress.md keeps only what was not verified or not done; the command stub and the documented counts are regenerated. --- CHANGELOG.md | 114 +++++- Progress.md | 382 ++++-------------- README.md | 10 +- README/README_zh-CN.md | 10 +- README/README_zh-TW.md | 10 +- architecture_explore.md | 70 ++-- docs/updates/2026-10.md | 84 ++++ docs/updates/README.md | 16 +- je_auto_control/actions.pyi | 15 +- .../wrapper/auto_control_window.py | 12 +- .../wrapper/window_backends/base.py | 6 + .../window_backends/windows_backend.py | 3 + .../test_window_lifecycle_contract.py | 11 + 13 files changed, 385 insertions(+), 358 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f78dd9825..d32a82885 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,38 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's ### Added +- Action files can be signed with an Ed25519 key pair, so an endpoint that + verifies does not hold what signs: `create_signing_keypair(private_path, + public_path)` / `AC_create_signing_keypair`, `private_key_path=` on + `sign_action_file`, `public_key_path=` on `verify_action_file`, and the + environment variables `JE_AUTOCONTROL_ACTION_SIGNING_PRIVATE_KEY`, + `JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY` and + `JE_AUTOCONTROL_ACCEPT_LEGACY_ACTION_SIGNATURES`. Nothing changes while none + of them is set. `CryptographyUnavailableError` (an `AutoControlException` and + a `RuntimeError`) replaces the bare `RuntimeError` raised without the + `cryptography` package. +- Opt-in roles for the REST API and the MCP HTTP transport: + `JE_AUTOCONTROL_RBAC_USERS=` (or `user_store=`, or `--users` + on the REST entry points). A bearer token then names a user, each route, + tool and privileged `AC_*` command needs a capability of that user's role, + and audit entries carry the user id. `UserStore`, `UserRecord`, + `AuthorizationContext`, `AuthorizationError`, `UserAuthError` and + `authorization_scope` are exported; `Capability.SIGN_ACTIONS` is new. +- Opt-in confinement of MCP file arguments: `JE_AUTOCONTROL_MCP_PATH_ROOTS`, + `JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT` and, for `env://` references, + `JE_AUTOCONTROL_MCP_ENV_REF_ALLOW`. Tool schemas mark file-path arguments + with `"format": "path"`. `PathPolicy` and `PathNotAllowedError` are exported. +- `execution_scope(variables=None)`: a fresh variable scope for a block of + runs. +- `JE_AUTOCONTROL_ALLOWED_PACKAGES` and `je_auto_control run --allow-package + NAME` allow packages at the package gate without Python glue. +- USB passthrough requests carry an optional `request_id` that a current host + echoes; `UsbClientDesynchronized`, `ClientHandle.reusable`, + `UsbPassthroughClient.reusable`. +- `remote_desktop_registry` records who opened each host and viewer + (`adopt`, `evict`, `release`, `owner_of`, `owned`; `owner=` on the stop and + disconnect methods), and every `*_status` result gains an `"owner"` key. +- `mark_screen` returns `image_origin`; `render_marks(..., origin=)`. - GUI: a navigation panel on the left of the main window lists every tab by category, with a search box (`Ctrl+K`; `Ctrl+B` hides the panel). `AutoControlGUIWidget.activate_tab(key)` opens a tab or brings it to the @@ -93,8 +125,8 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's in one call: the clicks land on one point, `interval` seconds apart. Defaults keep the single click, and a single click records the same action as before. `AC_click_mouse`, the `ac_click_mouse` MCP tool and the - Script Builder take both parameters. On macOS the clicks still arrive as - separate single clicks (`Progress.md`). + Script Builder take both parameters. On macOS the n-th click sets the + click-state field (unverified on hardware, `Progress.md`). - **Drag pacing: `tween_drag` / `drag_path` take `step_delay_s` and `settle_s`** (seconds, default 0). `step_delay_s` rests after each move; `settle_s` rests on the start before the press, after the press and before @@ -161,6 +193,84 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's ### Changed +- **Breaking: the package gate refuses by default.** + `AC_add_package_to_executor` / `AC_add_package_to_callback_executor` (and + the `package_manager` methods) no longer import a package that has not been + allowed; the `DeprecationWarning` release is over. Allow with + `executor.allow_packages(...)`, `JE_AUTOCONTROL_ALLOWED_PACKAGES` or + `--allow-package`; `executor.set_allow_arbitrary_packages(True)` restores + the old behaviour. `allow_packages` now rejects a name that is not a dotted + module name. +- **Breaking: a file a remote-desktop host pushes lands below the viewer's + download directory.** `host.send_file_to_viewers(src, dest_path)`: + `dest_path` is now relative to `~/Downloads/AutoControl` + (`JE_AUTOCONTROL_REMOTE_DOWNLOAD_DIR`); an absolute path, a drive or UNC + path, `..` or a link leading outside fails the transfer. Viewer → host is + unchanged, and `FileReceiver()` without `base_dir` is unconfined as before. +- **Variables no longer survive a run.** `execute_action_with_vars`, REST + `/execute` and `/execute_file`, every MCP tool call, each socket command, + scheduler jobs, trigger / hotkey / webhook / e-mail firings, ChatOps `/run` + and voice commands each get a fresh variable scope; a later `${name}` fails + with `Unknown variable` instead of reading the previous caller's value. + Direct `executor.execute_action(...)` from Python keeps the process scope. + An `AC_parallel` branch starts with a copy of the caller's variables, and + `AC_run_dag` local nodes run on the calling list's executor and scope. +- **Windows is per-monitor DPI aware.** Importing the package asks for + per-monitor v2 (falling back to system awareness). On a monitor whose scale + differs from the primary's, coordinates and capture sizes are now physical + pixels: a point at an offset from that monitor's corner moves by + `its scale / primary scale`, and its screenshots are no longer downscaled. + Coordinates, regions, window layouts and templates recorded on such a + monitor have to be recorded again; the primary monitor is unaffected. +- **Keyboard input on Windows and X11.** `write("Hi")` types `Hi` (it typed + `hi`); `is_shift=True` is honoured by `type_keyboard`, `hotkey`, `write` and + `press_keyboard_key`; CR LF is one Enter; `write_secret` and the Unicode + typing helpers press Return / Tab / Backspace as keys instead of sending + them as code points. +- **`mouse_scroll` defaults to `scroll_direction="scroll_up"`**, so a positive + amount scrolls up on every platform; on X11 / Wayland it used to scroll + down. NaN, infinite and non-numeric coordinates raise before anything moves, + and fractional coordinates are rounded rather than truncated. +- **Window management on Windows.** `focus_window` raises + `AutoControlActionException` when the window did not become the foreground + one; `show_window_by_title` and the z-order calls return the real result; + `list_windows` / `find_window` skip DWM-cloaked and zero-area windows; + `post_key` sends one `WM_CHAR` for a printable character (pass the integer + virtual-key code to get key messages) and `post_key_to_window` accepts + `"enter"` / `"esc"`; `save_window_layout` stores the rectangle `MoveWindow` + positions (re-save older layout files); snap / grid / cascade use the work + area; `wait_for_window` never sleeps past its timeout. +- Screenshots of a region on Windows (`pil_screenshot`, `screenshot`, + `AC_screenshot`, `keyword_screenshot`, `capture_window`) capture any + monitor instead of returning black outside the primary one; a region + entirely off every monitor raises `AutoControlScreenException`. + `grab_logical` clips a region to the desktop and returns the clipped + origin, and set-of-marks renders the whole virtual desktop. +- macOS: `click_mouse(clicks=n)` sets the click-state field on the n-th + click's events, `list_windows` includes minimised windows and `restore` + finds them, and `grab_logical` works in points across every display. None + of this has been run on a Mac yet (`Progress.md`). +- With an Ed25519 key configured, a legacy HMAC signature is refused unless + `JE_AUTOCONTROL_ACCEPT_LEGACY_ACTION_SIGNATURES` is set, and an endpoint + holding only the public key refuses `AC_sign_action_file`. +- A remote-desktop panel reads, drives and disconnects only the session it + opened; a panel whose session is replaced closes its window. Starting a + host no longer disconnects a viewer. +- The Anthropic agent backends no longer rewrite turns they have sent: past + three screenshots the conversation restarts from a summary of the goal and + the actions so far plus the latest screenshot. +- The `pytest11` entry point is the top-level module `je_auto_control_pytest`, + which imports only pytest; `je_auto_control.utils.pytest_plugin` re-exports + it. Reinstall for the new entry point to take effect. +- `cryptography>=50.0.0` (was `>=48.0.1`). There is no `macosx_10_9_x86_64` + wheel at that floor, so an Intel Mac builds it from source. +- Image and OCR: templates load from non-ASCII paths and may be grayscale; an + unreadable or unsupported template raises `ImageNotFoundException`; + `find_spans` finds a phrase that starts inside a long box; the centre of a + box with negative coordinates is floored. +- `keyboard_layout` reports `None` for a Shift half that prints no single + character and labels ISO / UK / ABNT keys; a clipboard format named `None` + normalises to `""`. - GUI tabs are built the first time they are opened. `AutoControlGUIWidget` registers all 48 tabs from `gui/tab_registry.py` but constructs only the three it opens on and its own forms; `list_registered_tabs()` builds diff --git a/Progress.md b/Progress.md index 7741dff80..2dfbd4945 100644 --- a/Progress.md +++ b/Progress.md @@ -10,7 +10,9 @@ 實作計畫：[分階段交付計畫](docs/superpowers/plans/2026-10-02-modernization-index.md)，待審閱。 現有 `[Answer]` 決策沿用。 -`WIP` — 計畫 F（GUI）：F1 的延遲分頁註冊與 F2 的導覽／搜尋／主題已交付（U-20261008-02），其餘各子計畫尚未開始。F 還缺： +`WIP` — 計畫 A（既有決策與執行契約）已交付，A2、A7 在 2026-10-08、其餘在 2026-10-09（U-20261009-01…14）； +各項沒驗到或沒做完的部分在本檔下面三節（輸入與視窗、macOS、這一輪修正留下的後續）。計畫 F（GUI）：F1 的延遲分頁註冊與 F2 的導覽／搜尋／主題已交付（U-20261008-02）。 +計畫 B（動作日誌、自愈量測、codegen）、C（持久化同步與遠端 session；其中 C4 的擁有者模型已做）、D（Wayland）、E（Android／iOS）、G（MCP 逐步揭露）、H（型別深化與總驗收）尚未開始。F 還缺： - **窄視窗的內容是被擠壓而不是可捲動**：`gui/main_widget.py` 給 `QTabWidget` 明確的最小尺寸讓視窗能縮到 640×420， 但分頁內容沒有包進 `QScrollArea`；包進去會改變 `tabs.indexOf(entry.widget)` 這個 PyBreeze 與測試都在用的關係，要一起設計。 @@ -106,7 +108,7 @@ sys_platform != 'win32' or platform_machine != 'ARM64' | 功能 | 缺的是 | 錯誤形式 | | --- | --- | --- | | 影像比對、截圖轉 BGR、螢幕錄影 | `opencv-python`／`je_open_cv` | `utils/cv2_utils/optional.py` 的 `require_cv2()`／`require_je_open_cv()` 拋 `RuntimeError` | -| 動作檔加密（`action_signing`） | `cryptography` | `_fernet_types()` 拋 `RuntimeError`（簽章本身是 HMAC，不受影響） | +| 動作檔加密與 Ed25519 簽章（`action_signing`） | `cryptography` | 拋 `CryptographyUnavailableError`（也是 `RuntimeError`；HMAC 簽章不受影響） | | 秘密金庫（`${secrets.NAME}`） | `cryptography` | 同上 | | ACME／TLS 發證、加密錄影 | `cryptography` | 模組層 `ImportError` 轉述（照 `webrtc_transport` 慣例） | @@ -115,7 +117,7 @@ sys_platform != 'win32' or platform_machine != 'ARM64' | 依賴 | win_arm64 | 實測（2026-08-20） | | --- | --- | --- | | `opencv-python>=4.8,<6` | **沒有** | 任何版本都沒有，pip 回的是 `from versions: none`。`je_open_cv` 自己是純 Python，但相依 opencv-python，所以一起卡——標記也必須一起下。 | -| `cryptography>=48.0.1` | **沒有** | wheel 只出到 **46.0.3**，46.0.4 起上游就不再發 win_arm64。而 `>=48.0.1` 是 347ec1e 為了 GHSA-537c-gmf6-5ccf（high）訂的**安全下限**，不能為了 arm64 降回去。 | +| `cryptography>=50.0.0` | **沒有** | wheel 只出到 **46.0.3**，46.0.4 起上游就不再發 win_arm64。`>=50.0.0` 是**安全下限**（GHSA-537c-gmf6-5ccf 與 GHSA-g6cj-pr64-35w5），不能為了 arm64 降回去。 | | `pillow==12.3.0` | 有 | `pillow-12.3.0-cp3xx-win_arm64.whl` 一直都在。**曾經被寫成卡點，那是猜的，它從來不是。** | | `mss`／`defusedxml` | 有 | 純 Python。這三個加上 Pillow 就是 arm64 實際裝到的全部。 | | `PySide6==6.11.1`／`qt-material==2.17` | 有 | `[gui]` extra 在 arm64 上裝得起來。 | @@ -124,7 +126,7 @@ sys_platform != 'win32' or platform_machine != 'ARM64' 重驗指令（不需要 arm64 機器，也不需要 runner）： ```bash -pip install --dry-run --only-binary=:all: --platform win_arm64 --python-version 3.12 --target /tmp/probe 'opencv-python>=4.8,<6' 'cryptography>=48.0.1' +pip install --dry-run --only-binary=:all: --platform win_arm64 --python-version 3.12 --target /tmp/probe 'opencv-python>=4.8,<6' 'cryptography>=50.0.0' ``` 兩行 `ERROR: No matching distribution` 就是現況。**哪天其中一行不見了，就把 @@ -175,105 +177,87 @@ pip install --dry-run --only-binary=:all: --platform win_arm64 --python-version --- -## 鍵盤與滑鼠 wrapper 的輸入修正：等 Jeffrey_RPA 批次停下 +## 輸入、視窗與擷取修正：還沒在真的桌面與 Jeffrey_RPA 上驗過 -`BLOCKED` — Jeffrey_RPA 以 editable install 載入這個工作樹，正式批次（`webrunner_novelai.py`）與 Discord bot 正在跑，並且經 `_gui_control.py` 呼叫 `ac.write`、`ac.hotkey`、`ac.mouse_scroll`；下面每一項都會改變它打出來的字或滾動方向，依工作區規則在它執行期間不動 +`BLOCKED` — 程式已改完（U-20261009-03、-04、-05），但驗證全部是假後端；Jeffrey_RPA 的正式批次在跑，不能在它的環境裡換版本 -2026-09-24 稽核用假後端重現： +2026-10-08 實測：Jeffrey_RPA（`NovelAI_RPA`）的 venv 是從 PyPI 裝的 `je_auto_control`，不是這個工作樹的 editable install， +所以這裡的修改不會直接影響正在跑的批次；它升級套件時才會拿到。升級前要做的事： -- **大寫字母打成小寫**：`wrapper/auto_control_keyboard.py:234` `write()` 在 Windows 送的是與小寫相同的虛擬鍵（`_platform_windows.py` 的表裡 `"A"` 與 `"a"` 同一個碼），沒有按 Shift，`"Hi"` 打成 `hi`；X11 很可能一樣。做法：需要 Shift 的字元改走 `_write_char_via_unicode`，或包一層 Shift 按下／放開。 -- **`is_shift` 在 Windows 與 X11 無效**：`auto_control_keyboard.py:73`、`:104` 只在 macOS 把它傳下去，其他平台直接忽略，docstring 卻寫「是否同時按下 Shift」。做法：在這一層按住 `keyboard_keys_table["shift"]`，`finally` 放開。 -- **`"\r\n"` 按兩次 Enter**：`write()` 把 `\r` 與 `\n` 都對到 `return`，從檔案讀進來的 Windows 換行每行多一個空行。做法：迴圈前把 `\r\n` 換成 `\n`。 -- **X11 預設滾動方向與 Windows／macOS 相反**：`wrapper/auto_control_mouse.py` `mouse_scroll(..., scroll_direction="scroll_down")`，正值在 X11 往下、其他平台往上，與 docstring「一份寫法各平台通用」不符。做法：預設改 `scroll_up`，或改 docstring 講清楚（重播路徑已在 U-20260924-14 明確傳 `scroll_up`）。 -- **`mouse_scroll` 的 NaN 座標被悄悄夾到桌面邊緣**：`auto_control_mouse.py` 的夾限在 `_coordinate()` 驗證之前，`mouse_scroll(3, x=nan, y=100)` 移到 `(-1920, 100)` 才滾；`set_mouse_position(nan, …)` 則正確丟例外。做法：夾限前先過 `_coordinate()`。 -- **座標截斷而非四捨五入**：`set_mouse_position(-0.6, 10.9)` 得到 `(0, 10)`，註解寫的是「rounded point」。做法：`int(round(value))`。 -- **`post_key` 打出三次同一字元**：`windows/window/windows_window_manage.py:347` 自己送 `WM_CHAR`，而目標的 `TranslateMessage` 又從 `WM_KEYDOWN` 與（`lParam=0` 被當成按下的）`WM_KEYUP` 各產生一次，`post_key_to_window(title, "a")` 打出 `aaa`。做法：可列印字元只送 `WM_CHAR`，其他鍵送 `WM_KEYDOWN`（`lParam = 1 | scan<<16`）與 `WM_KEYUP`（`0xC0000001 | scan<<16`）。`post_key_to_window(title, "enter")`／`"esc"` 在 Windows 丟 `unknown key name`（`wrapper/auto_control_window.py:170`），一併改走 `resolve_key_name`。 -- **焦點、顯示、z-order 失敗仍回報成功**：`windows_window_manage.py:191-229` 丟掉 `SetForegroundWindow`／`ShowWindow` 的回傳值，`window_zorder.py:50` 永遠回 `True`；Windows 的前景鎖常拒絕背景程序。做法：回傳 BOOL，`focus_window` 以 `GetForegroundWindow() == hwnd` 確認，否則丟 `AutoControlActionException`（`WindowManageBackend.bring_to_front` 已經這樣做）。 -- **列出看不見的視窗**：`windows_window_manage.py:88` 只看 `IsWindowVisible`，被 DWM cloak 的視窗（`Windows 輸入體驗`、背景的「設定」）與零面積視窗都算，`find_window` 可能選到它們。做法：略過 `DWMWA_CLOAKED` 非零與空矩形的視窗。 -- **視窗版面每次還原都偏移**：`utils/window_capture/window_capture.py:115` 存 DWM 可見框、還原時交給 `MoveWindow`（它定位的是含隱形邊框的完整矩形），每輪右移 7 px、縮小 14×7 px；最大化視窗與不同 DPI 的第二螢幕偏得更多。做法：存 `GetWindowRect`，或改用 `GetWindowPlacement`／`SetWindowPlacement`。同一檔的 snap／grid／cascade 用整個螢幕而非工作區，最底下 48 px 落在工作列下，一併改用 `SPI_GETWORKAREA`。 -- **`wait_for_window` 睡過逾時**：`wrapper/auto_control_window.py:79` 以 `poll` 整段睡，`poll=30` 就睡 30 秒，`poll=inf` 丟 `OverflowError`。做法：`clamp_poll_interval`，並只睡到截止時間。 -- **Windows 鍵表沒有標點鍵**：`plus`、`minus`、`comma`、`period`、`slash` 等沒有對應的 `VK_OEM_*`，computer use 的 `ctrl+minus` 在 Windows 失敗。做法：在 Windows 鍵表補上 `VK_OEM_PLUS`／`VK_OEM_MINUS`／`VK_OEM_COMMA`／`VK_OEM_PERIOD`／`VK_OEM_2` 等。 +- **在 Jeffrey_RPA 跑 `test/test_je_facade.py`**（原本的解除條件），並看 `_gui_control.py` 對下列行為變更有沒有依賴： + `write()` 大寫與 `is_shift` 在 Windows／X11 生效、`"\r\n"` 只按一次 Enter、`mouse_scroll` 預設方向改為 `scroll_up` + （X11／Wayland 上 `mouse_scroll(3)` 由往下變往上）、小數座標四捨五入、`focus_window` 在 Windows 沒拿到前景會丟例外、 + `list_windows` 不再列出 DWM cloak 與零面積視窗、`post_key` 對可列印字元只送 `WM_CHAR`。 +- **混合 DPI 螢幕上的座標與樣板要重錄**：Windows 改成 per-monitor v2 之後，縮放比例與主螢幕不同的螢幕上， + 座標變成原本的「該螢幕縮放／主螢幕縮放」倍（125% 副螢幕配 100% 主螢幕是 ×1.25），截圖是實體像素、不再是縮小的影像； + 在那些螢幕上錄的點擊座標、`screen_region`、視窗版面與裁出來的樣板都要重做。主螢幕上的不受影響。 + 這台機器現在只接一個螢幕，位移量是推導的，沒有量過；第三個螢幕接在縮放螢幕之後時，原點會不會也移動，未知。 +- **舊版存的視窗版面檔要重存**：`save_window_layout` 改存 `GetWindowRect`，舊檔還原會偏右 7 px、縮小 14×7 px。 +- **真的 Win32 行為沒有實測**：`post_key` 在真的編輯框是否只出現一個字、DWM cloak 過濾掉的是哪些視窗、 + `MoveWindow` 對最大化視窗與另一個 DPI 的螢幕是否能原樣還原，都只用假的 `user32` 驗過。 +- **X11／Wayland**：大寫與 Shift 標點的判斷假設 `keysym_to_keycode` 把大寫字母對到小寫的 keycode，沒有對著 X server 跑過。 -同一次稽核的影像與 OCR 部分也在它的路徑上（Discord bot 的 `!find_image`／`!find_text`），一併等： +**還沒決定的小事**：Windows 鍵表沒有 `slash` 這個名字。`plus`／`minus`／`comma`／`period` 與 `oem_1`…`oem_8` 都有， +但鍵表的註解刻意不替 `oem_*` 取好讀的別名，因為它們隨鍵盤配置而變（`oem_2` 只有在美式配置上是 `/`）。 +要加的話是 `wrapper/_platform_windows.py` 的 `keyboard_key_aliases` 一行。 -- **非 ASCII 路徑與灰階樣板**：`cv2_utils/template_detection.py:126` 經 `je_open_cv` 的 `cv2.imread` 讀樣板，`測試\t.png` 讀不到；2-D 陣列或 PIL `"L"` 樣板丟出 `cv2.error`，不在 `wrapper/auto_control_image.py` 的例外清單裡。做法：路徑改走 `cv2_utils/image_file.read_image`，2-D 直接用，`cv2.error` 包成 `ImageNotFoundException`。 -- **部分超出螢幕的 `screen_region` 被補黑**：`monitor_layout/logical_frame.py:143` 沒有先和畫面取交集，PIL `crop` 補零，可能回傳螢幕外的命中；寬或高為負時丟裸 `ValueError`。做法：先取交集（回傳裁過的原點），非正的寬高丟框架例外。 -- **OCR 跨框比對漏掉從長框中段開始的字串**：`ocr/text_span.py:330` 的視窗超過「目標長度＋40」就整個丟掉最左框，即使目標從那框開始；`"Save As"` 在長句框之後就找不到。做法：只有剩下的部分仍不短於目標時才丟左框。 -- **負座標的中心點差一**：`wrapper/auto_control_image.py:48`、`:73` 的 `int((x1 + x2) / 2)` 向零截斷。做法：`(x1 + x2) // 2`。 -- **Unicode 打字把換行與 Tab 當字元送**：`utils/text_unicode/text_unicode.py:49` `plan_unicode_keys("a\nb\tc")` 送出碼位 10 與 9，多數程式會丟掉 Unicode 的 LF；`write` 早就把它們對到 Return／Tab（`WRITE_CONTROL_KEYS`）。這個模組被 `wrapper/auto_control_keyboard.py:27` 載入。做法：控制空白改成按鍵。 -- **鍵盤配置表的 Shift 半邊與非美式鍵**：`utils/keyboard_layout/keyboard_layout.py:98-100` 的 Shift 半邊是死鍵時退回未按 Shift 的字（美式國際配置的 Shift+6 回 `'6'`，契約是回 `None`）；`:96` 只翻譯美式鍵碼，德／法／北歐鍵盤的 `VK_OEM_102`（0xE2）與英式 `VK_OEM_8` 永遠沒有標籤；`:68-74` 把原型設在全程序共用的 `ctypes.windll.user32` 上，之後別的呼叫者用 `c_ubyte` 陣列呼叫 `ToUnicodeEx` 會 `ArgumentError`。Jeffrey_RPA 的 `_gui_control.py:3322` 呼叫 `ac.foreground_keyboard_layout()`。做法：死鍵半邊回 `None`；候選鍵碼加上 0xDF、0xE1、0xE2…；改用私有的 `ctypes.WinDLL("user32")`。 -- **剪貼簿格式名稱 `None`**：`utils/clipboard_formats/clipboard_formats.py:46` `_coerce` 把 tuple／list 描述的 `None` 名稱變成字串 `"None"`，dict 形式卻是 `""`，`diff_formats` 因此回報有變動。Jeffrey_RPA 的 `_gui_control.py:1126` 呼叫 `ac.clipboard_formats()`。做法：兩種形式都把 `None` 正規化成 `""`。 - -**解除條件**：Jeffrey_RPA 沒有批次在跑（`webrunner.pid` 的行程不在、Discord bot 停止）；改完在 Jeffrey_RPA 跑 `test/test_je_facade.py`。 - ---- - -## macOS 上 `click_mouse(clicks=2)` 不是雙擊 - -`TODO` — `wrapper/auto_control_mouse.py` 的 `click_mouse` 在同一點連點 `clicks` 次。Windows 與 X11 依兩次點擊的時間差與位移判定雙擊，所以這樣就夠；macOS 的應用程式讀的是事件上的點擊次數欄位（`kCGMouseEventClickState`），`osx/mouse/osx_mouse.py` 的 `mouse_event` 從不設定它（一律是 1），所以 macOS 收到的是兩次單擊。文件（`docs/source/API/wrapper/mouse.rst`、`docs/source/{Eng,Zh}/doc/mouse/mouse_doc.rst`、`click_mouse` 的 docstring）照實寫了這個限制。 - -做法：讓 osx 後端的按下／放開帶點擊次數（第 n 次點擊設成 n），wrapper 在 macOS 分支把序號傳下去；要在 macOS 上實測（`quality.yml` 的 `macos-14` 可以跑 CI，但雙擊是否被應用程式認得要真機看），完成後把文件裡的限制拿掉。 - ---- - -## RBAC 還沒接到 REST API 與 MCP server - -`DECIDE` — 要不要把 `utils/rbac` 接上兩個伺服器，以及現有單一共用 token 怎麼過渡（維護者拍板） - -`utils/rbac/users.py` 有使用者、角色與權杖驗證（2026-09-24 已補上：壞檔不覆寫、權杖不得重複），但沒有任何程式 -import 它：`rest_api/rest_auth.py` 與 `mcp_server/http_transport.py` 都只比對一個共用 token，稽核 log 也沒有 -`user_id`。模組 docstring 已改成照實描述。 - -**做法**：`RestAuthGate.check` 改成先查 `UserStore.authenticate`、再依路由對應的 `Capability` 呼叫 `can()`； -MCP 的 bearer 比對同理；稽核寫入帶上 `user_id`。 - -**要先想清楚**：沒有任何使用者時是否退回共用 token（相容現有部署）；viewer／operator／admin 各能呼叫哪些路由與工具。 +**MCP 的 `_show_command`**（`utils/mcp_server/tools/_handlers_system.py`，`window_minimize` 等）仍不看 `show_window` 的新回傳值。 --- -## 能執行動作的人也能替檔案簽章 +## macOS 的修正只對著假的 Quartz 驗過 -`DECIDE` — `JE_AUTOCONTROL_REQUIRE_SIGNED_ACTIONS` 要防的是誰（維護者拍板） +`TODO` — 要一台真的 Mac（Retina，最好接第二個螢幕）；CI 的 macos-14 能跑其中一部分 -`AC_sign_action_file` 用預設的個人金鑰簽章，所以凡是能透過 socket、REST 或 MCP 執行動作的人，都能先簽一個檔再用 -`AC_execute_files` 執行它；內嵌的動作清單本來就不驗簽。現在的強制簽章只擋得住「能改檔案、但不能執行動作」的人。 +U-20261009-06 與 -04 改了四件事，全部在 Windows 上以假的 pyobjc／Quartz 物件測試： -**選項**：簽章指令在強制模式下只准本機 CLI 使用；或簽章金鑰與執行權限分開保存（簽章端不在執行端）。 +- **`click_mouse(clicks=2)`**：按下與放開現在帶 `kCGMouseEventClickState`（第 n 次點擊是 n）。應用程式是否因此認得雙擊，沒看過。 + `test_osx_mouse_click_state.py` 有三個只在 darwin 跑的測試會把欄位讀回來，第一次執行在 CI。 +- **還原最小化視窗**：`_info_for` 改用 `kCGWindowListOptionIncludingWindow`，`list_windows` 會附上最小化的視窗。 + `test_window_backend_macos_real.py` 會在 macOS CI 真的開一個視窗、最小化、列出、還原，**從沒執行過**； + `test_a_really_minimised_window_stays_in_the_listing` 依賴最小化視窗的 Quartz 邊界或標題仍對得上它的 AX 元素，最可能紅。 + 沒加 AX 逾時，沒回應的 app 會拖慢列出；同一行程裡與最小化視窗同原點或同標題的螢幕外輔助視窗可能被誤列。 +- **`grab_logical`**（`utils/monitor_layout/macos_frame.py`）：改成點座標、逐螢幕擷取後拼接。三個假設要實機確認： + `screencapture -R` 接受含負值的全域點座標、Pillow 的 `scale_down=True` 給出點尺寸的影像、`CGDisplayBounds` 與 Quartz 滑鼠事件同一個座標空間。 + 每一格要為每個螢幕各開一次 `screencapture`。 -[Answer] 簽章金鑰與執行權限分開保存 --- -## USB passthrough viewer 以種類配對回覆，逾時的回覆會交給下一個請求 - -`DECIDE` — 協定要不要加請求編號（線上格式改動，新舊版本相容要一起想） - -`utils/usb/passthrough/viewer_client.py:413`（`_on_opened`）與 `:466`（`_complete_pending`）只按 OPEN／LIST／claim -配對回覆，回覆沒有序號。請求逾時後，對同一種類的下一個請求會拿到遲到的舊回覆：`open(aaaa)` 逾時、`open(bbbb)` -收到 `aaaa` 的 OPENED，claim 綁錯裝置；bulk 讀逾時後，下一次傳輸拿到上一次的資料。host 接受最長 60 秒的 -`timeout_ms`，client 預設 10 秒就放棄，正常使用就會遇到（2026-09-24 稽核重現）。 - -**選項**：在 payload 加一個由 client 產生、host 原樣帶回的請求編號（舊 host 不帶就退回現在的配對）；或逾時後把該 -claim 標成需排空，丟掉下一個回覆——但 host 若根本沒回，會丟掉正確的回覆。 - -[Answer] 加，但是想辦法解決可能丟掉正確回復的問題 ---- - -## 全域 executor 的變數會留到下一次執行 - -`DECIDE` — 每次頂層執行要不要有自己的變數範圍（行為改動，維護者拍板） - -`execute_action_with_vars`（`utils/executor/action_executor.py`）把變數種進全域 `executor` 後從不清除，REST、MCP、 -socket server 的執行也都用同一個 `executor`；`for_each` 的迴圈變數與巨集參數同樣留著。下一次執行裡的 `${user}` -會安靜地取到前一個呼叫者的值，而不是報 `Unknown variable`（2026-09-24 稽核重現）。模組文件把這個範圍描述成 -「共用」，所以有人可能依賴它在執行之間傳值。 - -**選項**：`execute_action_with_vars` 與各伺服器入口每次開一個新的 `VariableScope`（`AC_set_var` 在單次執行內照舊）； -或保留共用，但在伺服器入口清空，並在文件寫明。 - -**附帶**：`AC_circuit_call`、`AC_bulkhead_run`、`AC_run_chaos`、`AC_run_dag` 的巢狀動作跑在全域 `executor` 上， -在 `AC_parallel` 分支裡因此用到父層的變數範圍，而不是分支自己的。 - -[Answer] `execute_action_with_vars` 與各伺服器入口每次開一個新的 `VariableScope`（`AC_set_var` 在單次執行內照舊 +## 這一輪修正留下的後續 + +`TODO` — 各項都是 2026-10-09 那批（U-20261009-01…14）交付時明確沒做的部分 + +- **Intel Mac 安裝要編譯 `cryptography`**：下限拉到 50 之後 `macosx_10_9_x86_64` 沒有 wheel（探測：最新只到 48.0.1）， + `pip install je_auto_control` 在 Intel Mac 需要 Rust 工具鏈。沒有在 Intel Mac 上實際編過。 +- **簽章分離擋不住能寫檔的人**：能執行寫檔指令（shell、檔案類指令）的人仍能換掉公鑰檔本身，保護那個檔是作業系統權限的事； + 私鑰是未加密的 PEM（0600，Windows 上沒驗權限位元），沒有通行碼選項；遷移模式開著時 HMAC 簽章會被接受。 + 強制簽章經 socket／REST／MCP 的端到端沒有跑過，覆蓋的是共用的 `read_executable_action_json` 與指令本身。 +- **RBAC**（`JE_AUTOCONTROL_RBAC_USERS`，預設關閉）： + - 延後執行的工作不帶角色：operator 註冊的排程、觸發器、熱鍵、watchdog 之後執行時沒有身分，裡面的特權指令不會被擋。 + - 使用者只能用 Python 管（`UserStore.add_user/set_role/rotate_token/remove_user`），沒有 `AC_*`、CLI 或 GUI；`Capability.MANAGE_USERS` 因此沒有任何路由或工具對應。 + - `gui/rest_api_tab.py` 仍顯示共用 token，RBAC 開啟後那個 token 會被拒絕。 + - viewer 拿得到所有標成唯讀的工具，包括 `ac_sql_query`、`ac_load_dotenv`、`ac_get_clipboard`、`ac_jwt_encode`，沒有重新分類。 + - 只實作 `check()` 的自製 REST gate 現在會 `AttributeError`（要有 `authenticate()`）。 + - 沒對真的 client（Claude Desktop、VS Code、內建 dashboard）試過；SSE、無狀態與 `subscriptions/listen` 路徑、TLS 加 RBAC 沒有測試。 +- **MCP 路徑根目錄**（`JE_AUTOCONTROL_MCP_PATH_ROOTS`，預設關閉）管不到的參數：執行動作清單的工具、 + 有時才是路徑的參數（`ac_open_path`／`ac_plan_open`／`ac_file_association`／`ac_act_in_view` 的 `target`，`ac_launch_process`／`ac_shell` 的 `argv`／`command`）、 + `ac_handle_file_dialog` 的 `path`、自由格式物件裡的路徑（`ac_run_suite` 的 `spec`、`ac_run_dag` 的 `definition`、`ac_assert_all` 的 `specs`）、沒標註的外掛工具。 + 檔案 symlink 的跳脫在這台機器上建不出來（只跑了目錄 junction）；POSIX 的 `:` 分隔與 `~` 沒在 Linux／macOS 跑。 + `roots/list` 來的根目錄要另外開 `JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT` 才算數——這是實作時定的，請維護者確認。 +- **USB passthrough**：client 仍在 `reply_timeout_s`（10 秒）放棄，即使呼叫端要的 `timeout_ms` 更長（host 接受到 60 秒）； + 對新 host 逾時已經無害，對舊 host 正常使用就會進入「需重連」。通道層的 ERROR（passthrough 關閉、bad frame、host 沒有 session）在 session 之前送出、不帶請求編號， + 呼叫仍是等到逾時。沒對真的舊版 host、真的 WebRTC 通道或真的 USB 裝置跑過。 +- **變數範圍**：`je_auto_control run --dry-run --var`、observer 回呼（`AC_observe_add` 與 MCP 的 observe bridge）、 + `utils/llm/planner.py` 與狀態機預設 runner 仍用行程層級的範圍。排程／觸發／熱鍵／webhook／e-mail 的隔離是在 `run_counting_failures` 測的，沒有把每個 daemon 真的觸發一次。 + free-threaded 版（執行緒繼承 context）上，一次執行裡另開的執行緒會看到該次執行的範圍，沒測。 +- **遠端桌面的擁有者模型**：Quick Connect 沒有狀態列，它的連線被取代時視窗只是關掉；主機的 Stop 會停任何擁有者開的 host（刻意的）； + 只用假的 viewer／host 測過，真的 viewer 被擠掉時會不會觸發 `on_error` 而多跳一個警告框，沒看過。 +- **套件閘門**：`execute_action` 會先驗完所有指令名稱才執行，所以「載入套件」與「用它的指令」寫在同一份清單裡會以 unknown command 失敗，與閘門無關（既有限制）。 + repo 根目錄的舊範例 `AutoControl/keyword/keyword1.json` 載入 `time`，現在預設會被拒絕。 +- **`pytest --cov` 沒有重新量過**：進入點搬到 `je_auto_control_pytest` 之後，它是否還少算，未知；規則仍是 `coverage run -m pytest`。 +- **快到 750 行上限的檔案**：`utils/usb/passthrough/viewer_client.py`（748）、`gui/remote_desktop/connection_screen.py`（729）、 + `utils/mcp_server/server.py`（734）、`utils/mcp_server/http_transport.py`（727）、`utils/agent/backends/anthropic_computer_use.py`（723）——下一次要加東西就得先拆。 +- **Sphinx 沒有建置過**：這批改了 35 個 `.rst`，都沒有渲染檢查。 --- @@ -290,45 +274,6 @@ socket server 的執行也都用同一個 `executor`；`for_each` 的迴圈變 --- -## mypy 2.4.0 在三個平台模組回報 `has-type` - -`TODO` — 讓型別契約在 mypy 2.4 也過，再把 CI 的 pin 往上提 - -`quality.yml` 的 `typing-stable-api` 固定 `mypy==2.3.0`，`test/verify/typing_contract_verify.py` 在它上面是 0 個失敗模組； -`dev_requirements.txt` 只寫下限 `mypy>=2.3.0`，新環境會裝到 2.4.0，同一個指令就多出三個不在豁免清單上的模組 -（2026-10-08 實測）：`wrapper/_platform_osx.py`（非 darwin 目標下 `osx_key_*` 全部 `Cannot determine type`）、 -`wrapper/_platform_windows.py:363`（`win32_recorder`）、`windows/message/window_message.py:7-8`（`user32`）。 - -**做法**：替這些跨平台名稱補上明確型別註記（或把平台分支改成 mypy 看得懂的 `sys.platform` 判斷）， -兩個版本都驗過後把 `quality.yml` 的 pin 提到 2.4.x。 - ---- - -## pytest11 進入點會把整個門面拉進每一次 pytest - -`DECIDE` — 要不要把進入點搬到一個精簡的頂層模組（打包層的改動，維護者拍板） - -`pyproject.toml` 的 `pytest11` 進入點指向 `je_auto_control.utils.pytest_plugin.plugin`。 -外掛模組本身很輕（只 import pytest，fixture 裡才 import 本套件），但它是**套件的子模組**， -所以 Python 會先跑 `je_auto_control/__init__.py`——量到 **1,355 個模組**。機器上任何一個 -安裝了本套件的環境，每一次 pytest 啟動都付這筆成本（Jeffrey_RPA 因此在 `pytest.ini` 用 -`-p no:je_auto_control` 擋掉它）。pytest 官方文件建議的形狀正是「進入點指向只 import pytest -的精簡模組」。 - -改法：新增頂層模組（例如 `je_auto_control_pytest.py`，`[tool.setuptools] py-modules`）， -進入點改指它，`utils/pytest_plugin/plugin.py` 轉為 re-export 以維持 -`pytest_plugins = ["je_auto_control.utils.pytest_plugin"]` 這條路。 - -**為什麼要拍板**：(1) 這是發佈產物的改動，會在 site-packages 多一個頂層名字； -(2) 進入點改了要重裝才生效（本機的 editable 安裝、CI 的 `pip install -e .`）； -(3) `test/unit_test/headless/test_coverage_measurement.py` 的前提會改變——它現在釘住 -「外掛載入時門面已經在 `sys.modules` 裡」，改完就不成立，那份說明與測試要一起改寫 -（CI 仍可繼續用 `coverage run -m pytest`）。 - -[Answer] 照你建議改寫 - ---- - ## libei 的 `ei_unref` 在半開交握上會 SIGSEGV `BLOCKED` — 上游（libei 1.3.901） @@ -348,174 +293,18 @@ socket server 的執行也都用同一個 `executor`；`for_each` 的迴圈變 --- -## `cryptography` 的安全下限要不要拉到 50 - -`DECIDE` — 要不要用 Intel Mac 的預編 wheel 換掉一個本套件沒用到的漏洞範圍 - -`pyproject.toml` 的 `cryptography>=48.0.1` 仍包含 GHSA-g6cj-pr64-35w5（high,`>=44.0.0, <50.0.0`, -PKCS#7 EnvelopedData 解密的 Bleichenbacher oracle）的範圍。本套件沒有呼叫 PKCS#7 解密 -（用的是 Fernet，以及 aiortc 的 DTLS），所以目前不受影響。`uv.lock` 已鎖在 50.0.1。 - -**為什麼要拍板**:49.0.0 起上游不再發 `macosx_10_9_universal2` wheel，只剩 `macosx_11_0_arm64`。 -下限拉到 `>=50.0.0` 之後，Intel Mac 上的 `pip install` 要從原始碼編譯（得先裝 Rust 工具鏈）。 -CI 只有 macos-14(arm64)，量不到這一點。重新檢查（不需要機器）: - -```bash -pip install --dry-run --only-binary=:all: --platform macosx_10_9_x86_64 \ - --python-version 3.12 --target /tmp/probe 'cryptography>=50' -``` - -[Answer] 可以 - ---- - -## Viewer 端要不要把 host 推來的檔案關在一個目錄裡 - -`DECIDE` — 這是改一個已寫進文件的功能，由維護者決定 - -`host.send_file_to_viewers(source, dest_path)` 由 **host** 指定 viewer 機器上的完整路徑 -（`docs/source/{Eng,Zh}/doc/new_features/new_features_doc.rst` 的範例是 `/tmp/from_host.bin`）， -viewer 端的 `FileReceiver`（`utils/remote_desktop/file_transfer.py`）照單全收：`expanduser`、 -建立父目錄、寫入。也就是被控端可以在控制端機器的任何可寫位置放檔案。模組說明的 -「trusted token holders == trusted users」只涵蓋 host 端；viewer 連上一台被入侵的 host 時沒有這層保護。 - -**做法**：`FileReceiver` 加 `base_dir`，viewer（`viewer.py` 的 `_ensure_file_receiver`、GUI 的 -`viewer_panel.py`）預設給一個下載目錄，只保留相對路徑並拒絕跳出 `base_dir`；host 端維持現狀。 - -**為什麼要拍板**：`dest_path` 的語意會從「viewer 上的絕對路徑」變成「viewer 下載目錄裡的相對路徑」， -現有腳本與文件範例都要跟著改。 - -[Answer] 可以改 - ---- - -## macOS 無法還原最小化的視窗 - -`TODO` — 需要在 macOS 上驗證，離線的 pyobjc 替身抓不到 - -`wrapper/window_backends/macos_backend.py` 的 `_info_for` 只搜「在螢幕上」的視窗，最小化的不在其中： -`minimize(77)` 成功後 `list_windows` 看不到它、`restore(77)` 丟「請授權 Accessibility」（即使已授權）。 -Windows 的 `list_windows` 則包含最小化視窗。 - -**做法**：`_info_for` 改用 `CGWindowListCopyWindowInfo(kCGWindowListOptionIncludingWindow, window_id)`； -在 macOS CI（TCC 已授權）加一個真的最小化再還原的測試。 - ---- - -## Agent 的截圖修剪會改寫較早的回合 - -`TODO` — 需要付費實機跑一次多步驟任務驗證，不能只靠離線測試 - -`utils/agent/backends/base.py` 的 `prune_old_screenshots` 每一步把較舊的截圖換成文字，改的是已送出過的訊息。 -Claude Fable 5.1 與 Opus 5.5 的 thinking 區塊綁定它之前的整段對話，2026-08-31 之後建立的帳號會直接回 400 -（"block is bound to a different conversation"），約在第 4 步中斷；其他模型則是每一步都讓 prompt cache 失效。 -不修剪也不行：每步重送全部截圖會超過 32 MB 的請求上限。 - -**做法（擇一，依 claude-api 文件的 append-only 對照表）**：用戶端「簡單壓縮」——截圖數超過上限時，以一則摘要 -（目標、已執行的動作）加最新截圖開新對話，不重播舊回合；或送 -`thinking.block_binding.prefix_mismatch_behavior: "drop_block"`（beta `thinking-binding-controls-2026-08-01`）， -讓被改到的 thinking 區塊被丟棄而不是 400。伺服器端 tool-result clearing 不會縮小請求本身，擋不住 32 MB。 - -**要動的地方**：`anthropic.py`、`anthropic_computer_use.py`（兩條路徑）呼叫 `prune_old_screenshots` 之處； -OpenAI 後端沒有這個綁定，照舊。 - ---- - -## MCP 工具的檔案路徑參數要不要限制在工作區根目錄裡 - -`DECIDE` — 限制範圍與預設值由維護者決定 - -MCP 工具的檔案參數（`path`、`file_path`、`db`、`image_path`、`golden_path`、`output_path`… 約 100 個） -不受任何根目錄限制；只有 `resources/read` 關在 `roots/list` 的根目錄裡。完整模式下這不是新的權限 -（`ac_execute_actions` 本來就能做任何事），但 `JE_AUTOCONTROL_MCP_READONLY=1` 的部署仍能讀到根目錄外的 -任意檔案：例如 `ac_load_dotenv` 會把任何檔案解析成 KEY=VALUE 回給模型，`ac_read_document`、 -`ac_extract_pdf_text` 也一樣。2026 年 MCP 伺服器通報最多的一類就是這種路徑越界。 - -**做法**：在 `utils/mcp_server/tools/_factories.py` 的 schema 裡把真正是檔案路徑的屬性標上 -`"format": "path"`（不能照名字判斷：`ac_json_query` 的 `path` 是 JSON 路徑，`template`／`source`／ -`target` 有時是檔案有時不是），`server.py` 的 `_prepare_tool_call` 在設定了根目錄時先 `realpath` -再檢查是否落在根目錄內，不在就回工具執行錯誤（`isError`，和其他參數驗證失敗一樣）。 - -**為什麼要拍板**：根目錄從哪來（新的環境變數、沿用 `roots/list`、或兩者），唯讀模式要不要預設開啟； -預設開啟會讓現有讀取工作區外檔案的用法失效。 - -同一個問題也在 `ac_resolve_ref`／`ac_resolve_refs`（`_factories.py:7182`，標為 `READ_ONLY`）：`file://` 沒有 -`base_dir` 限制，`env://` 可讀任何環境變數，包括放 API 金鑰的那些，結果直接回給模型。`secret://` 已經拒絕； -`env://` 要不要改成允許清單、`file://` 要不要套同一個根目錄，跟上面一起決定。 - -[Answer] 兩者，不要預設開啟唯獨 - ---- - -## Windows 的 DPI 感知是系統層級，混合 DPI 的螢幕座標被虛擬化 +## Agent 的歷史壓縮還沒對真的 API 跑過 -`DECIDE` — 改成 per-monitor 會移動那些螢幕上的所有座標，Jeffrey_RPA 在那些螢幕上錄的座標與樣板要重錄 +`TODO` — 需要付費實機跑一次多步驟任務；離線測試只驗得到請求的形狀 -`windows/screen/win32_screen.py:50` 在 import 時呼叫 `SetProcessDPIAware()`，那是系統 DPI 感知，不是 -per-monitor。DPI 與主螢幕不同的螢幕會被 Windows 虛擬化：本機第二螢幕 125%，實際 1920×1080，但 Win32、 -`mss` 與 Qt 都回報 `(1920, -164, 1536, 864)`，截圖是 Windows 縮小過的影像，那個螢幕上的樣板比對與 OCR -用的是模糊的畫面。同檔註解說之後「所有 Win32 座標查詢都會拿到實體像素」，只在主螢幕 DPI 的螢幕上成立。 +U-20261009-11 之後，Anthropic 的兩條路徑（`anthropic.py`、`anthropic_computer_use.py` 的 beta 與 GA toolset）不再改寫已送出的回合： +截圖超過 3 張（或 base64 超過 20,000,000 字元）時，以一則「目標＋已執行的動作」摘要加最新截圖開新對話。OpenAI 後端照舊就地修剪。 +假 client 驗到的是：前綴只增不改、重開後只有一則 `[image, text]`、沒有孤兒 `tool_result`、沒有重播 thinking。沒驗到的： -**做法**：先呼叫 `SetProcessDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2)`，失敗再退回 -`SetProcessDPIAware()`，並改正註解；`utils/monitor_layout` 的換算與 `gui/_screen_geometry.py` 一起檢查。 - -**為什麼要拍板**：這個檔在 Jeffrey_RPA 正在跑的截圖路徑上。換成 per-monitor 之後，縮放螢幕上的座標與截圖 -尺寸都會變，既有的樣板和錄好的座標在那些螢幕上會失準。 - -[Answer] 換並修好 Jeffrey_RPA - ---- - -## `pil_screenshot`／`screenshot` 的區域擷取在 Windows 只看得到主螢幕 - -`BLOCKED` — 要改的是 Jeffrey_RPA 正在跑的截圖路徑（`cv2_utils/screenshot.py`、`wrapper/auto_control_screen.py`、`utils/window_capture/window_capture.py`），依工作區規則在它執行期間不動 - -`cv2_utils/screenshot.py:58` 把 `screen_region` 交給 `ImageGrab.grab(bbox=...)`，Pillow 在 Windows 沒帶 -`all_screens=True` 時只擷取主螢幕再裁切，主螢幕外的部分補黑。分析類指令已改走 -`cv2_utils/region_capture.grab_screen_region`（顏色、直方圖、SSIM、對比、顏色等待、QR、VLM、MCP 截圖）， -下面這些仍是舊路徑，在主螢幕左側或上方的螢幕得到全黑影像： - -- `pil_screenshot(screen_region=...)`、`screenshot(screen_region=...)` 與 `AC_screenshot`。 -- `utils/pytest_plugin/keywords.py:41` `keyword_screenshot`（與 `AC_screenshot` 同一語意，一起改）。 -- `utils/window_capture/window_capture.py:66` `capture_window`：視窗在副螢幕時截到黑的。 -- `utils/set_of_marks/set_of_marks.py:121` 把標記畫在 `pil_screenshot()`（只有主螢幕）上，副螢幕的元件沒有標記。 - -**做法**：`pil_screenshot` 的區域路徑在 Windows 改走 `grab_screen_region`（它已處理 DPI 與負座標）， -`capture_window` 同樣；set-of-marks 改用 `grab_logical(None)` 並把原點加回標記座標。 - ---- - -## macOS 的 `grab_logical` 在 Retina 上是像素座標，而且只看得到主螢幕 - -`BLOCKED` — `utils/monitor_layout/logical_frame.py` 在 Jeffrey_RPA 正在跑的截圖路徑上，依工作區規則在它執行期間不動 - -`grab_logical` 在 macOS 呼叫 `ImageGrab.grab(all_screens=True)`。讀 Pillow 12.3.0 的 darwin 分支：`all_screens` -不被使用，`screencapture -x` 只擷取主螢幕；Retina 螢幕的影像是點座標的 2 倍（Pillow 文件：「screen captures will -be at 2x if on a Retina screen」，`scale_down=True` 只在帶 `bbox` 時生效）。`logical_virtual_rect` 只讀 Windows 的 -`GetSystemMetrics`，所以 macOS 不縮放：樣板比對、OCR 與其他走 `grab_logical` 的定位，在 Retina 上回傳的座標是 -滑鼠（Quartz，點座標）的 2 倍，副螢幕上的目標則找不到。GitHub 的 macOS runner 是 1x 虛擬螢幕，CI 測不到。 - -**做法**：darwin 上以 `CGDisplayBounds`／`CGGetActiveDisplayList` 取得各螢幕的點座標範圍；有 `region` 時交給 -`ImageGrab.grab(bbox=..., scale_down=True)`（`screencapture -R` 接受全域點座標，包括負值），整個桌面則逐螢幕擷取、 -各自縮到點座標後拼接，原點取所有螢幕的最小 x／y。需要在 Retina Mac 上實測。 - ---- - -## 遠端桌面的 viewer 槽位由各面板共用 - -`DECIDE` — 要改 `registry` 的擁有權模型 - -`utils/remote_desktop/registry.py` 的 TCP 與 WS viewer 各只有一個槽位，快速連線（`gui/remote_desktop/connection_screen.py`）、 -舊式 viewer 分頁（`viewer_panel.py`）與 `AC_remote_connect` 都寫同一格。每一方連線前先 `registry.disconnect_viewer()`， -於是在一邊連線會切斷另一邊的連線，被切斷的面板卻不知道：它的彈出視窗仍停在最後一格畫面， -「中斷」按鈕則會切斷別人的連線。快速連線的「開始被遠端」也一樣會停掉主機分頁開的 host。 - -**做法**：registry 記錄每個 viewer／host 由誰開的（owner token），`disconnect_*` 只在 owner 相符時動作； -被別人取代時通知原本的面板收掉自己的視窗。或是反過來讓每個面板持有自己的 viewer，不經 registry。 - -**為什麼要拍板**：`AC_remote_*` 指令與 MCP 工具依賴「registry 裡就是那一個 viewer」，改成多槽位要一起改它們的語意。 - -[Answer] 一起改沒問題 +- 重開的歷史在強制 thinking 綁定的帳號（Opus 5.5／Fable 5.1）上是否被接受。 +- 模型只憑摘要能不能接著做——上限是 3，大約每 3 張截圖就壓縮一次，摘要最多列最新 60 個動作、每個截到 240 字元。 +- 真的桌面 PNG 的請求大小離 32 MB 多遠。 +- 兩個後端都沒送 `cache_control`，所以現在本來就沒有 prompt cache；要開的話是另一個請求形狀的改動。 --- @@ -525,10 +314,3 @@ be at 2x if on a Retina screen」，`scale_down=True` 只在帶 `bbox` 時生效 --- -## 套件閘門的預設改成拒絕 - -`BLOCKED` — 等含警告的版本出去之後再發兩版 - -`AC_add_package_to_executor`／`AC_add_package_to_callback_executor` 前面已有套件閘門（工作區 X-12），但沒設定時仍會載入任何套件、只發 `DeprecationWarning`。兩個版本之後，在 `utils/package_manager/package_manager_class.py` 的 `PackageManager.__init__` 把 `allow_arbitrary_packages` 改成 `False`，拿掉 `_check_allowed` 裡的警告分支，並更新三份 README 的「Package gate」段落、`docs/source/{Eng,Zh}/doc/keyword_and_executor/keyword_and_executor_doc.rst` 與 `docs/source/API/utils/package_manager.rst`，`CHANGELOG.md` 記成破壞性變更。 - -**先決定**：只跑動作檔、沒有 Python 宿主程式的使用者（`je_auto_control` CLI、socket／REST／MCP server、排程器）要怎麼放行套件。 diff --git a/README.md b/README.md index ba7d30e84..f062d3d07 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ from JSON files / CLI / servers, and a **GUI tab**. Nothing is GUI-only. - **One API, seven platforms.** `wrapper/platform_wrapper.py` picks the backend at import time; your script does not change between Windows, macOS, X11, and Wayland. -- **Scriptable without Python.** 778 `AC_*` commands cover the whole feature set, so a +- **Scriptable without Python.** 779 `AC_*` commands cover the whole feature set, so a JSON file can do anything the library can — including loops, branches, try/catch, macros, and variables. - **Headless by default.** `import je_auto_control` never loads Qt. The GUI is an @@ -231,7 +231,7 @@ desktop app; tab commands live in the window's **Actions** menu. | Natural-language planner | `plan_actions`, `run_from_description` | `AC_llm_plan` | LLM Planner | | Computer-use agent | `AgentLoop`, `run_agent` | `AC_run_agent` | Computer Use | | Record & replay | `record`, `stop_record` | `AC_record`, `AC_stop_record` | Record | -| JSON scripting | `execute_action`, `execute_files` | all 778 commands | Script, Script Builder | +| JSON scripting | `execute_action`, `execute_files` | all 779 commands | Script, Script Builder | | Variables & flow control | `execute_action_with_vars` | `AC_set_var`, `AC_loop`, `AC_for_each`, `AC_try`, `AC_retry` | Variables | | Data-driven runs | — | `AC_for_each_row` (CSV / JSON / SQLite / Excel) | Data Sources | | Assertions | `assert_text`, `assert_image` | `AC_assert_text` + 20 more | Assertions | @@ -259,7 +259,7 @@ per-module map is in **[architecture_explore.md](architecture_explore.md)**. ## Command-line interface ```bash -je_auto_control run script.json [--var name=value] [--dry-run] +je_auto_control run script.json [--var name=value] [--dry-run] [--allow-package NAME] je_auto_control validate script.json # alias: lint je_auto_control fmt script.json [--check] je_auto_control list-commands [--filter mouse] [--json] @@ -296,7 +296,9 @@ under a relative name are written below that directory. All servers bind to `127.0.0.1` unless you opt in explicitly. -**Package gate.** `AC_add_package_to_executor` and `AC_add_package_to_callback_executor` import a Python package and register its members as commands, so an action list arriving over any of these surfaces could load `os` or `subprocess`. The host program decides what may load: `executor.allow_packages("name", …)` lists the packages (submodules included) and `executor.set_allow_arbitrary_packages(False)` refuses the rest before importing them. Neither is an `AC_*` command, so an action list cannot open its own gate. A refused package fails that action with `AutoControlExecuteActionException`. Until the host calls either switch, any package still loads but raises a `DeprecationWarning`: a future release will refuse unlisted packages by default. +**Package gate.** `AC_add_package_to_executor` and `AC_add_package_to_callback_executor` import a Python package and register its members as commands, so an action list arriving over any of these surfaces could load `os` or `subprocess`. No package loads unless it has been allowed: a package that is not on the allowlist is refused before it is imported, and that action fails with `AutoControlExecuteActionException`. Allow packages (submodules included) from Python with `executor.allow_packages("name", …)`, for every entry point — both CLIs, the socket / REST / MCP servers and the scheduler — with the `JE_AUTOCONTROL_ALLOWED_PACKAGES` environment variable (comma-separated names, read when the process starts), or for one CLI run with `je_auto_control run script.json --allow-package NAME` (repeatable). `executor.set_allow_arbitrary_packages(True)` opens the gate for every package, which is what earlier releases did by default (with a `DeprecationWarning`). None of these is an `AC_*` command, so an action list cannot open its own gate. + +**Opt-in hardening.** Each of these is off until configured, and a server without them behaves as before. `JE_AUTOCONTROL_RBAC_USERS=` makes the REST API and the MCP HTTP transport resolve the bearer token to a user and authorise each route, tool and privileged `AC_*` command by role (viewer / operator / admin); the shared token is then refused. `JE_AUTOCONTROL_MCP_PATH_ROOTS` (directories separated by the OS path separator) confines every MCP tool argument that is a file path to those directories, `JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT=1` adds the client's `roots/list`, and `JE_AUTOCONTROL_MCP_ENV_REF_ALLOW` limits which `env://` names `ac_resolve_ref` may read. `JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY` makes an endpoint verify Ed25519-signed action files without being able to sign them (`create_signing_keypair`; the private key stays on the signing machine). A remote-desktop viewer writes files pushed by a host only below `~/Downloads/AutoControl` (`JE_AUTOCONTROL_REMOTE_DOWNLOAD_DIR`); that one is on by default. ### How the remote-desktop wire protocol works diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index 429fdd30b..a5380bab6 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -19,7 +19,7 @@ - **一套 API，七个平台。** `wrapper/platform_wrapper.py` 在导入时挑选后端；同一份脚本在 Windows、macOS、X11 与 Wayland 上都不需要改写。 -- **不写 Python 也能脚本化。** 778 个 `AC_*` 命令覆盖全部功能，因此一个 JSON 文件能做到库 +- **不写 Python 也能脚本化。** 779 个 `AC_*` 命令覆盖全部功能，因此一个 JSON 文件能做到库 能做的任何事——包含循环、分支、try/catch、宏与变量。 - **默认无头运行。** `import je_auto_control` 绝不会加载 Qt。GUI 是可选包，包在同一个无头内核之外。 - **四种定位方式。** 模板匹配、OCR、无障碍树、视觉语言模型——可通过锚点定位器与自愈回退串接组合。 @@ -219,7 +219,7 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" | 自然语言规划 | `plan_actions`、`run_from_description` | `AC_llm_plan` | LLM Planner | | Computer-use agent | `AgentLoop`、`run_agent` | `AC_run_agent` | Computer Use | | 录制与回放 | `record`、`stop_record` | `AC_record`、`AC_stop_record` | Record | -| JSON 脚本 | `execute_action`、`execute_files` | 全部 778 个命令 | Script、Script Builder | +| JSON 脚本 | `execute_action`、`execute_files` | 全部 779 个命令 | Script、Script Builder | | 变量与流程控制 | `execute_action_with_vars` | `AC_set_var`、`AC_loop`、`AC_for_each`、`AC_try`、`AC_retry` | Variables | | 数据驱动执行 | — | `AC_for_each_row`（CSV／JSON／SQLite／Excel） | Data Sources | | 断言 | `assert_text`、`assert_image` | `AC_assert_text` 等 21 个 | Assertions | @@ -246,7 +246,7 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" ## 命令行界面 ```bash -je_auto_control run script.json [--var name=value] [--dry-run] +je_auto_control run script.json [--var name=value] [--dry-run] [--allow-package NAME] je_auto_control validate script.json # 别名：lint je_auto_control fmt script.json [--check] je_auto_control list-commands [--filter mouse] [--json] @@ -277,7 +277,9 @@ je_auto_control version 除非明确指定，所有服务器都绑定在 `127.0.0.1`。 -**包闸门。** `AC_add_package_to_executor` 与 `AC_add_package_to_callback_executor` 会导入 Python 包并把成员注册成命令，所以从上面任何一个入口送来的动作列表都可能加载 `os` 或 `subprocess`。哪些包可以加载，由宿主程序决定：`executor.allow_packages("name", …)` 列出可以加载的包（含子模块），`executor.set_allow_arbitrary_packages(False)` 会在导入前拒绝其他包。这两个都不是 `AC_*` 命令，所以动作列表不能自己打开闸门。被拒绝的包会让该动作以 `AutoControlExecuteActionException` 失败。宿主程序调用任一个开关之前，任何包仍会加载，但会发出 `DeprecationWarning`：之后的版本会默认拒绝清单以外的包。 +**包闸门。** `AC_add_package_to_executor` 与 `AC_add_package_to_callback_executor` 会导入 Python 包并把成员注册成命令，所以从上面任何一个入口送来的动作列表都可能加载 `os` 或 `subprocess`。没有被放行的包不会加载：不在允许清单上的包在导入前就被拒绝，该动作以 `AutoControlExecuteActionException` 失败。放行包（含子模块）的方式：在 Python 里调用 `executor.allow_packages("name", …)`；对所有入口——两个 CLI、socket／REST／MCP 服务器与调度器——设置环境变量 `JE_AUTOCONTROL_ALLOWED_PACKAGES`（以逗号分隔的名称，进程启动时读取）；或只对一次 CLI 执行使用 `je_auto_control run script.json --allow-package NAME`（可重复）。`executor.set_allow_arbitrary_packages(True)` 会对所有包打开闸门，也就是先前版本的默认行为（当时会发 `DeprecationWarning`）。这些都不是 `AC_*` 命令，所以动作列表不能自己打开闸门。 + +**可选的加固设置。** 下列各项在设置之前都是关闭的，没设置的服务器行为与先前相同。`JE_AUTOCONTROL_RBAC_USERS=<用户文件>` 让 REST API 与 MCP HTTP 传输把 bearer token 对应到用户，并按角色（viewer／operator／admin）授权每一个路由、工具与特权 `AC_*` 命令；此时共用 token 会被拒绝。`JE_AUTOCONTROL_MCP_PATH_ROOTS`（以操作系统的路径分隔符分开的目录）把每一个属于文件路径的 MCP 工具参数限制在这些目录里，`JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT=1` 再加上 client 的 `roots/list`，`JE_AUTOCONTROL_MCP_ENV_REF_ALLOW` 限制 `ac_resolve_ref` 可以读哪些 `env://` 名称。`JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY` 让端点只能验证 Ed25519 签名的动作文件而不能签名（`create_signing_keypair`；私钥留在签名的机器上）。远程桌面的 viewer 只把 host 推来的文件写在 `~/Downloads/AutoControl` 下面（`JE_AUTOCONTROL_REMOTE_DOWNLOAD_DIR`）；这一项默认就是开的。 ### 远程桌面的线路协议 diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index b4e4520fc..5519d8c4b 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -18,7 +18,7 @@ - **一套 API，七個平台。** `wrapper/platform_wrapper.py` 在匯入時挑選後端；同一份腳本在 Windows、macOS、X11 與 Wayland 上都不需要改寫。 -- **不寫 Python 也能腳本化。** 778 個 `AC_*` 指令涵蓋全部功能，因此一個 JSON 檔能做到函式庫 +- **不寫 Python 也能腳本化。** 779 個 `AC_*` 指令涵蓋全部功能，因此一個 JSON 檔能做到函式庫 能做的任何事——包含迴圈、分支、try/catch、巨集與變數。 - **預設無頭執行。** `import je_auto_control` 絕不會載入 Qt。GUI 是選用套件，包在同一個無頭核心之外。 - **四種定位方式。** 樣板比對、OCR、無障礙樹、視覺語言模型——可透過錨點定位器與自癒後備串接組合。 @@ -218,7 +218,7 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" | 自然語言規劃 | `plan_actions`、`run_from_description` | `AC_llm_plan` | LLM Planner | | Computer-use agent | `AgentLoop`、`run_agent` | `AC_run_agent` | Computer Use | | 錄製與重播 | `record`、`stop_record` | `AC_record`、`AC_stop_record` | Record | -| JSON 腳本 | `execute_action`、`execute_files` | 全部 778 個指令 | Script、Script Builder | +| JSON 腳本 | `execute_action`、`execute_files` | 全部 779 個指令 | Script、Script Builder | | 變數與流程控制 | `execute_action_with_vars` | `AC_set_var`、`AC_loop`、`AC_for_each`、`AC_try`、`AC_retry` | Variables | | 資料驅動執行 | — | `AC_for_each_row`（CSV／JSON／SQLite／Excel） | Data Sources | | 斷言 | `assert_text`、`assert_image` | `AC_assert_text` 等 21 個 | Assertions | @@ -245,7 +245,7 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" ## 命令列介面 ```bash -je_auto_control run script.json [--var name=value] [--dry-run] +je_auto_control run script.json [--var name=value] [--dry-run] [--allow-package NAME] je_auto_control validate script.json # 別名：lint je_auto_control fmt script.json [--check] je_auto_control list-commands [--filter mouse] [--json] @@ -276,7 +276,9 @@ je_auto_control version 除非明確指定，所有伺服器都綁在 `127.0.0.1`。 -**套件閘門。** `AC_add_package_to_executor` 與 `AC_add_package_to_callback_executor` 會匯入 Python 套件並把成員註冊成命令，所以從上面任何一個入口送來的動作清單都可能載入 `os` 或 `subprocess`。哪些套件可以載入，由宿主程式決定：`executor.allow_packages("name", …)` 列出可以載入的套件（含子模組），`executor.set_allow_arbitrary_packages(False)` 會在匯入前拒絕其他套件。這兩個都不是 `AC_*` 命令，所以動作清單不能自己打開閘門。被拒絕的套件會讓該動作以 `AutoControlExecuteActionException` 失敗。宿主程式呼叫任一個開關之前，任何套件仍會載入，但會發出 `DeprecationWarning`：之後的版本會預設拒絕清單以外的套件。 +**套件閘門。** `AC_add_package_to_executor` 與 `AC_add_package_to_callback_executor` 會匯入 Python 套件並把成員註冊成命令，所以從上面任何一個入口送來的動作清單都可能載入 `os` 或 `subprocess`。沒有被放行的套件不會載入：不在允許清單上的套件在匯入前就被拒絕，該動作以 `AutoControlExecuteActionException` 失敗。放行套件（含子模組）的方式：在 Python 裡呼叫 `executor.allow_packages("name", …)`；對所有入口——兩個 CLI、socket／REST／MCP 伺服器與排程器——設定環境變數 `JE_AUTOCONTROL_ALLOWED_PACKAGES`（以逗號分隔的名稱，行程啟動時讀取）；或只對一次 CLI 執行使用 `je_auto_control run script.json --allow-package NAME`（可重複）。`executor.set_allow_arbitrary_packages(True)` 會對所有套件打開閘門，也就是先前版本的預設行為（當時會發 `DeprecationWarning`）。這些都不是 `AC_*` 命令，所以動作清單不能自己打開閘門。 + +**選用的強化設定。** 下列各項在設定之前都是關閉的，沒設定的伺服器行為與先前相同。`JE_AUTOCONTROL_RBAC_USERS=<使用者檔>` 讓 REST API 與 MCP HTTP 傳輸把 bearer token 對到使用者，並依角色（viewer／operator／admin）授權每一個路由、工具與特權 `AC_*` 命令；此時共用 token 會被拒絕。`JE_AUTOCONTROL_MCP_PATH_ROOTS`（以作業系統的路徑分隔字元分開的目錄）把每一個屬於檔案路徑的 MCP 工具參數限制在這些目錄裡，`JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT=1` 再加上 client 的 `roots/list`，`JE_AUTOCONTROL_MCP_ENV_REF_ALLOW` 限制 `ac_resolve_ref` 可以讀哪些 `env://` 名稱。`JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY` 讓端點只能驗證 Ed25519 簽章的動作檔而不能簽章（`create_signing_keypair`；私鑰留在簽章的機器上）。遠端桌面的 viewer 只把 host 推來的檔案寫在 `~/Downloads/AutoControl` 底下（`JE_AUTOCONTROL_REMOTE_DOWNLOAD_DIR`）；這一項預設就是開的。 ### 遠端桌面的線路協定 diff --git a/architecture_explore.md b/architecture_explore.md index 561375631..28ba2c2fe 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -19,10 +19,10 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | -| Python 模組總數（含周邊子專案） | 1,074 | -| 程式碼總行數 | 160,861 | +| Python 模組總數（含周邊子專案） | 1,077 | +| 程式碼總行數 | 161,543 | | `je_auto_control/utils/` 子套件數 | 310 | -| `AC_*` 動作指令數（`known_commands()` 實測） | 778 | +| `AC_*` 動作指令數（`known_commands()` 實測） | 779 | | 套件門面 `__all__` 公開名稱數 | 1,244 | | GUI 分頁數（`main_widget` 註冊） | 48 | | MCP 工具數（`build_default_tool_registry()` 實測） | 680 | @@ -92,7 +92,7 @@ USB/IP 協定、Prometheus 指標），以維持這條輕相依基線。 | 模式 | 落點 | 說明 | | --- | --- | --- | | **Strategy** | `wrapper/platform_wrapper.py` | 依 `sys.platform` 只匯入當前 OS 的 `keyboard`／`mouse`／`screen`／`recorder` 實作；Linux 再細分 Wayland／X11，Wayland 後端不可用時自動退回 XWayland 並記警告。新增平台不需要改 wrapper。 | -| **Facade** | `je_auto_control/__init__.py` | 把 1,251 個公開名稱集中再匯出，使用者只 `import je_auto_control`。`api/core.py` 另提供一個小而穩定的版本化門面（7 個名稱）給新整合使用。 | +| **Facade** | `je_auto_control/__init__.py` | 把 1,264 個公開名稱集中再匯出，使用者只 `import je_auto_control`。`api/core.py` 另提供一個小而穩定的版本化門面（7 個名稱）給新整合使用。 | | **Command** | `utils/executor/action_executor.py` | `Executor.event_dict` 是字串 → callable 的分派表；JSON 動作檔即指令序列，因此可錄製、序列化、重播、簽章。 | | **Observer** | `utils/callback/`、`utils/observer/`、`utils/triggers/` | 動作完成後觸發回呼；畫面出現／消失／變化與外部事件（webhook／IMAP／檔案）驅動腳本。 | | **Template Method** | `utils/generate_report/` | HTML／JSON／XML 三個產生器共用「收集紀錄 → 格式化 → 寫檔」骨架，各自實作渲染。 | @@ -154,7 +154,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | 模組 | 行數 | 職責 | | --- | ---: | --- | -| `je_auto_control/__init__.py` | 1,986 | **套件門面**。集中匯入並再匯出 1,251 個公開名稱，以功能區塊註解分段（callback／exception／executor／a11y／vision／clipboard…）。 | +| `je_auto_control/__init__.py` | 1,994 | **套件門面**。集中匯入並再匯出 1,264 個公開名稱，以功能區塊註解分段（callback／exception／executor／a11y／vision／clipboard…）。 | | `je_auto_control/__main__.py` | 98 | 舊版 argparse 進入點：`-e` 執行單檔、`-d` 執行整個目錄、`--execute_str` 執行 JSON 字串、`-c` 建立專案。 | | `je_auto_control/cli.py` | 372 | **主 CLI**（`je_auto_control` console script）。子命令：`run`（含 `--var`／`--dry-run`）、`validate`／`lint`、`list-commands`、`fmt`、`record`、`codegen`、`failure-bundle`、`list-jobs`、`start-server`、`start-rest`、`version`。所有子命令延遲匯入，確保不碰 Qt。 | | `je_auto_control/api/__init__.py` | 22 | 版本化整合進入點。 | @@ -182,8 +182,8 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `wrapper/auto_control_screen.py` | 111 | 螢幕 API：`screen_size`、`screenshot`（可指定區域）、`get_pixel`。 | | `wrapper/auto_control_image.py` | 85 | 影像 API：`locate_all_image`、`locate_image_center`、`locate_and_click`。 | | `wrapper/auto_control_record.py` | 124 | 錄製 API：`record`／`stop_record`／`record_to_json`（支援 stop event 與逾時）。 | -| `wrapper/auto_control_window.py` | 329 | 視窗管理門面：列舉、尋找、聚焦、等待、關閉、顯示狀態、幾何、所屬行程 PID、依行程列舉／最小化視窗、不搶焦點的投遞式輸入（目前僅 Windows 實作）。 | -| `wrapper/window_backends/` | 1,100 | 視窗管理的平台縫（`base` / `windows_backend` / `x11_backend` / `macos_backend` / `null_backend`）。放在 `wrapper/` 而不是 `utils/`，因為它必須 import `windows/`、`linux_with_x11/`、`osx/`，而 `utils/` 在分層上在那三者之上。 | +| `wrapper/auto_control_window.py` | 331 | 視窗管理門面：列舉、尋找、聚焦、等待、關閉、顯示狀態、幾何、所屬行程 PID、依行程列舉／最小化視窗、不搶焦點的投遞式輸入（目前僅 Windows 實作）。 | +| `wrapper/window_backends/` | 1,109 | 視窗管理的平台縫（`base` / `windows_backend` / `x11_backend` / `macos_backend` / `null_backend`）。放在 `wrapper/` 而不是 `utils/`，因為它必須 import `windows/`、`linux_with_x11/`、`osx/`，而 `utils/` 在分層上在那三者之上。 | ### 5.3 平台後端 @@ -272,7 +272,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.1 執行引擎與腳本資產 -> 24 個套件、約 15,216 行。 +> 24 個套件、約 15,223 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -283,7 +283,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/dag/` | 559 | 跨主機 DAG 編排器（圖模型 + runner） | | `utils/decision_table/` | 112 | DMN 風格決策表：規則 + 命中策略，把分支外部化 | | `utils/deterministic/` | 116 | 決定性執行控制：固定亂數種子 + 凍結時鐘 | -| `utils/executor/` | 9,659 | **核心**。`Executor` 指令分派表（778 個 `AC_*`）、參數插值、乾跑、逐步 callback；`flow_control` 提供 34 個區塊指令（迴圈／分支／try／巨集／變數） | +| `utils/executor/` | 9,666 | **核心**。`Executor` 指令分派表（779 個 `AC_*`）、參數插值、乾跑、逐步 callback；`flow_control` 提供 34 個區塊指令（迴圈／分支／try／巨集／變數） | | `utils/flow_debugger/` | 166 | action list 的單步除錯器與追蹤器 | | `utils/input_macro/` | 462 | 定時輸入事件：錄製結果的整形（`timeline`／`InputRecorder`，Windows 與 macOS 共用）、重播與宣告式輸入序列 DSL | | `utils/json/` | 99 | action JSON 檔讀寫與正規化格式化（`fmt --check` 的後端） | @@ -494,7 +494,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.9 AI / Agent / LLM -> 13 個套件、約 23,540 行。 +> 13 個套件、約 23,676 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -507,7 +507,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/cua_action/` | 204 | 標準化 computer-use 動作結構（Anthropic／OpenAI → `AC_*`） | | `utils/llm/` | 365 | 自然語言 → action list 規劃器 + Anthropic／null 後端 | | `utils/mcp_registry/` | 97 | MCP registry `server.json` 資訊清單產生（可被發現） | -| `utils/mcp_server/` | 19,025 | **無頭 MCP 伺服器**（16K LOC，預設註冊 680 個工具＝661 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | +| `utils/mcp_server/` | 19,161 | **無頭 MCP 伺服器**（16K LOC，預設註冊 680 個工具＝661 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | | `utils/tool_use_schema/` | 195 | 把 `AC_*` 指令匯出成 Claude／OpenAI 的 tool-use schema | | `utils/trajectory_eval/` | 132 | agent 軌跡評估：依評分規準為一次執行打分 | | `utils/vision/` | 538 | VLM 元素定位器（依描述找元素）+ Anthropic／OpenAI／null 後端 | @@ -527,7 +527,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.11 伺服器、網路協定與外部整合 -> 24 個套件、約 6,947 行。 +> 24 個套件、約 7,101 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -549,7 +549,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/otp/` | 37 | TOTP 一次性密碼產生（自動化 2FA 登入） | | `utils/outbox/` | 107 | 交易式 outbox，保證至少一次的事件投遞 | | `utils/pytest_plugin/` | 283 | pytest 外掛 + BDD step library（`pytest11` entry point） | -| `utils/rest_api/` | 1,885 | 純標準庫 REST 前端：路由、Bearer 驗證、限流、Prometheus 指標、OpenAPI 3.1 產生 | +| `utils/rest_api/` | 2,039 | 純標準庫 REST 前端：路由、Bearer 驗證、限流、Prometheus 指標、OpenAPI 3.1 產生 | | `utils/socket_server/` | 163 | 執行 action JSON 的執行緒式 TCP 指令伺服器（預設綁 127.0.0.1） | | `utils/sse_client/` | 142 | Server-Sent Events 用戶端解析 | | `utils/tls_acme/` | 473 | TLS 自動化：HTTP-01 挑戰伺服器、金鑰／CSR、自動續期 | @@ -630,7 +630,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.14 安全、機密與合規 -> 13 個套件、約 3,035 行。 +> 13 個套件、約 3,401 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -639,7 +639,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/governance/` | 242 | 治理：maker-checker 核准閘門與即時憑證租約 | | `utils/license_policy/` | 240 | 以 SBOM 元件評估 SPDX 授權允許／拒絕政策 | | `utils/provenance/` | 126 | SLSA 建置來源證明（in-toto v1） | -| `utils/rbac/` | 302 | 角色型存取控制：使用者、角色與權杖驗證（尚未接到 REST／MCP） | +| `utils/rbac/` | 668 | 角色型存取控制：使用者、角色與權杖驗證（尚未接到 REST／MCP） | | `utils/redaction/` | 508 | 截圖遮蔽層：規則偵測 + 政策 + 協調器（上傳 VLM 前先遮） | | `utils/sbom/` | 170 | SBOM（CycloneDX）產生 | | `utils/secret_ref/` | 238 | URI scheme 形式的值參照解析 | @@ -696,19 +696,19 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 上表以子套件為單位；以下把行數最大的幾個子系統展開到檔案層。 -#### `utils/executor/`（9,659 行）— 執行核心 +#### `utils/executor/`（9,666 行）— 執行核心 | 檔案 | 行數 | 職責 | | --- | ---: | --- | -| `action_executor.py` | 8,433 | `Executor` 類別與 `event_dict` 分派表（778 個指令），另含數百個把 utils 能力接成指令的 adapter 函式；全域單例 `executor` 與 `add_command_to_executor()` 擴充點。 | -| `flow_control.py` | 651 | 真正的流程控制：`AC_loop`／`AC_for_each`／`AC_while_*`／`AC_if_*`／`AC_try`／`AC_retry`／`AC_parallel`／`AC_define_macro`／`AC_call_macro`／變數指令（`AC_set_var`／`AC_get_var`／`AC_inc_var`）。`LoopBreak`／`LoopContinue` 以例外實作。34 個區塊指令的分派表 `BLOCK_COMMANDS` 也在這裡，含下一列匯入的資料來源指令。 | +| `action_executor.py` | 8,435 | `Executor` 類別與 `event_dict` 分派表（779 個指令），另含數百個把 utils 能力接成指令的 adapter 函式；全域單例 `executor` 與 `add_command_to_executor()` 擴充點。 | +| `flow_control.py` | 656 | 真正的流程控制：`AC_loop`／`AC_for_each`／`AC_while_*`／`AC_if_*`／`AC_try`／`AC_retry`／`AC_parallel`／`AC_define_macro`／`AC_call_macro`／變數指令（`AC_set_var`／`AC_get_var`／`AC_inc_var`）。`LoopBreak`／`LoopContinue` 以例外實作。34 個區塊指令的分派表 `BLOCK_COMMANDS` 也在這裡，含下一列匯入的資料來源指令。 | | `flow_data_commands.py` | 272 | `AC_*_to_var` 資料來源與轉換指令：shell、時鐘、亂數、PDF、TOTP、SQL、檔案、HTTP、OCR，加上 `AC_assert_var`／`AC_assert_db`／`AC_assert_duration`／`AC_transform_var`。都不執行巢狀 action list，所以沒有迴圈／分支語意。 | | `action_schema.py` | 159 | action list 的結構驗證：形狀、參數型別、未知指令拒絕。單一走訪同時支援兩種消費方式：`validate_actions()` 遇到第一個問題就拋、`unknown_command_names()` 收齊全部不認得的名字（REST `/execute` 用它回 400）。 | | `action_redaction.py` | 86 | 記錄與紀錄鍵用的遮蔽：`AC_secret_*` 的參數（金庫通行碼、機密值）在寫進 log、當成結果紀錄的鍵之前換成 `***`，巢狀在區塊指令裡的也一樣。 | | `mouse_aliases.py` | 39 | 單鍵點擊別名（`AC_click_left` 等），executor 與 callback executor 共用。 | | `flags.py` | 19 | 旗標讀取 `as_bool`：依拼法（`"true"`／`"yes"`／`"on"`／`"1"`）而非 Python 真值判斷，所以 JSON、CLI、MCP 傳來的 `"false"` 是關；所有 adapter 與流程指令共用。 | -#### `utils/mcp_server/`（19,025 行，680 個工具）— 最大子系統 +#### `utils/mcp_server/`（19,161 行，680 個工具）— 最大子系統 | 檔案 | 行數 | 職責 | | --- | ---: | --- | @@ -724,8 +724,8 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `tools/_handlers_executor_bridge.py` | 1,429 | 252 個純委派（中位數 3 行，最長的 16 行全是參數簽章）：每個都是 `from action_executor import _x` 再 `return _x(...)`，沒有分支邏輯。超過 750 行,理由記在 `Progress.md` 的豁免表（再切只能照 MCP 工廠領域分,會把同一種委派散進十幾個沒有語意邊界的檔）。 | | `tools/_handlers_locators.py` | 436 | 同一種 adapter，定位主題：無障礙樹、智慧等待、自我修復、螢幕觀察、座標空間、視覺與 OCR、影像去重、元件倉庫、A/B 定位。 | | `tools/_handlers_operations.py` | 629 | 同一種 adapter，營運主題：agent 與其記憶／追蹤、治理與合規、成本與遙測、失敗掛鉤、看門狗、速率限制、檢查點、核可、產物與資產、測試選擇與分片、佇列與 saga。 | -| `server.py` | 732 | JSON-RPC 2.0 over stdio 的最小 MCP 伺服器：連線範圍狀態、行內／併發分派、工具與 resource／prompt 處理器；握手時代的方法表（`_run_method`），兩個協定時代的逐請求分派在 `_stateless.py`。 | -| `http_transport.py` | 719 | MCP 的 HTTP 傳輸；宣告 2026-07-28 的請求走 `_http_stateless.py` 的標頭規則，不發 session。 | +| `server.py` | 734 | JSON-RPC 2.0 over stdio 的最小 MCP 伺服器：連線範圍狀態、行內／併發分派、工具與 resource／prompt 處理器；握手時代的方法表（`_run_method`），兩個協定時代的逐請求分派在 `_stateless.py`。 | +| `http_transport.py` | 727 | MCP 的 HTTP 傳輸；宣告 2026-07-28 的請求走 `_http_stateless.py` 的標頭規則，不發 session。 | | `_http_stateless.py` | 185 | MCP 2026-07-28 在 Streamable HTTP 上的規則：`MCP-Protocol-Version`／`Mcp-Method`／`Mcp-Name` 必須與 body 相符（`=?base64?…?=` 先解碼），不符是 400＋`HeaderMismatch`；版本與中繼資料錯誤 400、未知方法 404。純函式，由 `http_transport.py` 回覆。 | | `http_sessions.py` | 247 | MCP 的 HTTP 傳輸用的 session 身分:`Mcp-Session-Id` 註冊表,以及每個 session 那條常駐的 server→client SSE 串流。 | | `_client_requests.py` | 274 | 伺服器主動送出的請求：`roots/list`／`elicitation/create`／`sampling/createMessage`,對應表與回應路由,以及破壞性工具的確認交握。只屬於握手時代：無狀態請求裡送出會丟例外。 | @@ -741,7 +741,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `tools/_validation.py` | 122 | MCP 工具用到的 JSON Schema 子集驗證器。 | | `tools/plugin_tools.py` | 89 | 把外掛載入的 `AC_*` callable 包成 `MCPTool`。 | | `log_bridge.py` | 118 | 把 Python logging 記錄橋接成 MCP `notifications/message`；2026-07-28 的請求只收到自己設了 `logLevel` 時產生的記錄。 | -| `audit.py` | 94 | MCP 工具呼叫稽核記錄。 | +| `audit.py` | 112 | MCP 工具呼叫稽核記錄。 | | `context.py` | 71 | 傳給 opt-in 工具處理器的每次呼叫上下文。 | | `rate_limit.py` | 48 | 工具呼叫的 token bucket 限流。 | | `__main__.py` | 92 | `je_auto_control_mcp` console script 進入點。 | @@ -820,17 +820,17 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `usbip/libusb_backend.py` | 224 | 以 PyUSB／libusb 執行 URB 的正式後端。 | | `usbip/backend.py` | 87 | 可插拔 URB 執行後端。 | -#### `utils/rest_api/`（1,885 行） +#### `utils/rest_api/`（2,039 行） | 檔案 | 行數 | 職責 | | --- | ---: | --- | -| `rest_server.py` | 549 | HTTP 前端主體。 | +| `rest_server.py` | 634 | HTTP 前端主體。 | | `rest_handlers.py` | 528 | 端點實作。 | -| `rest_openapi.py` | 431 | 走訪路由表產生 OpenAPI 3.1 規格。 | -| `rest_auth.py` | 157 | Bearer token 驗證 + 逐 client 限流閘門。 | +| `rest_openapi.py` | 443 | 走訪路由表產生 OpenAPI 3.1 規格。 | +| `rest_auth.py` | 202 | Bearer token 驗證 + 逐 client 限流閘門。 | | `rest_metrics.py` | 75 | Prometheus 曝露端點。 | -| `rest_registry.py` | 75 | 保存執行中 REST 伺服器的行程級單例。 | -| `__main__.py` | 56 | `python -m je_auto_control.utils.rest_api` 進入點。 | +| `rest_registry.py` | 77 | 保存執行中 REST 伺服器的行程級單例。 | +| `__main__.py` | 66 | `python -m je_auto_control.utils.rest_api` 進入點。 | #### 其他多檔子套件 @@ -1080,15 +1080,15 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | 層／子系統 | 檔案數 | 行數 | | --- | ---: | ---: | | `gui/` | 98 | 28,507 | -| `utils/mcp_server/` | 36 | 19,025 | +| `utils/mcp_server/` | 37 | 19,161 | | `utils/remote_desktop/` | 56 | 13,301 | -| `utils/executor/` | 8 | 9,659 | +| `utils/executor/` | 8 | 9,666 | | `utils/usb/` | 17 | 4,816 | -| `je_auto_control/`（頂層 3 檔） | 3 | 2,456 | +| `je_auto_control/`（頂層 3 檔） | 3 | 2,464 | | `utils/accessibility/` | 14 | 3,143 | -| `wrapper/` | 19 | 4,083 | +| `wrapper/` | 19 | 4,094 | | `windows/` | 23 | 2,154 | -| `utils/rest_api/` | 8 | 1,885 | +| `utils/rest_api/` | 8 | 2,039 | | `utils/agent/` | 9 | 2,142 | | `linux_with_x11/` | 19 | 1,281 | | `linux_wayland/` | 17 | 2,921 | @@ -1099,6 +1099,6 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `osx/` | 17 | 981 | | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 852 | -| 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 687 | 58,187 | -| **總計** | **1,068** | **160,796** | +| 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 689 | 58,553 | +| **總計** | **1,071** | **161,478** | diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 1ff1082b7..43b28bdb0 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -218,3 +218,87 @@ Jeffrey_RPA kept its own `KEY_ALIASES` (40 friendly names) and `_EXTRA_KEY_CODES - **Docs**: the three READMEs (extras table), `docs/source/{Eng,Zh}/doc/new_features/v223_features_doc.rst`, `architecture.md` §2 and §5, `architecture_explore.md` (GUI file table, extension points), `CHANGELOG.md`. - **Files**: `je_auto_control/gui/{tab_registry,navigation,theme}.py` (new), `je_auto_control/gui/{main_widget,main_window}.py`, the four `gui/language_wrapper/` catalogues (8 keys each), the tests above. - **Open items**: in `Progress.md` › 跨平台與 GUI 全面改版 — page content is squeezed, not scrolled, in a narrow window; the theme and panel state are not remembered between runs; `qt-material` is still in the `[gui]` extra; tab close buttons are Fusion's; F3 (workers, cancel, close) and F4 (benchmarks, parity) have not started. + +## U-20261009-01 · 2026-10-09 · pytest plugin loads from a top-level module; start-up no longer imports the facade · #done #packaging #pytest #A1 + +- **What**: `je_auto_control_pytest.py` holds the plugin and imports only pytest; `utils/pytest_plugin/plugin.py` re-exports the same objects (private helpers included). `pyproject.toml` and `dev.toml`: `py-modules`, the `pytest11` entry point, coverage `source`. `quality.yml` lints and scans the new module. `CLAUDE.md`, `architecture.md` and `test_coverage_measurement.py` describe the old import as history; `coverage run -m pytest` stays the rule. +- **Tests**: `test_pytest_entrypoint_light.py` (4, child interpreters). A wheel and an sdist built with `uv build` contain the module and `[pytest11] je_auto_control = je_auto_control_pytest`. +- **Open items**: `pytest --cov` was not re-measured against the new entry point. + +## U-20261009-02 · 2026-10-09 · Typing contract passes on mypy 2.4; CI checks with 2.4.0 · #done #typing + +- **What**: mypy 2.4 resolves `sys.platform not in [...]` against the target, so past those guards a foreign-platform module is unreachable and unannotated names have no type for importers. `osx_vk.py` defines its tables before the platform check; `user32`, `win32_recorder` and `osx_recorder` are annotated. `quality.yml` pins `mypy==2.4.0`, `dev_requirements.txt` asks for `>=2.4.0` (#506). +- **Tests**: `typing_contract_verify.py`: 0 failing modules on win32 / linux / darwin with mypy 2.3.0 and with 2.4.0. +- **Open items**: none. + +## U-20261009-03 · 2026-10-09 · Keyboard and mouse wrappers type and scroll what was asked · #done #input #A13 + +- **What**: `write` holds Shift for capitals (and US-shifted punctuation on X11), `is_shift` works on Windows and X11 and is released in `finally`, CR LF is one Enter; `mouse_scroll` defaults to `scroll_up`, validates before clamping, rounds coordinates; `text_unicode` presses control whitespace as keys; `keyboard_layout` reports dead-key Shift halves as `None`, covers OEM_8 / OEM_102, uses a private `WinDLL`; `clipboard_formats` normalises a `None` name. +- **Tests**: `test_input_wrapper_contract.py` (42), recording fakes only; four existing files updated for the scroll default. +- **Open items**: nothing typed for real; Jeffrey_RPA's facade test not run; the `slash` alias is undecided (`Progress.md`). + +## U-20261009-04 · 2026-10-09 · Windows: per-monitor DPI, every-monitor capture, honest focus and layout; macOS frame in points · #done #windows #dpi #capture #A8 #A12 + +- **What**: `win32_screen.enable_dpi_awareness` (per-monitor v2, fallback, idempotent); region screenshots, `capture_window`, `keyword_screenshot` and set-of-marks go through the virtual-desktop capture; `grab_logical` clips and rejects empty regions; `focus_window` confirms the foreground on the backend that can (`confirms_foreground`, Windows); show / z-order return real results; cloaked and zero-area windows are not listed; `post_key` sends one message per character; layouts store `GetWindowRect`; snap / grid / cascade use the work area; `wait_for_window` is bounded. `utils/monitor_layout/macos_frame.py` captures each display in points and stitches. +- **Tests**: `test_window_lifecycle_contract.py` (39) and `test_platform_coordinate_contract.py` (26), fakes only. +- **Open items**: no mixed-DPI desktop, real Win32 window or Mac was used (`Progress.md`). + +## U-20261009-05 · 2026-10-09 · Templates at any path and in grayscale; OCR keeps the left box; negative centres floor · #done #image #ocr #A14 + +- **What**: `template_detection` reads through `image_file.read_image`, accepts 2-D / `L` / `1` templates and raises `ImageNotFoundException` for an unreadable template or a `cv2.error`; `text_span.find_spans` drops a leading box only when what follows still covers the target; centres use `//`. +- **Tests**: `test_image_ocr_coordinate_contract.py` (19); the non-ASCII path ran with real OpenCV on this machine. +- **Open items**: none. + +## U-20261009-06 · 2026-10-09 · macOS: click count on mouse events, minimised windows listed and restorable · #done #macos + +- **What**: `osx_mouse` writes `kCGMouseEventClickState` (n-th click sets n, within the system double-click interval); `macos_backend._info_for` asks Quartz by window id, `list_windows` appends minimised windows. +- **Tests**: `test_osx_mouse_click_state.py` (21) and 16 new cases in `test_window_backend_macos.py` against fake Quartz; `test_window_backend_macos_real.py` (2) runs only on macOS CI. +- **Open items**: never executed on macOS (`Progress.md`). + +## U-20261009-07 · 2026-10-09 · Every top-level run gets its own variable scope · #done #executor #A3 + +- **What**: `utils/script_vars/execution.py`: `execution_scope`, a `ContextVar` binding restored in `finally`. `execute_action_with_vars`, REST, MCP tool calls, socket commands, `run_counting_failures` (scheduler, triggers, hotkeys, webhooks, e-mail), ChatOps and voice open one. `AC_parallel` branches fork the caller's variables; `AC_run_dag` local nodes use the calling executor and scope. Direct `executor.execute_action` keeps the process scope. +- **Tests**: `test_execution_scope_isolation.py` (26); 20 failed before the change. +- **Open items**: three Python-API paths still use the process scope (`Progress.md`). + +## U-20261009-08 · 2026-10-09 · Ed25519 action signatures keep the signing key away from execution; cryptography floor 50 · #done #security #signing #A5 #A11 + +- **What**: Version-2 sidecar (JSON, Ed25519, key id), `create_signing_keypair`, `AC_create_signing_keypair`, three environment variables; a verify-only endpoint refuses to sign; a legacy HMAC sidecar needs the explicit migration variable once a key pair is configured. `cryptography>=50.0.0` in `pyproject.toml`, `dev.toml` and `uv.lock`; `CryptographyUnavailableError`. +- **Tests**: `test_signing_execution_roles.py` (30), `test_crypto_fifty_install_contract.py` (6). Wheel probes: `macosx_10_9_x86_64` has none at 50 (newest 48.0.1), `macosx_11_0_arm64` resolves 50.0.2, `win_arm64` stops at 46.0.3. +- **Open items**: see `Progress.md` (file-writing commands, unencrypted private key, Intel Mac source build). + +## U-20261009-09 · 2026-10-09 · Opt-in roots for MCP file arguments; viewer keeps pushed files in its download directory · #done #security #mcp #remote-desktop #A6 + +- **What**: `utils/path_guard/policy.py` (`PathPolicy`), `utils/mcp_server/_argument_policy.py`; 168 argument positions in 143 tools carry `"format": "path"`; `file://` and `env://` references follow the same policy and an allowlist. `FileReceiver(base_dir=)`, `default_download_dir()`; the TCP and WebSocket viewers and the viewer panel confine by default. +- **Tests**: `test_file_boundary_policy.py` (103 cases; 2 skipped: this account cannot create file symlinks, the junction halves ran). +- **Open items**: arguments the schema cannot classify (`Progress.md`). + +## U-20261009-10 · 2026-10-09 · USB passthrough replies are paired by an echoed request id · #done #usb #protocol #A4 + +- **What**: Optional `request_id` in the JSON payload, generated by the viewer, echoed by the host on every reply; a timed-out request leaves a tombstone so only its own late reply is discarded. Against a host that echoes nothing, a timeout marks the claim (or the client, for OPEN / LIST) not reusable and raises `UsbClientDesynchronized` instead of guessing. A late OPENED releases its claim. +- **Tests**: `test_usb_request_correlation.py` (36); 23 failed before the change. +- **Open items**: client timeout versus `timeout_ms`; channel-level errors carry no id (`Progress.md`). + +## U-20261009-11 · 2026-10-09 · Anthropic agent history is append-only; long runs restart from a summary · #agent #A9 + +- **What**: `backends/base.py`: `compact_history`, `summarise_steps`, `needs_compaction`. `anthropic.py` and both paths of `anthropic_computer_use.py` stop calling `prune_old_screenshots`; past 3 screenshots (or 20,000,000 base64 characters) the next request is one `[image, text]` user message. The OpenAI backend is unchanged. +- **Tests**: `test_agent_append_only_history.py` (11), fake clients. +- **Open items**: no live API run (`Progress.md`); not `#done` for that reason. + +## U-20261009-12 · 2026-10-09 · Remote-desktop hosts and viewers have owners · #done #remote-desktop #gui #C4 + +- **What**: `registry.adopt / evict / release / owner_of / owned`; panels own what they open and are told, on the GUI thread, when replaced. `AC_remote_*`, `AC_ws_*` and the MCP tools act as owner `script` on the active session, as before. +- **Tests**: `test_remote_registry_owners.py` (55), `test_remote_registry_owners_gui.py` (20, offscreen, fakes). +- **Open items**: see `Progress.md`. + +## U-20261009-13 · 2026-10-09 · REST routes and MCP tools are authorised by role when a user store is configured · #done #security #rbac #A10 + +- **What**: `utils/rbac/authorization.py` and `policy.py`, `utils/mcp_server/_authz.py`. Off unless `JE_AUTOCONTROL_RBAC_USERS`, `user_store=` or `--users` names a store; then the shared token is refused, routes / tools / privileged commands need a capability, `tools/list` shows only what the caller may call, and audit entries carry the user. +- **Tests**: `test_rbac_server_wiring.py` (26). +- **Open items**: deferred jobs carry no role; no user management surface (`Progress.md`). + +## U-20261009-14 · 2026-10-09 · The package gate refuses unlisted packages by default · #done #security #breaking + +- **What**: `PackageManager.allow_arbitrary_packages` defaults to `False`; `JE_AUTOCONTROL_ALLOWED_PACKAGES` is read where the package manager is created, so every entry point honours it; `je_auto_control run --allow-package`. +- **Tests**: `test_package_gate.py` (12). +- **Open items**: none beyond the pre-existing same-list limitation (`Progress.md`). diff --git a/docs/updates/README.md b/docs/updates/README.md index 73e4ad061..e6dbf4fcf 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,20 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261009-14 | 2026-10-09 | The package gate refuses unlisted packages by default | #done #security #breaking | [2026-10](2026-10.md) | +| U-20261009-13 | 2026-10-09 | REST routes and MCP tools are authorised by role when a user store is configured | #done #security #rbac #A10 | [2026-10](2026-10.md) | +| U-20261009-12 | 2026-10-09 | Remote-desktop hosts and viewers have owners | #done #remote-desktop #gui #C4 | [2026-10](2026-10.md) | +| U-20261009-11 | 2026-10-09 | Anthropic agent history is append-only; long runs restart from a summary | #agent #A9 | [2026-10](2026-10.md) | +| U-20261009-10 | 2026-10-09 | USB passthrough replies are paired by an echoed request id | #done #usb #protocol #A4 | [2026-10](2026-10.md) | +| U-20261009-09 | 2026-10-09 | Opt-in roots for MCP file arguments; viewer keeps pushed files in its download directory | #done #security #mcp #remote-desktop #A6 | [2026-10](2026-10.md) | +| U-20261009-08 | 2026-10-09 | Ed25519 action signatures keep the signing key away from execution; cryptography floor 50 | #done #security #signing #A5 #A11 | [2026-10](2026-10.md) | +| U-20261009-07 | 2026-10-09 | Every top-level run gets its own variable scope | #done #executor #A3 | [2026-10](2026-10.md) | +| U-20261009-06 | 2026-10-09 | macOS: click count on mouse events, minimised windows listed and restorable | #done #macos | [2026-10](2026-10.md) | +| U-20261009-05 | 2026-10-09 | Templates at any path and in grayscale; OCR keeps the left box; negative centres floor | #done #image #ocr #A14 | [2026-10](2026-10.md) | +| U-20261009-04 | 2026-10-09 | Windows: per-monitor DPI, every-monitor capture, honest focus and layout; macOS frame in points | #done #windows #dpi #capture #A8 #A12 | [2026-10](2026-10.md) | +| U-20261009-03 | 2026-10-09 | Keyboard and mouse wrappers type and scroll what was asked | #done #input #A13 | [2026-10](2026-10.md) | +| U-20261009-02 | 2026-10-09 | Typing contract passes on mypy 2.4; CI checks with 2.4.0 | #done #typing | [2026-10](2026-10.md) | +| U-20261009-01 | 2026-10-09 | pytest plugin loads from a top-level module; start-up no longer imports the facade | #done #packaging #pytest #A1 | [2026-10](2026-10.md) | | U-20261008-02 | 2026-10-08 | GUI: navigation panel with search, dark and light themes from tokens, tabs built on first open | #feature #gui #F1 #F2 | [2026-10](2026-10.md) | | U-20261008-01 | 2026-10-08 | The legacy CLI flags exit 1 when an action failed; reports go below TestPioneer's artifact directory | #feature #cli #report #testpioneer | [2026-10](2026-10.md) | | U-20261006-01 | 2026-10-06 | Make AutoControl discoverable and safer for AI agents | #feature #ai #mcp #docs | [2026-10](2026-10.md) | @@ -349,7 +363,7 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 13 | +| [2026-10.md](2026-10.md) | 2026-10 | 27 | | [2026-09-e.md](2026-09-e.md) | 2026-09 | 7 | | [2026-09-d.md](2026-09-d.md) | 2026-09 | 55 | | [2026-09-c.md](2026-09-c.md) | 2026-09 | 38 | diff --git a/je_auto_control/actions.pyi b/je_auto_control/actions.pyi index f382bf1e4..35e4e39ee 100644 --- a/je_auto_control/actions.pyi +++ b/je_auto_control/actions.pyi @@ -808,6 +808,9 @@ def AC_costs_summary(limit: int = ...) -> Dict[str, Any]: def AC_create_project(project_path: str | None = ..., parent_name: str = ...) -> None: """Create project directory structure and templates.""" +def AC_create_signing_keypair(private_path: str, public_path: str) -> Dict[str, Any]: + """Executor adapter: create an Ed25519 action-signing key pair.""" + def AC_critical_steps(steps: Any, top: Any = ...) -> Dict[str, Any]: """Adapter: the steps that dominate a run's time (bottlenecks).""" @@ -2688,8 +2691,8 @@ def AC_shard_suite( def AC_shell_command(shell_command: str | List[str] | None = ..., *, command: str | List[str] | None = ...) -> None: """Execute shell command with shell=False.""" -def AC_sign_action_file(path: str, key: str | None = ...) -> Dict[str, Any]: - """Executor adapter: write an HMAC-SHA256 signature sidecar for a file.""" +def AC_sign_action_file(path: str, key: str | None = ..., private_key_path: str | None = ...) -> Dict[str, Any]: + """Executor adapter: write an Ed25519 or HMAC signature sidecar for a file.""" def AC_simhash(text: str, bits: Any = ...) -> Dict[str, Any]: """Adapter: SimHash fingerprint of text (as int).""" @@ -2752,6 +2755,7 @@ def AC_start_mcp_http_server( mcp: Any = ..., auth_token: str | None = ..., ssl_context: Any = ..., + user_store: Any = ..., ) -> Any: """Start and return an :class:`HttpMCPServer`; convenience wrapper.""" @@ -2987,7 +2991,12 @@ def AC_validate_json(data: Any, schema: Any) -> Dict[str, Any]: def AC_validate_rows(rows: List[Dict[str, Any]], schema: Dict[str, Any]) -> Dict[str, Any]: """Adapter: validate rows against a declarative schema.""" -def AC_verify_action_file(path: str, key: str | None = ..., raise_on_fail: bool = ...) -> Dict[str, Any]: +def AC_verify_action_file( + path: str, + key: str | None = ..., + raise_on_fail: bool = ..., + public_key_path: str | None = ..., +) -> Dict[str, Any]: """Executor adapter: verify an action file against its signature sidecar.""" def AC_verify_artifact(name: str, content: Any, approvals_dir: str = ..., extension: str = ...) -> Dict[str, Any]: diff --git a/je_auto_control/wrapper/auto_control_window.py b/je_auto_control/wrapper/auto_control_window.py index c413483d7..b450c50f8 100644 --- a/je_auto_control/wrapper/auto_control_window.py +++ b/je_auto_control/wrapper/auto_control_window.py @@ -70,10 +70,11 @@ def _became_foreground(backend, hwnd: int, settle_s: float) -> bool: def focus_window(title_substring: str, case_sensitive: bool = False) -> int: """Bring the first matching window to the foreground; return its hwnd. - Raises ``AutoControlActionException`` when the window is not the foreground - one afterwards. Windows' foreground lock refuses a background process - often, and reporting success then sends the caller's next keystrokes to - whatever window the user has active. + On a backend that can confirm it (Windows), raises + ``AutoControlActionException`` when the window is not the foreground one + afterwards. Windows' foreground lock refuses a background process often, + and reporting success then sends the caller's next keystrokes to whatever + window the user has active. """ hit = find_window(title_substring, case_sensitive) if hit is None: @@ -88,7 +89,8 @@ def focus_window(title_substring: str, case_sensitive: bool = False) -> int: if backend.is_minimized(hwnd): backend.restore(hwnd) backend.set_foreground(hwnd) - if not _became_foreground(backend, hwnd, _FOCUS_SETTLE_S): + confirms = getattr(backend, "confirms_foreground", False) + if confirms and not _became_foreground(backend, hwnd, _FOCUS_SETTLE_S): raise AutoControlActionException( f"focus_window: {title!r} (hwnd={hwnd}) did not become the " f"foreground window; the system refused the request") diff --git a/je_auto_control/wrapper/window_backends/base.py b/je_auto_control/wrapper/window_backends/base.py index dd7c23014..689ba5f51 100644 --- a/je_auto_control/wrapper/window_backends/base.py +++ b/je_auto_control/wrapper/window_backends/base.py @@ -25,6 +25,12 @@ class WindowManageBackend: answers, and a caller that cannot tell them apart will retry forever. """ + #: Whether ``foreground_window()`` right after ``set_foreground(id)`` is a + #: reliable answer to "did it work". Only then does ``focus_window`` turn a + #: mismatch into an error; elsewhere a mismatch may only mean the platform + #: reports its foreground window under another id. + confirms_foreground: bool = False + name: str = "abstract" available: bool = False diff --git a/je_auto_control/wrapper/window_backends/windows_backend.py b/je_auto_control/wrapper/window_backends/windows_backend.py index 9772a0f6f..c10ec800b 100644 --- a/je_auto_control/wrapper/window_backends/windows_backend.py +++ b/je_auto_control/wrapper/window_backends/windows_backend.py @@ -13,6 +13,9 @@ class WindowsWindowBackend(WindowManageBackend): """ name = "win32" + # GetForegroundWindow answers with the HWND set_foreground was given, so + # a refused request can be told from a granted one. + confirms_foreground = True def __init__(self) -> None: self.available = sys.platform in ("win32", "cygwin", "msys") diff --git a/test/unit_test/headless/test_window_lifecycle_contract.py b/test/unit_test/headless/test_window_lifecycle_contract.py index db8f474c6..7bdd62cd7 100644 --- a/test/unit_test/headless/test_window_lifecycle_contract.py +++ b/test/unit_test/headless/test_window_lifecycle_contract.py @@ -154,6 +154,8 @@ def test_an_unknown_key_name_is_still_refused(monkeypatch): class _FocusBackend: """A desktop whose foreground lock refuses unless ``allow`` is set.""" + confirms_foreground = True + def __init__(self, allow): self.allow = allow self.foreground = 99 @@ -184,6 +186,15 @@ def test_foreground_failure_propagates(monkeypatch): w.focus_window("Editor") +def test_a_backend_that_cannot_confirm_the_foreground_is_not_second_guessed(monkeypatch): + # macOS and X11 may report the foreground window under another id; a + # mismatch there is not evidence that the request was refused. + backend = _FocusBackend(allow=False) + backend.confirms_foreground = False + monkeypatch.setattr(w, "get_backend", lambda: backend) + assert w.focus_window("Editor") == 11 + + def test_focus_window_returns_the_handle_once_it_is_in_front(monkeypatch): backend = _FocusBackend(allow=True) monkeypatch.setattr(w, "get_backend", lambda: backend) From cd7be7ac866cedbd3de8d8efa656df445e379f06 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:42:08 +0800 Subject: [PATCH 24/26] Find a minimised macOS window through the full list when the by-id query misses it, and settle the analyser findings The macos-14 runner showed the by-id Quartz query returning nothing for a window the same process had just minimised. The DAG runner's default no longer rebinds a parameter through an import, which the analyser read as a possible None. --- architecture_explore.md | 18 +++++++++--------- je_auto_control/utils/dag/runner.py | 3 ++- je_auto_control/utils/rbac/users.py | 4 +++- .../wrapper/window_backends/macos_backend.py | 15 ++++++++++----- test/unit_test/headless/test_package_gate.py | 2 +- .../headless/test_window_backend_macos_real.py | 2 +- 6 files changed, 26 insertions(+), 18 deletions(-) diff --git a/architecture_explore.md b/architecture_explore.md index 28ba2c2fe..e4f1efc4a 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -20,7 +20,7 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | | Python 模組總數（含周邊子專案） | 1,077 | -| 程式碼總行數 | 161,543 | +| 程式碼總行數 | 161,551 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 779 | | 套件門面 `__all__` 公開名稱數 | 1,244 | @@ -183,7 +183,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `wrapper/auto_control_image.py` | 85 | 影像 API：`locate_all_image`、`locate_image_center`、`locate_and_click`。 | | `wrapper/auto_control_record.py` | 124 | 錄製 API：`record`／`stop_record`／`record_to_json`（支援 stop event 與逾時）。 | | `wrapper/auto_control_window.py` | 331 | 視窗管理門面：列舉、尋找、聚焦、等待、關閉、顯示狀態、幾何、所屬行程 PID、依行程列舉／最小化視窗、不搶焦點的投遞式輸入（目前僅 Windows 實作）。 | -| `wrapper/window_backends/` | 1,109 | 視窗管理的平台縫（`base` / `windows_backend` / `x11_backend` / `macos_backend` / `null_backend`）。放在 `wrapper/` 而不是 `utils/`，因為它必須 import `windows/`、`linux_with_x11/`、`osx/`，而 `utils/` 在分層上在那三者之上。 | +| `wrapper/window_backends/` | 1,114 | 視窗管理的平台縫（`base` / `windows_backend` / `x11_backend` / `macos_backend` / `null_backend`）。放在 `wrapper/` 而不是 `utils/`，因為它必須 import `windows/`、`linux_with_x11/`、`osx/`，而 `utils/` 在分層上在那三者之上。 | ### 5.3 平台後端 @@ -272,7 +272,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.1 執行引擎與腳本資產 -> 24 個套件、約 15,223 行。 +> 24 個套件、約 15,224 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -280,7 +280,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/action_signing/` | 716 | action 檔 HMAC-SHA256 簽章與 Fernet 加密，`execute_files` 會強制驗簽 | | `utils/checkpoint/` | 129 | 流程檢查點與續跑，讓長 action list 具持久性 | | `utils/codegen/` | 294 | 由 action list 產生可執行的 pytest / python / robot 測試碼 | -| `utils/dag/` | 559 | 跨主機 DAG 編排器（圖模型 + runner） | +| `utils/dag/` | 560 | 跨主機 DAG 編排器（圖模型 + runner） | | `utils/decision_table/` | 112 | DMN 風格決策表：規則 + 命中策略，把分支外部化 | | `utils/deterministic/` | 116 | 決定性執行控制：固定亂數種子 + 凍結時鐘 | | `utils/executor/` | 9,666 | **核心**。`Executor` 指令分派表（779 個 `AC_*`）、參數插值、乾跑、逐步 callback；`flow_control` 提供 34 個區塊指令（迴圈／分支／try／巨集／變數） | @@ -630,7 +630,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.14 安全、機密與合規 -> 13 個套件、約 3,401 行。 +> 13 個套件、約 3,403 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -639,7 +639,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/governance/` | 242 | 治理：maker-checker 核准閘門與即時憑證租約 | | `utils/license_policy/` | 240 | 以 SBOM 元件評估 SPDX 授權允許／拒絕政策 | | `utils/provenance/` | 126 | SLSA 建置來源證明（in-toto v1） | -| `utils/rbac/` | 668 | 角色型存取控制：使用者、角色與權杖驗證（尚未接到 REST／MCP） | +| `utils/rbac/` | 670 | 角色型存取控制：使用者、角色與權杖驗證（尚未接到 REST／MCP） | | `utils/redaction/` | 508 | 截圖遮蔽層：規則偵測 + 政策 + 協調器（上傳 VLM 前先遮） | | `utils/sbom/` | 170 | SBOM（CycloneDX）產生 | | `utils/secret_ref/` | 238 | URI scheme 形式的值參照解析 | @@ -1086,7 +1086,7 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `utils/usb/` | 17 | 4,816 | | `je_auto_control/`（頂層 3 檔） | 3 | 2,464 | | `utils/accessibility/` | 14 | 3,143 | -| `wrapper/` | 19 | 4,094 | +| `wrapper/` | 19 | 4,099 | | `windows/` | 23 | 2,154 | | `utils/rest_api/` | 8 | 2,039 | | `utils/agent/` | 9 | 2,142 | @@ -1099,6 +1099,6 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `osx/` | 17 | 981 | | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 852 | -| 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 689 | 58,553 | -| **總計** | **1,071** | **161,478** | +| 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 689 | 58,556 | +| **總計** | **1,071** | **161,486** | diff --git a/je_auto_control/utils/dag/runner.py b/je_auto_control/utils/dag/runner.py index 48443b76f..1a53092a9 100644 --- a/je_auto_control/utils/dag/runner.py +++ b/je_auto_control/utils/dag/runner.py @@ -316,7 +316,8 @@ def _default_local_runner(node: DagNode, _definition: DagDefinition, """Run a local node's actions on ``runner`` (default: the module executor).""" from je_auto_control.utils.json.json_file import read_executable_action_json if runner is None: - from je_auto_control.utils.executor.action_executor import executor as runner + from je_auto_control.utils.executor.action_executor import executor + runner = executor # raise_on_error=True: by default a failed action is only recorded, so # a node whose actions all failed counted as succeeded and its # dependants ran anyway. diff --git a/je_auto_control/utils/rbac/users.py b/je_auto_control/utils/rbac/users.py index ca5b2aea0..e88eccdd8 100644 --- a/je_auto_control/utils/rbac/users.py +++ b/je_auto_control/utils/rbac/users.py @@ -231,7 +231,9 @@ def _read(self) -> Tuple[Dict[str, UserRecord], Optional[str]]: if not self._path.exists(): return {}, None try: - body = json.loads(self._path.read_text(encoding="utf-8")) + # The path is the operator's own choice of user store (constructor, + # JE_AUTOCONTROL_RBAC_USERS or --users); no request can set it. + body = json.loads(self._path.read_text(encoding="utf-8")) # NOSONAR pythonsecurity:S8707 # reason: operator-configured file, never request data except (OSError, ValueError) as error: # ValueError: bad JSON or not UTF-8 return {}, repr(error) entries = body.get("users") if isinstance(body, dict) else None diff --git a/je_auto_control/wrapper/window_backends/macos_backend.py b/je_auto_control/wrapper/window_backends/macos_backend.py index b8f087df4..806186fce 100644 --- a/je_auto_control/wrapper/window_backends/macos_backend.py +++ b/je_auto_control/wrapper/window_backends/macos_backend.py @@ -139,11 +139,16 @@ def _info_for(self, window_id: int) -> Optional[dict]: wanted = int(window_id) if wanted <= 0: return None - found = Quartz.CGWindowListCopyWindowInfo( - Quartz.kCGWindowListOptionIncludingWindow, wanted) - for info in found or []: - if _number(info) == wanted: - return info + # Asked by id first. On a real window server (macos-14) that query + # came back without a window the same process had just minimised, so + # the complete list is the second source rather than a failure. + for option, relative_to in ( + (Quartz.kCGWindowListOptionIncludingWindow, wanted), + (Quartz.kCGWindowListOptionAll, Quartz.kCGNullWindowID)): + found = Quartz.CGWindowListCopyWindowInfo(option, relative_to) + for info in found or []: + if _number(info) == wanted: + return info return None def foreground_window(self) -> int: diff --git a/test/unit_test/headless/test_package_gate.py b/test/unit_test/headless/test_package_gate.py index 974d7b75a..133a6b7e1 100644 --- a/test/unit_test/headless/test_package_gate.py +++ b/test/unit_test/headless/test_package_gate.py @@ -84,7 +84,7 @@ def test_environment_variable_reaches_the_gate_every_entry_point_shares(tmp_path tree = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname( os.path.abspath(package_manager_class.__file__))))) env["PYTHONPATH"] = os.pathsep.join(filter(None, [tree, env.get("PYTHONPATH")])) - done = subprocess.run( # nosec B603 # reason: fixed argv, this interpreter + done = subprocess.run( # nosec B603 # nosemgrep # reason: fixed argv, this interpreter [sys.executable, "-c", code], env=env, cwd=tmp_path, capture_output=True, text=True, timeout=120, check=True) assert json.loads(done.stdout.strip().splitlines()[-1]) == [["my_plugins", "time"], False] diff --git a/test/unit_test/headless/test_window_backend_macos_real.py b/test/unit_test/headless/test_window_backend_macos_real.py index b32ae56af..848bd74c0 100644 --- a/test/unit_test/headless/test_window_backend_macos_real.py +++ b/test/unit_test/headless/test_window_backend_macos_real.py @@ -93,7 +93,7 @@ def owned_window(): backend = MacOSWindowBackend() if not backend.available: pytest.skip("the macOS window backend is unavailable here") - child = subprocess.Popen( # nosec B603 # reason: this interpreter, fixed argv, no shell + child = subprocess.Popen( # nosec B603 # nosemgrep # reason: this interpreter, fixed argv, no shell [sys.executable, "-c", _CHILD], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) try: From 6abb2a3e6e0ff7ac33a002162f3d9b7929f9cf1a Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 00:42:43 +0800 Subject: [PATCH 25/26] Keep the analyser marker within the line limit --- je_auto_control/utils/rbac/users.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/je_auto_control/utils/rbac/users.py b/je_auto_control/utils/rbac/users.py index e88eccdd8..898021165 100644 --- a/je_auto_control/utils/rbac/users.py +++ b/je_auto_control/utils/rbac/users.py @@ -233,7 +233,7 @@ def _read(self) -> Tuple[Dict[str, UserRecord], Optional[str]]: try: # The path is the operator's own choice of user store (constructor, # JE_AUTOCONTROL_RBAC_USERS or --users); no request can set it. - body = json.loads(self._path.read_text(encoding="utf-8")) # NOSONAR pythonsecurity:S8707 # reason: operator-configured file, never request data + body = json.loads(self._path.read_text(encoding="utf-8")) # NOSONAR pythonsecurity:S8707 except (OSError, ValueError) as error: # ValueError: bad JSON or not UTF-8 return {}, repr(error) entries = body.get("users") if isinstance(body, dict) else None From 95e3d9c4720b8decf4cbe2099ef7f570748640eb Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 01:01:53 +0800 Subject: [PATCH 26/26] Record what the macOS runners verified and what still needs a Mac --- Progress.md | 24 +++++++++++++----------- docs/updates/2026-10.md | 4 ++-- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/Progress.md b/Progress.md index 2dfbd4945..dd14e3088 100644 --- a/Progress.md +++ b/Progress.md @@ -205,21 +205,23 @@ pip install --dry-run --only-binary=:all: --platform win_arm64 --python-version --- -## macOS 的修正只對著假的 Quartz 驗過 +## macOS 的修正：CI 驗到一半，其餘要一台真的 Mac -`TODO` — 要一台真的 Mac（Retina，最好接第二個螢幕）；CI 的 macos-14 能跑其中一部分 +`TODO` — 要一台真的 Mac（Retina，最好接第二個螢幕） -U-20261009-06 與 -04 改了四件事，全部在 Windows 上以假的 pyobjc／Quartz 物件測試： +U-20261009-06 與 -04 的 macOS 部分是在 Windows 上對假的 pyobjc／Quartz 寫的。2026-10-09 在 CI 的 macos-14（3.10 與 3.14）跑過的： -- **`click_mouse(clicks=2)`**：按下與放開現在帶 `kCGMouseEventClickState`（第 n 次點擊是 n）。應用程式是否因此認得雙擊，沒看過。 - `test_osx_mouse_click_state.py` 有三個只在 darwin 跑的測試會把欄位讀回來，第一次執行在 CI。 -- **還原最小化視窗**：`_info_for` 改用 `kCGWindowListOptionIncludingWindow`，`list_windows` 會附上最小化的視窗。 - `test_window_backend_macos_real.py` 會在 macOS CI 真的開一個視窗、最小化、列出、還原，**從沒執行過**； - `test_a_really_minimised_window_stays_in_the_listing` 依賴最小化視窗的 Quartz 邊界或標題仍對得上它的 AX 元素，最可能紅。 - 沒加 AX 逾時，沒回應的 app 會拖慢列出；同一行程裡與最小化視窗同原點或同標題的螢幕外輔助視窗可能被誤列。 -- **`grab_logical`**（`utils/monitor_layout/macos_frame.py`）：改成點座標、逐螢幕擷取後拼接。三個假設要實機確認： +- **還原最小化視窗**：`test_window_backend_macos_real.py` 的兩個測試（真的開視窗、最小化、以 id 查、列出、還原）通過。 + 第一次執行時是紅的：`kCGWindowListOptionIncludingWindow` 以 id 查不到同一行程剛最小化的視窗，`_info_for` 因此加了以完整清單查找的後備路徑。 +- **點擊次數欄位**：`test_osx_mouse_click_state.py` 的 darwin 專屬測試通過——真的 Quartz 事件會保存寫進去的 `kCGMouseEventClickState`，`doubleClickInterval()` 是正數。 + +還沒驗的： + +- **應用程式是否把 `click_mouse(clicks=2)` 認成雙擊**：只確認了欄位的值，沒有對真的應用程式送事件。 +- **`grab_logical`**（`utils/monitor_layout/macos_frame.py`，點座標、逐螢幕擷取後拼接）完全沒在 macOS 跑過。三個假設要實機確認： `screencapture -R` 接受含負值的全域點座標、Pillow 的 `scale_down=True` 給出點尺寸的影像、`CGDisplayBounds` 與 Quartz 滑鼠事件同一個座標空間。 - 每一格要為每個螢幕各開一次 `screencapture`。 + 每一格要為每個螢幕各開一次 `screencapture`。CI 的 runner 是 1x 單螢幕，測不到 Retina 與副螢幕。 +- **`list_windows` 在真的桌面上的成本與誤列**：沒加 AX 逾時，沒回應的 app 會拖慢列出；同一行程裡與最小化視窗同原點或同標題的螢幕外輔助視窗可能被誤列。 --- diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 43b28bdb0..e2f1d8e36 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -252,8 +252,8 @@ Jeffrey_RPA kept its own `KEY_ALIASES` (40 friendly names) and `_EXTRA_KEY_CODES ## U-20261009-06 · 2026-10-09 · macOS: click count on mouse events, minimised windows listed and restorable · #done #macos - **What**: `osx_mouse` writes `kCGMouseEventClickState` (n-th click sets n, within the system double-click interval); `macos_backend._info_for` asks Quartz by window id, `list_windows` appends minimised windows. -- **Tests**: `test_osx_mouse_click_state.py` (21) and 16 new cases in `test_window_backend_macos.py` against fake Quartz; `test_window_backend_macos_real.py` (2) runs only on macOS CI. -- **Open items**: never executed on macOS (`Progress.md`). +- **Tests**: `test_osx_mouse_click_state.py` (21) and 16 new cases in `test_window_backend_macos.py` against fake Quartz; `test_window_backend_macos_real.py` (2) runs only on macOS CI, where its first run failed: the by-id Quartz query did not return a window the same process had just minimised, so `_info_for` falls back to the full list. Both real tests and the darwin-only click-state cases then passed on macos-14 (3.10 and 3.14). +- **Open items**: application-level double-click recognition and the point-based capture are unverified (`Progress.md`). ## U-20261009-07 · 2026-10-09 · Every top-level run gets its own variable scope · #done #executor #A3