Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's

### Added

- GUI: a navigation panel on the left of the main window lists every tab by
category, with a search box (`Ctrl+K`; `Ctrl+B` hides the panel).
`AutoControlGUIWidget.activate_tab(key)` opens a tab or brings it to the
front, and `current_tab_key()` names the tab on screen.
- GUI: **View → Theme** switches between a dark and a light theme, both built
from the tokens in `gui/theme.py` (`AutoControlGUIUI.set_theme(name)`).
- AI-agent documentation and a dedicated `AI.md` explain computer-use positioning, MCP aliases, safe tool selection, and OpenAI integration.
- `AC_run_agent` now uses a focused computer-use allow-list by default instead of exposing the full `AC_*` command catalogue to the model.
- `write_secret(secret)` / `AC_write_secret` (`secret`): type a password or
Expand Down Expand Up @@ -155,6 +161,17 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's

### Changed

- GUI tabs are built the first time they are opened. `AutoControlGUIWidget`
registers all 48 tabs from `gui/tab_registry.py` but constructs only the
three it opens on and its own forms; `list_registered_tabs()` builds
nothing, and reading `entry.widget` on a `_tab_entries` row builds that tab.
An embedder that relied on every tab existing after construction (a tab's
timer or listener started at start-up) has to open the tab first.
- The main window no longer imports `qt_material`; it styles itself from
`gui/theme.py`. The `[gui]` extra still lists `qt-material` for now.
- GUI text size "Auto" is 10 / 11 / 13 pt by screen height (was 12 / 14 / 16),
and the default window is 1280×800 (was 1000×760). The font family is the
platform's UI font instead of Lato.
- `je_auto_control_dev`, the dev-channel package, declares what
`je_auto_control` declares: the same pinned dependencies and platform
markers (`defusedxml`, `cryptography` and the `opencv-python` bound are new
Expand Down
18 changes: 16 additions & 2 deletions Progress.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,21 @@
`utils/{config_sync,remote_desktop,mcp_server,self_healing,codegen,executor}/` 與型別/文件驗證。
核准設計:[跨平台自動化與 GUI 改版](docs/superpowers/specs/2026-10-02-platform-gui-modernization-design.md)。
實作計畫:[分階段交付計畫](docs/superpowers/plans/2026-10-02-modernization-index.md),待審閱。
現有 `[Answer]` 決策沿用;產品實作尚未開始。
現有 `[Answer]` 決策沿用。

`WIP` — 計畫 F(GUI):F1 的延遲分頁註冊與 F2 的導覽/搜尋/主題已交付(U-20261008-02),其餘各子計畫尚未開始。F 還缺:

- **窄視窗的內容是被擠壓而不是可捲動**:`gui/main_widget.py` 給 `QTabWidget` 明確的最小尺寸讓視窗能縮到 640×420,
但分頁內容沒有包進 `QScrollArea`;包進去會改變 `tabs.indexOf(entry.widget)` 這個 PyBreeze 與測試都在用的關係,要一起設計。
- **主題與面板狀態不會記住**:`AutoControlGUIUI.set_theme`、導覽面板的顯示與寬度、字級都只活在當次執行;
要用 `QSettings` 存,並讓測試不寫到使用者的設定。
- **`qt-material` 還在 `[gui]` extra**:`gui/main_window.py` 已不匯入它。移除要同時改 `pyproject.toml`、`dev.toml`、
`requirements.txt`、`uv.lock` 與 mypy 的 override,並先確認 PyBreeze 沒有靠這個 extra 取得它。
- **分頁的關閉鈕是 Fusion 內建圖示**:計畫規定主題不新增點陣圖相依,換圖示要用 Qt 內建向量或既有資產。
- **Remote Desktop 與 Script Builder 仍在啟動時建立**(預設開啟),啟動時間 2.6–2.7 秒裡大半是它們與門面匯入。
- **F1 的 `TabRegistry.open/close` 介面與 `close` 釋放訂閱**:現在關閉分頁只是從分頁列移除,widget 留著。
- **F3**(共用 worker、取消、關閉時不碰已銷毀物件、`webrtc_panel.py` 拆分)與 **F4**(啟動/記憶體基準、mixed-DPI、
功能對等測試)尚未開始。

**只記未完成的事。** 完成的工作記在 [docs/updates/](docs/updates/README.md)(每月一個批次檔,
索引與查詢指令在它的 README),相容性變更寫進 [CHANGELOG.md](CHANGELOG.md);完成的項目
Expand Down Expand Up @@ -507,7 +521,7 @@ be at 2x if on a Retina screen」,`scale_down=True` 只在帶 `bbox` 時生效

## `test_usb_acl_prompt.py` 讓 Python 3.10 的 headless 測試間歇 segfault

`TODO` — `test/unit_test/headless/test_usb_acl_prompt.py::test_bridge_remember_persists_acl_rule` 在 `coverage run -m pytest` 下讓行程 SIGSEGV(exit 139),整個 `pytest-headless` job 因此失敗:2026-09-26 連續三次 AutoControl Code Quality(ubuntu-22.04/3.10),2026-09-30 一次(macos-14/3.10);同一次其他版本都過,之後的 run 又過,所以是間歇的。原因還沒查:先在 3.10 開 `faulthandler` 重跑這一支,看崩在哪個原生呼叫。
`TODO` — `test/unit_test/headless/test_usb_acl_prompt.py::test_bridge_remember_persists_acl_rule` 在 `coverage run -m pytest` 下讓行程 SIGSEGV(exit 139),整個 `pytest-headless` job 因此失敗:2026-09-26 連續三次 AutoControl Code Quality(ubuntu-22.04/3.10),2026-09-30 一次(macos-14/3.10),2026-10-08 一次(ubuntu-22.04/3.10,PR #501,重跑該 job 後通過);同一次其他版本都過,之後的 run 又過,所以是間歇的。原因還沒查:先在 3.10 開 `faulthandler` 重跑這一支,看崩在哪個原生呼叫。

---

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ Optional extras, installed only when you need them:

| Extra | Enables |
|---|---|
| `gui` | PySide6 desktop application (48 tabs) |
| `gui` | PySide6 desktop application (48 tabs): a searchable navigation panel (`Ctrl+K`) lists every feature by category, tabs are built the first time they are opened, and **View → Theme** switches dark / light |
| `webrtc` | WebRTC remote desktop, USB passthrough (`aiortc`, `av`) |
| `signaling` | Standalone signaling / rendezvous server (`fastapi`, `uvicorn`) |
| `discovery` | mDNS / Zeroconf LAN host discovery |
Expand Down
2 changes: 1 addition & 1 deletion README/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ pip install je_auto_control[gui] # 加上 PySide6 桌面应用

| Extra | 启用的功能 |
|---|---|
| `gui` | PySide6 桌面应用(48 个标签页) |
| `gui` | PySide6 桌面应用(48 个标签页):左侧可搜索的导航面板(`Ctrl+K`)按分类列出全部功能,标签页在第一次打开时才创建,**View → Theme** 切换深色/浅色 |
| `webrtc` | WebRTC 远程桌面、USB 直通(`aiortc`、`av`) |
| `signaling` | 独立的信令/rendezvous 服务器(`fastapi`、`uvicorn`) |
| `discovery` | mDNS / Zeroconf 局域网主机发现 |
Expand Down
2 changes: 1 addition & 1 deletion README/README_zh-TW.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ pip install je_auto_control[gui] # 加上 PySide6 桌面應用程式

| Extra | 啟用的功能 |
|---|---|
| `gui` | PySide6 桌面應用程式(48 個分頁) |
| `gui` | PySide6 桌面應用程式(48 個分頁):左側可搜尋的導覽面板(`Ctrl+K`)依分類列出全部功能,分頁在第一次開啟時才建立,**View → Theme** 切換深色/淺色 |
| `webrtc` | WebRTC 遠端桌面、USB 直通(`aiortc`、`av`) |
| `signaling` | 獨立的訊令/rendezvous 伺服器(`fastapi`、`uvicorn`) |
| `discovery` | mDNS / Zeroconf 區網主機探索 |
Expand Down
7 changes: 4 additions & 3 deletions architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ entry points → execution core (`utils/executor/`) → headless capabilities (`
| `je_auto_control/wrapper/` | Platform-neutral API (`auto_control_mouse/keyboard/screen/image/record/window.py`); `platform_wrapper.py` picks the backend; `backend_contract.py` types the seam; `window_backends/`. |
| `je_auto_control/{windows,osx,linux_with_x11,linux_wayland}/` | Desktop OS backends; only the running OS's backend is imported. |
| `je_auto_control/{android,ios}/` | Mobile device control (adb / uiautomator2, WebDriverAgent). |
| `je_auto_control/gui/` | Optional PySide6 GUI (`[gui]` extra): `main_window.py`, tab registry `main_widget.py`, `script_builder/`, `remote_desktop/`, `language_wrapper/`. |
| `je_auto_control/gui/` | Optional PySide6 GUI (`[gui]` extra): `main_window.py` (menus, navigation dock, theme), `main_widget.py` (the tabbed workspace), the tab table `tab_registry.py` (tabs are built on first open), `navigation.py`, `theme.py`, `script_builder/`, `remote_desktop/`, `language_wrapper/`. |
| `autocontrol-lsp/` | Separate distribution: language server for `AC_*` action JSON, plus a `vscode/` client. |
| `test/` | `unit_test/headless/` (CI gate), `unit_test/flow_control/`, `integrated_test/`, `gui_test/`, `manual_test/`, `verify/`. |
| `docs/` | Sphinx docs, `API_LIFECYCLE.md`, `CAPABILITY_MATRIX.md`. |
Expand Down Expand Up @@ -108,8 +108,9 @@ wrapper/auto_control_record.record → OS listener (e.g. windows/record/win32_in
`_handlers_scheduling.py`, `_handlers_remote.py`, `_handlers_locators.py`, `_handlers_operations.py`,
`_handlers_qa.py`, `_handlers_executor_bridge.py` (a three-line delegation to an executor function), or
`_handlers.py` for data, text and the WebRunner bridge.
6. GUI: thin widget in `gui/`, registered in `gui/main_widget.py` (`_add_tab`) with commands exposed through
`menu_actions()`; strings in every `gui/language_wrapper/*.py` catalogue.
6. GUI: thin widget in `gui/`, registered by one `TabSpec` row in `gui/tab_registry.py` (module and class name, so
it is imported only when opened) with commands exposed through `menu_actions()`; strings in every
`gui/language_wrapper/*.py` catalogue.
7. Headless test in `test/unit_test/headless/`.
8. Update `architecture_explore.md` (and `README.md` + `README/` translations if a quoted count changes), then run
`python test/unit_test/headless/test_doc_line_counts.py --fix`. Regenerate the typed stub with
Expand Down
23 changes: 13 additions & 10 deletions architecture_explore.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ iOS(WebDriverAgent)。核心能力是滑鼠/鍵盤控制、影像辨識、

| 指標 | 數值 |
| --- | ---: |
| Python 模組總數(含周邊子專案) | 1,065 |
| 程式碼總行數 | 157,741 |
| Python 模組總數(含周邊子專案) | 1,068 |
| 程式碼總行數 | 158,287 |
| `je_auto_control/utils/` 子套件數 | 310 |
| `AC_*` 動作指令數(`known_commands()` 實測) | 778 |
| 套件門面 `__all__` 公開名稱數 | 1,244 |
Expand Down Expand Up @@ -880,9 +880,12 @@ GUI 是**選用 extra**(`pip install je_auto_control[gui]`,PySide6 + qt-mate

| 模組 | 行數 | 職責 |
| --- | ---: | --- |
| `gui/__init__.py` | 23 | `start_autocontrol_gui()`:**唯一**會延遲匯入 PySide6 的地方,維持頂層套件 Qt-free。 |
| `main_window.py` | 301 | `QMainWindow`:選單列(File/Actions/View/…)、可關閉分頁、即時語言切換、字級預設、qt-material 主題。分頁分為 core/editing/detection/automation/system 五類。 |
| `main_widget.py` | 437 | 擁有 `QTabWidget`,註冊 48 個分頁,並暴露 show/hide/list API 給選單列。核心分頁在註冊時直接宣告 `(label_key, handler)` 動作對;分頁本體都在下列 mixin。 |
| `gui/__init__.py` | 25 | `start_autocontrol_gui()`:**唯一**會延遲匯入 PySide6 的地方,維持頂層套件 Qt-free。 |
| `main_window.py` | 379 | `QMainWindow`:選單列(File/Actions/View/…)、左側導覽面板 dock(`Ctrl+K` 搜尋、`Ctrl+B` 收合)、即時語言切換、字級預設、深色/淺色主題(`theme.py` 的 token,不再用 qt-material)。分頁分為 core/editing/detection/automation/system 五類。 |
| `main_widget.py` | 368 | 工作區:擁有 `QTabWidget`,依 `tab_registry.TAB_SPECS` 註冊 48 個分頁,並暴露 show/hide/activate/list API 給選單列與導覽面板。只有預設開啟的三個分頁與自己的 mixin 表單在啟動時建立,其餘第一次開啟才匯入模組、建立 widget。核心分頁在 `_own_tab_builders` 宣告 `(label_key, handler)` 動作對。 |
| `tab_registry.py` | 125 | 分頁表:每個分頁一筆 `TabSpec`(鍵、標題鍵、分類、模組與類別名),`TabEntry` 在第一次存取 `widget` 時才呼叫 factory。不匯入 Qt。 |
| `navigation.py` | 200 | `NavigationPanel`:搜尋框 + 依分類的功能樹,列出每個已註冊分頁(開啟中的以粗體標示),只回報被選的鍵,開啟分頁仍由視窗負責。 |
| `theme.py` | 178 | 設計 token(`ThemeTokens`:顏色、圓角、間距、字族)、深色與淺色兩組值、由 token 產生的樣式表與對應的 `QPalette`;不載入圖檔或字型檔。 |
| `_auto_click_tab.py` | 291 | 自動點擊分頁的 mixin 建構器。 |
| `_screenshot_tab.py` | 137 | 截圖/取像素分頁 mixin。 |
| `_image_detect_tab.py` | 115 | 影像偵測分頁 mixin。 |
Expand All @@ -894,11 +897,11 @@ GUI 是**選用 extra**(`pip install je_auto_control[gui]`,PySide6 + qt-mate
| `_screen_geometry.py` | 52 | Qt 邏輯座標與截圖用的原生像素互轉:`native_region()`、`screen_at_native()`、`logical_point()`(每個螢幕的左上角在兩者相同,螢幕內依 device pixel ratio 縮放)。區域選取與主機端標註覆蓋層都用它。 |
| `_daemon_thread.py` | 79 | `DaemonThread`:`QThread` 的替代品,保留遠端桌面 worker 用到的介面(`start`/`run`/`isRunning`/`wait`/`requestInterruption`/`started`/`finished`),但 `run()` 跑在 daemon `threading.Thread` 上,刪除物件或程式結束都不會銷毀執行中的執行緒。 |
| `_worker_thread.py` | 216 | `start_worker()`:在 daemon `threading.Thread` 上執行 `QObject` worker 的 `run()`(沒有 `QThread` 可被銷毀),並經由分頁擁有的中繼物件回報結果(回呼一律在 GUI 執行緒;worker 沒處理的例外也送到 `on_fail`);worker 留在模組登錄表直到 GUI 執行緒看到它結束,回傳 `WorkerHandle`(`isRunning()`);程式結束時先呼叫 worker 的 `request_stop()`,最多等 10 秒,仍在跑的隨行程結束。 |
| `language_wrapper/` | 5,031 | 四語系字典(英/日/簡中/繁中)+ `multi_language_wrapper` 執行期切換器與監聽註冊表。 |
| `language_wrapper/` | 5,063 | 四語系字典(英/日/簡中/繁中)+ `multi_language_wrapper` 執行期切換器與監聽註冊表。 |
| `selector/` | 216 | 拖曳選取螢幕區域的半透明全螢幕覆蓋層與樣板裁切工具(互動式,但都有對應的程式化 API)。 |

> **分頁指令一律走 Actions 選單**:分頁本身只放輸入、表格與結果檢視,指令由視窗層選單暴露。
> 核心分頁在 `main_widget.py` 註冊時宣告動作;功能分頁實作 `menu_actions()`(目前 40 個檔案有此 hook)。
> 核心分頁在 `main_widget.py` 的 `_own_tab_builders` 宣告動作;功能分頁實作 `menu_actions()`(目前 40 個檔案有此 hook)。
> `test/unit_test/headless/test_actions_menu_gui.py` 會守住這個契約——沒有動作宣告的新分頁會讓 CI 失敗。

#### 48 個分頁
Expand Down Expand Up @@ -1005,7 +1008,7 @@ GUI 是**選用 extra**(`pip install je_auto_control[gui]`,PySide6 + qt-mate
| **新平台後端** | 新增 `je_auto_control/<platform>/` 實作 backend 介面,並在 `wrapper/platform_wrapper.py` 加一個分支 | 所有 wrapper 模組與上層 |
| **新 `AC_*` 指令** | 在 `utils/` 寫無頭實作 → 加進 `Executor.event_dict` → 加進 `gui/script_builder/command_schema.py` | executor 分派邏輯本身 |
| **執行期外掛指令** | `add_command_to_executor({"AC_x": fn})`,或用 `utils/plugin_loader`(掃描目錄)/`utils/plugin_sdk`(entry points) | 核心程式碼 |
| **新 GUI 分頁** | 在 `gui/` 新增 widget(只做 UI 翻譯)→ 在 `main_widget.py` `_add_tab` 註冊 → 提供 `menu_actions()` | 主視窗選單建構邏輯 |
| **新 GUI 分頁** | 在 `gui/` 新增 widget(只做 UI 翻譯)→ 在 `gui/tab_registry.py` 的 `TAB_SPECS` 加一筆 `TabSpec` → 提供 `menu_actions()` | 主視窗選單建構邏輯 |
| **新 OCR/VLM/LLM/a11y 後端** | 在對應 `backends/` 實作 base 協定 | 呼叫端 |
| **新報表格式** | 仿 `generate_report/` 既有三者的骨架新增產生器 | 執行紀錄收集 |
| **新 MCP 工具** | 在 `mcp_server/tools/_factories.py` 加工廠、`_handlers.py` 加 adapter(QA 主題加在 `_handlers_qa.py`) | 傳輸層 |
Expand Down Expand Up @@ -1076,7 +1079,7 @@ socket 預設綁 `127.0.0.1`;資源一律用 `with`。

| 層/子系統 | 檔案數 | 行數 |
| --- | ---: | ---: |
| `gui/` | 95 | 27,853 |
| `gui/` | 98 | 28,399 |
| `utils/mcp_server/` | 35 | 18,898 |
| `utils/remote_desktop/` | 56 | 13,014 |
| `utils/executor/` | 8 | 9,606 |
Expand All @@ -1097,5 +1100,5 @@ socket 預設綁 `127.0.0.1`;資源一律用 `with`。
| `autocontrol-lsp/` | 8 | 744 |
| `utils/hotkey/` | 7 | 852 |
| 其餘模組(約 286 個 `utils/` 子套件 + `android/`/`ios/`/周邊小工具) | 682 | 57,156 |
| **總計** | **1,059** | **157,676** |
| **總計** | **1,062** | **158,222** |

35 changes: 34 additions & 1 deletion docs/source/Eng/doc/new_features/v223_features_doc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,36 @@ replace stay in place: per-page browse buttons inside stacked trigger forms,
the visibility-toggled data-source browse button, and stateful auto-refresh
checkboxes.

The navigation panel
--------------------

Every registered tab is listed on the left of the window, grouped by the same
five categories, whether it is open or not; open tabs are shown in bold.
Click a feature to open it (or bring it to the front). The search box filters
the list as you type — by title, by key (``usb_devices``) or by category — and
**Return** opens the first match. ``Ctrl+K`` (**View → Search Features...**)
puts the cursor in the search box from anywhere, and ``Ctrl+B``
(**View → Navigation Panel**) hides or shows the panel.

A tab is built the first time it is opened: the window starts with the three
default tabs and the forms the main widget owns, and imports the module of
any other tab only when you open it.

The navigation panel
--------------------

Every registered tab is listed on the left of the window, grouped by the same
five categories, whether it is open or not; open tabs are shown in bold.
Click a feature to open it (or bring it to the front). The search box filters
the list as you type — by title, by key (``usb_devices``) or by category — and
**Return** opens the first match. ``Ctrl+K`` (**View → Search Features...**)
puts the cursor in the search box from anywhere, and ``Ctrl+B``
(**View → Navigation Panel**) hides or shows the panel.

A tab is built the first time it is opened: the window starts with the three
default tabs and the forms the main widget owns, and imports the module of
any other tab only when you open it.

The View menu
-------------

Expand All @@ -33,8 +63,11 @@ The View menu
default layout opens with just Record, Script Builder, and Remote Desktop;
everything else is one menu click away. Tabs are closable — closing one is
the same as unchecking it in the View menu.
* **View → Theme** switches between the dark and the light theme. Both come
from one set of design tokens in ``gui/theme.py``; the window no longer
uses ``qt-material``.
* **View → Text Size** offers auto (screen-height based) and preset font
sizes applied live.
sizes applied live, on top of the active theme.

The contract test
-----------------
Expand Down
Loading
Loading