Skip to content

feat: implement the FileAutomation 1.0 roadmap - #109

Open
JE-Chen wants to merge 40 commits into
devfrom
feat/universal-storage-layer
Open

JE-Chen wants to merge 40 commits into
devfrom
feat/universal-storage-layer

Conversation

@JE-Chen

@JE-Chen JE-Chen commented Oct 8, 2026

Copy link
Copy Markdown
Member

Implements the FileAutomation 1.0 roadmap (#107) on top of dev. Nothing was removed: every FA_* action, facade name and command-line flag of 0.0.x still works. docs/source/Eng/usage/migration.rst lists the few behaviours that changed.

What is in it

  • Universal storage layer: File / Storage over one StorageBackend contract and one URI syntax. Twelve backends: local, in-memory, S3, Azure Blob, Google Drive, OneDrive, Dropbox, SFTP, FTP/FTPS, WebDAV, SMB, fsspec. An 88-case contract suite that every backend passes; copy_between now runs on the layer.
  • Extras: the base install carries no cloud SDK and no GUI toolkit; each backend, Parquet and the GUI is an extra ([all] for everything).
  • IntegrityMonitor 2.0: baselines and verification of a tree in any backend, six kinds of change, watching, opt-in remediation.
  • Pipeline runtime: retry, timeout, cancellation, conditions, idempotency, checkpoint and resume, dry run, history, YAML/JSON definitions.
  • Scheduler v2: cron with a time zone, manual, file, event and pipeline triggers; a record of every run.
  • Events, notification router, audit schema v2: the router and the audit trail are opt-in.
  • Semantic MCP tools: fourteen tools, read-only and confined to named roots by default, with a dry run.
  • UI 2.0 on an application layer (automation_file.app), which the web UI renders too.
  • CLI: storage, integrity, pipeline and audit subcommands.
  • Docs in three languages: storage, events, integrity, audit, pipelines, scheduler, MCP, GUI, application layer, public API policy, migration, deployment, integration tests.

Problems fixed on the way

  • A move between two views of one store (two roots, two prefixes) overwrote the file and then deleted it.
  • WebDAVClient could be sent, credentials included, to another host by an absolute path or a redirect.
  • ActionACL and the MCP --allowed-actions list did not check actions nested in the arguments of another action.
  • FA_storage_verify could not fail a pipeline task; it takes strict=True now.

Decisions made here, open to change before 1.0

  • OneDrive is the eleventh remote backend of the contract; Box stays action-only.
  • The public API and deprecation policy (docs/source/Eng/usage/api_policy.rst).
  • A MINOR or MAJOR release is made by writing X.Y.0 in stable.toml and dev.toml in the pull request to main (scripts/stable_release.py).
  • tzdata is a base dependency on Windows, for time zones in the scheduler.

Checks

Locally, on Windows with Python 3.14: 5763 passed and 257 skipped with every extra, 3879 passed and 137 skipped with the base dependencies only, ruff check, ruff format --check and mypy automation_file clean.

Not verified: please read

  • .github/workflows/integration.yml and tests/integration/ have never run. There is no Docker on the development machine, so this pull request is their first run. The container options, the Samba and FTP ones most of all, may need adjusting. These jobs do not gate publishing.
  • No storage adapter has met a real service; all were tested against stand-ins.
  • The unit tests have not run on Linux or macOS, nor on Python 3.10 to 3.13.
  • UI 2.0 has only run on Qt's offscreen platform, never on a real display.
  • The Sphinx build of the new pages was not run.

progress.md lists what remains open; docs/updates/2026-10.md has one entry per piece of work (U-20261008-01 to -32).

JE-Chen added 30 commits October 8, 2026 11:45
SFTP, FTP/FTPS, Google Drive, OneDrive and Dropbox resolve by URI through the shared clients; WebDAV, SMB and fsspec are mounted.

A copy or a move of a file onto itself through two views of one store is refused (file identity), the WebDAV client only talks to its own server and no longer leaves redirects to requests, and the SFTP, OneDrive and SMB clients name the extra to install when their SDK is missing.
The integrity package compares a tree at any storage URI with an approved baseline: snapshot, verify, watch and continuous modes, six kinds of change, a versioned manifest, one IntegrityViolation event per pass that finds drift, and opt-in remediation. The first monitor's call, summary and notification are kept.
Routes deliver events to named sinks with deduplication and a rate limit per route, and the audit trail records one row per event and per storage operation in a searchable store. Both are opt-in. While the router is active, notify_on_failure and the integrity monitor leave the direct notification to it.
Pipelines run tasks in dependency order with retry, timeout, cancellation, conditions, idempotency, checkpoint and resume, a dry run and an execution history, from Python or from a versioned YAML or JSON definition.

FA_storage_verify can raise on a mismatch (strict), so a verification can fail its task. The action ACL and the MCP server now check action names nested in the arguments of another action.
Says what is public, the stability levels, what a version number promises and how a name is retired, and adds the helper every deprecation goes through.
The storage contract suite runs against MinIO, Azurite, OpenSSH, FTP, WebDAV and Samba in containers, and the unit tests on Linux and macOS. Nothing here has run yet: there is no Docker on the development machine, so the first workflow run is the test.
Cron with a time zone, manual, file, event and pipeline triggers; action-list and pipeline targets; a record of every run with seven states; overlap protection, timeouts and cancellation.
One plain-Python service per navigation entry, a sidebar GUI built on them with a pipeline editor, and a web UI that renders the same services. The older tabs stay under Advanced.
@codacy-production

codacy-production Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 9000 complexity · 81 duplication

Metric Results
Complexity 9000
Duplication 81

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

JE-Chen added 10 commits October 8, 2026 16:04
MinIO from quay.io, a writable FTP root, Samba on port 445 and a replace that removes the target first, a 400 from Apache for a path below a file, a versioning name too long to ask about on macOS, the package job on the locked tools, and equality tests written so that a scanner does not read them as self-comparisons.
File-level pylint disables in the test modules for the rules pytest idioms trip, each with its reason; markers with reasons where a scanner reads a test digest or a constant SQL statement as a risk; nine long lines wrapped; one variable renamed. The S3 integration job uses S3Mock, since MinIO's image can no longer be pulled.
@sonarqubecloud

sonarqubecloud Bot commented Oct 8, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant