Merge branch 'main' into refactor/migrate-react-window-to-tanstack-virtual

# Conflicts:
#	web/pnpm-lock.yaml

.agents/skills/backend-code-review/SKILL.md

@@ -1,168 +0,0 @@
----
-name: backend-code-review
-description: Review backend code for quality, security, maintainability, and best practices based on established checklist rules. Use when the user requests a review, analysis, or improvement of backend files (e.g., `.py`) under the `api/` directory. Do NOT use for frontend files (e.g., `.tsx`, `.ts`, `.js`). Supports pending-change review, code snippets review, and file-focused review.
----
-
-# Backend Code Review
-
-## When to use this skill
-
-Use this skill whenever the user asks to **review, analyze, or improve** backend code (e.g., `.py`) under the `api/` directory. Supports the following review modes:
-
-- **Pending-change review**: when the user asks to review current changes (inspect staged/working-tree files slated for commit to get the changes).
-- **Code snippets review**: when the user pastes code snippets (e.g., a function/class/module excerpt) into the chat and asks for a review.
-- **File-focused review**: when the user points to specific files and asks for a review of those files (one file or a small, explicit set of files, e.g., `api/...`, `api/app.py`).
-
-Do NOT use this skill when:
-
-- The request is about frontend code or UI (e.g., `.tsx`, `.ts`, `.js`, `web/`).
-- The user is not asking for a review/analysis/improvement of backend code.
-- The scope is not under `api/` (unless the user explicitly asks to review backend-related changes outside `api/`).
-
-## How to use this skill
-
-Follow these steps when using this skill:
-
-1. **Identify the review mode** (pending-change vs snippet vs file-focused) based on the user’s input. Keep the scope tight: review only what the user provided or explicitly referenced.
-2. Follow the rules defined in **Checklist** to perform the review. If no Checklist rule matches, apply **General Review Rules** as a fallback to perform the best-effort review.
-3. Compose the final output strictly follow the **Required Output Format**.
-
-Notes when using this skill:
-- Always include actionable fixes or suggestions (including possible code snippets).
-- Use best-effort `File:Line` references when a file path and line numbers are available; otherwise, use the most specific identifier you can.
-
-## Checklist
-
-- db schema design: if the review scope includes code/files under `api/models/` or `api/migrations/`, follow [references/db-schema-rule.md](references/db-schema-rule.md) to perform the review
-- architecture: if the review scope involves controller/service/core-domain/libs/model layering, dependency direction, or moving responsibilities across modules, follow [references/architecture-rule.md](references/architecture-rule.md) to perform the review
-- repositories abstraction: if the review scope contains table/model operations (e.g., `select(...)`, `session.execute(...)`, joins, CRUD) and is not under `api/repositories`, `api/core/repositories`, or `api/extensions/*/repositories/`, follow [references/repositories-rule.md](references/repositories-rule.md) to perform the review
-- sqlalchemy patterns: if the review scope involves SQLAlchemy session/query usage, db transaction/crud usage, or raw SQL usage, follow [references/sqlalchemy-rule.md](references/sqlalchemy-rule.md) to perform the review
-
-## General Review Rules
-
-### 1. Security Review
-
-Check for:
-- SQL injection vulnerabilities
-- Server-Side Request Forgery (SSRF)
-- Command injection
-- Insecure deserialization
-- Hardcoded secrets/credentials
-- Improper authentication/authorization
-- Insecure direct object references
-
-### 2. Performance Review
-
-Check for:
-- N+1 queries
-- Missing database indexes
-- Memory leaks
-- Blocking operations in async code
-- Missing caching opportunities
-
-### 3. Code Quality Review
-
-Check for:
-- Code forward compatibility
-- Code duplication (DRY violations)
-- Functions doing too much (SRP violations)
-- Deep nesting / complex conditionals
-- Magic numbers/strings
-- Poor naming
-- Missing error handling
-- Incomplete type coverage
-
-### 4. Testing Review
-
-Check for:
-- Missing test coverage for new code
-- Tests that don't test behavior
-- Flaky test patterns
-- Missing edge cases
-
-## Required Output Format
-
-When this skill invoked, the response must exactly follow one of the two templates:
-
-### Template A (any findings)
-
-```markdown
-# Code Review Summary
-
-Found <X> critical issues need to be fixed:
-
-## 🔴 Critical (Must Fix)
-
-### 1. <brief description of the issue>
-
-FilePath: <path> line <line>
-<relevant code snippet or pointer>
-
-#### Explanation
-
-<detailed explanation and references of the issue>
-
-#### Suggested Fix
-
-1. <brief description of suggested fix>
-2. <code example> (optional, omit if not applicable)
-
----
-... (repeat for each critical issue) ...
-
-Found <Y> suggestions for improvement:
-
-## 🟡 Suggestions (Should Consider)
-
-### 1. <brief description of the suggestion>
-
-FilePath: <path> line <line>
-<relevant code snippet or pointer>
-
-#### Explanation
-
-<detailed explanation and references of the suggestion>
-
-#### Suggested Fix
-
-1. <brief description of suggested fix>
-2. <code example> (optional, omit if not applicable)
-
----
-... (repeat for each suggestion) ...
-
-Found <Z> optional nits:
-
-## 🟢 Nits (Optional)
-### 1. <brief description of the nit>
-
-FilePath: <path> line <line>
-<relevant code snippet or pointer>
-
-#### Explanation
-
-<explanation and references of the optional nit>
-
-#### Suggested Fix
-
-- <minor suggestions>
-
----
-... (repeat for each nits) ...
-
-## ✅ What's Good
-
-- <Positive feedback on good patterns>
-```
-
-- If there are no critical issues or suggestions or option nits or good points, just omit that section.
-- If the issue number is more than 10, summarize as "Found 10+ critical issues/suggestions/optional nits" and only output the first 10 items.
-- Don't compress the blank lines between sections; keep them as-is for readability.
-- If there is any issue requires code changes, append a brief follow-up question to ask whether the user wants to apply the fix(es) after the structured output. For example: "Would you like me to use the Suggested fix(es) to address these issues?"
-
-### Template B (no issues)
-
-```markdown
-## Code Review Summary
-✅ No issues found.
-```

.agents/skills/backend-code-review/references/architecture-rule.md

@@ -1,91 +0,0 @@
-# Rule Catalog — Architecture
-
-## Scope
-- Covers: controller/service/core-domain/libs/model layering, dependency direction, responsibility placement, observability-friendly flow.
-
-## Rules
-
-### Keep business logic out of controllers
-- Category: maintainability
-- Severity: critical
-- Description: Controllers should parse input, call services, and return serialized responses. Business decisions inside controllers make behavior hard to reuse and test.
-- Suggested fix: Move domain/business logic into the service or core/domain layer. Keep controller handlers thin and orchestration-focused.
-- Example:
-  - Bad:
-    ```python
-    @bp.post("/apps/<app_id>/publish")
-    def publish_app(app_id: str):
-        payload = request.get_json() or {}
-        if payload.get("force") and current_user.role != "admin":
-            raise ValueError("only admin can force publish")
-        app = App.query.get(app_id)
-        app.status = "published"
-        db.session.commit()
-        return {"result": "ok"}
-    ```
-  - Good:
-    ```python
-    @bp.post("/apps/<app_id>/publish")
-    def publish_app(app_id: str):
-        payload = PublishRequest.model_validate(request.get_json() or {})
-        app_service.publish_app(app_id=app_id, force=payload.force, actor_id=current_user.id)
-        return {"result": "ok"}
-    ```
-
-### Preserve layer dependency direction
-- Category: best practices
-- Severity: critical
-- Description: Controllers may depend on services, and services may depend on core/domain abstractions. Reversing this direction (for example, core importing controller/web modules) creates cycles and leaks transport concerns into domain code.
-- Suggested fix: Extract shared contracts into core/domain or service-level modules and make upper layers depend on lower, not the reverse.
-- Example:
-  - Bad:
-    ```python
-    # core/policy/publish_policy.py
-    from controllers.console.app import request_context
-
-    def can_publish() -> bool:
-        return request_context.current_user.is_admin
-    ```
-  - Good:
-    ```python
-    # core/policy/publish_policy.py
-    def can_publish(role: str) -> bool:
-        return role == "admin"
-
-    # service layer adapts web/user context to domain input
-    allowed = can_publish(role=current_user.role)
-    ```
-
-### Keep libs business-agnostic
-- Category: maintainability
-- Severity: critical
-- Description: Modules under `api/libs/` should remain reusable, business-agnostic building blocks. They must not encode product/domain-specific rules, workflow orchestration, or business decisions.
-- Suggested fix:
-  - If business logic appears in `api/libs/`, extract it into the appropriate `services/` or `core/` module and keep `libs` focused on generic, cross-cutting helpers.
-  - Keep `libs` dependencies clean: avoid importing service/controller/domain-specific modules into `api/libs/`.
-- Example:
-  - Bad:
-    ```python
-    # api/libs/conversation_filter.py
-    from services.conversation_service import ConversationService
-
-    def should_archive_conversation(conversation, tenant_id: str) -> bool:
-        # Domain policy and service dependency are leaking into libs.
-        service = ConversationService()
-        if service.has_paid_plan(tenant_id):
-            return conversation.idle_days > 90
-        return conversation.idle_days > 30
-    ```
-  - Good:
-    ```python
-    # api/libs/datetime_utils.py (business-agnostic helper)
-    def older_than_days(idle_days: int, threshold_days: int) -> bool:
-        return idle_days > threshold_days
-
-    # services/conversation_service.py (business logic stays in service/core)
-    from libs.datetime_utils import older_than_days
-
-    def should_archive_conversation(conversation, tenant_id: str) -> bool:
-        threshold_days = 90 if has_paid_plan(tenant_id) else 30
-        return older_than_days(conversation.idle_days, threshold_days)
-    ```

.agents/skills/backend-code-review/references/db-schema-rule.md

@@ -1,157 +0,0 @@
-# Rule Catalog — DB Schema Design
-
-## Scope
-- Covers: model/base inheritance, schema boundaries in model properties, tenant-aware schema design, index redundancy checks, dialect portability in models, and cross-database compatibility in migrations.
-- Does NOT cover: session lifecycle, transaction boundaries, and query execution patterns (handled by `sqlalchemy-rule.md`).
-
-## Rules
-
-### Do not query other tables inside `@property`
-- Category: [maintainability, performance]
-- Severity: critical
-- Description: A model `@property` must not open sessions or query other tables. This hides dependencies across models, tightly couples schema objects to data access, and can cause N+1 query explosions when iterating collections.
-- Suggested fix:
-  - Keep model properties pure and local to already-loaded fields.
-  - Move cross-table data fetching to service/repository methods.
-  - For list/batch reads, fetch required related data explicitly (join/preload/bulk query) before rendering derived values.
-- Example:
-  - Bad:
-    ```python
-    class Conversation(TypeBase):
-        __tablename__ = "conversations"
-
-        @property
-        def app_name(self) -> str:
-            with Session(db.engine, expire_on_commit=False) as session:
-                app = session.execute(select(App).where(App.id == self.app_id)).scalar_one()
-                return app.name
-    ```
-  - Good:
-    ```python
-    class Conversation(TypeBase):
-        __tablename__ = "conversations"
-
-        @property
-        def display_title(self) -> str:
-            return self.name or "Untitled"
-
-
-    # Service/repository layer performs explicit batch fetch for related App rows.
-    ```
-
-### Prefer including `tenant_id` in model definitions
-- Category: maintainability
-- Severity: suggestion
-- Description: In multi-tenant domains, include `tenant_id` in schema definitions whenever the entity belongs to tenant-owned data. This improves data isolation safety and keeps future partitioning/sharding strategies practical as data volume grows.
-- Suggested fix:
-  - Add a `tenant_id` column and ensure related unique/index constraints include tenant dimension when applicable.
-  - Propagate `tenant_id` through service/repository contracts to keep access paths tenant-aware.
-  - Exception: if a table is explicitly designed as non-tenant-scoped global metadata, document that design decision clearly.
-- Example:
-  - Bad:
-    ```python
-    from sqlalchemy.orm import Mapped
-
-    class Dataset(TypeBase):
-        __tablename__ = "datasets"
-        id: Mapped[str] = mapped_column(StringUUID, primary_key=True)
-        name: Mapped[str] = mapped_column(sa.String(255), nullable=False)
-    ```
-  - Good:
-    ```python
-    from sqlalchemy.orm import Mapped
-
-    class Dataset(TypeBase):
-        __tablename__ = "datasets"
-        id: Mapped[str] = mapped_column(StringUUID, primary_key=True)
-        tenant_id: Mapped[str] = mapped_column(StringUUID, nullable=False, index=True)
-        name: Mapped[str] = mapped_column(sa.String(255), nullable=False)
-    ```
-
-### Detect and avoid duplicate/redundant indexes
-- Category: performance
-- Severity: suggestion
-- Description: Review index definitions for leftmost-prefix redundancy. For example, index `(a, b, c)` can safely cover most lookups for `(a, b)`. Keeping both may increase write overhead and can mislead the optimizer into suboptimal execution plans.
-- Suggested fix:
-  - Before adding an index, compare against existing composite indexes by leftmost-prefix rules.
-  - Drop or avoid creating redundant prefixes unless there is a proven query-pattern need.
-  - Apply the same review standard in both model `__table_args__` and migration index DDL.
-- Example:
-  - Bad:
-    ```python
-    __table_args__ = (
-        sa.Index("idx_msg_tenant_app", "tenant_id", "app_id"),
-        sa.Index("idx_msg_tenant_app_created", "tenant_id", "app_id", "created_at"),
-    )
-    ```
-  - Good:
-    ```python
-    __table_args__ = (
-        # Keep the wider index unless profiling proves a dedicated short index is needed.
-        sa.Index("idx_msg_tenant_app_created", "tenant_id", "app_id", "created_at"),
-    )
-    ```
-
-### Avoid PostgreSQL-only dialect usage in models; wrap in `models.types`
-- Category: maintainability
-- Severity: critical
-- Description: Model/schema definitions should avoid PostgreSQL-only constructs directly in business models. When database-specific behavior is required, encapsulate it in `api/models/types.py` using both PostgreSQL and MySQL dialect implementations, then consume that abstraction from model code.
-- Suggested fix:
-  - Do not directly place dialect-only types/operators in model columns when a portable wrapper can be used.
-  - Add or extend wrappers in `models.types` (for example, `AdjustedJSON`, `LongText`, `BinaryData`) to normalize behavior across PostgreSQL and MySQL.
-- Example:
-  - Bad:
-    ```python
-    from sqlalchemy.dialects.postgresql import JSONB
-    from sqlalchemy.orm import Mapped
-
-    class ToolConfig(TypeBase):
-        __tablename__ = "tool_configs"
-        config: Mapped[dict] = mapped_column(JSONB, nullable=False)
-    ```
-  - Good:
-    ```python
-    from sqlalchemy.orm import Mapped
-
-    from models.types import AdjustedJSON
-
-    class ToolConfig(TypeBase):
-        __tablename__ = "tool_configs"
-        config: Mapped[dict] = mapped_column(AdjustedJSON(), nullable=False)
-    ```
-
-### Guard migration incompatibilities with dialect checks and shared types
-- Category: maintainability
-- Severity: critical
-- Description: Migration scripts under `api/migrations/versions/` must account for PostgreSQL/MySQL incompatibilities explicitly. For dialect-sensitive DDL or defaults, branch on the active dialect (for example, `conn.dialect.name == "postgresql"`), and prefer reusable compatibility abstractions from `models.types` where applicable.
-- Suggested fix:
-  - In migration upgrades/downgrades, bind connection and branch by dialect for incompatible SQL fragments.
-  - Reuse `models.types` wrappers in column definitions when that keeps behavior aligned with runtime models.
-  - Avoid one-dialect-only migration logic unless there is a documented, deliberate compatibility exception.
-- Example:
-  - Bad:
-    ```python
-    with op.batch_alter_table("dataset_keyword_tables") as batch_op:
-        batch_op.add_column(
-            sa.Column(
-                "data_source_type",
-                sa.String(255),
-                server_default=sa.text("'database'::character varying"),
-                nullable=False,
-            )
-        )
-    ```
-  - Good:
-    ```python
-    def _is_pg(conn) -> bool:
-        return conn.dialect.name == "postgresql"
-
-
-    conn = op.get_bind()
-    default_expr = sa.text("'database'::character varying") if _is_pg(conn) else sa.text("'database'")
-
-    with op.batch_alter_table("dataset_keyword_tables") as batch_op:
-        batch_op.add_column(
-            sa.Column("data_source_type", sa.String(255), server_default=default_expr, nullable=False)
-        )
-    ```

.agents/skills/backend-code-review/references/repositories-rule.md

@@ -1,61 +0,0 @@
-# Rule Catalog - Repositories Abstraction
-
-## Scope
-- Covers: when to reuse existing repository abstractions, when to introduce new repositories, and how to preserve dependency direction between service/core and infrastructure implementations.
-- Does NOT cover: SQLAlchemy session lifecycle and query-shape specifics (handled by `sqlalchemy-rule.md`), and table schema/migration design (handled by `db-schema-rule.md`).
-
-## Rules
-
-### Introduce repositories abstraction
-- Category: maintainability
-- Severity: suggestion
-- Description: If a table/model already has a repository abstraction, all reads/writes/queries for that table should use the existing repository. If no repository exists, introduce one only when complexity justifies it, such as large/high-volume tables, repeated complex query logic, or likely storage-strategy variation.
-- Suggested fix:
-  - First check  `api/repositories`, `api/core/repositories`, and `api/extensions/*/repositories/` to verify whether the table/model already has a repository abstraction. If it exists, route all operations through it and add missing repository methods instead of bypassing it with ad-hoc SQLAlchemy access.
-  - If no repository exists, add one only when complexity warrants it (for example, repeated complex queries, large data domains, or multiple storage strategies), while preserving dependency direction (service/core depends on abstraction; infra provides implementation).
-- Example:
-  - Bad:
-    ```python
-    # Existing repository is ignored and service uses ad-hoc table queries.
-    class AppService:
-        def archive_app(self, app_id: str, tenant_id: str) -> None:
-            app = self.session.execute(
-                select(App).where(App.id == app_id, App.tenant_id == tenant_id)
-            ).scalar_one()
-            app.archived = True
-            self.session.commit()
-    ```
-  - Good:
-    ```python
-    # Case A: Existing repository must be reused for all table operations.
-    class AppService:
-        def archive_app(self, app_id: str, tenant_id: str) -> None:
-            app = self.app_repo.get_by_id(app_id=app_id, tenant_id=tenant_id)
-            app.archived = True
-            self.app_repo.save(app)
-
-    # If the query is missing, extend the existing abstraction.
-    active_apps = self.app_repo.list_active_for_tenant(tenant_id=tenant_id)
-    ```
-  - Bad:
-    ```python
-    # No repository exists, but large-domain query logic is scattered in service code.
-    class ConversationService:
-        def list_recent_for_app(self, app_id: str, tenant_id: str, limit: int) -> list[Conversation]:
-            ...
-            # many filters/joins/pagination variants duplicated across services
-    ```
-  - Good:
-    ```python
-    # Case B: Introduce repository for large/complex domains or storage variation.
-    class ConversationRepository(Protocol):
-        def list_recent_for_app(self, app_id: str, tenant_id: str, limit: int) -> list[Conversation]: ...
-
-    class SqlAlchemyConversationRepository:
-        def list_recent_for_app(self, app_id: str, tenant_id: str, limit: int) -> list[Conversation]:
-            ...
-
-    class ConversationService:
-        def __init__(self, conversation_repo: ConversationRepository):
-            self.conversation_repo = conversation_repo
-    ```

.agents/skills/backend-code-review/references/sqlalchemy-rule.md

@@ -1,139 +0,0 @@
-# Rule Catalog — SQLAlchemy Patterns
-
-## Scope
-- Covers: SQLAlchemy session and transaction lifecycle, query construction, tenant scoping, raw SQL boundaries, and write-path concurrency safeguards.
-- Does NOT cover: table/model schema and migration design details (handled by `db-schema-rule.md`).
-
-## Rules
-
-### Use Session context manager with explicit transaction control behavior
-- Category: best practices
-- Severity: critical
-- Description: Session and transaction lifecycle must be explicit and bounded on write paths. Missing commits can silently drop intended updates, while ad-hoc or long-lived transactions increase contention, lock duration, and deadlock risk.
-- Suggested fix:
-  - Use **explicit `session.commit()`** after completing a related write unit.
-  - Or use **`session.begin()` context manager** for automatic commit/rollback on a scoped block.
-  - Keep transaction windows short: avoid network I/O, heavy computation, or unrelated work inside the transaction.
-- Example:
-  - Bad:
-    ```python
-    # Missing commit: write may never be persisted.
-    with Session(db.engine, expire_on_commit=False) as session:
-        run = session.get(WorkflowRun, run_id)
-        run.status = "cancelled"
-
-    # Long transaction: external I/O inside a DB transaction.
-    with Session(db.engine, expire_on_commit=False) as session, session.begin():
-        run = session.get(WorkflowRun, run_id)
-        run.status = "cancelled"
-        call_external_api()
-    ```
-  - Good:
-    ```python
-    # Option 1: explicit commit.
-    with Session(db.engine, expire_on_commit=False) as session:
-        run = session.get(WorkflowRun, run_id)
-        run.status = "cancelled"
-        session.commit()
-
-    # Option 2: scoped transaction with automatic commit/rollback.
-    with Session(db.engine, expire_on_commit=False) as session, session.begin():
-        run = session.get(WorkflowRun, run_id)
-        run.status = "cancelled"
-
-    # Keep non-DB work outside transaction scope.
-    call_external_api()
-    ```
-
-### Enforce tenant_id scoping on shared-resource queries
-- Category: security
-- Severity: critical
-- Description: Reads and writes against shared tables must be scoped by `tenant_id` to prevent cross-tenant data leakage or corruption.
-- Suggested fix: Add `tenant_id` predicate to all tenant-owned entity queries and propagate tenant context through service/repository interfaces.
-- Example:
-  - Bad:
-    ```python
-    stmt = select(Workflow).where(Workflow.id == workflow_id)
-    workflow = session.execute(stmt).scalar_one_or_none()
-    ```
-  - Good:
-    ```python
-    stmt = select(Workflow).where(
-        Workflow.id == workflow_id,
-        Workflow.tenant_id == tenant_id,
-    )
-    workflow = session.execute(stmt).scalar_one_or_none()
-    ```
-
-### Prefer SQLAlchemy expressions over raw SQL by default
-- Category: maintainability
-- Severity: suggestion
-- Description: Raw SQL should be exceptional. ORM/Core expressions are easier to evolve, safer to compose, and more consistent with the codebase.
-- Suggested fix: Rewrite straightforward raw SQL into SQLAlchemy `select/update/delete` expressions; keep raw SQL only when required by clear technical constraints.
-- Example:
-  - Bad:
-    ```python
-    row = session.execute(
-        text("SELECT * FROM workflows WHERE id = :id AND tenant_id = :tenant_id"),
-        {"id": workflow_id, "tenant_id": tenant_id},
-    ).first()
-    ```
-  - Good:
-    ```python
-    stmt = select(Workflow).where(
-        Workflow.id == workflow_id,
-        Workflow.tenant_id == tenant_id,
-    )
-    row = session.execute(stmt).scalar_one_or_none()
-    ```
-
-### Protect write paths with concurrency safeguards
-- Category: quality
-- Severity: critical
-- Description: Multi-writer paths without explicit concurrency control can silently overwrite data. Choose the safeguard based on contention level, lock scope, and throughput cost instead of defaulting to one strategy.
-- Suggested fix:
-  - **Optimistic locking**: Use when contention is usually low and retries are acceptable. Add a version (or updated_at) guard in `WHERE` and treat `rowcount == 0` as a conflict.
-  - **Redis distributed lock**: Use when the critical section spans multiple steps/processes (or includes non-DB side effects) and you need cross-worker mutual exclusion.
-  - **SELECT ... FOR UPDATE**: Use when contention is high on the same rows and strict in-transaction serialization is required. Keep transactions short to reduce lock wait/deadlock risk.
-  - In all cases, scope by `tenant_id` and verify affected row counts for conditional writes.
-- Example:
-  - Bad:
-    ```python
-    # No tenant scope, no conflict detection, and no lock on a contested write path.
-    session.execute(update(WorkflowRun).where(WorkflowRun.id == run_id).values(status="cancelled"))
-    session.commit()  # silently overwrites concurrent updates
-    ```
-  - Good:
-    ```python
-    # 1) Optimistic lock (low contention, retry on conflict)
-    result = session.execute(
-        update(WorkflowRun)
-        .where(
-            WorkflowRun.id == run_id,
-            WorkflowRun.tenant_id == tenant_id,
-            WorkflowRun.version == expected_version,
-        )
-        .values(status="cancelled", version=WorkflowRun.version + 1)
-    )
-    if result.rowcount == 0:
-        raise WorkflowStateConflictError("stale version, retry")
-
-    # 2) Redis distributed lock (cross-worker critical section)
-    lock_name = f"workflow_run_lock:{tenant_id}:{run_id}"
-    with redis_client.lock(lock_name, timeout=20):
-        session.execute(
-            update(WorkflowRun)
-            .where(WorkflowRun.id == run_id, WorkflowRun.tenant_id == tenant_id)
-            .values(status="cancelled")
-        )
-        session.commit()
-
-    # 3) Pessimistic lock with SELECT ... FOR UPDATE (high contention)
-    run = session.execute(
-        select(WorkflowRun)
-        .where(WorkflowRun.id == run_id, WorkflowRun.tenant_id == tenant_id)
-        .with_for_update()
-    ).scalar_one()
-    run.status = "cancelled"
-    session.commit()
-    ```

.claude/skills/backend-code-review

@@ -1 +0,0 @@
-../../.agents/skills/backend-code-review

.codex/skills/component-refactoring

@@ -0,0 +1 @@
+../../.agents/skills/component-refactoring

.codex/skills/frontend-code-review

@@ -0,0 +1 @@
+../../.agents/skills/frontend-code-review

.codex/skills/frontend-testing

@@ -0,0 +1 @@
+../../.agents/skills/frontend-testing

.codex/skills/orpc-contract-first

@@ -0,0 +1 @@
+../../.agents/skills/orpc-contract-first

.github/CODEOWNERS

@@ -9,9 +9,6 @@
 # CODEOWNERS file
 /.github/CODEOWNERS @laipz8200 @crazywoola
 
-# Agents
-/.agents/skills/ @hyoban
-
 # Docs
 /docs/ @crazywoola
 
@@ -24,10 +21,6 @@
 /api/services/tools/mcp_tools_manage_service.py @Nov1c444
 /api/controllers/mcp/ @Nov1c444
 /api/controllers/console/app/mcp_server.py @Nov1c444
-
-# Backend - Tests
-/api/tests/ @laipz8200 @QuantumGhost
-
 /api/tests/**/*mcp* @Nov1c444
 
 # Backend - Workflow - Engine (Core graph execution engine)
@@ -238,9 +231,6 @@
 # Frontend - Base Components
 /web/app/components/base/ @iamjoel @zxhlyh
 
-# Frontend - Base Components Tests
-/web/app/components/base/**/*.spec.tsx @hyoban @CodingOnStar
-
 # Frontend - Utils and Hooks
 /web/utils/classnames.ts @iamjoel @zxhlyh
 /web/utils/time.ts @iamjoel @zxhlyh

.github/dependabot.yml

@@ -1,25 +1,12 @@
 version: 2
-
-multi-ecosystem-groups:
-  python:
-    schedule:
-      interval: "weekly"  # or whatever schedule you want
-
 updates:
-  - package-ecosystem: "pip"
-    directory: "/api"
-    open-pull-requests-limit: 2
-    patterns: ["*"]
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "uv"
-    directory: "/api"
-    open-pull-requests-limit: 2
-    patterns: ["*"]
-    schedule:
-      interval: "weekly"
   - package-ecosystem: "npm"
     directory: "/web"
     schedule:
       interval: "weekly"
     open-pull-requests-limit: 2
+  - package-ecosystem: "uv"
+    directory: "/api"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 2

.github/workflows/autofix.yml

@@ -79,6 +79,29 @@ jobs:
           find . -name "*.py" -type f -exec sed -i.bak -E 's/"([^"]+)" \| None/Optional["\1"]/g; s/'"'"'([^'"'"']+)'"'"' \| None/Optional['"'"'\1'"'"']/g' {} \;
           find . -name "*.py.bak" -type f -delete
 
+      - name: Install pnpm
+        uses: pnpm/action-setup@v4
+        with:
+          package_json_file: web/package.json
+          run_install: false
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v6
+        with:
+          node-version: 24
+          cache: pnpm
+          cache-dependency-path: ./web/pnpm-lock.yaml
+
+      - name: Install web dependencies
+        run: |
+          cd web
+          pnpm install --frozen-lockfile
+
+      - name: ESLint autofix
+        run: |
+          cd web
+          pnpm lint:fix || true
+
       # mdformat breaks YAML front matter in markdown files. Add --exclude for directories containing YAML front matter.
       - name: mdformat
         run: |

.github/workflows/build-push.yml

@@ -8,7 +8,6 @@ on:
       - "build/**"
       - "release/e-*"
       - "hotfix/**"
-      - "feat/hitl-backend"
     tags:
       - "*"
 

.github/workflows/deploy-hitl.yml

@@ -4,7 +4,8 @@ on:
   workflow_run:
     workflows: ["Build and Push API & Web"]
     branches:
-      - "build/feat/hitl"
+      - "feat/hitl-frontend"
+      - "feat/hitl-backend"
     types:
       - completed
 
@@ -13,7 +14,10 @@ jobs:
     runs-on: ubuntu-latest
     if: |
       github.event.workflow_run.conclusion == 'success' &&
-      github.event.workflow_run.head_branch == 'build/feat/hitl'
+      (
+        github.event.workflow_run.head_branch == 'feat/hitl-frontend' ||
+        github.event.workflow_run.head_branch == 'feat/hitl-backend'
+      )
     steps:
       - name: Deploy to server
         uses: appleboy/ssh-action@v1

.github/workflows/pyrefly-diff-comment.yml

@@ -1,88 +0,0 @@
-name: Comment with Pyrefly Diff
-
-on:
-  workflow_run:
-    workflows:
-      - Pyrefly Diff Check
-    types:
-      - completed
-
-permissions: {}
-
-jobs:
-  comment:
-    name: Comment PR with pyrefly diff
-    runs-on: ubuntu-latest
-    permissions:
-      actions: read
-      contents: read
-      issues: write
-      pull-requests: write
-    if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.pull_requests[0].head.repo.full_name != github.repository }}
-    steps:
-      - name: Download pyrefly diff artifact
-        uses: actions/github-script@v8
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          script: |
-            const fs = require('fs');
-            const artifacts = await github.rest.actions.listWorkflowRunArtifacts({
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              run_id: ${{ github.event.workflow_run.id }},
-            });
-            const match = artifacts.data.artifacts.find((artifact) =>
-              artifact.name === 'pyrefly_diff'
-            );
-            if (!match) {
-              throw new Error('pyrefly_diff artifact not found');
-            }
-            const download = await github.rest.actions.downloadArtifact({
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              artifact_id: match.id,
-              archive_format: 'zip',
-            });
-            fs.writeFileSync('pyrefly_diff.zip', Buffer.from(download.data));
-
-      - name: Unzip artifact
-        run: unzip -o pyrefly_diff.zip
-
-      - name: Post comment
-        uses: actions/github-script@v8
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          script: |
-            const fs = require('fs');
-            let diff = fs.readFileSync('pyrefly_diff.txt', { encoding: 'utf8' });
-            let prNumber = null;
-            try {
-              prNumber = parseInt(fs.readFileSync('pr_number.txt', { encoding: 'utf8' }), 10);
-            } catch (err) {
-              // Fallback to workflow_run payload if artifact is missing or incomplete.
-              const prs = context.payload.workflow_run.pull_requests || [];
-              if (prs.length > 0 && prs[0].number) {
-                prNumber = prs[0].number;
-              }
-            }
-            if (!prNumber) {
-              throw new Error('PR number not found in artifact or workflow_run payload');
-            }
-
-            const MAX_CHARS = 65000;
-            if (diff.length > MAX_CHARS) {
-              diff = diff.slice(0, MAX_CHARS);
-              diff = diff.slice(0, diff.lastIndexOf('\\n'));
-              diff += '\\n\\n... (truncated) ...';
-            }
-
-            const body = diff.trim()
-              ? '### Pyrefly Diff\n<details>\n<summary>base → PR</summary>\n\n```diff\n' + diff + '\n```\n</details>'
-              : '### Pyrefly Diff\nNo changes detected.';
-
-            await github.rest.issues.createComment({
-              issue_number: prNumber,
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              body,
-            });

.github/workflows/pyrefly-diff.yml

@@ -1,94 +0,0 @@
-name: Pyrefly Diff Check
-
-on:
-  pull_request:
-    paths:
-      - 'api/**/*.py'
-
-permissions:
-  contents: read
-
-jobs:
-  pyrefly-diff:
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      issues: write
-      pull-requests: write
-    steps:
-      - name: Checkout PR branch
-        uses: actions/checkout@v6
-        with:
-          fetch-depth: 0
-
-      - name: Setup Python & UV
-        uses: astral-sh/setup-uv@v5
-        with:
-          enable-cache: true
-
-      - name: Install dependencies
-        run: uv sync --project api --dev
-
-      - name: Run pyrefly on PR branch
-        run: |
-          uv run --directory api pyrefly check > /tmp/pyrefly_pr.txt 2>&1 || true
-
-      - name: Checkout base branch
-        run: git checkout ${{ github.base_ref }}
-
-      - name: Run pyrefly on base branch
-        run: |
-          uv run --directory api pyrefly check > /tmp/pyrefly_base.txt 2>&1 || true
-
-      - name: Compute diff
-        run: |
-          diff /tmp/pyrefly_base.txt /tmp/pyrefly_pr.txt > pyrefly_diff.txt || true
-
-      - name: Save PR number
-        run: |
-          echo ${{ github.event.pull_request.number }} > pr_number.txt
-
-      - name: Upload pyrefly diff
-        uses: actions/upload-artifact@v4
-        with:
-          name: pyrefly_diff
-          path: |
-            pyrefly_diff.txt
-            pr_number.txt
-
-      - name: Comment PR with pyrefly diff
-        if: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
-        uses: actions/github-script@v8
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          script: |
-            const fs = require('fs');
-            let diff = fs.readFileSync('pyrefly_diff.txt', { encoding: 'utf8' });
-            const prNumber = context.payload.pull_request.number;
-
-            const MAX_CHARS = 65000;
-            if (diff.length > MAX_CHARS) {
-              diff = diff.slice(0, MAX_CHARS);
-              diff = diff.slice(0, diff.lastIndexOf('\n'));
-              diff += '\n\n... (truncated) ...';
-            }
-
-            const body = diff.trim()
-              ? [
-                  '### Pyrefly Diff',
-                  '<details>',
-                  '<summary>base → PR</summary>',
-                  '',
-                  '```diff',
-                  diff,
-                  '```',
-                  '</details>',
-                ].join('\n')
-              : '### Pyrefly Diff\nNo changes detected.';
-
-            await github.rest.issues.createComment({
-              issue_number: prNumber,
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              body,
-            });

.github/workflows/web-tests.yml

@@ -3,22 +3,14 @@ name: Web Tests
 on:
   workflow_call:
 
-permissions:
-  contents: read
-
 concurrency:
   group: web-tests-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 
 jobs:
   test:
-    name: Web Tests (${{ matrix.shardIndex }}/${{ matrix.shardTotal }})
+    name: Web Tests
     runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        shardIndex: [1, 2, 3, 4]
-        shardTotal: [4]
     defaults:
       run:
         shell: bash
@@ -47,58 +39,7 @@ jobs:
         run: pnpm install --frozen-lockfile
 
       - name: Run tests
-        run: pnpm vitest run --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --coverage
-
-      - name: Upload blob report
-        if: ${{ !cancelled() }}
-        uses: actions/upload-artifact@v6
-        with:
-          name: blob-report-${{ matrix.shardIndex }}
-          path: web/.vitest-reports/*
-          include-hidden-files: true
-          retention-days: 1
-
-  merge-reports:
-    name: Merge Test Reports
-    if: ${{ !cancelled() }}
-    needs: [test]
-    runs-on: ubuntu-latest
-    defaults:
-      run:
-        shell: bash
-        working-directory: ./web
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Install pnpm
-        uses: pnpm/action-setup@v4
-        with:
-          package_json_file: web/package.json
-          run_install: false
-
-      - name: Setup Node.js
-        uses: actions/setup-node@v6
-        with:
-          node-version: 24
-          cache: pnpm
-          cache-dependency-path: ./web/pnpm-lock.yaml
-
-      - name: Install dependencies
-        run: pnpm install --frozen-lockfile
-
-      - name: Download blob reports
-        uses: actions/download-artifact@v6
-        with:
-          path: web/.vitest-reports
-          pattern: blob-report-*
-          merge-multiple: true
-
-      - name: Merge reports
-        run: pnpm vitest --merge-reports --coverage --silent=passed-only
+        run: pnpm test:coverage
 
       - name: Coverage Summary
         if: always()

.vscode/launch.json.template

@@ -37,7 +37,7 @@
                 "-c",
                 "1",
                 "-Q",
-                "dataset,priority_dataset,priority_pipeline,pipeline,mail,ops_trace,app_deletion,plugin,workflow_storage,conversation,workflow,schedule_poller,schedule_executor,triggered_workflow_dispatcher,trigger_refresh_executor,retention,workflow_based_app_execution",
+                "dataset,priority_dataset,priority_pipeline,pipeline,mail,ops_trace,app_deletion,plugin,workflow_storage,conversation,workflow,schedule_poller,schedule_executor,triggered_workflow_dispatcher,trigger_refresh_executor,retention",
                 "--loglevel",
                 "INFO"
             ],

README.md

@@ -1,5 +1,9 @@
 ![cover-v5-optimized](./images/GitHub_README_if.png)
 
+<p align="center">
+  📌 <a href="https://dify.ai/blog/introducing-dify-workflow-file-upload-a-demo-on-ai-podcast">Introducing Dify Workflow File Upload: Recreate Google NotebookLM Podcast</a>
+</p>
+
 <p align="center">
   <a href="https://cloud.dify.ai">Dify Cloud</a> ·
   <a href="https://docs.dify.ai/getting-started/install-self-hosted">Self-hosting</a> ·

api/.env.example

@@ -553,8 +553,6 @@ WORKFLOW_LOG_CLEANUP_ENABLED=false
 WORKFLOW_LOG_RETENTION_DAYS=30
 # Batch size for workflow log cleanup operations (default: 100)
 WORKFLOW_LOG_CLEANUP_BATCH_SIZE=100
-# Comma-separated list of workflow IDs to clean logs for
-WORKFLOW_LOG_CLEANUP_SPECIFIC_WORKFLOW_IDS=
 
 # App configuration
 APP_MAX_EXECUTION_TIME=1200
@@ -717,31 +715,5 @@ ANNOTATION_IMPORT_MAX_CONCURRENT=5
 # Sandbox expired records clean configuration
 SANDBOX_EXPIRED_RECORDS_CLEAN_GRACEFUL_PERIOD=21
 SANDBOX_EXPIRED_RECORDS_CLEAN_BATCH_SIZE=1000
-SANDBOX_EXPIRED_RECORDS_CLEAN_BATCH_MAX_INTERVAL=200
 SANDBOX_EXPIRED_RECORDS_RETENTION_DAYS=30
 SANDBOX_EXPIRED_RECORDS_CLEAN_TASK_LOCK_TTL=90000
-
-
-# Redis URL used for PubSub between API and
-# celery worker
-# defaults to url constructed from `REDIS_*`
-# configurations
-PUBSUB_REDIS_URL=
-# Pub/sub channel type for streaming events.
-# valid options are:
-#
-# - pubsub: for normal Pub/Sub
-# - sharded: for sharded Pub/Sub
-#
-# It's highly recommended to use sharded Pub/Sub AND redis cluster
-# for large deployments.
-PUBSUB_REDIS_CHANNEL_TYPE=pubsub
-# Whether to use Redis cluster mode while running
-# PubSub.
-#  It's highly recommended to enable this for large deployments.
-PUBSUB_REDIS_USE_CLUSTERS=false
-
-# Whether to Enable human input timeout check task
-ENABLE_HUMAN_INPUT_TIMEOUT_TASK=true
-# Human input timeout check interval in minutes
-HUMAN_INPUT_TIMEOUT_TASK_INTERVAL=1

api/.importlinter

@@ -36,8 +36,6 @@ ignore_imports =
     core.workflow.nodes.loop.loop_node -> core.workflow.graph_engine
     core.workflow.nodes.loop.loop_node -> core.workflow.graph
     core.workflow.nodes.loop.loop_node -> core.workflow.graph_engine.command_channels
-    # TODO(QuantumGhost): fix the import violation later
-    core.workflow.entities.pause_reason -> core.workflow.nodes.human_input.entities
 
 [importlinter:contract:workflow-infrastructure-dependencies]
 name = Workflow Infrastructure Dependencies
@@ -50,13 +48,16 @@ forbidden_modules =
 allow_indirect_imports = True
 ignore_imports =
     core.workflow.nodes.agent.agent_node -> extensions.ext_database
+    core.workflow.nodes.datasource.datasource_node -> extensions.ext_database
     core.workflow.nodes.knowledge_index.knowledge_index_node -> extensions.ext_database
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> extensions.ext_database
     core.workflow.nodes.llm.file_saver -> extensions.ext_database
     core.workflow.nodes.llm.llm_utils -> extensions.ext_database
     core.workflow.nodes.llm.node -> extensions.ext_database
     core.workflow.nodes.tool.tool_node -> extensions.ext_database
-    # TODO(QuantumGhost): use DI to avoid depending on global DB.
-    core.workflow.nodes.human_input.human_input_node -> extensions.ext_database
+    core.workflow.graph_engine.command_channels.redis_channel -> extensions.ext_redis
+    core.workflow.graph_engine.manager -> extensions.ext_redis
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> extensions.ext_redis
 
 [importlinter:contract:workflow-external-imports]
 name = Workflow External Imports
@@ -88,6 +89,7 @@ forbidden_modules =
     core.logging
     core.mcp
     core.memory
+    core.model_manager
     core.moderation
     core.ops
     core.plugin
@@ -101,23 +103,40 @@ forbidden_modules =
     core.variables
 ignore_imports =
     core.workflow.nodes.loop.loop_node -> core.app.workflow.node_factory
+    core.workflow.graph_engine.command_channels.redis_channel -> extensions.ext_redis
     core.workflow.workflow_entry -> core.app.workflow.layers.observability
     core.workflow.nodes.agent.agent_node -> core.model_manager
     core.workflow.nodes.agent.agent_node -> core.provider_manager
     core.workflow.nodes.agent.agent_node -> core.tools.tool_manager
+    core.workflow.nodes.code.code_node -> core.helper.code_executor.code_executor
+    core.workflow.nodes.datasource.datasource_node -> models.model
+    core.workflow.nodes.datasource.datasource_node -> models.tools
+    core.workflow.nodes.datasource.datasource_node -> services.datasource_provider_service
+    core.workflow.nodes.document_extractor.node -> configs
+    core.workflow.nodes.document_extractor.node -> core.file.file_manager
     core.workflow.nodes.document_extractor.node -> core.helper.ssrf_proxy
+    core.workflow.nodes.http_request.entities -> configs
+    core.workflow.nodes.http_request.executor -> configs
+    core.workflow.nodes.http_request.executor -> core.file.file_manager
+    core.workflow.nodes.http_request.node -> configs
     core.workflow.nodes.http_request.node -> core.tools.tool_file_manager
     core.workflow.nodes.iteration.iteration_node -> core.app.workflow.node_factory
     core.workflow.nodes.knowledge_index.knowledge_index_node -> core.rag.index_processor.index_processor_factory
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.rag.datasource.retrieval_service
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.rag.retrieval.dataset_retrieval
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> models.dataset
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> services.feature_service
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.model_runtime.model_providers.__base.large_language_model
     core.workflow.nodes.llm.llm_utils -> configs
     core.workflow.nodes.llm.llm_utils -> core.app.entities.app_invoke_entities
+    core.workflow.nodes.llm.llm_utils -> core.file.models
     core.workflow.nodes.llm.llm_utils -> core.model_manager
-    core.workflow.nodes.llm.protocols -> core.model_manager
     core.workflow.nodes.llm.llm_utils -> core.model_runtime.model_providers.__base.large_language_model
     core.workflow.nodes.llm.llm_utils -> models.model
     core.workflow.nodes.llm.llm_utils -> models.provider
     core.workflow.nodes.llm.llm_utils -> services.credit_pool_service
     core.workflow.nodes.llm.node -> core.tools.signature
+    core.workflow.nodes.template_transform.template_transform_node -> configs
     core.workflow.nodes.tool.tool_node -> core.callback_handler.workflow_tool_callback_handler
     core.workflow.nodes.tool.tool_node -> core.tools.tool_engine
     core.workflow.nodes.tool.tool_node -> core.tools.tool_manager
@@ -126,9 +145,9 @@ ignore_imports =
     core.workflow.nodes.agent.agent_node -> core.agent.entities
     core.workflow.nodes.agent.agent_node -> core.agent.plugin_entities
     core.workflow.nodes.base.node -> core.app.entities.app_invoke_entities
-    core.workflow.nodes.human_input.human_input_node -> core.app.entities.app_invoke_entities
     core.workflow.nodes.knowledge_index.knowledge_index_node -> core.app.entities.app_invoke_entities
     core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.app.app_config.entities
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.app.entities.app_invoke_entities
     core.workflow.nodes.llm.node -> core.app.entities.app_invoke_entities
     core.workflow.nodes.parameter_extractor.parameter_extractor_node -> core.app.entities.app_invoke_entities
     core.workflow.nodes.parameter_extractor.parameter_extractor_node -> core.prompt.advanced_prompt_transform
@@ -142,18 +161,50 @@ ignore_imports =
     core.workflow.workflow_entry -> core.app.apps.exc
     core.workflow.workflow_entry -> core.app.entities.app_invoke_entities
     core.workflow.workflow_entry -> core.app.workflow.node_factory
+    core.workflow.nodes.datasource.datasource_node -> core.datasource.datasource_manager
+    core.workflow.nodes.datasource.datasource_node -> core.datasource.utils.message_transformer
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.entities.agent_entities
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.entities.model_entities
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.model_manager
     core.workflow.nodes.llm.llm_utils -> core.entities.provider_entities
     core.workflow.nodes.parameter_extractor.parameter_extractor_node -> core.model_manager
     core.workflow.nodes.question_classifier.question_classifier_node -> core.model_manager
+    core.workflow.node_events.node -> core.file
+    core.workflow.nodes.agent.agent_node -> core.file
+    core.workflow.nodes.datasource.datasource_node -> core.file
+    core.workflow.nodes.datasource.datasource_node -> core.file.enums
+    core.workflow.nodes.document_extractor.node -> core.file
+    core.workflow.nodes.http_request.executor -> core.file.enums
+    core.workflow.nodes.http_request.node -> core.file
+    core.workflow.nodes.http_request.node -> core.file.file_manager
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.file.models
+    core.workflow.nodes.list_operator.node -> core.file
+    core.workflow.nodes.llm.file_saver -> core.file
     core.workflow.nodes.llm.llm_utils -> core.variables.segments
+    core.workflow.nodes.llm.node -> core.file
+    core.workflow.nodes.llm.node -> core.file.file_manager
+    core.workflow.nodes.llm.node -> core.file.models
     core.workflow.nodes.loop.entities -> core.variables.types
+    core.workflow.nodes.parameter_extractor.parameter_extractor_node -> core.file
+    core.workflow.nodes.protocols -> core.file
+    core.workflow.nodes.question_classifier.question_classifier_node -> core.file.models
+    core.workflow.nodes.tool.tool_node -> core.file
     core.workflow.nodes.tool.tool_node -> core.tools.utils.message_transformer
     core.workflow.nodes.tool.tool_node -> models
+    core.workflow.nodes.trigger_webhook.node -> core.file
+    core.workflow.runtime.variable_pool -> core.file
+    core.workflow.runtime.variable_pool -> core.file.file_manager
+    core.workflow.system_variable -> core.file.models
+    core.workflow.utils.condition.processor -> core.file
+    core.workflow.utils.condition.processor -> core.file.file_manager
+    core.workflow.workflow_entry -> core.file.models
+    core.workflow.workflow_type_encoder -> core.file.models
     core.workflow.nodes.agent.agent_node -> models.model
     core.workflow.nodes.code.code_node -> core.helper.code_executor.code_node_provider
     core.workflow.nodes.code.code_node -> core.helper.code_executor.javascript.javascript_code_provider
     core.workflow.nodes.code.code_node -> core.helper.code_executor.python3.python3_code_provider
     core.workflow.nodes.code.entities -> core.helper.code_executor.code_executor
+    core.workflow.nodes.datasource.datasource_node -> core.variables.variables
     core.workflow.nodes.http_request.executor -> core.helper.ssrf_proxy
     core.workflow.nodes.http_request.node -> core.helper.ssrf_proxy
     core.workflow.nodes.llm.file_saver -> core.helper.ssrf_proxy
@@ -163,6 +214,7 @@ ignore_imports =
     core.workflow.nodes.llm.node -> core.llm_generator.output_parser.structured_output
     core.workflow.nodes.llm.node -> core.model_manager
     core.workflow.nodes.agent.entities -> core.prompt.entities.advanced_prompt_entities
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.prompt.simple_prompt_transform
     core.workflow.nodes.llm.entities -> core.prompt.entities.advanced_prompt_entities
     core.workflow.nodes.llm.llm_utils -> core.prompt.entities.advanced_prompt_entities
     core.workflow.nodes.llm.node -> core.prompt.entities.advanced_prompt_entities
@@ -178,6 +230,7 @@ ignore_imports =
     core.workflow.nodes.knowledge_index.knowledge_index_node -> services.summary_index_service
     core.workflow.nodes.knowledge_index.knowledge_index_node -> tasks.generate_summary_index_task
     core.workflow.nodes.knowledge_index.knowledge_index_node -> core.rag.index_processor.processor.paragraph_index_processor
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> core.rag.retrieval.retrieval_methods
     core.workflow.nodes.llm.node -> models.dataset
     core.workflow.nodes.agent.agent_node -> core.tools.utils.message_transformer
     core.workflow.nodes.llm.file_saver -> core.tools.signature
@@ -190,11 +243,11 @@ ignore_imports =
     core.workflow.nodes.code.code_node -> core.variables.segments
     core.workflow.nodes.code.code_node -> core.variables.types
     core.workflow.nodes.code.entities -> core.variables.types
+    core.workflow.nodes.datasource.datasource_node -> core.variables.segments
     core.workflow.nodes.document_extractor.node -> core.variables
     core.workflow.nodes.document_extractor.node -> core.variables.segments
     core.workflow.nodes.http_request.executor -> core.variables.segments
     core.workflow.nodes.http_request.node -> core.variables.segments
-    core.workflow.nodes.human_input.entities -> core.variables.consts
     core.workflow.nodes.iteration.iteration_node -> core.variables
     core.workflow.nodes.iteration.iteration_node -> core.variables.segments
     core.workflow.nodes.iteration.iteration_node -> core.variables.variables
@@ -231,14 +284,16 @@ ignore_imports =
     core.workflow.variable_loader -> core.variables
     core.workflow.variable_loader -> core.variables.consts
     core.workflow.workflow_type_encoder -> core.variables
+    core.workflow.graph_engine.manager -> extensions.ext_redis
     core.workflow.nodes.agent.agent_node -> extensions.ext_database
+    core.workflow.nodes.datasource.datasource_node -> extensions.ext_database
     core.workflow.nodes.knowledge_index.knowledge_index_node -> extensions.ext_database
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> extensions.ext_database
+    core.workflow.nodes.knowledge_retrieval.knowledge_retrieval_node -> extensions.ext_redis
     core.workflow.nodes.llm.file_saver -> extensions.ext_database
     core.workflow.nodes.llm.llm_utils -> extensions.ext_database
     core.workflow.nodes.llm.node -> extensions.ext_database
     core.workflow.nodes.tool.tool_node -> extensions.ext_database
-    core.workflow.nodes.human_input.human_input_node -> extensions.ext_database
-    core.workflow.nodes.human_input.human_input_node -> core.repositories.human_input_repository
     core.workflow.workflow_entry -> extensions.otel.runtime
     core.workflow.nodes.agent.agent_node -> models
     core.workflow.nodes.base.node -> models.enums
@@ -247,11 +302,6 @@ ignore_imports =
     core.workflow.workflow_entry -> models.enums
     core.workflow.nodes.agent.agent_node -> services
     core.workflow.nodes.tool.tool_node -> services
-    core.workflow.nodes.agent.agent_node -> core.model_runtime.token_buffer_memory
-    core.workflow.nodes.llm.llm_utils -> core.model_runtime.token_buffer_memory
-    core.workflow.nodes.llm.node -> core.model_runtime.token_buffer_memory
-    core.workflow.nodes.parameter_extractor.parameter_extractor_node -> core.model_runtime.token_buffer_memory
-    core.workflow.nodes.question_classifier.question_classifier_node -> core.model_runtime.token_buffer_memory
 
 [importlinter:contract:model-runtime-no-internal-imports]
 name = Model Runtime Internal Imports
@@ -304,13 +354,6 @@ ignore_imports =
     core.model_runtime.model_providers.model_provider_factory -> configs
     core.model_runtime.model_providers.model_provider_factory -> extensions.ext_redis
     core.model_runtime.model_providers.model_provider_factory -> models.provider_ids
-    core.model_runtime.token_buffer_memory -> core.app.app_config.features.file_upload.manager
-    core.model_runtime.token_buffer_memory -> core.model_manager
-    core.model_runtime.token_buffer_memory -> core.prompt.utils.extract_thread_messages
-    core.model_runtime.token_buffer_memory -> core.workflow.file.file_manager
-    core.model_runtime.token_buffer_memory -> extensions.ext_database
-    core.model_runtime.token_buffer_memory -> models.model
-    core.model_runtime.token_buffer_memory -> models.workflow
 
 [importlinter:contract:rsc]
 name = RSC

api/.vscode/launch.json.example

@@ -54,7 +54,7 @@
                 "--loglevel",
                 "DEBUG",
                 "-Q",
-                "dataset,priority_pipeline,pipeline,mail,ops_trace,app_deletion,plugin,workflow_storage,conversation,workflow,workflow_based_app_execution,schedule_poller,schedule_executor,triggered_workflow_dispatcher,trigger_refresh_executor"
+                "dataset,priority_pipeline,pipeline,mail,ops_trace,app_deletion,plugin,workflow_storage,conversation,workflow,schedule_poller,schedule_executor,triggered_workflow_dispatcher,trigger_refresh_executor"
             ]
         }
     ]

api/README.md

@@ -42,7 +42,7 @@ The scripts resolve paths relative to their location, so you can run them from a
 
 1. Set up your application by visiting `http://localhost:3000`.
 
-1. Start the worker service (async and scheduler tasks, runs from `api`).
+1. Optional: start the worker service (async tasks, runs from `api`).
 
    ```bash
    ./dev/start-worker
@@ -54,6 +54,86 @@ The scripts resolve paths relative to their location, so you can run them from a
    ./dev/start-beat
    ```
 
+### Manual commands
+
+<details>
+<summary>Show manual setup and run steps</summary>
+
+These commands assume you start from the repository root.
+
+1. Start the docker-compose stack.
+
+   The backend requires middleware, including PostgreSQL, Redis, and Weaviate, which can be started together using `docker-compose`.
+
+   ```bash
+   cp docker/middleware.env.example docker/middleware.env
+   # Use mysql or another vector database profile if you are not using postgres/weaviate.
+   docker compose -f docker/docker-compose.middleware.yaml --profile postgresql --profile weaviate -p dify up -d
+   ```
+
+1. Copy env files.
+
+   ```bash
+   cp api/.env.example api/.env
+   cp web/.env.example web/.env.local
+   ```
+
+1. Install UV if needed.
+
+   ```bash
+   pip install uv
+   # Or on macOS
+   brew install uv
+   ```
+
+1. Install API dependencies.
+
+   ```bash
+   cd api
+   uv sync --group dev
+   ```
+
+1. Install web dependencies.
+
+   ```bash
+   cd web
+   pnpm install
+   cd ..
+   ```
+
+1. Start backend (runs migrations first, in a new terminal).
+
+   ```bash
+   cd api
+   uv run flask db upgrade
+   uv run flask run --host 0.0.0.0 --port=5001 --debug
+   ```
+
+1. Start Dify [web](../web) service (in a new terminal).
+
+   ```bash
+   cd web
+   pnpm dev:inspect
+   ```
+
+1. Set up your application by visiting `http://localhost:3000`.
+
+1. Optional: start the worker service (async tasks, in a new terminal).
+
+   ```bash
+   cd api
+   uv run celery -A app.celery worker -P threads -c 2 --loglevel INFO -Q dataset,priority_dataset,priority_pipeline,pipeline,mail,ops_trace,app_deletion,plugin,workflow_storage,conversation,workflow,schedule_poller,schedule_executor,triggered_workflow_dispatcher,trigger_refresh_executor,retention
+   ```
+
+1. Optional: start Celery Beat (scheduled tasks, in a new terminal).
+
+   ```bash
+   cd api
+   uv run celery -A app.celery beat
+   ```
+
+</details>
+
 ### Environment notes
 
 > [!IMPORTANT]

api/commands.py

@@ -30,7 +30,6 @@ from extensions.ext_redis import redis_client
 from extensions.ext_storage import storage
 from extensions.storage.opendal_storage import OpenDALStorage
 from extensions.storage.storage_type import StorageType
-from libs.db_migration_lock import DbMigrationAutoRenewLock
 from libs.helper import email as email_validate
 from libs.password import hash_password, password_pattern, valid_password
 from libs.rsa import generate_key_pair
@@ -55,8 +54,6 @@ from tasks.remove_app_and_related_data_task import delete_draft_variables_batch
 
 logger = logging.getLogger(__name__)
 
-DB_UPGRADE_LOCK_TTL_SECONDS = 60
-
 
 @click.command("reset-password", help="Reset the account password.")
 @click.option("--email", prompt=True, help="Account email to reset password for")
@@ -730,15 +727,8 @@ def create_tenant(email: str, language: str | None = None, name: str | None = No
 @click.command("upgrade-db", help="Upgrade the database")
 def upgrade_db():
     click.echo("Preparing database migration...")
-    lock = DbMigrationAutoRenewLock(
-        redis_client=redis_client,
-        name="db_upgrade_lock",
-        ttl_seconds=DB_UPGRADE_LOCK_TTL_SECONDS,
-        logger=logger,
-        log_context="db_migration",
-    )
+    lock = redis_client.lock(name="db_upgrade_lock", timeout=60)
     if lock.acquire(blocking=False):
-        migration_succeeded = False
         try:
             click.echo(click.style("Starting database migration.", fg="green"))
 
@@ -747,16 +737,12 @@ def upgrade_db():
 
             flask_migrate.upgrade()
 
-            migration_succeeded = True
             click.echo(click.style("Database migration successful!", fg="green"))
 
-        except Exception as e:
+        except Exception:
             logger.exception("Failed to execute database migration")
-            click.echo(click.style(f"Database migration failed: {e}", fg="red"))
-            raise SystemExit(1)
         finally:
-            status = "successful" if migration_succeeded else "failed"
-            lock.release_safely(status=status)
+            lock.release()
     else:
         click.echo("Database migration skipped")
 
@@ -1464,58 +1450,54 @@ def clear_orphaned_file_records(force: bool):
         all_ids_in_tables = []
         for ids_table in ids_tables:
             query = ""
-            match ids_table["type"]:
-                case "uuid":
-                    click.echo(
-                        click.style(
-                            f"- Listing file ids in column {ids_table['column']} in table {ids_table['table']}",
-                            fg="white",
-                        )
+            if ids_table["type"] == "uuid":
+                click.echo(
+                    click.style(
+                        f"- Listing file ids in column {ids_table['column']} in table {ids_table['table']}", fg="white"
                     )
-                    c = ids_table["column"]
-                    query = f"SELECT {c} FROM {ids_table['table']} WHERE {c} IS NOT NULL"
-                    with db.engine.begin() as conn:
-                        rs = conn.execute(sa.text(query))
-                    for i in rs:
-                        all_ids_in_tables.append({"table": ids_table["table"], "id": str(i[0])})
-                case "text":
-                    t = ids_table["table"]
-                    click.echo(
-                        click.style(
-                            f"- Listing file-id-like strings in column {ids_table['column']} in table {t}",
-                            fg="white",
-                        )
+                )
+                query = (
+                    f"SELECT {ids_table['column']} FROM {ids_table['table']} WHERE {ids_table['column']} IS NOT NULL"
+                )
+                with db.engine.begin() as conn:
+                    rs = conn.execute(sa.text(query))
+                for i in rs:
+                    all_ids_in_tables.append({"table": ids_table["table"], "id": str(i[0])})
+            elif ids_table["type"] == "text":
+                click.echo(
+                    click.style(
+                        f"- Listing file-id-like strings in column {ids_table['column']} in table {ids_table['table']}",
+                        fg="white",
                     )
-                    query = (
-                        f"SELECT regexp_matches({ids_table['column']}, '{guid_regexp}', 'g') AS extracted_id "
-                        f"FROM {ids_table['table']}"
+                )
+                query = (
+                    f"SELECT regexp_matches({ids_table['column']}, '{guid_regexp}', 'g') AS extracted_id "
+                    f"FROM {ids_table['table']}"
+                )
+                with db.engine.begin() as conn:
+                    rs = conn.execute(sa.text(query))
+                for i in rs:
+                    for j in i[0]:
+                        all_ids_in_tables.append({"table": ids_table["table"], "id": j})
+            elif ids_table["type"] == "json":
+                click.echo(
+                    click.style(
+                        (
+                            f"- Listing file-id-like JSON string in column {ids_table['column']} "
+                            f"in table {ids_table['table']}"
+                        ),
+                        fg="white",
                     )
-                    with db.engine.begin() as conn:
-                        rs = conn.execute(sa.text(query))
-                    for i in rs:
-                        for j in i[0]:
-                            all_ids_in_tables.append({"table": ids_table["table"], "id": j})
-                case "json":
-                    click.echo(
-                        click.style(
-                            (
-                                f"- Listing file-id-like JSON string in column {ids_table['column']} "
-                                f"in table {ids_table['table']}"
-                            ),
-                            fg="white",
-                        )
-                    )
-                    query = (
-                        f"SELECT regexp_matches({ids_table['column']}::text, '{guid_regexp}', 'g') AS extracted_id "
-                        f"FROM {ids_table['table']}"
-                    )
-                    with db.engine.begin() as conn:
-                        rs = conn.execute(sa.text(query))
-                    for i in rs:
-                        for j in i[0]:
-                            all_ids_in_tables.append({"table": ids_table["table"], "id": j})
-                case _:
-                    pass
+                )
+                query = (
+                    f"SELECT regexp_matches({ids_table['column']}::text, '{guid_regexp}', 'g') AS extracted_id "
+                    f"FROM {ids_table['table']}"
+                )
+                with db.engine.begin() as conn:
+                    rs = conn.execute(sa.text(query))
+                for i in rs:
+                    for j in i[0]:
+                        all_ids_in_tables.append({"table": ids_table["table"], "id": j})
         click.echo(click.style(f"Found {len(all_ids_in_tables)} file ids in tables.", fg="white"))
 
     except Exception as e:
@@ -1755,18 +1737,59 @@ def file_usage(
                 if src_filter != src:
                     continue
 
-        match ids_table["type"]:
-            case "uuid":
-                # Direct UUID match
-                query = (
-                    f"SELECT {ids_table['pk_column']}, {ids_table['column']} "
-                    f"FROM {ids_table['table']} WHERE {ids_table['column']} IS NOT NULL"
-                )
-                with db.engine.begin() as conn:
-                    rs = conn.execute(sa.text(query))
-                    for row in rs:
-                        record_id = str(row[0])
-                        ref_file_id = str(row[1])
+        if ids_table["type"] == "uuid":
+            # Direct UUID match
+            query = (
+                f"SELECT {ids_table['pk_column']}, {ids_table['column']} "
+                f"FROM {ids_table['table']} WHERE {ids_table['column']} IS NOT NULL"
+            )
+            with db.engine.begin() as conn:
+                rs = conn.execute(sa.text(query))
+                for row in rs:
+                    record_id = str(row[0])
+                    ref_file_id = str(row[1])
+                    if ref_file_id not in file_key_map:
+                        continue
+                    storage_key = file_key_map[ref_file_id]
+
+                    # Apply filters
+                    if file_id and ref_file_id != file_id:
+                        continue
+                    if key and not storage_key.endswith(key):
+                        continue
+
+                    # Only collect items within the requested page range
+                    if offset <= total_count < offset + limit:
+                        paginated_usages.append(
+                            {
+                                "src": f"{ids_table['table']}.{ids_table['column']}",
+                                "record_id": record_id,
+                                "file_id": ref_file_id,
+                                "key": storage_key,
+                            }
+                        )
+                    total_count += 1
+
+        elif ids_table["type"] in ("text", "json"):
+            # Extract UUIDs from text/json content
+            column_cast = f"{ids_table['column']}::text" if ids_table["type"] == "json" else ids_table["column"]
+            query = (
+                f"SELECT {ids_table['pk_column']}, {column_cast} "
+                f"FROM {ids_table['table']} WHERE {ids_table['column']} IS NOT NULL"
+            )
+            with db.engine.begin() as conn:
+                rs = conn.execute(sa.text(query))
+                for row in rs:
+                    record_id = str(row[0])
+                    content = str(row[1])
+
+                    # Find all UUIDs in the content
+                    import re
+
+                    uuid_pattern = re.compile(guid_regexp, re.IGNORECASE)
+                    matches = uuid_pattern.findall(content)
+
+                    for ref_file_id in matches:
                         if ref_file_id not in file_key_map:
                             continue
                         storage_key = file_key_map[ref_file_id]
@@ -1789,50 +1812,6 @@ def file_usage(
                             )
                         total_count += 1
 
-            case "text" | "json":
-                # Extract UUIDs from text/json content
-                column_cast = f"{ids_table['column']}::text" if ids_table["type"] == "json" else ids_table["column"]
-                query = (
-                    f"SELECT {ids_table['pk_column']}, {column_cast} "
-                    f"FROM {ids_table['table']} WHERE {ids_table['column']} IS NOT NULL"
-                )
-                with db.engine.begin() as conn:
-                    rs = conn.execute(sa.text(query))
-                    for row in rs:
-                        record_id = str(row[0])
-                        content = str(row[1])
-
-                        # Find all UUIDs in the content
-                        import re
-
-                        uuid_pattern = re.compile(guid_regexp, re.IGNORECASE)
-                        matches = uuid_pattern.findall(content)
-
-                        for ref_file_id in matches:
-                            if ref_file_id not in file_key_map:
-                                continue
-                            storage_key = file_key_map[ref_file_id]
-
-                            # Apply filters
-                            if file_id and ref_file_id != file_id:
-                                continue
-                            if key and not storage_key.endswith(key):
-                                continue
-
-                            # Only collect items within the requested page range
-                            if offset <= total_count < offset + limit:
-                                paginated_usages.append(
-                                    {
-                                        "src": f"{ids_table['table']}.{ids_table['column']}",
-                                        "record_id": record_id,
-                                        "file_id": ref_file_id,
-                                        "key": storage_key,
-                                    }
-                                )
-                            total_count += 1
-            case _:
-                pass
-
     # Output results
     if output_json:
         result = {

api/configs/feature/__init__.py

@@ -1,4 +1,3 @@
-from datetime import timedelta
 from enum import StrEnum
 from typing import Literal
 
@@ -49,16 +48,6 @@ class SecurityConfig(BaseSettings):
         default=5,
     )
 
-    WEB_FORM_SUBMIT_RATE_LIMIT_MAX_ATTEMPTS: PositiveInt = Field(
-        description="Maximum number of web form submissions allowed per IP within the rate limit window",
-        default=30,
-    )
-
-    WEB_FORM_SUBMIT_RATE_LIMIT_WINDOW_SECONDS: PositiveInt = Field(
-        description="Time window in seconds for web form submission rate limiting",
-        default=60,
-    )
-
     LOGIN_DISABLED: bool = Field(
         description="Whether to disable login checks",
         default=False,
@@ -93,12 +82,6 @@ class AppExecutionConfig(BaseSettings):
         default=0,
     )
 
-    HUMAN_INPUT_GLOBAL_TIMEOUT_SECONDS: PositiveInt = Field(
-        description="Maximum seconds a workflow run can stay paused waiting for human input before global timeout.",
-        default=int(timedelta(days=7).total_seconds()),
-        ge=1,
-    )
-
 
 class CodeExecutionSandboxConfig(BaseSettings):
     """
@@ -265,11 +248,6 @@ class PluginConfig(BaseSettings):
         default=60 * 60,
     )
 
-    PLUGIN_MAX_FILE_SIZE: PositiveInt = Field(
-        description="Maximum allowed size (bytes) for plugin-generated files",
-        default=50 * 1024 * 1024,
-    )
-
 
 class MarketplaceConfig(BaseSettings):
     """
@@ -1156,14 +1134,6 @@ class CeleryScheduleTasksConfig(BaseSettings):
         description="Enable queue monitor task",
         default=False,
     )
-    ENABLE_HUMAN_INPUT_TIMEOUT_TASK: bool = Field(
-        description="Enable human input timeout check task",
-        default=True,
-    )
-    HUMAN_INPUT_TIMEOUT_TASK_INTERVAL: PositiveInt = Field(
-        description="Human input timeout check interval in minutes",
-        default=1,
-    )
     ENABLE_CHECK_UPGRADABLE_PLUGIN_TASK: bool = Field(
         description="Enable check upgradable plugin task",
         default=True,
@@ -1185,16 +1155,6 @@ class CeleryScheduleTasksConfig(BaseSettings):
         default=0,
     )
 
-    # API token last_used_at batch update
-    ENABLE_API_TOKEN_LAST_USED_UPDATE_TASK: bool = Field(
-        description="Enable periodic batch update of API token last_used_at timestamps",
-        default=True,
-    )
-    API_TOKEN_LAST_USED_UPDATE_INTERVAL: int = Field(
-        description="Interval in minutes for batch updating API token last_used_at (default 30)",
-        default=30,
-    )
-
     # Trigger provider refresh (simple version)
     ENABLE_TRIGGER_PROVIDER_REFRESH_TASK: bool = Field(
         description="Enable trigger provider refresh poller",
@@ -1319,9 +1279,6 @@ class WorkflowLogConfig(BaseSettings):
     WORKFLOW_LOG_CLEANUP_BATCH_SIZE: int = Field(
         default=100, description="Batch size for workflow run log cleanup operations"
     )
-    WORKFLOW_LOG_CLEANUP_SPECIFIC_WORKFLOW_IDS: str = Field(
-        default="", description="Comma-separated list of workflow IDs to clean logs for"
-    )
 
 
 class SwaggerUIConfig(BaseSettings):
@@ -1352,10 +1309,6 @@ class SandboxExpiredRecordsCleanConfig(BaseSettings):
         description="Maximum number of records to process in each batch",
         default=1000,
     )
-    SANDBOX_EXPIRED_RECORDS_CLEAN_BATCH_MAX_INTERVAL: PositiveInt = Field(
-        description="Maximum interval in milliseconds between batches",
-        default=200,
-    )
     SANDBOX_EXPIRED_RECORDS_RETENTION_DAYS: PositiveInt = Field(
         description="Retention days for sandbox expired workflow_run records and message records",
         default=30,

api/configs/middleware/__init__.py

@@ -6,7 +6,6 @@ from pydantic import Field, NonNegativeFloat, NonNegativeInt, PositiveFloat, Pos
 from pydantic_settings import BaseSettings
 
 from .cache.redis_config import RedisConfig
-from .cache.redis_pubsub_config import RedisPubSubConfig
 from .storage.aliyun_oss_storage_config import AliyunOSSStorageConfig
 from .storage.amazon_s3_storage_config import S3StorageConfig
 from .storage.azure_blob_storage_config import AzureBlobStorageConfig
@@ -259,20 +258,11 @@ class CeleryConfig(DatabaseConfig):
         description="Password of the Redis Sentinel master.",
         default=None,
     )
-
     CELERY_SENTINEL_SOCKET_TIMEOUT: PositiveFloat | None = Field(
         description="Timeout for Redis Sentinel socket operations in seconds.",
         default=0.1,
     )
 
-    CELERY_TASK_ANNOTATIONS: dict[str, Any] | None = Field(
-        description=(
-            "Annotations for Celery tasks as a JSON mapping of task name -> options "
-            "(for example, rate limits or other task-specific settings)."
-        ),
-        default=None,
-    )
-
     @computed_field
     def CELERY_RESULT_BACKEND(self) -> str | None:
         if self.CELERY_BACKEND in ("database", "rabbitmq"):
@@ -327,7 +317,6 @@ class MiddlewareConfig(
     CeleryConfig,  # Note: CeleryConfig already inherits from DatabaseConfig
     KeywordStoreConfig,
     RedisConfig,
-    RedisPubSubConfig,
     # configs of storage and storage providers
     StorageConfig,
     AliyunOSSStorageConfig,

api/configs/middleware/cache/redis_pubsub_config.py

@@ -1,96 +0,0 @@
-from typing import Literal, Protocol
-from urllib.parse import quote_plus, urlunparse
-
-from pydantic import Field
-from pydantic_settings import BaseSettings
-
-
-class RedisConfigDefaults(Protocol):
-    REDIS_HOST: str
-    REDIS_PORT: int
-    REDIS_USERNAME: str | None
-    REDIS_PASSWORD: str | None
-    REDIS_DB: int
-    REDIS_USE_SSL: bool
-    REDIS_USE_SENTINEL: bool | None
-    REDIS_USE_CLUSTERS: bool
-
-
-class RedisConfigDefaultsMixin:
-    def _redis_defaults(self: RedisConfigDefaults) -> RedisConfigDefaults:
-        return self
-
-
-class RedisPubSubConfig(BaseSettings, RedisConfigDefaultsMixin):
-    """
-    Configuration settings for Redis pub/sub streaming.
-    """
-
-    PUBSUB_REDIS_URL: str | None = Field(
-        alias="PUBSUB_REDIS_URL",
-        description=(
-            "Redis connection URL for pub/sub streaming events between API "
-            "and celery worker, defaults to url constructed from "
-            "`REDIS_*` configurations"
-        ),
-        default=None,
-    )
-
-    PUBSUB_REDIS_USE_CLUSTERS: bool = Field(
-        description=(
-            "Enable Redis Cluster mode for pub/sub streaming. It's highly "
-            "recommended to enable this for large deployments."
-        ),
-        default=False,
-    )
-
-    PUBSUB_REDIS_CHANNEL_TYPE: Literal["pubsub", "sharded"] = Field(
-        description=(
-            "Pub/sub channel type for streaming events. "
-            "Valid options are:\n"
-            "\n"
-            " - pubsub: for normal Pub/Sub\n"
-            " - sharded: for sharded Pub/Sub\n"
-            "\n"
-            "It's highly recommended to use sharded Pub/Sub AND redis cluster "
-            "for large deployments."
-        ),
-        default="pubsub",
-    )
-
-    def _build_default_pubsub_url(self) -> str:
-        defaults = self._redis_defaults()
-        if not defaults.REDIS_HOST or not defaults.REDIS_PORT:
-            raise ValueError("PUBSUB_REDIS_URL must be set when default Redis URL cannot be constructed")
-
-        scheme = "rediss" if defaults.REDIS_USE_SSL else "redis"
-        username = defaults.REDIS_USERNAME or None
-        password = defaults.REDIS_PASSWORD or None
-
-        userinfo = ""
-        if username:
-            userinfo = quote_plus(username)
-        if password:
-            password_part = quote_plus(password)
-            userinfo = f"{userinfo}:{password_part}" if userinfo else f":{password_part}"
-        if userinfo:
-            userinfo = f"{userinfo}@"
-
-        host = defaults.REDIS_HOST
-        port = defaults.REDIS_PORT
-        db = defaults.REDIS_DB
-
-        netloc = f"{userinfo}{host}:{port}"
-        return urlunparse((scheme, netloc, f"/{db}", "", "", ""))
-
-    @property
-    def normalized_pubsub_redis_url(self) -> str:
-        pubsub_redis_url = self.PUBSUB_REDIS_URL
-        if pubsub_redis_url:
-            cleaned = pubsub_redis_url.strip()
-            pubsub_redis_url = cleaned or None
-
-        if pubsub_redis_url:
-            return pubsub_redis_url
-
-        return self._build_default_pubsub_url()

api/configs/middleware/vdb/oceanbase_config.py

@@ -1,5 +1,3 @@
-from typing import Literal
-
 from pydantic import Field, PositiveInt
 from pydantic_settings import BaseSettings
 
@@ -51,43 +49,3 @@ class OceanBaseVectorConfig(BaseSettings):
         ),
         default="ik",
     )
-
-    OCEANBASE_VECTOR_BATCH_SIZE: PositiveInt = Field(
-        description="Number of documents to insert per batch",
-        default=100,
-    )
-
-    OCEANBASE_VECTOR_METRIC_TYPE: Literal["l2", "cosine", "inner_product"] = Field(
-        description="Distance metric type for vector index: l2, cosine, or inner_product",
-        default="l2",
-    )
-
-    OCEANBASE_HNSW_M: PositiveInt = Field(
-        description="HNSW M parameter (max number of connections per node)",
-        default=16,
-    )
-
-    OCEANBASE_HNSW_EF_CONSTRUCTION: PositiveInt = Field(
-        description="HNSW efConstruction parameter (index build-time search width)",
-        default=256,
-    )
-
-    OCEANBASE_HNSW_EF_SEARCH: int = Field(
-        description="HNSW efSearch parameter (query-time search width, -1 uses server default)",
-        default=-1,
-    )
-
-    OCEANBASE_VECTOR_POOL_SIZE: PositiveInt = Field(
-        description="SQLAlchemy connection pool size",
-        default=5,
-    )
-
-    OCEANBASE_VECTOR_MAX_OVERFLOW: int = Field(
-        description="SQLAlchemy connection pool max overflow connections",
-        default=10,
-    )
-
-    OCEANBASE_HNSW_REFRESH_THRESHOLD: int = Field(
-        description="Minimum number of inserted documents to trigger an automatic HNSW index refresh (0 to disable)",
-        default=1000,
-    )

api/constants/languages.py

@@ -21,7 +21,6 @@ language_timezone_mapping = {
     "th-TH": "Asia/Bangkok",
     "id-ID": "Asia/Jakarta",
     "ar-TN": "Africa/Tunis",
-    "nl-NL": "Europe/Amsterdam",
 }
 
 languages = list(language_timezone_mapping.keys())

api/controllers/common/fields.py

@@ -4,7 +4,7 @@ from typing import Any, TypeAlias
 
 from pydantic import BaseModel, ConfigDict, computed_field
 
-from core.workflow.file import helpers as file_helpers
+from core.file import helpers as file_helpers
 from models.model import IconType
 
 JSONValue: TypeAlias = str | int | float | bool | None | dict[str, Any] | list[Any]

api/controllers/common/schema.py

@@ -5,6 +5,8 @@ from enum import StrEnum
 from flask_restx import Namespace
 from pydantic import BaseModel, TypeAdapter
 
+from controllers.console import console_ns
+
 DEFAULT_REF_TEMPLATE_SWAGGER_2_0 = "#/definitions/{model}"
 
 
@@ -22,9 +24,6 @@ def register_schema_models(namespace: Namespace, *models: type[BaseModel]) -> No
 
 
 def get_or_create_model(model_name: str, field_def):
-    # Import lazily to avoid circular imports between console controllers and schema helpers.
-    from controllers.console import console_ns
-
     existing = console_ns.models.get(model_name)
     if existing is None:
         existing = console_ns.model(model_name, field_def)

api/controllers/console/__init__.py

@@ -37,7 +37,6 @@ from . import (
     apikey,
     extension,
     feature,
-    human_input_form,
     init_validate,
     ping,
     setup,
@@ -172,7 +171,6 @@ __all__ = [
     "forgot_password",
     "generator",
     "hit_testing",
-    "human_input_form",
     "init_validate",
     "installed_app",
     "load_balancing_config",

api/controllers/console/apikey.py

@@ -10,7 +10,6 @@ from libs.helper import TimestampField
 from libs.login import current_account_with_tenant, login_required
 from models.dataset import Dataset
 from models.model import ApiToken, App
-from services.api_token_service import ApiTokenCache
 
 from . import console_ns
 from .wraps import account_initialization_required, edit_permission_required, setup_required
@@ -132,11 +131,6 @@ class BaseApiKeyResource(Resource):
         if key is None:
             flask_restx.abort(HTTPStatus.NOT_FOUND, message="API key not found")
 
-        # Invalidate cache before deleting from database
-        # Type assertion: key is guaranteed to be non-None here because abort() raises
-        assert key is not None  # nosec - for type checker only
-        ApiTokenCache.delete(key.token, key.type)
-
         db.session.query(ApiToken).where(ApiToken.id == api_key_id).delete()
         db.session.commit()
 

api/controllers/console/app/annotation.py

@@ -1,11 +1,10 @@
 from typing import Any, Literal
 
 from flask import abort, make_response, request
-from flask_restx import Resource
-from pydantic import BaseModel, Field, TypeAdapter, field_validator
+from flask_restx import Resource, fields, marshal, marshal_with
+from pydantic import BaseModel, Field, field_validator
 
 from controllers.common.errors import NoFileUploadedError, TooManyFilesError
-from controllers.common.schema import register_schema_models
 from controllers.console import console_ns
 from controllers.console.wraps import (
     account_initialization_required,
@@ -17,11 +16,9 @@ from controllers.console.wraps import (
 )
 from extensions.ext_redis import redis_client
 from fields.annotation_fields import (
-    Annotation,
-    AnnotationExportList,
-    AnnotationHitHistory,
-    AnnotationHitHistoryList,
-    AnnotationList,
+    annotation_fields,
+    annotation_hit_history_fields,
+    build_annotation_model,
 )
 from libs.helper import uuid_value
 from libs.login import login_required
@@ -92,14 +89,6 @@ reg(CreateAnnotationPayload)
 reg(UpdateAnnotationPayload)
 reg(AnnotationReplyStatusQuery)
 reg(AnnotationFilePayload)
-register_schema_models(
-    console_ns,
-    Annotation,
-    AnnotationList,
-    AnnotationExportList,
-    AnnotationHitHistory,
-    AnnotationHitHistoryList,
-)
 
 
 @console_ns.route("/apps/<uuid:app_id>/annotation-reply/<string:action>")
@@ -118,11 +107,10 @@ class AnnotationReplyActionApi(Resource):
     def post(self, app_id, action: Literal["enable", "disable"]):
         app_id = str(app_id)
         args = AnnotationReplyPayload.model_validate(console_ns.payload)
-        match action:
-            case "enable":
-                result = AppAnnotationService.enable_app_annotation(args.model_dump(), app_id)
-            case "disable":
-                result = AppAnnotationService.disable_app_annotation(app_id)
+        if action == "enable":
+            result = AppAnnotationService.enable_app_annotation(args.model_dump(), app_id)
+        elif action == "disable":
+            result = AppAnnotationService.disable_app_annotation(app_id)
         return result, 200
 
 
@@ -213,33 +201,33 @@ class AnnotationApi(Resource):
 
         app_id = str(app_id)
         annotation_list, total = AppAnnotationService.get_annotation_list_by_app_id(app_id, page, limit, keyword)
-        annotation_models = TypeAdapter(list[Annotation]).validate_python(annotation_list, from_attributes=True)
-        response = AnnotationList(
-            data=annotation_models,
-            has_more=len(annotation_list) == limit,
-            limit=limit,
-            total=total,
-            page=page,
-        )
-        return response.model_dump(mode="json"), 200
+        response = {
+            "data": marshal(annotation_list, annotation_fields),
+            "has_more": len(annotation_list) == limit,
+            "limit": limit,
+            "total": total,
+            "page": page,
+        }
+        return response, 200
 
     @console_ns.doc("create_annotation")
     @console_ns.doc(description="Create a new annotation for an app")
     @console_ns.doc(params={"app_id": "Application ID"})
     @console_ns.expect(console_ns.models[CreateAnnotationPayload.__name__])
-    @console_ns.response(201, "Annotation created successfully", console_ns.models[Annotation.__name__])
+    @console_ns.response(201, "Annotation created successfully", build_annotation_model(console_ns))
     @console_ns.response(403, "Insufficient permissions")
     @setup_required
     @login_required
     @account_initialization_required
     @cloud_edition_billing_resource_check("annotation")
+    @marshal_with(annotation_fields)
     @edit_permission_required
     def post(self, app_id):
         app_id = str(app_id)
         args = CreateAnnotationPayload.model_validate(console_ns.payload)
         data = args.model_dump(exclude_none=True)
         annotation = AppAnnotationService.up_insert_app_annotation_from_message(data, app_id)
-        return Annotation.model_validate(annotation, from_attributes=True).model_dump(mode="json")
+        return annotation
 
     @setup_required
     @login_required
@@ -276,7 +264,7 @@ class AnnotationExportApi(Resource):
     @console_ns.response(
         200,
         "Annotations exported successfully",
-        console_ns.models[AnnotationExportList.__name__],
+        console_ns.model("AnnotationList", {"data": fields.List(fields.Nested(build_annotation_model(console_ns)))}),
     )
     @console_ns.response(403, "Insufficient permissions")
     @setup_required
@@ -286,8 +274,7 @@ class AnnotationExportApi(Resource):
     def get(self, app_id):
         app_id = str(app_id)
         annotation_list = AppAnnotationService.export_annotation_list_by_app_id(app_id)
-        annotation_models = TypeAdapter(list[Annotation]).validate_python(annotation_list, from_attributes=True)
-        response_data = AnnotationExportList(data=annotation_models).model_dump(mode="json")
+        response_data = {"data": marshal(annotation_list, annotation_fields)}
 
         # Create response with secure headers for CSV export
         response = make_response(response_data, 200)
@@ -302,7 +289,7 @@ class AnnotationUpdateDeleteApi(Resource):
     @console_ns.doc("update_delete_annotation")
     @console_ns.doc(description="Update or delete an annotation")
     @console_ns.doc(params={"app_id": "Application ID", "annotation_id": "Annotation ID"})
-    @console_ns.response(200, "Annotation updated successfully", console_ns.models[Annotation.__name__])
+    @console_ns.response(200, "Annotation updated successfully", build_annotation_model(console_ns))
     @console_ns.response(204, "Annotation deleted successfully")
     @console_ns.response(403, "Insufficient permissions")
     @console_ns.expect(console_ns.models[UpdateAnnotationPayload.__name__])
@@ -311,6 +298,7 @@ class AnnotationUpdateDeleteApi(Resource):
     @account_initialization_required
     @cloud_edition_billing_resource_check("annotation")
     @edit_permission_required
+    @marshal_with(annotation_fields)
     def post(self, app_id, annotation_id):
         app_id = str(app_id)
         annotation_id = str(annotation_id)
@@ -318,7 +306,7 @@ class AnnotationUpdateDeleteApi(Resource):
         annotation = AppAnnotationService.update_app_annotation_directly(
             args.model_dump(exclude_none=True), app_id, annotation_id
         )
-        return Annotation.model_validate(annotation, from_attributes=True).model_dump(mode="json")
+        return annotation
 
     @setup_required
     @login_required
@@ -426,7 +414,14 @@ class AnnotationHitHistoryListApi(Resource):
     @console_ns.response(
         200,
         "Hit histories retrieved successfully",
-        console_ns.models[AnnotationHitHistoryList.__name__],
+        console_ns.model(
+            "AnnotationHitHistoryList",
+            {
+                "data": fields.List(
+                    fields.Nested(console_ns.model("AnnotationHitHistoryItem", annotation_hit_history_fields))
+                )
+            },
+        ),
     )
     @console_ns.response(403, "Insufficient permissions")
     @setup_required
@@ -441,14 +436,11 @@ class AnnotationHitHistoryListApi(Resource):
         annotation_hit_history_list, total = AppAnnotationService.get_annotation_hit_histories(
             app_id, annotation_id, page, limit
         )
-        history_models = TypeAdapter(list[AnnotationHitHistory]).validate_python(
-            annotation_hit_history_list, from_attributes=True
-        )
-        response = AnnotationHitHistoryList(
-            data=history_models,
-            has_more=len(annotation_hit_history_list) == limit,
-            limit=limit,
-            total=total,
-            page=page,
-        )
-        return response.model_dump(mode="json")
+        response = {
+            "data": marshal(annotation_hit_history_list, annotation_hit_history_fields),
+            "has_more": len(annotation_hit_history_list) == limit,
+            "limit": limit,
+            "total": total,
+            "page": page,
+        }
+        return response

api/controllers/console/app/app.py

@@ -1,4 +1,3 @@
-import logging
 import uuid
 from datetime import datetime
 from typing import Any, Literal, TypeAlias
@@ -23,10 +22,10 @@ from controllers.console.wraps import (
     is_admin_or_owner_required,
     setup_required,
 )
+from core.file import helpers as file_helpers
 from core.ops.ops_trace_manager import OpsTraceManager
 from core.rag.retrieval.retrieval_methods import RetrievalMethod
 from core.workflow.enums import NodeType, WorkflowExecutionStatus
-from core.workflow.file import helpers as file_helpers
 from extensions.ext_database import db
 from libs.login import current_account_with_tenant, login_required
 from models import App, DatasetPermissionEnum, Workflow
@@ -55,8 +54,6 @@ ALLOW_CREATE_APP_MODES = ["chat", "agent-chat", "advanced-chat", "workflow", "co
 
 register_enum_models(console_ns, IconType)
 
-_logger = logging.getLogger(__name__)
-
 
 class AppListQuery(BaseModel):
     page: int = Field(default=1, ge=1, le=99999, description="Page number (1-99999)")
@@ -502,7 +499,6 @@ class AppListApi(Resource):
                     select(Workflow).where(
                         Workflow.version == Workflow.VERSION_DRAFT,
                         Workflow.app_id.in_(workflow_capable_app_ids),
-                        Workflow.tenant_id == current_tenant_id,
                     )
                 )
                 .scalars()
@@ -514,14 +510,12 @@ class AppListApi(Resource):
                 NodeType.TRIGGER_PLUGIN,
             }
             for workflow in draft_workflows:
-                node_id = None
                 try:
-                    for node_id, node_data in workflow.walk_nodes():
+                    for _, node_data in workflow.walk_nodes():
                         if node_data.get("type") in trigger_node_types:
                             draft_trigger_app_ids.add(str(workflow.app_id))
                             break
                 except Exception:
-                    _logger.exception("error while walking nodes, workflow_id=%s, node_id=%s", workflow.id, node_id)
                     continue
 
         for app in app_pagination.items:
@@ -660,19 +654,6 @@ class AppCopyApi(Resource):
             )
             session.commit()
 
-            # Inherit web app permission from original app
-            if result.app_id and FeatureService.get_system_features().webapp_auth.enabled:
-                try:
-                    # Get the original app's access mode
-                    original_settings = EnterpriseService.WebAppAuth.get_app_access_mode_by_id(app_model.id)
-                    access_mode = original_settings.access_mode
-                except Exception:
-                    # If original app has no settings (old app), default to public to match fallback behavior
-                    access_mode = "public"
-
-                # Apply the same access mode to the copied app
-                EnterpriseService.WebAppAuth.update_app_access_mode(result.app_id, access_mode)
-
             stmt = select(App).where(App.id == result.app_id)
             app = session.scalar(stmt)
 

api/controllers/console/app/audio.py

@@ -6,7 +6,6 @@ from pydantic import BaseModel, Field
 from werkzeug.exceptions import InternalServerError
 
 import services
-from controllers.common.schema import register_schema_models
 from controllers.console import console_ns
 from controllers.console.app.error import (
     AppUnavailableError,
@@ -34,6 +33,7 @@ from services.errors.audio import (
 )
 
 logger = logging.getLogger(__name__)
+DEFAULT_REF_TEMPLATE_SWAGGER_2_0 = "#/definitions/{model}"
 
 
 class TextToSpeechPayload(BaseModel):
@@ -47,11 +47,13 @@ class TextToSpeechVoiceQuery(BaseModel):
     language: str = Field(..., description="Language code")
 
 
-class AudioTranscriptResponse(BaseModel):
-    text: str = Field(description="Transcribed text from audio")
-
-
-register_schema_models(console_ns, AudioTranscriptResponse, TextToSpeechPayload, TextToSpeechVoiceQuery)
+console_ns.schema_model(
+    TextToSpeechPayload.__name__, TextToSpeechPayload.model_json_schema(ref_template=DEFAULT_REF_TEMPLATE_SWAGGER_2_0)
+)
+console_ns.schema_model(
+    TextToSpeechVoiceQuery.__name__,
+    TextToSpeechVoiceQuery.model_json_schema(ref_template=DEFAULT_REF_TEMPLATE_SWAGGER_2_0),
+)
 
 
 @console_ns.route("/apps/<uuid:app_id>/audio-to-text")
@@ -62,7 +64,7 @@ class ChatMessageAudioApi(Resource):
     @console_ns.response(
         200,
         "Audio transcription successful",
-        console_ns.models[AudioTranscriptResponse.__name__],
+        console_ns.model("AudioTranscriptResponse", {"text": fields.String(description="Transcribed text from audio")}),
     )
     @console_ns.response(400, "Bad request - No audio uploaded or unsupported type")
     @console_ns.response(413, "Audio file too large")

api/controllers/console/app/conversation.py

@@ -89,7 +89,6 @@ status_count_model = console_ns.model(
         "success": fields.Integer,
         "failed": fields.Integer,
         "partial_success": fields.Integer,
-        "paused": fields.Integer,
     },
 )
 
@@ -509,19 +508,16 @@ class ChatConversationApi(Resource):
                 case "created_at" | "-created_at" | _:
                     query = query.where(Conversation.created_at <= end_datetime_utc)
 
-        match args.annotation_status:
-            case "annotated":
-                query = query.options(joinedload(Conversation.message_annotations)).join(  # type: ignore
-                    MessageAnnotation, MessageAnnotation.conversation_id == Conversation.id
-                )
-            case "not_annotated":
-                query = (
-                    query.outerjoin(MessageAnnotation, MessageAnnotation.conversation_id == Conversation.id)
-                    .group_by(Conversation.id)
-                    .having(func.count(MessageAnnotation.id) == 0)
-                )
-            case "all":
-                pass
+        if args.annotation_status == "annotated":
+            query = query.options(joinedload(Conversation.message_annotations)).join(  # type: ignore
+                MessageAnnotation, MessageAnnotation.conversation_id == Conversation.id
+            )
+        elif args.annotation_status == "not_annotated":
+            query = (
+                query.outerjoin(MessageAnnotation, MessageAnnotation.conversation_id == Conversation.id)
+                .group_by(Conversation.id)
+                .having(func.count(MessageAnnotation.id) == 0)
+            )
 
         if app_model.mode == AppMode.ADVANCED_CHAT:
             query = query.where(Conversation.invoke_from != InvokeFrom.DEBUGGER)
@@ -599,12 +595,7 @@ def _get_conversation(app_model, conversation_id):
     db.session.execute(
         sa.update(Conversation)
         .where(Conversation.id == conversation_id, Conversation.read_at.is_(None))
-        # Keep updated_at unchanged when only marking a conversation as read.
-        .values(
-            read_at=naive_utc_now(),
-            read_account_id=current_user.id,
-            updated_at=Conversation.updated_at,
-        )
+        .values(read_at=naive_utc_now(), read_account_id=current_user.id)
     )
     db.session.commit()
     db.session.refresh(conversation)

api/controllers/console/app/message.py

@@ -7,7 +7,6 @@ from pydantic import BaseModel, Field, field_validator
 from sqlalchemy import exists, select
 from werkzeug.exceptions import InternalServerError, NotFound
 
-from controllers.common.schema import register_schema_models
 from controllers.console import console_ns
 from controllers.console.app.error import (
     CompletionRequestError,
@@ -33,9 +32,10 @@ from libs.login import current_account_with_tenant, login_required
 from models.model import AppMode, Conversation, Message, MessageAnnotation, MessageFeedback
 from services.errors.conversation import ConversationNotExistsError
 from services.errors.message import MessageNotExistsError, SuggestedQuestionsAfterAnswerDisabledError
-from services.message_service import MessageService, attach_message_extra_contents
+from services.message_service import MessageService
 
 logger = logging.getLogger(__name__)
+DEFAULT_REF_TEMPLATE_SWAGGER_2_0 = "#/definitions/{model}"
 
 
 class ChatMessagesQuery(BaseModel):
@@ -90,22 +90,13 @@ class FeedbackExportQuery(BaseModel):
         raise ValueError("has_comment must be a boolean value")
 
 
-class AnnotationCountResponse(BaseModel):
-    count: int = Field(description="Number of annotations")
+def reg(cls: type[BaseModel]):
+    console_ns.schema_model(cls.__name__, cls.model_json_schema(ref_template=DEFAULT_REF_TEMPLATE_SWAGGER_2_0))
 
 
-class SuggestedQuestionsResponse(BaseModel):
-    data: list[str] = Field(description="Suggested question")
-
-
-register_schema_models(
-    console_ns,
-    ChatMessagesQuery,
-    MessageFeedbackPayload,
-    FeedbackExportQuery,
-    AnnotationCountResponse,
-    SuggestedQuestionsResponse,
-)
+reg(ChatMessagesQuery)
+reg(MessageFeedbackPayload)
+reg(FeedbackExportQuery)
 
 # Register models for flask_restx to avoid dict type issues in Swagger
 # Register in dependency order: base models first, then dependent models
@@ -207,7 +198,6 @@ message_detail_model = console_ns.model(
         "created_at": TimestampField,
         "agent_thoughts": fields.List(fields.Nested(agent_thought_model)),
         "message_files": fields.List(fields.Nested(message_file_model)),
-        "extra_contents": fields.List(fields.Raw),
         "metadata": fields.Raw(attribute="message_metadata_dict"),
         "status": fields.String,
         "error": fields.String,
@@ -241,7 +231,7 @@ class ChatMessageListApi(Resource):
     @marshal_with(message_infinite_scroll_pagination_model)
     @edit_permission_required
     def get(self, app_model):
-        args = ChatMessagesQuery.model_validate(request.args.to_dict())
+        args = ChatMessagesQuery.model_validate(request.args.to_dict(flat=True))  # type: ignore
 
         conversation = (
             db.session.query(Conversation)
@@ -300,7 +290,6 @@ class ChatMessageListApi(Resource):
             has_more = False
 
         history_messages = list(reversed(history_messages))
-        attach_message_extra_contents(history_messages)
 
         return InfiniteScrollPagination(data=history_messages, limit=args.limit, has_more=has_more)
 
@@ -367,7 +356,7 @@ class MessageAnnotationCountApi(Resource):
     @console_ns.response(
         200,
         "Annotation count retrieved successfully",
-        console_ns.models[AnnotationCountResponse.__name__],
+        console_ns.model("AnnotationCountResponse", {"count": fields.Integer(description="Number of annotations")}),
     )
     @get_app_model
     @setup_required
@@ -387,7 +376,9 @@ class MessageSuggestedQuestionApi(Resource):
     @console_ns.response(
         200,
         "Suggested questions retrieved successfully",
-        console_ns.models[SuggestedQuestionsResponse.__name__],
+        console_ns.model(
+            "SuggestedQuestionsResponse", {"data": fields.List(fields.String(description="Suggested question"))}
+        ),
     )
     @console_ns.response(404, "Message or conversation not found")
     @setup_required
@@ -437,7 +428,7 @@ class MessageFeedbackExportApi(Resource):
     @login_required
     @account_initialization_required
     def get(self, app_model):
-        args = FeedbackExportQuery.model_validate(request.args.to_dict())
+        args = FeedbackExportQuery.model_validate(request.args.to_dict(flat=True))  # type: ignore
 
         # Import the service function
         from services.feedback_service import FeedbackService
@@ -483,5 +474,4 @@ class MessageApi(Resource):
         if not message:
             raise NotFound("Message Not Exists.")
 
-        attach_message_extra_contents([message])
         return message

api/controllers/console/app/workflow.py

@@ -20,6 +20,7 @@ from core.app.app_config.features.file_upload.manager import FileUploadConfigMan
 from core.app.apps.base_app_queue_manager import AppQueueManager
 from core.app.apps.workflow.app_generator import SKIP_PREPARE_USER_INPUTS_KEY
 from core.app.entities.app_invoke_entities import InvokeFrom
+from core.file.models import File
 from core.helper.trace_id_helper import get_external_trace_id
 from core.model_runtime.utils.encoders import jsonable_encoder
 from core.plugin.impl.exc import PluginInvokeError
@@ -30,10 +31,8 @@ from core.trigger.debug.event_selectors import (
     select_trigger_debug_events,
 )
 from core.workflow.enums import NodeType
-from core.workflow.file.models import File
 from core.workflow.graph_engine.manager import GraphEngineManager
 from extensions.ext_database import db
-from extensions.ext_redis import redis_client
 from factories import file_factory, variable_factory
 from fields.member_fields import simple_account_fields
 from fields.workflow_fields import workflow_fields, workflow_pagination_fields
@@ -508,179 +507,6 @@ class WorkflowDraftRunLoopNodeApi(Resource):
             raise InternalServerError()
 
 
-class HumanInputFormPreviewPayload(BaseModel):
-    inputs: dict[str, Any] = Field(
-        default_factory=dict,
-        description="Values used to fill missing upstream variables referenced in form_content",
-    )
-
-
-class HumanInputFormSubmitPayload(BaseModel):
-    form_inputs: dict[str, Any] = Field(..., description="Values the user provides for the form's own fields")
-    inputs: dict[str, Any] = Field(
-        ...,
-        description="Values used to fill missing upstream variables referenced in form_content",
-    )
-    action: str = Field(..., description="Selected action ID")
-
-
-class HumanInputDeliveryTestPayload(BaseModel):
-    delivery_method_id: str = Field(..., description="Delivery method ID")
-    inputs: dict[str, Any] = Field(
-        default_factory=dict,
-        description="Values used to fill missing upstream variables referenced in form_content",
-    )
-
-
-reg(HumanInputFormPreviewPayload)
-reg(HumanInputFormSubmitPayload)
-reg(HumanInputDeliveryTestPayload)
-
-
-@console_ns.route("/apps/<uuid:app_id>/advanced-chat/workflows/draft/human-input/nodes/<string:node_id>/form/preview")
-class AdvancedChatDraftHumanInputFormPreviewApi(Resource):
-    @console_ns.doc("get_advanced_chat_draft_human_input_form")
-    @console_ns.doc(description="Get human input form preview for advanced chat workflow")
-    @console_ns.doc(params={"app_id": "Application ID", "node_id": "Node ID"})
-    @console_ns.expect(console_ns.models[HumanInputFormPreviewPayload.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @get_app_model(mode=[AppMode.ADVANCED_CHAT])
-    @edit_permission_required
-    def post(self, app_model: App, node_id: str):
-        """
-        Preview human input form content and placeholders
-        """
-        current_user, _ = current_account_with_tenant()
-        args = HumanInputFormPreviewPayload.model_validate(console_ns.payload or {})
-        inputs = args.inputs
-
-        workflow_service = WorkflowService()
-        preview = workflow_service.get_human_input_form_preview(
-            app_model=app_model,
-            account=current_user,
-            node_id=node_id,
-            inputs=inputs,
-        )
-        return jsonable_encoder(preview)
-
-
-@console_ns.route("/apps/<uuid:app_id>/advanced-chat/workflows/draft/human-input/nodes/<string:node_id>/form/run")
-class AdvancedChatDraftHumanInputFormRunApi(Resource):
-    @console_ns.doc("submit_advanced_chat_draft_human_input_form")
-    @console_ns.doc(description="Submit human input form preview for advanced chat workflow")
-    @console_ns.doc(params={"app_id": "Application ID", "node_id": "Node ID"})
-    @console_ns.expect(console_ns.models[HumanInputFormSubmitPayload.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @get_app_model(mode=[AppMode.ADVANCED_CHAT])
-    @edit_permission_required
-    def post(self, app_model: App, node_id: str):
-        """
-        Submit human input form preview
-        """
-        current_user, _ = current_account_with_tenant()
-        args = HumanInputFormSubmitPayload.model_validate(console_ns.payload or {})
-        workflow_service = WorkflowService()
-        result = workflow_service.submit_human_input_form_preview(
-            app_model=app_model,
-            account=current_user,
-            node_id=node_id,
-            form_inputs=args.form_inputs,
-            inputs=args.inputs,
-            action=args.action,
-        )
-        return jsonable_encoder(result)
-
-
-@console_ns.route("/apps/<uuid:app_id>/workflows/draft/human-input/nodes/<string:node_id>/form/preview")
-class WorkflowDraftHumanInputFormPreviewApi(Resource):
-    @console_ns.doc("get_workflow_draft_human_input_form")
-    @console_ns.doc(description="Get human input form preview for workflow")
-    @console_ns.doc(params={"app_id": "Application ID", "node_id": "Node ID"})
-    @console_ns.expect(console_ns.models[HumanInputFormPreviewPayload.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @get_app_model(mode=[AppMode.WORKFLOW])
-    @edit_permission_required
-    def post(self, app_model: App, node_id: str):
-        """
-        Preview human input form content and placeholders
-        """
-        current_user, _ = current_account_with_tenant()
-        args = HumanInputFormPreviewPayload.model_validate(console_ns.payload or {})
-        inputs = args.inputs
-
-        workflow_service = WorkflowService()
-        preview = workflow_service.get_human_input_form_preview(
-            app_model=app_model,
-            account=current_user,
-            node_id=node_id,
-            inputs=inputs,
-        )
-        return jsonable_encoder(preview)
-
-
-@console_ns.route("/apps/<uuid:app_id>/workflows/draft/human-input/nodes/<string:node_id>/form/run")
-class WorkflowDraftHumanInputFormRunApi(Resource):
-    @console_ns.doc("submit_workflow_draft_human_input_form")
-    @console_ns.doc(description="Submit human input form preview for workflow")
-    @console_ns.doc(params={"app_id": "Application ID", "node_id": "Node ID"})
-    @console_ns.expect(console_ns.models[HumanInputFormSubmitPayload.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @get_app_model(mode=[AppMode.WORKFLOW])
-    @edit_permission_required
-    def post(self, app_model: App, node_id: str):
-        """
-        Submit human input form preview
-        """
-        current_user, _ = current_account_with_tenant()
-        workflow_service = WorkflowService()
-        args = HumanInputFormSubmitPayload.model_validate(console_ns.payload or {})
-        result = workflow_service.submit_human_input_form_preview(
-            app_model=app_model,
-            account=current_user,
-            node_id=node_id,
-            form_inputs=args.form_inputs,
-            inputs=args.inputs,
-            action=args.action,
-        )
-        return jsonable_encoder(result)
-
-
-@console_ns.route("/apps/<uuid:app_id>/workflows/draft/human-input/nodes/<string:node_id>/delivery-test")
-class WorkflowDraftHumanInputDeliveryTestApi(Resource):
-    @console_ns.doc("test_workflow_draft_human_input_delivery")
-    @console_ns.doc(description="Test human input delivery for workflow")
-    @console_ns.doc(params={"app_id": "Application ID", "node_id": "Node ID"})
-    @console_ns.expect(console_ns.models[HumanInputDeliveryTestPayload.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @get_app_model(mode=[AppMode.WORKFLOW, AppMode.ADVANCED_CHAT])
-    @edit_permission_required
-    def post(self, app_model: App, node_id: str):
-        """
-        Test human input delivery
-        """
-        current_user, _ = current_account_with_tenant()
-        workflow_service = WorkflowService()
-        args = HumanInputDeliveryTestPayload.model_validate(console_ns.payload or {})
-        workflow_service.test_human_input_delivery(
-            app_model=app_model,
-            account=current_user,
-            node_id=node_id,
-            delivery_method_id=args.delivery_method_id,
-            inputs=args.inputs,
-        )
-        return jsonable_encoder({})
-
-
 @console_ns.route("/apps/<uuid:app_id>/workflows/draft/run")
 class DraftWorkflowRunApi(Resource):
     @console_ns.doc("run_draft_workflow")
@@ -741,7 +567,7 @@ class WorkflowTaskStopApi(Resource):
         AppQueueManager.set_stop_flag_no_user_check(task_id)
 
         # New graph engine command channel mechanism
-        GraphEngineManager(redis_client).send_stop_command(task_id)
+        GraphEngineManager.send_stop_command(task_id)
 
         return {"result": "success"}
 

api/controllers/console/app/workflow_draft_variable.py

@@ -15,11 +15,11 @@ from controllers.console.app.error import (
 from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import account_initialization_required, edit_permission_required, setup_required
 from controllers.web.error import InvalidArgumentError, NotFoundError
+from core.file import helpers as file_helpers
 from core.variables.segment_group import SegmentGroup
 from core.variables.segments import ArrayFileSegment, FileSegment, Segment
 from core.variables.types import SegmentType
 from core.workflow.constants import CONVERSATION_VARIABLE_NODE_ID, SYSTEM_VARIABLE_NODE_ID
-from core.workflow.file import helpers as file_helpers
 from extensions.ext_database import db
 from factories.file_factory import build_from_mapping, build_from_mappings
 from factories.variable_factory import build_segment_with_type
@@ -112,11 +112,11 @@ _WORKFLOW_DRAFT_VARIABLE_WITHOUT_VALUE_FIELDS = {
     "is_truncated": fields.Boolean(attribute=lambda model: model.file_id is not None),
 }
 
-_WORKFLOW_DRAFT_VARIABLE_FIELDS = {
-    **_WORKFLOW_DRAFT_VARIABLE_WITHOUT_VALUE_FIELDS,
-    "value": fields.Raw(attribute=_serialize_var_value),
-    "full_content": fields.Raw(attribute=_serialize_full_content),
-}
+_WORKFLOW_DRAFT_VARIABLE_FIELDS = dict(
+    _WORKFLOW_DRAFT_VARIABLE_WITHOUT_VALUE_FIELDS,
+    value=fields.Raw(attribute=_serialize_var_value),
+    full_content=fields.Raw(attribute=_serialize_full_content),
+)
 
 _WORKFLOW_DRAFT_ENV_VARIABLE_FIELDS = {
     "id": fields.String,

api/controllers/console/app/workflow_run.py

@@ -5,15 +5,10 @@ from flask import request
 from flask_restx import Resource, fields, marshal_with
 from pydantic import BaseModel, Field, field_validator
 from sqlalchemy import select
-from sqlalchemy.orm import sessionmaker
 
-from configs import dify_config
 from controllers.console import console_ns
 from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import account_initialization_required, setup_required
-from controllers.web.error import NotFoundError
-from core.workflow.entities.pause_reason import HumanInputRequired
-from core.workflow.enums import WorkflowExecutionStatus
 from extensions.ext_database import db
 from fields.end_user_fields import simple_end_user_fields
 from fields.member_fields import simple_account_fields
@@ -32,21 +27,9 @@ from libs.custom_inputs import time_duration
 from libs.helper import uuid_value
 from libs.login import current_user, login_required
 from models import Account, App, AppMode, EndUser, WorkflowArchiveLog, WorkflowRunTriggeredFrom
-from models.workflow import WorkflowRun
-from repositories.factory import DifyAPIRepositoryFactory
 from services.retention.workflow_run.constants import ARCHIVE_BUNDLE_NAME
 from services.workflow_run_service import WorkflowRunService
 
-
-def _build_backstage_input_url(form_token: str | None) -> str | None:
-    if not form_token:
-        return None
-    base_url = dify_config.APP_WEB_URL
-    if not base_url:
-        return None
-    return f"{base_url.rstrip('/')}/form/{form_token}"
-
-
 # Workflow run status choices for filtering
 WORKFLOW_RUN_STATUS_CHOICES = ["running", "succeeded", "failed", "stopped", "partial-succeeded"]
 EXPORT_SIGNED_URL_EXPIRE_SECONDS = 3600
@@ -457,68 +440,3 @@ class WorkflowRunNodeExecutionListApi(Resource):
         )
 
         return {"data": node_executions}
-
-
-@console_ns.route("/workflow/<string:workflow_run_id>/pause-details")
-class ConsoleWorkflowPauseDetailsApi(Resource):
-    """Console API for getting workflow pause details."""
-
-    @setup_required
-    @login_required
-    @account_initialization_required
-    def get(self, workflow_run_id: str):
-        """
-        Get workflow pause details.
-
-        GET /console/api/workflow/<workflow_run_id>/pause-details
-
-        Returns information about why and where the workflow is paused.
-        """
-
-        # Query WorkflowRun to determine if workflow is suspended
-        session_maker = sessionmaker(bind=db.engine)
-        workflow_run_repo = DifyAPIRepositoryFactory.create_api_workflow_run_repository(session_maker=session_maker)
-
-        workflow_run = db.session.get(WorkflowRun, workflow_run_id)
-        if not workflow_run:
-            raise NotFoundError("Workflow run not found")
-
-        if workflow_run.tenant_id != current_user.current_tenant_id:
-            raise NotFoundError("Workflow run not found")
-
-        # Check if workflow is suspended
-        is_paused = workflow_run.status == WorkflowExecutionStatus.PAUSED
-        if not is_paused:
-            return {
-                "paused_at": None,
-                "paused_nodes": [],
-            }, 200
-
-        pause_entity = workflow_run_repo.get_workflow_pause(workflow_run_id)
-        pause_reasons = pause_entity.get_pause_reasons() if pause_entity else []
-
-        # Build response
-        paused_at = pause_entity.paused_at if pause_entity else None
-        paused_nodes = []
-        response = {
-            "paused_at": paused_at.isoformat() + "Z" if paused_at else None,
-            "paused_nodes": paused_nodes,
-        }
-
-        for reason in pause_reasons:
-            if isinstance(reason, HumanInputRequired):
-                paused_nodes.append(
-                    {
-                        "node_id": reason.node_id,
-                        "node_title": reason.node_title,
-                        "pause_type": {
-                            "type": "human_input",
-                            "form_id": reason.form_id,
-                            "backstage_input_url": _build_backstage_input_url(reason.form_token),
-                        },
-                    }
-                )
-            else:
-                raise AssertionError("unimplemented.")
-
-        return response, 200

api/controllers/console/auth/data_source_oauth.py

@@ -2,11 +2,9 @@ import logging
 
 import httpx
 from flask import current_app, redirect, request
-from flask_restx import Resource
-from pydantic import BaseModel, Field
+from flask_restx import Resource, fields
 
 from configs import dify_config
-from controllers.common.schema import register_schema_models
 from libs.login import login_required
 from libs.oauth_data_source import NotionOAuth
 
@@ -16,26 +14,6 @@ from ..wraps import account_initialization_required, is_admin_or_owner_required,
 logger = logging.getLogger(__name__)
 
 
-class OAuthDataSourceResponse(BaseModel):
-    data: str = Field(description="Authorization URL or 'internal' for internal setup")
-
-
-class OAuthDataSourceBindingResponse(BaseModel):
-    result: str = Field(description="Operation result")
-
-
-class OAuthDataSourceSyncResponse(BaseModel):
-    result: str = Field(description="Operation result")
-
-
-register_schema_models(
-    console_ns,
-    OAuthDataSourceResponse,
-    OAuthDataSourceBindingResponse,
-    OAuthDataSourceSyncResponse,
-)
-
-
 def get_oauth_providers():
     with current_app.app_context():
         notion_oauth = NotionOAuth(
@@ -56,7 +34,10 @@ class OAuthDataSource(Resource):
     @console_ns.response(
         200,
         "Authorization URL or internal setup success",
-        console_ns.models[OAuthDataSourceResponse.__name__],
+        console_ns.model(
+            "OAuthDataSourceResponse",
+            {"data": fields.Raw(description="Authorization URL or 'internal' for internal setup")},
+        ),
     )
     @console_ns.response(400, "Invalid provider")
     @console_ns.response(403, "Admin privileges required")
@@ -120,7 +101,7 @@ class OAuthDataSourceBinding(Resource):
     @console_ns.response(
         200,
         "Data source binding success",
-        console_ns.models[OAuthDataSourceBindingResponse.__name__],
+        console_ns.model("OAuthDataSourceBindingResponse", {"result": fields.String(description="Operation result")}),
     )
     @console_ns.response(400, "Invalid provider or code")
     def get(self, provider: str):
@@ -152,7 +133,7 @@ class OAuthDataSourceSync(Resource):
     @console_ns.response(
         200,
         "Data source sync success",
-        console_ns.models[OAuthDataSourceSyncResponse.__name__],
+        console_ns.model("OAuthDataSourceSyncResponse", {"result": fields.String(description="Operation result")}),
     )
     @console_ns.response(400, "Invalid provider or sync failed")
     @setup_required

api/controllers/console/auth/forgot_password.py

@@ -2,11 +2,10 @@ import base64
 import secrets
 
 from flask import request
-from flask_restx import Resource
+from flask_restx import Resource, fields
 from pydantic import BaseModel, Field, field_validator
 from sqlalchemy.orm import Session
 
-from controllers.common.schema import register_schema_models
 from controllers.console import console_ns
 from controllers.console.auth.error import (
     EmailCodeError,
@@ -49,31 +48,8 @@ class ForgotPasswordResetPayload(BaseModel):
         return valid_password(value)
 
 
-class ForgotPasswordEmailResponse(BaseModel):
-    result: str = Field(description="Operation result")
-    data: str | None = Field(default=None, description="Reset token")
-    code: str | None = Field(default=None, description="Error code if account not found")
-
-
-class ForgotPasswordCheckResponse(BaseModel):
-    is_valid: bool = Field(description="Whether code is valid")
-    email: EmailStr = Field(description="Email address")
-    token: str = Field(description="New reset token")
-
-
-class ForgotPasswordResetResponse(BaseModel):
-    result: str = Field(description="Operation result")
-
-
-register_schema_models(
-    console_ns,
-    ForgotPasswordSendPayload,
-    ForgotPasswordCheckPayload,
-    ForgotPasswordResetPayload,
-    ForgotPasswordEmailResponse,
-    ForgotPasswordCheckResponse,
-    ForgotPasswordResetResponse,
-)
+for model in (ForgotPasswordSendPayload, ForgotPasswordCheckPayload, ForgotPasswordResetPayload):
+    console_ns.schema_model(model.__name__, model.model_json_schema(ref_template=DEFAULT_REF_TEMPLATE_SWAGGER_2_0))
 
 
 @console_ns.route("/forgot-password")
@@ -84,7 +60,14 @@ class ForgotPasswordSendEmailApi(Resource):
     @console_ns.response(
         200,
         "Email sent successfully",
-        console_ns.models[ForgotPasswordEmailResponse.__name__],
+        console_ns.model(
+            "ForgotPasswordEmailResponse",
+            {
+                "result": fields.String(description="Operation result"),
+                "data": fields.String(description="Reset token"),
+                "code": fields.String(description="Error code if account not found"),
+            },
+        ),
     )
     @console_ns.response(400, "Invalid email or rate limit exceeded")
     @setup_required
@@ -123,7 +106,14 @@ class ForgotPasswordCheckApi(Resource):
     @console_ns.response(
         200,
         "Code verified successfully",
-        console_ns.models[ForgotPasswordCheckResponse.__name__],
+        console_ns.model(
+            "ForgotPasswordCheckResponse",
+            {
+                "is_valid": fields.Boolean(description="Whether code is valid"),
+                "email": fields.String(description="Email address"),
+                "token": fields.String(description="New reset token"),
+            },
+        ),
     )
     @console_ns.response(400, "Invalid code or token")
     @setup_required
@@ -173,7 +163,7 @@ class ForgotPasswordResetApi(Resource):
     @console_ns.response(
         200,
         "Password reset successfully",
-        console_ns.models[ForgotPasswordResetResponse.__name__],
+        console_ns.model("ForgotPasswordResetResponse", {"result": fields.String(description="Operation result")}),
     )
     @console_ns.response(400, "Invalid token or password mismatch")
     @setup_required

api/controllers/console/auth/oauth_server.py

@@ -155,43 +155,43 @@ class OAuthServerUserTokenApi(Resource):
             grant_type = OAuthGrantType(payload.grant_type)
         except ValueError:
             raise BadRequest("invalid grant_type")
-        match grant_type:
-            case OAuthGrantType.AUTHORIZATION_CODE:
-                if not payload.code:
-                    raise BadRequest("code is required")
 
-                if payload.client_secret != oauth_provider_app.client_secret:
-                    raise BadRequest("client_secret is invalid")
+        if grant_type == OAuthGrantType.AUTHORIZATION_CODE:
+            if not payload.code:
+                raise BadRequest("code is required")
 
-                if payload.redirect_uri not in oauth_provider_app.redirect_uris:
-                    raise BadRequest("redirect_uri is invalid")
+            if payload.client_secret != oauth_provider_app.client_secret:
+                raise BadRequest("client_secret is invalid")
 
-                access_token, refresh_token = OAuthServerService.sign_oauth_access_token(
-                    grant_type, code=payload.code, client_id=oauth_provider_app.client_id
-                )
-                return jsonable_encoder(
-                    {
-                        "access_token": access_token,
-                        "token_type": "Bearer",
-                        "expires_in": OAUTH_ACCESS_TOKEN_EXPIRES_IN,
-                        "refresh_token": refresh_token,
-                    }
-                )
-            case OAuthGrantType.REFRESH_TOKEN:
-                if not payload.refresh_token:
-                    raise BadRequest("refresh_token is required")
+            if payload.redirect_uri not in oauth_provider_app.redirect_uris:
+                raise BadRequest("redirect_uri is invalid")
 
-                access_token, refresh_token = OAuthServerService.sign_oauth_access_token(
-                    grant_type, refresh_token=payload.refresh_token, client_id=oauth_provider_app.client_id
-                )
-                return jsonable_encoder(
-                    {
-                        "access_token": access_token,
-                        "token_type": "Bearer",
-                        "expires_in": OAUTH_ACCESS_TOKEN_EXPIRES_IN,
-                        "refresh_token": refresh_token,
-                    }
-                )
+            access_token, refresh_token = OAuthServerService.sign_oauth_access_token(
+                grant_type, code=payload.code, client_id=oauth_provider_app.client_id
+            )
+            return jsonable_encoder(
+                {
+                    "access_token": access_token,
+                    "token_type": "Bearer",
+                    "expires_in": OAUTH_ACCESS_TOKEN_EXPIRES_IN,
+                    "refresh_token": refresh_token,
+                }
+            )
+        elif grant_type == OAuthGrantType.REFRESH_TOKEN:
+            if not payload.refresh_token:
+                raise BadRequest("refresh_token is required")
+
+            access_token, refresh_token = OAuthServerService.sign_oauth_access_token(
+                grant_type, refresh_token=payload.refresh_token, client_id=oauth_provider_app.client_id
+            )
+            return jsonable_encoder(
+                {
+                    "access_token": access_token,
+                    "token_type": "Bearer",
+                    "expires_in": OAUTH_ACCESS_TOKEN_EXPIRES_IN,
+                    "refresh_token": refresh_token,
+                }
+            )
 
 
 @console_ns.route("/oauth/provider/account")

api/controllers/console/datasets/data_source.py

@@ -1,6 +1,6 @@
 import json
 from collections.abc import Generator
-from typing import Any, Literal, cast
+from typing import Any, cast
 
 from flask import request
 from flask_restx import Resource, fields, marshal_with
@@ -157,8 +157,9 @@ class DataSourceApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    def patch(self, binding_id, action: Literal["enable", "disable"]):
+    def patch(self, binding_id, action):
         binding_id = str(binding_id)
+        action = str(action)
         with Session(db.engine) as session:
             data_source_binding = session.execute(
                 select(DataSourceOauthBinding).filter_by(id=binding_id)
@@ -166,24 +167,23 @@ class DataSourceApi(Resource):
         if data_source_binding is None:
             raise NotFound("Data source binding not found.")
         # enable binding
-        match action:
-            case "enable":
-                if data_source_binding.disabled:
-                    data_source_binding.disabled = False
-                    data_source_binding.updated_at = naive_utc_now()
-                    db.session.add(data_source_binding)
-                    db.session.commit()
-                else:
-                    raise ValueError("Data source is not disabled.")
-            # disable binding
-            case "disable":
-                if not data_source_binding.disabled:
-                    data_source_binding.disabled = True
-                    data_source_binding.updated_at = naive_utc_now()
-                    db.session.add(data_source_binding)
-                    db.session.commit()
-                else:
-                    raise ValueError("Data source is disabled.")
+        if action == "enable":
+            if data_source_binding.disabled:
+                data_source_binding.disabled = False
+                data_source_binding.updated_at = naive_utc_now()
+                db.session.add(data_source_binding)
+                db.session.commit()
+            else:
+                raise ValueError("Data source is not disabled.")
+        # disable binding
+        if action == "disable":
+            if not data_source_binding.disabled:
+                data_source_binding.disabled = True
+                data_source_binding.updated_at = naive_utc_now()
+                db.session.add(data_source_binding)
+                db.session.commit()
+            else:
+                raise ValueError("Data source is disabled.")
         return {"result": "success"}, 200
 
 

api/controllers/console/datasets/datasets.py

@@ -55,7 +55,6 @@ from libs.login import current_account_with_tenant, login_required
 from models import ApiToken, Dataset, Document, DocumentSegment, UploadFile
 from models.dataset import DatasetPermissionEnum
 from models.provider_ids import ModelProviderID
-from services.api_token_service import ApiTokenCache
 from services.dataset_service import DatasetPermissionService, DatasetService, DocumentService
 
 # Register models for flask_restx to avoid dict type issues in Swagger
@@ -821,11 +820,6 @@ class DatasetApiDeleteApi(Resource):
         if key is None:
             console_ns.abort(404, message="API key not found")
 
-        # Invalidate cache before deleting from database
-        # Type assertion: key is guaranteed to be non-None here because abort() raises
-        assert key is not None  # nosec - for type checker only
-        ApiTokenCache.delete(key.token, key.type)
-
         db.session.query(ApiToken).where(ApiToken.id == api_key_id).delete()
         db.session.commit()
 

api/controllers/console/datasets/datasets_document.py

@@ -576,62 +576,63 @@ class DocumentBatchIndexingEstimateApi(DocumentResource):
             if document.indexing_status in {"completed", "error"}:
                 raise DocumentAlreadyFinishedError()
             data_source_info = document.data_source_info_dict
-            match document.data_source_type:
-                case "upload_file":
-                    if not data_source_info:
-                        continue
-                    file_id = data_source_info["upload_file_id"]
-                    file_detail = (
-                        db.session.query(UploadFile)
-                        .where(UploadFile.tenant_id == current_tenant_id, UploadFile.id == file_id)
-                        .first()
-                    )
 
-                    if file_detail is None:
-                        raise NotFound("File not found.")
+            if document.data_source_type == "upload_file":
+                if not data_source_info:
+                    continue
+                file_id = data_source_info["upload_file_id"]
+                file_detail = (
+                    db.session.query(UploadFile)
+                    .where(UploadFile.tenant_id == current_tenant_id, UploadFile.id == file_id)
+                    .first()
+                )
 
-                    extract_setting = ExtractSetting(
-                        datasource_type=DatasourceType.FILE, upload_file=file_detail, document_model=document.doc_form
-                    )
-                    extract_settings.append(extract_setting)
-                case "notion_import":
-                    if not data_source_info:
-                        continue
-                    extract_setting = ExtractSetting(
-                        datasource_type=DatasourceType.NOTION,
-                        notion_info=NotionInfo.model_validate(
-                            {
-                                "credential_id": data_source_info.get("credential_id"),
-                                "notion_workspace_id": data_source_info["notion_workspace_id"],
-                                "notion_obj_id": data_source_info["notion_page_id"],
-                                "notion_page_type": data_source_info["type"],
-                                "tenant_id": current_tenant_id,
-                            }
-                        ),
-                        document_model=document.doc_form,
-                    )
-                    extract_settings.append(extract_setting)
-                case "website_crawl":
-                    if not data_source_info:
-                        continue
-                    extract_setting = ExtractSetting(
-                        datasource_type=DatasourceType.WEBSITE,
-                        website_info=WebsiteInfo.model_validate(
-                            {
-                                "provider": data_source_info["provider"],
-                                "job_id": data_source_info["job_id"],
-                                "url": data_source_info["url"],
-                                "tenant_id": current_tenant_id,
-                                "mode": data_source_info["mode"],
-                                "only_main_content": data_source_info["only_main_content"],
-                            }
-                        ),
-                        document_model=document.doc_form,
-                    )
-                    extract_settings.append(extract_setting)
+                if file_detail is None:
+                    raise NotFound("File not found.")
 
-                case _:
-                    raise ValueError("Data source type not support")
+                extract_setting = ExtractSetting(
+                    datasource_type=DatasourceType.FILE, upload_file=file_detail, document_model=document.doc_form
+                )
+                extract_settings.append(extract_setting)
+
+            elif document.data_source_type == "notion_import":
+                if not data_source_info:
+                    continue
+                extract_setting = ExtractSetting(
+                    datasource_type=DatasourceType.NOTION,
+                    notion_info=NotionInfo.model_validate(
+                        {
+                            "credential_id": data_source_info.get("credential_id"),
+                            "notion_workspace_id": data_source_info["notion_workspace_id"],
+                            "notion_obj_id": data_source_info["notion_page_id"],
+                            "notion_page_type": data_source_info["type"],
+                            "tenant_id": current_tenant_id,
+                        }
+                    ),
+                    document_model=document.doc_form,
+                )
+                extract_settings.append(extract_setting)
+            elif document.data_source_type == "website_crawl":
+                if not data_source_info:
+                    continue
+                extract_setting = ExtractSetting(
+                    datasource_type=DatasourceType.WEBSITE,
+                    website_info=WebsiteInfo.model_validate(
+                        {
+                            "provider": data_source_info["provider"],
+                            "job_id": data_source_info["job_id"],
+                            "url": data_source_info["url"],
+                            "tenant_id": current_tenant_id,
+                            "mode": data_source_info["mode"],
+                            "only_main_content": data_source_info["only_main_content"],
+                        }
+                    ),
+                    document_model=document.doc_form,
+                )
+                extract_settings.append(extract_setting)
+
+            else:
+                raise ValueError("Data source type not support")
             indexing_runner = IndexingRunner()
             try:
                 response = indexing_runner.indexing_estimate(
@@ -953,24 +954,23 @@ class DocumentProcessingApi(DocumentResource):
         if not current_user.is_dataset_editor:
             raise Forbidden()
 
-        match action:
-            case "pause":
-                if document.indexing_status != "indexing":
-                    raise InvalidActionError("Document not in indexing state.")
+        if action == "pause":
+            if document.indexing_status != "indexing":
+                raise InvalidActionError("Document not in indexing state.")
 
-                document.paused_by = current_user.id
-                document.paused_at = naive_utc_now()
-                document.is_paused = True
-                db.session.commit()
+            document.paused_by = current_user.id
+            document.paused_at = naive_utc_now()
+            document.is_paused = True
+            db.session.commit()
 
-            case "resume":
-                if document.indexing_status not in {"paused", "error"}:
-                    raise InvalidActionError("Document not in paused or error state.")
+        elif action == "resume":
+            if document.indexing_status not in {"paused", "error"}:
+                raise InvalidActionError("Document not in paused or error state.")
 
-                document.paused_by = None
-                document.paused_at = None
-                document.is_paused = False
-                db.session.commit()
+            document.paused_by = None
+            document.paused_at = None
+            document.is_paused = False
+            db.session.commit()
 
         return {"result": "success"}, 200
 
@@ -1339,18 +1339,6 @@ class DocumentGenerateSummaryApi(Resource):
             missing_ids = set(document_list) - found_ids
             raise NotFound(f"Some documents not found: {list(missing_ids)}")
 
-        # Update need_summary to True for documents that don't have it set
-        # This handles the case where documents were created when summary_index_setting was disabled
-        documents_to_update = [doc for doc in documents if not doc.need_summary and doc.doc_form != "qa_model"]
-
-        if documents_to_update:
-            document_ids_to_update = [str(doc.id) for doc in documents_to_update]
-            DocumentService.update_documents_need_summary(
-                dataset_id=dataset_id,
-                document_ids=document_ids_to_update,
-                need_summary=True,
-            )
-
         # Dispatch async tasks for each document
         for document in documents:
             # Skip qa_model documents as they don't generate summaries

api/controllers/console/datasets/metadata.py

@@ -126,11 +126,10 @@ class DatasetMetadataBuiltInFieldActionApi(Resource):
             raise NotFound("Dataset not found.")
         DatasetService.check_dataset_permission(dataset, current_user)
 
-        match action:
-            case "enable":
-                MetadataService.enable_built_in_field(dataset)
-            case "disable":
-                MetadataService.disable_built_in_field(dataset)
+        if action == "enable":
+            MetadataService.enable_built_in_field(dataset)
+        elif action == "disable":
+            MetadataService.disable_built_in_field(dataset)
         return {"result": "success"}, 200
 
 

api/controllers/console/explore/trial.py

@@ -1,16 +1,15 @@
 import logging
-from typing import Any, Literal, cast
+from typing import Any, cast
 
 from flask import request
-from flask_restx import Resource, fields, marshal, marshal_with
-from pydantic import BaseModel
+from flask_restx import Resource, fields, marshal, marshal_with, reqparse
 from werkzeug.exceptions import Forbidden, InternalServerError, NotFound
 
 import services
 from controllers.common.fields import Parameters as ParametersResponse
 from controllers.common.fields import Site as SiteResponse
 from controllers.common.schema import get_or_create_model
-from controllers.console import console_ns
+from controllers.console import api, console_ns
 from controllers.console.app.error import (
     AppUnavailableError,
     AudioTooLargeError,
@@ -44,7 +43,6 @@ from core.errors.error import (
 from core.model_runtime.errors.invoke import InvokeError
 from core.workflow.graph_engine.manager import GraphEngineManager
 from extensions.ext_database import db
-from extensions.ext_redis import redis_client
 from fields.app_fields import (
     app_detail_fields_with_site,
     deleted_tool_fields,
@@ -53,7 +51,7 @@ from fields.app_fields import (
     tag_fields,
 )
 from fields.dataset_fields import dataset_fields
-from fields.member_fields import simple_account_fields
+from fields.member_fields import build_simple_account_model
 from fields.workflow_fields import (
     conversation_variable_fields,
     pipeline_variable_fields,
@@ -105,7 +103,7 @@ app_detail_fields_with_site_copy["tags"] = fields.List(fields.Nested(tag_model))
 app_detail_fields_with_site_copy["site"] = fields.Nested(site_model)
 app_detail_with_site_model = get_or_create_model("TrialAppDetailWithSite", app_detail_fields_with_site_copy)
 
-simple_account_model = get_or_create_model("SimpleAccount", simple_account_fields)
+simple_account_model = build_simple_account_model(console_ns)
 conversation_variable_model = get_or_create_model("TrialConversationVariable", conversation_variable_fields)
 pipeline_variable_model = get_or_create_model("TrialPipelineVariable", pipeline_variable_fields)
 
@@ -119,56 +117,7 @@ workflow_fields_copy["rag_pipeline_variables"] = fields.List(fields.Nested(pipel
 workflow_model = get_or_create_model("TrialWorkflow", workflow_fields_copy)
 
 
-# Pydantic models for request validation
-DEFAULT_REF_TEMPLATE_SWAGGER_2_0 = "#/definitions/{model}"
-
-
-class WorkflowRunRequest(BaseModel):
-    inputs: dict
-    files: list | None = None
-
-
-class ChatRequest(BaseModel):
-    inputs: dict
-    query: str
-    files: list | None = None
-    conversation_id: str | None = None
-    parent_message_id: str | None = None
-    retriever_from: str = "explore_app"
-
-
-class TextToSpeechRequest(BaseModel):
-    message_id: str | None = None
-    voice: str | None = None
-    text: str | None = None
-    streaming: bool | None = None
-
-
-class CompletionRequest(BaseModel):
-    inputs: dict
-    query: str = ""
-    files: list | None = None
-    response_mode: Literal["blocking", "streaming"] | None = None
-    retriever_from: str = "explore_app"
-
-
-# Register schemas for Swagger documentation
-console_ns.schema_model(
-    WorkflowRunRequest.__name__, WorkflowRunRequest.model_json_schema(ref_template=DEFAULT_REF_TEMPLATE_SWAGGER_2_0)
-)
-console_ns.schema_model(
-    ChatRequest.__name__, ChatRequest.model_json_schema(ref_template=DEFAULT_REF_TEMPLATE_SWAGGER_2_0)
-)
-console_ns.schema_model(
-    TextToSpeechRequest.__name__, TextToSpeechRequest.model_json_schema(ref_template=DEFAULT_REF_TEMPLATE_SWAGGER_2_0)
-)
-console_ns.schema_model(
-    CompletionRequest.__name__, CompletionRequest.model_json_schema(ref_template=DEFAULT_REF_TEMPLATE_SWAGGER_2_0)
-)
-
-
 class TrialAppWorkflowRunApi(TrialAppResource):
-    @console_ns.expect(console_ns.models[WorkflowRunRequest.__name__])
     def post(self, trial_app):
         """
         Run workflow
@@ -180,8 +129,10 @@ class TrialAppWorkflowRunApi(TrialAppResource):
         if app_mode != AppMode.WORKFLOW:
             raise NotWorkflowAppError()
 
-        request_data = WorkflowRunRequest.model_validate(console_ns.payload)
-        args = request_data.model_dump()
+        parser = reqparse.RequestParser()
+        parser.add_argument("inputs", type=dict, required=True, nullable=False, location="json")
+        parser.add_argument("files", type=list, required=False, location="json")
+        args = parser.parse_args()
         assert current_user is not None
         try:
             app_id = app_model.id
@@ -226,13 +177,12 @@ class TrialAppWorkflowTaskStopApi(TrialAppResource):
         AppQueueManager.set_stop_flag_no_user_check(task_id)
 
         # New graph engine command channel mechanism
-        GraphEngineManager(redis_client).send_stop_command(task_id)
+        GraphEngineManager.send_stop_command(task_id)
 
         return {"result": "success"}
 
 
 class TrialChatApi(TrialAppResource):
-    @console_ns.expect(console_ns.models[ChatRequest.__name__])
     @trial_feature_enable
     def post(self, trial_app):
         app_model = trial_app
@@ -240,14 +190,14 @@ class TrialChatApi(TrialAppResource):
         if app_mode not in {AppMode.CHAT, AppMode.AGENT_CHAT, AppMode.ADVANCED_CHAT}:
             raise NotChatAppError()
 
-        request_data = ChatRequest.model_validate(console_ns.payload)
-        args = request_data.model_dump()
-
-        # Validate UUID values if provided
-        if args.get("conversation_id"):
-            args["conversation_id"] = uuid_value(args["conversation_id"])
-        if args.get("parent_message_id"):
-            args["parent_message_id"] = uuid_value(args["parent_message_id"])
+        parser = reqparse.RequestParser()
+        parser.add_argument("inputs", type=dict, required=True, location="json")
+        parser.add_argument("query", type=str, required=True, location="json")
+        parser.add_argument("files", type=list, required=False, location="json")
+        parser.add_argument("conversation_id", type=uuid_value, location="json")
+        parser.add_argument("parent_message_id", type=uuid_value, required=False, location="json")
+        parser.add_argument("retriever_from", type=str, required=False, default="explore_app", location="json")
+        args = parser.parse_args()
 
         args["auto_generate_name"] = False
 
@@ -370,16 +320,20 @@ class TrialChatAudioApi(TrialAppResource):
 
 
 class TrialChatTextApi(TrialAppResource):
-    @console_ns.expect(console_ns.models[TextToSpeechRequest.__name__])
     @trial_feature_enable
     def post(self, trial_app):
         app_model = trial_app
         try:
-            request_data = TextToSpeechRequest.model_validate(console_ns.payload)
+            parser = reqparse.RequestParser()
+            parser.add_argument("message_id", type=str, required=False, location="json")
+            parser.add_argument("voice", type=str, location="json")
+            parser.add_argument("text", type=str, location="json")
+            parser.add_argument("streaming", type=bool, location="json")
+            args = parser.parse_args()
 
-            message_id = request_data.message_id
-            text = request_data.text
-            voice = request_data.voice
+            message_id = args.get("message_id", None)
+            text = args.get("text", None)
+            voice = args.get("voice", None)
             if not isinstance(current_user, Account):
                 raise ValueError("current_user must be an Account instance")
 
@@ -417,15 +371,19 @@ class TrialChatTextApi(TrialAppResource):
 
 
 class TrialCompletionApi(TrialAppResource):
-    @console_ns.expect(console_ns.models[CompletionRequest.__name__])
     @trial_feature_enable
     def post(self, trial_app):
         app_model = trial_app
         if app_model.mode != "completion":
             raise NotCompletionAppError()
 
-        request_data = CompletionRequest.model_validate(console_ns.payload)
-        args = request_data.model_dump()
+        parser = reqparse.RequestParser()
+        parser.add_argument("inputs", type=dict, required=True, location="json")
+        parser.add_argument("query", type=str, location="json", default="")
+        parser.add_argument("files", type=list, required=False, location="json")
+        parser.add_argument("response_mode", type=str, choices=["blocking", "streaming"], location="json")
+        parser.add_argument("retriever_from", type=str, required=False, default="explore_app", location="json")
+        args = parser.parse_args()
 
         streaming = args["response_mode"] == "streaming"
         args["auto_generate_name"] = False
@@ -470,7 +428,7 @@ class TrialSitApi(Resource):
     """Resource for trial app sites."""
 
     @trial_feature_enable
-    @get_app_model_with_trial(None)
+    @get_app_model_with_trial
     def get(self, app_model):
         """Retrieve app site info.
 
@@ -492,7 +450,7 @@ class TrialAppParameterApi(Resource):
     """Resource for app variables."""
 
     @trial_feature_enable
-    @get_app_model_with_trial(None)
+    @get_app_model_with_trial
     def get(self, app_model):
         """Retrieve app parameters."""
 
@@ -521,7 +479,7 @@ class TrialAppParameterApi(Resource):
 
 class AppApi(Resource):
     @trial_feature_enable
-    @get_app_model_with_trial(None)
+    @get_app_model_with_trial
     @marshal_with(app_detail_with_site_model)
     def get(self, app_model):
         """Get app detail"""
@@ -534,7 +492,7 @@ class AppApi(Resource):
 
 class AppWorkflowApi(Resource):
     @trial_feature_enable
-    @get_app_model_with_trial(None)
+    @get_app_model_with_trial
     @marshal_with(workflow_model)
     def get(self, app_model):
         """Get workflow detail"""
@@ -553,7 +511,7 @@ class AppWorkflowApi(Resource):
 
 class DatasetListApi(Resource):
     @trial_feature_enable
-    @get_app_model_with_trial(None)
+    @get_app_model_with_trial
     def get(self, app_model):
         page = request.args.get("page", default=1, type=int)
         limit = request.args.get("limit", default=20, type=int)
@@ -571,31 +529,27 @@ class DatasetListApi(Resource):
         return response
 
 
-console_ns.add_resource(TrialChatApi, "/trial-apps/<uuid:app_id>/chat-messages", endpoint="trial_app_chat_completion")
+api.add_resource(TrialChatApi, "/trial-apps/<uuid:app_id>/chat-messages", endpoint="trial_app_chat_completion")
 
-console_ns.add_resource(
+api.add_resource(
     TrialMessageSuggestedQuestionApi,
     "/trial-apps/<uuid:app_id>/messages/<uuid:message_id>/suggested-questions",
     endpoint="trial_app_suggested_question",
 )
 
-console_ns.add_resource(TrialChatAudioApi, "/trial-apps/<uuid:app_id>/audio-to-text", endpoint="trial_app_audio")
-console_ns.add_resource(TrialChatTextApi, "/trial-apps/<uuid:app_id>/text-to-audio", endpoint="trial_app_text")
+api.add_resource(TrialChatAudioApi, "/trial-apps/<uuid:app_id>/audio-to-text", endpoint="trial_app_audio")
+api.add_resource(TrialChatTextApi, "/trial-apps/<uuid:app_id>/text-to-audio", endpoint="trial_app_text")
 
-console_ns.add_resource(
-    TrialCompletionApi, "/trial-apps/<uuid:app_id>/completion-messages", endpoint="trial_app_completion"
-)
+api.add_resource(TrialCompletionApi, "/trial-apps/<uuid:app_id>/completion-messages", endpoint="trial_app_completion")
 
-console_ns.add_resource(TrialSitApi, "/trial-apps/<uuid:app_id>/site")
+api.add_resource(TrialSitApi, "/trial-apps/<uuid:app_id>/site")
 
-console_ns.add_resource(TrialAppParameterApi, "/trial-apps/<uuid:app_id>/parameters", endpoint="trial_app_parameters")
+api.add_resource(TrialAppParameterApi, "/trial-apps/<uuid:app_id>/parameters", endpoint="trial_app_parameters")
 
-console_ns.add_resource(AppApi, "/trial-apps/<uuid:app_id>", endpoint="trial_app")
+api.add_resource(AppApi, "/trial-apps/<uuid:app_id>", endpoint="trial_app")
 
-console_ns.add_resource(
-    TrialAppWorkflowRunApi, "/trial-apps/<uuid:app_id>/workflows/run", endpoint="trial_app_workflow_run"
-)
-console_ns.add_resource(TrialAppWorkflowTaskStopApi, "/trial-apps/<uuid:app_id>/workflows/tasks/<string:task_id>/stop")
+api.add_resource(TrialAppWorkflowRunApi, "/trial-apps/<uuid:app_id>/workflows/run", endpoint="trial_app_workflow_run")
+api.add_resource(TrialAppWorkflowTaskStopApi, "/trial-apps/<uuid:app_id>/workflows/tasks/<string:task_id>/stop")
 
-console_ns.add_resource(AppWorkflowApi, "/trial-apps/<uuid:app_id>/workflows", endpoint="trial_app_workflow")
-console_ns.add_resource(DatasetListApi, "/trial-apps/<uuid:app_id>/datasets", endpoint="trial_app_datasets")
+api.add_resource(AppWorkflowApi, "/trial-apps/<uuid:app_id>/workflows", endpoint="trial_app_workflow")
+api.add_resource(DatasetListApi, "/trial-apps/<uuid:app_id>/datasets", endpoint="trial_app_datasets")

api/controllers/console/explore/workflow.py

@@ -23,7 +23,6 @@ from core.errors.error import (
 )
 from core.model_runtime.errors.invoke import InvokeError
 from core.workflow.graph_engine.manager import GraphEngineManager
-from extensions.ext_redis import redis_client
 from libs import helper
 from libs.login import current_account_with_tenant
 from models.model import AppMode, InstalledApp
@@ -101,6 +100,6 @@ class InstalledAppWorkflowTaskStopApi(InstalledAppResource):
         AppQueueManager.set_stop_flag_no_user_check(task_id)
 
         # New graph engine command channel mechanism
-        GraphEngineManager(redis_client).send_stop_command(task_id)
+        GraphEngineManager.send_stop_command(task_id)
 
         return {"result": "success"}

api/controllers/console/explore/wraps.py

@@ -105,9 +105,9 @@ def trial_app_required(view: Callable[Concatenate[App, P], R] | None = None):
     return decorator
 
 
-def trial_feature_enable(view: Callable[P, R]):
+def trial_feature_enable(view: Callable[..., R]) -> Callable[..., R]:
     @wraps(view)
-    def decorated(*args: P.args, **kwargs: P.kwargs):
+    def decorated(*args, **kwargs):
         features = FeatureService.get_system_features()
         if not features.enable_trial_app:
             abort(403, "Trial app feature is not enabled.")
@@ -116,9 +116,9 @@ def trial_feature_enable(view: Callable[P, R]):
     return decorated
 
 
-def explore_banner_enabled(view: Callable[P, R]):
+def explore_banner_enabled(view: Callable[..., R]) -> Callable[..., R]:
     @wraps(view)
-    def decorated(*args: P.args, **kwargs: P.kwargs):
+    def decorated(*args, **kwargs):
         features = FeatureService.get_system_features()
         if not features.enable_explore_banner:
             abort(403, "Explore banner feature is not enabled.")

api/controllers/console/feature.py

@@ -1,60 +1,58 @@
-from flask_restx import Resource, fields
+from pydantic import BaseModel, Field
 from werkzeug.exceptions import Unauthorized
 
+from controllers.fastopenapi import console_router
 from libs.login import current_account_with_tenant, current_user, login_required
-from services.feature_service import FeatureService
+from services.feature_service import FeatureModel, FeatureService, SystemFeatureModel
 
-from . import console_ns
 from .wraps import account_initialization_required, cloud_utm_record, setup_required
 
 
-@console_ns.route("/features")
-class FeatureApi(Resource):
-    @console_ns.doc("get_tenant_features")
-    @console_ns.doc(description="Get feature configuration for current tenant")
-    @console_ns.response(
-        200,
-        "Success",
-        console_ns.model("FeatureResponse", {"features": fields.Raw(description="Feature configuration object")}),
-    )
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @cloud_utm_record
-    def get(self):
-        """Get feature configuration for current tenant"""
-        _, current_tenant_id = current_account_with_tenant()
-
-        return FeatureService.get_features(current_tenant_id).model_dump()
+class FeatureResponse(BaseModel):
+    features: FeatureModel = Field(description="Feature configuration object")
 
 
-@console_ns.route("/system-features")
-class SystemFeatureApi(Resource):
-    @console_ns.doc("get_system_features")
-    @console_ns.doc(description="Get system-wide feature configuration")
-    @console_ns.response(
-        200,
-        "Success",
-        console_ns.model(
-            "SystemFeatureResponse", {"features": fields.Raw(description="System feature configuration object")}
-        ),
-    )
-    def get(self):
-        """Get system-wide feature configuration
+class SystemFeatureResponse(BaseModel):
+    features: SystemFeatureModel = Field(description="System feature configuration object")
 
-        NOTE: This endpoint is unauthenticated by design, as it provides system features
-        data required for dashboard initialization.
 
-        Authentication would create circular dependency (can't login without dashboard loading).
+@console_router.get(
+    "/features",
+    response_model=FeatureResponse,
+    tags=["console"],
+)
+@setup_required
+@login_required
+@account_initialization_required
+@cloud_utm_record
+def get_tenant_features() -> FeatureResponse:
+    """Get feature configuration for current tenant."""
+    _, current_tenant_id = current_account_with_tenant()
 
-        Only non-sensitive configuration data should be returned by this endpoint.
-        """
-        # NOTE(QuantumGhost): ideally we should access `current_user.is_authenticated`
-        # without a try-catch. However, due to the implementation of user loader (the `load_user_from_request`
-        # in api/extensions/ext_login.py), accessing `current_user.is_authenticated` will
-        # raise `Unauthorized` exception if authentication token is not provided.
-        try:
-            is_authenticated = current_user.is_authenticated
-        except Unauthorized:
-            is_authenticated = False
-        return FeatureService.get_system_features(is_authenticated=is_authenticated).model_dump()
+    return FeatureResponse(features=FeatureService.get_features(current_tenant_id))
+
+
+@console_router.get(
+    "/system-features",
+    response_model=SystemFeatureResponse,
+    tags=["console"],
+)
+def get_system_features() -> SystemFeatureResponse:
+    """Get system-wide feature configuration
+
+    NOTE: This endpoint is unauthenticated by design, as it provides system features
+    data required for dashboard initialization.
+
+    Authentication would create circular dependency (can't login without dashboard loading).
+
+    Only non-sensitive configuration data should be returned by this endpoint.
+    """
+    # NOTE(QuantumGhost): ideally we should access `current_user.is_authenticated`
+    # without a try-catch. However, due to the implementation of user loader (the `load_user_from_request`
+    # in api/extensions/ext_login.py), accessing `current_user.is_authenticated` will
+    # raise `Unauthorized` exception if authentication token is not provided.
+    try:
+        is_authenticated = current_user.is_authenticated
+    except Unauthorized:
+        is_authenticated = False
+    return SystemFeatureResponse(features=FeatureService.get_system_features(is_authenticated=is_authenticated))

api/controllers/console/human_input_form.py

@@ -1,217 +0,0 @@
-"""
-Console/Studio Human Input Form APIs.
-"""
-
-import json
-import logging
-from collections.abc import Generator
-
-from flask import Response, jsonify, request
-from flask_restx import Resource, reqparse
-from sqlalchemy import select
-from sqlalchemy.orm import Session, sessionmaker
-
-from controllers.console import console_ns
-from controllers.console.wraps import account_initialization_required, setup_required
-from controllers.web.error import InvalidArgumentError, NotFoundError
-from core.app.apps.advanced_chat.app_generator import AdvancedChatAppGenerator
-from core.app.apps.common.workflow_response_converter import WorkflowResponseConverter
-from core.app.apps.message_generator import MessageGenerator
-from core.app.apps.workflow.app_generator import WorkflowAppGenerator
-from extensions.ext_database import db
-from libs.login import current_account_with_tenant, login_required
-from models import App
-from models.enums import CreatorUserRole
-from models.human_input import RecipientType
-from models.model import AppMode
-from models.workflow import WorkflowRun
-from repositories.factory import DifyAPIRepositoryFactory
-from services.human_input_service import Form, HumanInputService
-from services.workflow_event_snapshot_service import build_workflow_event_stream
-
-logger = logging.getLogger(__name__)
-
-
-def _jsonify_form_definition(form: Form) -> Response:
-    payload = form.get_definition().model_dump()
-    payload["expiration_time"] = int(form.expiration_time.timestamp())
-    return Response(json.dumps(payload, ensure_ascii=False), mimetype="application/json")
-
-
-@console_ns.route("/form/human_input/<string:form_token>")
-class ConsoleHumanInputFormApi(Resource):
-    """Console API for getting human input form definition."""
-
-    @staticmethod
-    def _ensure_console_access(form: Form):
-        _, current_tenant_id = current_account_with_tenant()
-
-        if form.tenant_id != current_tenant_id:
-            raise NotFoundError("App not found")
-
-    @setup_required
-    @login_required
-    @account_initialization_required
-    def get(self, form_token: str):
-        """
-        Get human input form definition by form token.
-
-        GET /console/api/form/human_input/<form_token>
-        """
-        service = HumanInputService(db.engine)
-        form = service.get_form_definition_by_token_for_console(form_token)
-        if form is None:
-            raise NotFoundError(f"form not found, token={form_token}")
-
-        self._ensure_console_access(form)
-
-        return _jsonify_form_definition(form)
-
-    @account_initialization_required
-    @login_required
-    def post(self, form_token: str):
-        """
-        Submit human input form by form token.
-
-        POST /console/api/form/human_input/<form_token>
-
-        Request body:
-        {
-            "inputs": {
-                "content": "User input content"
-            },
-            "action": "Approve"
-        }
-        """
-        parser = reqparse.RequestParser()
-        parser.add_argument("inputs", type=dict, required=True, location="json")
-        parser.add_argument("action", type=str, required=True, location="json")
-        args = parser.parse_args()
-        current_user, _ = current_account_with_tenant()
-
-        service = HumanInputService(db.engine)
-        form = service.get_form_by_token(form_token)
-        if form is None:
-            raise NotFoundError(f"form not found, token={form_token}")
-
-        self._ensure_console_access(form)
-
-        recipient_type = form.recipient_type
-        if recipient_type not in {RecipientType.CONSOLE, RecipientType.BACKSTAGE}:
-            raise NotFoundError(f"form not found, token={form_token}")
-        # The type checker is not smart enought to validate the following invariant.
-        # So we need to assert it manually.
-        assert recipient_type is not None, "recipient_type cannot be None here."
-
-        service.submit_form_by_token(
-            recipient_type=recipient_type,
-            form_token=form_token,
-            selected_action_id=args["action"],
-            form_data=args["inputs"],
-            submission_user_id=current_user.id,
-        )
-
-        return jsonify({})
-
-
-@console_ns.route("/workflow/<string:workflow_run_id>/events")
-class ConsoleWorkflowEventsApi(Resource):
-    """Console API for getting workflow execution events after resume."""
-
-    @account_initialization_required
-    @login_required
-    def get(self, workflow_run_id: str):
-        """
-        Get workflow execution events stream after resume.
-
-        GET /console/api/workflow/<workflow_run_id>/events
-
-        Returns Server-Sent Events stream.
-        """
-
-        user, tenant_id = current_account_with_tenant()
-        session_maker = sessionmaker(db.engine)
-        repo = DifyAPIRepositoryFactory.create_api_workflow_run_repository(session_maker)
-        workflow_run = repo.get_workflow_run_by_id_and_tenant_id(
-            tenant_id=tenant_id,
-            run_id=workflow_run_id,
-        )
-        if workflow_run is None:
-            raise NotFoundError(f"WorkflowRun not found, id={workflow_run_id}")
-
-        if workflow_run.created_by_role != CreatorUserRole.ACCOUNT:
-            raise NotFoundError(f"WorkflowRun not created by account, id={workflow_run_id}")
-
-        if workflow_run.created_by != user.id:
-            raise NotFoundError(f"WorkflowRun not created by the current account, id={workflow_run_id}")
-
-        with Session(expire_on_commit=False, bind=db.engine) as session:
-            app = _retrieve_app_for_workflow_run(session, workflow_run)
-
-        if workflow_run.finished_at is not None:
-            # TODO(QuantumGhost): should we modify the handling for finished workflow run here?
-            response = WorkflowResponseConverter.workflow_run_result_to_finish_response(
-                task_id=workflow_run.id,
-                workflow_run=workflow_run,
-                creator_user=user,
-            )
-
-            payload = response.model_dump(mode="json")
-            payload["event"] = response.event.value
-
-            def _generate_finished_events() -> Generator[str, None, None]:
-                yield f"data: {json.dumps(payload)}\n\n"
-
-            event_generator = _generate_finished_events
-
-        else:
-            msg_generator = MessageGenerator()
-            if app.mode == AppMode.ADVANCED_CHAT:
-                generator = AdvancedChatAppGenerator()
-            elif app.mode == AppMode.WORKFLOW:
-                generator = WorkflowAppGenerator()
-            else:
-                raise InvalidArgumentError(f"cannot subscribe to workflow run, workflow_run_id={workflow_run.id}")
-
-            include_state_snapshot = request.args.get("include_state_snapshot", "false").lower() == "true"
-
-            def _generate_stream_events():
-                if include_state_snapshot:
-                    return generator.convert_to_event_stream(
-                        build_workflow_event_stream(
-                            app_mode=AppMode(app.mode),
-                            workflow_run=workflow_run,
-                            tenant_id=workflow_run.tenant_id,
-                            app_id=workflow_run.app_id,
-                            session_maker=session_maker,
-                        )
-                    )
-                return generator.convert_to_event_stream(
-                    msg_generator.retrieve_events(AppMode(app.mode), workflow_run.id),
-                )
-
-            event_generator = _generate_stream_events
-
-        return Response(
-            event_generator(),
-            mimetype="text/event-stream",
-            headers={
-                "Cache-Control": "no-cache",
-                "Connection": "keep-alive",
-            },
-        )
-
-
-def _retrieve_app_for_workflow_run(session: Session, workflow_run: WorkflowRun):
-    query = select(App).where(
-        App.id == workflow_run.app_id,
-        App.tenant_id == workflow_run.tenant_id,
-    )
-    app = session.scalars(query).first()
-    if app is None:
-        raise AssertionError(
-            f"App not found for WorkflowRun, workflow_run_id={workflow_run.id}, "
-            f"app_id={workflow_run.app_id}, tenant_id={workflow_run.tenant_id}"
-        )
-
-    return app

api/controllers/console/remote_files.py

@@ -1,7 +1,6 @@
 import urllib.parse
 
 import httpx
-from flask_restx import Resource
 from pydantic import BaseModel, Field
 
 import services
@@ -11,12 +10,12 @@ from controllers.common.errors import (
     RemoteFileUploadError,
     UnsupportedFileTypeError,
 )
-from controllers.console import console_ns
+from controllers.fastopenapi import console_router
+from core.file import helpers as file_helpers
 from core.helper import ssrf_proxy
-from core.workflow.file import helpers as file_helpers
 from extensions.ext_database import db
 from fields.file_fields import FileWithSignedUrl, RemoteFileInfo
-from libs.login import current_account_with_tenant, login_required
+from libs.login import current_account_with_tenant
 from services.file_service import FileService
 
 
@@ -24,73 +23,69 @@ class RemoteFileUploadPayload(BaseModel):
     url: str = Field(..., description="URL to fetch")
 
 
-@console_ns.route("/remote-files/<path:url>")
-class GetRemoteFileInfo(Resource):
-    @login_required
-    def get(self, url: str):
-        decoded_url = urllib.parse.unquote(url)
-        resp = ssrf_proxy.head(decoded_url)
+@console_router.get(
+    "/remote-files/<path:url>",
+    response_model=RemoteFileInfo,
+    tags=["console"],
+)
+def get_remote_file_info(url: str) -> RemoteFileInfo:
+    decoded_url = urllib.parse.unquote(url)
+    resp = ssrf_proxy.head(decoded_url)
+    if resp.status_code != httpx.codes.OK:
+        resp = ssrf_proxy.get(decoded_url, timeout=3)
+    resp.raise_for_status()
+    return RemoteFileInfo(
+        file_type=resp.headers.get("Content-Type", "application/octet-stream"),
+        file_length=int(resp.headers.get("Content-Length", 0)),
+    )
+
+
+@console_router.post(
+    "/remote-files/upload",
+    response_model=FileWithSignedUrl,
+    tags=["console"],
+    status_code=201,
+)
+def upload_remote_file(payload: RemoteFileUploadPayload) -> FileWithSignedUrl:
+    url = payload.url
+
+    try:
+        resp = ssrf_proxy.head(url=url)
         if resp.status_code != httpx.codes.OK:
-            resp = ssrf_proxy.get(decoded_url, timeout=3)
-        resp.raise_for_status()
-        return RemoteFileInfo(
-            file_type=resp.headers.get("Content-Type", "application/octet-stream"),
-            file_length=int(resp.headers.get("Content-Length", 0)),
-        ).model_dump(mode="json")
+            resp = ssrf_proxy.get(url=url, timeout=3, follow_redirects=True)
+        if resp.status_code != httpx.codes.OK:
+            raise RemoteFileUploadError(f"Failed to fetch file from {url}: {resp.text}")
+    except httpx.RequestError as e:
+        raise RemoteFileUploadError(f"Failed to fetch file from {url}: {str(e)}")
 
+    file_info = helpers.guess_file_info_from_response(resp)
 
-@console_ns.route("/remote-files/upload")
-class RemoteFileUpload(Resource):
-    @login_required
-    def post(self):
-        payload = RemoteFileUploadPayload.model_validate(console_ns.payload)
-        url = payload.url
+    if not FileService.is_file_size_within_limit(extension=file_info.extension, file_size=file_info.size):
+        raise FileTooLargeError
 
-        # Try to fetch remote file metadata/content first
-        try:
-            resp = ssrf_proxy.head(url=url)
-            if resp.status_code != httpx.codes.OK:
-                resp = ssrf_proxy.get(url=url, timeout=3, follow_redirects=True)
-            if resp.status_code != httpx.codes.OK:
-                # Normalize into a user-friendly error message expected by tests
-                raise RemoteFileUploadError(f"Failed to fetch file from {url}: {resp.text}")
-        except httpx.RequestError as e:
-            raise RemoteFileUploadError(f"Failed to fetch file from {url}: {str(e)}")
+    content = resp.content if resp.request.method == "GET" else ssrf_proxy.get(url).content
 
-        file_info = helpers.guess_file_info_from_response(resp)
-
-        # Enforce file size limit with 400 (Bad Request) per tests' expectation
-        if not FileService.is_file_size_within_limit(extension=file_info.extension, file_size=file_info.size):
-            raise FileTooLargeError()
-
-        # Load content if needed
-        content = resp.content if resp.request.method == "GET" else ssrf_proxy.get(url).content
-
-        try:
-            user, _ = current_account_with_tenant()
-            upload_file = FileService(db.engine).upload_file(
-                filename=file_info.filename,
-                content=content,
-                mimetype=file_info.mimetype,
-                user=user,
-                source_url=url,
-            )
-        except services.errors.file.FileTooLargeError as file_too_large_error:
-            raise FileTooLargeError(file_too_large_error.description)
-        except services.errors.file.UnsupportedFileTypeError:
-            raise UnsupportedFileTypeError()
-
-        # Success: return created resource with 201 status
-        return (
-            FileWithSignedUrl(
-                id=upload_file.id,
-                name=upload_file.name,
-                size=upload_file.size,
-                extension=upload_file.extension,
-                url=file_helpers.get_signed_file_url(upload_file_id=upload_file.id),
-                mime_type=upload_file.mime_type,
-                created_by=upload_file.created_by,
-                created_at=int(upload_file.created_at.timestamp()),
-            ).model_dump(mode="json"),
-            201,
+    try:
+        user, _ = current_account_with_tenant()
+        upload_file = FileService(db.engine).upload_file(
+            filename=file_info.filename,
+            content=content,
+            mimetype=file_info.mimetype,
+            user=user,
+            source_url=url,
         )
+    except services.errors.file.FileTooLargeError as file_too_large_error:
+        raise FileTooLargeError(file_too_large_error.description)
+    except services.errors.file.UnsupportedFileTypeError:
+        raise UnsupportedFileTypeError()
+
+    return FileWithSignedUrl(
+        id=upload_file.id,
+        name=upload_file.name,
+        size=upload_file.size,
+        extension=upload_file.extension,
+        url=file_helpers.get_signed_file_url(upload_file_id=upload_file.id),
+        mime_type=upload_file.mime_type,
+        created_by=upload_file.created_by,
+        created_at=int(upload_file.created_at.timestamp()),
+    )

api/controllers/console/setup.py

@@ -42,15 +42,7 @@ class SetupResponse(BaseModel):
     tags=["console"],
 )
 def get_setup_status_api() -> SetupStatusResponse:
-    """Get system setup status.
-
-    NOTE: This endpoint is unauthenticated by design.
-
-    During first-time bootstrap there is no admin account yet, so frontend initialization must be
-    able to query setup progress before any login flow exists.
-
-    Only bootstrap-safe status information should be returned by this endpoint.
-    """
+    """Get system setup status."""
     if dify_config.EDITION == "SELF_HOSTED":
         setup_status = get_setup_status()
         if setup_status and not isinstance(setup_status, bool):
@@ -69,12 +61,7 @@ def get_setup_status_api() -> SetupStatusResponse:
 )
 @only_edition_self_hosted
 def setup_system(payload: SetupRequestPayload) -> SetupResponse:
-    """Initialize system setup with admin account.
-
-    NOTE: This endpoint is unauthenticated by design for first-time bootstrap.
-    Access is restricted by deployment mode (`SELF_HOSTED`), one-time setup guards,
-    and init-password validation rather than user session authentication.
-    """
+    """Initialize system setup with admin account."""
     if get_setup_status():
         raise AlreadySetupError()
 

api/controllers/console/tag/tags.py

@@ -1,27 +1,14 @@
 from typing import Literal
+from uuid import UUID
 
-from flask import request
-from flask_restx import Namespace, Resource, fields, marshal_with
 from pydantic import BaseModel, Field
 from werkzeug.exceptions import Forbidden
 
-from controllers.common.schema import register_schema_models
-from controllers.console import console_ns
 from controllers.console.wraps import account_initialization_required, edit_permission_required, setup_required
+from controllers.fastopenapi import console_router
 from libs.login import current_account_with_tenant, login_required
 from services.tag_service import TagService
 
-dataset_tag_fields = {
-    "id": fields.String,
-    "name": fields.String,
-    "type": fields.String,
-    "binding_count": fields.String,
-}
-
-
-def build_dataset_tag_fields(api_or_ns: Namespace):
-    return api_or_ns.model("DataSetTag", dataset_tag_fields)
-
 
 class TagBasePayload(BaseModel):
     name: str = Field(description="Tag name", min_length=1, max_length=50)
@@ -45,115 +32,129 @@ class TagListQueryParam(BaseModel):
     keyword: str | None = Field(None, description="Search keyword")
 
 
-register_schema_models(
-    console_ns,
-    TagBasePayload,
-    TagBindingPayload,
-    TagBindingRemovePayload,
-    TagListQueryParam,
+class TagResponse(BaseModel):
+    id: str = Field(description="Tag ID")
+    name: str = Field(description="Tag name")
+    type: str = Field(description="Tag type")
+    binding_count: int = Field(description="Number of bindings")
+
+
+class TagBindingResult(BaseModel):
+    result: Literal["success"] = Field(description="Operation result", examples=["success"])
+
+
+@console_router.get(
+    "/tags",
+    response_model=list[TagResponse],
+    tags=["console"],
 )
+@setup_required
+@login_required
+@account_initialization_required
+def list_tags(query: TagListQueryParam) -> list[TagResponse]:
+    _, current_tenant_id = current_account_with_tenant()
+    tags = TagService.get_tags(query.type, current_tenant_id, query.keyword)
+
+    return [
+        TagResponse(
+            id=tag.id,
+            name=tag.name,
+            type=tag.type,
+            binding_count=int(tag.binding_count),
+        )
+        for tag in tags
+    ]
 
 
-@console_ns.route("/tags")
-class TagListApi(Resource):
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @console_ns.doc(
-        params={"type": 'Tag type filter. Can be "knowledge" or "app".', "keyword": "Search keyword for tag name."}
-    )
-    @marshal_with(dataset_tag_fields)
-    def get(self):
-        _, current_tenant_id = current_account_with_tenant()
-        raw_args = request.args.to_dict()
-        param = TagListQueryParam.model_validate(raw_args)
-        tags = TagService.get_tags(param.type, current_tenant_id, param.keyword)
+@console_router.post(
+    "/tags",
+    response_model=TagResponse,
+    tags=["console"],
+)
+@setup_required
+@login_required
+@account_initialization_required
+def create_tag(payload: TagBasePayload) -> TagResponse:
+    current_user, _ = current_account_with_tenant()
+    # The role of the current user in the tag table must be admin, owner, or editor
+    if not (current_user.has_edit_permission or current_user.is_dataset_editor):
+        raise Forbidden()
 
-        return tags, 200
+    tag = TagService.save_tags(payload.model_dump())
 
-    @console_ns.expect(console_ns.models[TagBasePayload.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    def post(self):
-        current_user, _ = current_account_with_tenant()
-        # The role of the current user in the ta table must be admin, owner, or editor
-        if not (current_user.has_edit_permission or current_user.is_dataset_editor):
-            raise Forbidden()
-
-        payload = TagBasePayload.model_validate(console_ns.payload or {})
-        tag = TagService.save_tags(payload.model_dump())
-
-        response = {"id": tag.id, "name": tag.name, "type": tag.type, "binding_count": 0}
-
-        return response, 200
+    return TagResponse(id=tag.id, name=tag.name, type=tag.type, binding_count=0)
 
 
-@console_ns.route("/tags/<uuid:tag_id>")
-class TagUpdateDeleteApi(Resource):
-    @console_ns.expect(console_ns.models[TagBasePayload.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    def patch(self, tag_id):
-        current_user, _ = current_account_with_tenant()
-        tag_id = str(tag_id)
-        # The role of the current user in the ta table must be admin, owner, or editor
-        if not (current_user.has_edit_permission or current_user.is_dataset_editor):
-            raise Forbidden()
+@console_router.patch(
+    "/tags/<uuid:tag_id>",
+    response_model=TagResponse,
+    tags=["console"],
+)
+@setup_required
+@login_required
+@account_initialization_required
+def update_tag(tag_id: UUID, payload: TagBasePayload) -> TagResponse:
+    current_user, _ = current_account_with_tenant()
+    tag_id_str = str(tag_id)
+    # The role of the current user in the ta table must be admin, owner, or editor
+    if not (current_user.has_edit_permission or current_user.is_dataset_editor):
+        raise Forbidden()
 
-        payload = TagBasePayload.model_validate(console_ns.payload or {})
-        tag = TagService.update_tags(payload.model_dump(), tag_id)
+    tag = TagService.update_tags(payload.model_dump(), tag_id_str)
 
-        binding_count = TagService.get_tag_binding_count(tag_id)
+    binding_count = TagService.get_tag_binding_count(tag_id_str)
 
-        response = {"id": tag.id, "name": tag.name, "type": tag.type, "binding_count": binding_count}
-
-        return response, 200
-
-    @setup_required
-    @login_required
-    @account_initialization_required
-    @edit_permission_required
-    def delete(self, tag_id):
-        tag_id = str(tag_id)
-
-        TagService.delete_tag(tag_id)
-
-        return "", 204
+    return TagResponse(id=tag.id, name=tag.name, type=tag.type, binding_count=binding_count)
 
 
-@console_ns.route("/tag-bindings/create")
-class TagBindingCreateApi(Resource):
-    @console_ns.expect(console_ns.models[TagBindingPayload.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    def post(self):
-        current_user, _ = current_account_with_tenant()
-        # The role of the current user in the ta table must be admin, owner, editor, or dataset_operator
-        if not (current_user.has_edit_permission or current_user.is_dataset_editor):
-            raise Forbidden()
+@console_router.delete(
+    "/tags/<uuid:tag_id>",
+    tags=["console"],
+    status_code=204,
+)
+@setup_required
+@login_required
+@account_initialization_required
+@edit_permission_required
+def delete_tag(tag_id: UUID) -> None:
+    tag_id_str = str(tag_id)
 
-        payload = TagBindingPayload.model_validate(console_ns.payload or {})
-        TagService.save_tag_binding(payload.model_dump())
-
-        return {"result": "success"}, 200
+    TagService.delete_tag(tag_id_str)
 
 
-@console_ns.route("/tag-bindings/remove")
-class TagBindingDeleteApi(Resource):
-    @console_ns.expect(console_ns.models[TagBindingRemovePayload.__name__])
-    @setup_required
-    @login_required
-    @account_initialization_required
-    def post(self):
-        current_user, _ = current_account_with_tenant()
-        # The role of the current user in the ta table must be admin, owner, editor, or dataset_operator
-        if not (current_user.has_edit_permission or current_user.is_dataset_editor):
-            raise Forbidden()
+@console_router.post(
+    "/tag-bindings/create",
+    response_model=TagBindingResult,
+    tags=["console"],
+)
+@setup_required
+@login_required
+@account_initialization_required
+def create_tag_binding(payload: TagBindingPayload) -> TagBindingResult:
+    current_user, _ = current_account_with_tenant()
+    # The role of the current user in the tag table must be admin, owner, editor, or dataset_operator
+    if not (current_user.has_edit_permission or current_user.is_dataset_editor):
+        raise Forbidden()
 
-        payload = TagBindingRemovePayload.model_validate(console_ns.payload or {})
-        TagService.delete_tag_binding(payload.model_dump())
+    TagService.save_tag_binding(payload.model_dump())
 
-        return {"result": "success"}, 200
+    return TagBindingResult(result="success")
+
+
+@console_router.post(
+    "/tag-bindings/remove",
+    response_model=TagBindingResult,
+    tags=["console"],
+)
+@setup_required
+@login_required
+@account_initialization_required
+def delete_tag_binding(payload: TagBindingRemovePayload) -> TagBindingResult:
+    current_user, _ = current_account_with_tenant()
+    # The role of the current user in the tag table must be admin, owner, editor, or dataset_operator
+    if not (current_user.has_edit_permission or current_user.is_dataset_editor):
+        raise Forbidden()
+
+    TagService.delete_tag_binding(payload.model_dump())
+
+    return TagBindingResult(result="success")

api/controllers/console/workspace/account.py

@@ -12,7 +12,6 @@ from sqlalchemy.orm import Session
 
 from configs import dify_config
 from constants.languages import supported_language
-from controllers.common.schema import register_schema_models
 from controllers.console import console_ns
 from controllers.console.auth.error import (
     EmailAlreadyInUseError,
@@ -38,7 +37,7 @@ from controllers.console.wraps import (
     setup_required,
 )
 from extensions.ext_database import db
-from fields.member_fields import Account as AccountResponse
+from fields.member_fields import account_fields
 from libs.datetime_utils import naive_utc_now
 from libs.helper import EmailStr, TimestampField, extract_remote_ip, timezone
 from libs.login import current_account_with_tenant, login_required
@@ -171,12 +170,6 @@ reg(ChangeEmailSendPayload)
 reg(ChangeEmailValidityPayload)
 reg(ChangeEmailResetPayload)
 reg(CheckEmailUniquePayload)
-register_schema_models(console_ns, AccountResponse)
-
-
-def _serialize_account(account) -> dict:
-    return AccountResponse.model_validate(account, from_attributes=True).model_dump(mode="json")
-
 
 integrate_fields = {
     "provider": fields.String,
@@ -243,11 +236,11 @@ class AccountProfileApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountResponse.__name__])
+    @marshal_with(account_fields)
     @enterprise_license_required
     def get(self):
         current_user, _ = current_account_with_tenant()
-        return _serialize_account(current_user)
+        return current_user
 
 
 @console_ns.route("/account/name")
@@ -256,14 +249,14 @@ class AccountNameApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountResponse.__name__])
+    @marshal_with(account_fields)
     def post(self):
         current_user, _ = current_account_with_tenant()
         payload = console_ns.payload or {}
         args = AccountNamePayload.model_validate(payload)
         updated_account = AccountService.update_account(current_user, name=args.name)
 
-        return _serialize_account(updated_account)
+        return updated_account
 
 
 @console_ns.route("/account/avatar")
@@ -272,7 +265,7 @@ class AccountAvatarApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountResponse.__name__])
+    @marshal_with(account_fields)
     def post(self):
         current_user, _ = current_account_with_tenant()
         payload = console_ns.payload or {}
@@ -280,7 +273,7 @@ class AccountAvatarApi(Resource):
 
         updated_account = AccountService.update_account(current_user, avatar=args.avatar)
 
-        return _serialize_account(updated_account)
+        return updated_account
 
 
 @console_ns.route("/account/interface-language")
@@ -289,7 +282,7 @@ class AccountInterfaceLanguageApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountResponse.__name__])
+    @marshal_with(account_fields)
     def post(self):
         current_user, _ = current_account_with_tenant()
         payload = console_ns.payload or {}
@@ -297,7 +290,7 @@ class AccountInterfaceLanguageApi(Resource):
 
         updated_account = AccountService.update_account(current_user, interface_language=args.interface_language)
 
-        return _serialize_account(updated_account)
+        return updated_account
 
 
 @console_ns.route("/account/interface-theme")
@@ -306,7 +299,7 @@ class AccountInterfaceThemeApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountResponse.__name__])
+    @marshal_with(account_fields)
     def post(self):
         current_user, _ = current_account_with_tenant()
         payload = console_ns.payload or {}
@@ -314,7 +307,7 @@ class AccountInterfaceThemeApi(Resource):
 
         updated_account = AccountService.update_account(current_user, interface_theme=args.interface_theme)
 
-        return _serialize_account(updated_account)
+        return updated_account
 
 
 @console_ns.route("/account/timezone")
@@ -323,7 +316,7 @@ class AccountTimezoneApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountResponse.__name__])
+    @marshal_with(account_fields)
     def post(self):
         current_user, _ = current_account_with_tenant()
         payload = console_ns.payload or {}
@@ -331,7 +324,7 @@ class AccountTimezoneApi(Resource):
 
         updated_account = AccountService.update_account(current_user, timezone=args.timezone)
 
-        return _serialize_account(updated_account)
+        return updated_account
 
 
 @console_ns.route("/account/password")
@@ -340,7 +333,7 @@ class AccountPasswordApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountResponse.__name__])
+    @marshal_with(account_fields)
     def post(self):
         current_user, _ = current_account_with_tenant()
         payload = console_ns.payload or {}
@@ -351,7 +344,7 @@ class AccountPasswordApi(Resource):
         except ServiceCurrentPasswordIncorrectError:
             raise CurrentPasswordIncorrectError()
 
-        return _serialize_account(current_user)
+        return {"result": "success"}
 
 
 @console_ns.route("/account/integrates")
@@ -627,7 +620,7 @@ class ChangeEmailResetApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountResponse.__name__])
+    @marshal_with(account_fields)
     def post(self):
         payload = console_ns.payload or {}
         args = ChangeEmailResetPayload.model_validate(payload)
@@ -656,7 +649,7 @@ class ChangeEmailResetApi(Resource):
             email=normalized_new_email,
         )
 
-        return _serialize_account(updated_account)
+        return updated_account
 
 
 @console_ns.route("/account/change-email/check-email-unique")

api/controllers/console/workspace/endpoint.py

@@ -1,10 +1,9 @@
 from typing import Any
 
 from flask import request
-from flask_restx import Resource
+from flask_restx import Resource, fields
 from pydantic import BaseModel, Field
 
-from controllers.common.schema import register_schema_models
 from controllers.console import console_ns
 from controllers.console.wraps import account_initialization_required, is_admin_or_owner_required, setup_required
 from core.model_runtime.utils.encoders import jsonable_encoder
@@ -39,53 +38,15 @@ class EndpointListForPluginQuery(EndpointListQuery):
     plugin_id: str
 
 
-class EndpointCreateResponse(BaseModel):
-    success: bool = Field(description="Operation success")
-
-
-class EndpointListResponse(BaseModel):
-    endpoints: list[dict[str, Any]] = Field(description="Endpoint information")
-
-
-class PluginEndpointListResponse(BaseModel):
-    endpoints: list[dict[str, Any]] = Field(description="Endpoint information")
-
-
-class EndpointDeleteResponse(BaseModel):
-    success: bool = Field(description="Operation success")
-
-
-class EndpointUpdateResponse(BaseModel):
-    success: bool = Field(description="Operation success")
-
-
-class EndpointEnableResponse(BaseModel):
-    success: bool = Field(description="Operation success")
-
-
-class EndpointDisableResponse(BaseModel):
-    success: bool = Field(description="Operation success")
-
-
 def reg(cls: type[BaseModel]):
     console_ns.schema_model(cls.__name__, cls.model_json_schema(ref_template=DEFAULT_REF_TEMPLATE_SWAGGER_2_0))
 
 
-register_schema_models(
-    console_ns,
-    EndpointCreatePayload,
-    EndpointIdPayload,
-    EndpointUpdatePayload,
-    EndpointListQuery,
-    EndpointListForPluginQuery,
-    EndpointCreateResponse,
-    EndpointListResponse,
-    PluginEndpointListResponse,
-    EndpointDeleteResponse,
-    EndpointUpdateResponse,
-    EndpointEnableResponse,
-    EndpointDisableResponse,
-)
+reg(EndpointCreatePayload)
+reg(EndpointIdPayload)
+reg(EndpointUpdatePayload)
+reg(EndpointListQuery)
+reg(EndpointListForPluginQuery)
 
 
 @console_ns.route("/workspaces/current/endpoints/create")
@@ -96,7 +57,7 @@ class EndpointCreateApi(Resource):
     @console_ns.response(
         200,
         "Endpoint created successfully",
-        console_ns.models[EndpointCreateResponse.__name__],
+        console_ns.model("EndpointCreateResponse", {"success": fields.Boolean(description="Operation success")}),
     )
     @console_ns.response(403, "Admin privileges required")
     @setup_required
@@ -130,7 +91,9 @@ class EndpointListApi(Resource):
     @console_ns.response(
         200,
         "Success",
-        console_ns.models[EndpointListResponse.__name__],
+        console_ns.model(
+            "EndpointListResponse", {"endpoints": fields.List(fields.Raw(description="Endpoint information"))}
+        ),
     )
     @setup_required
     @login_required
@@ -163,7 +126,9 @@ class EndpointListForSinglePluginApi(Resource):
     @console_ns.response(
         200,
         "Success",
-        console_ns.models[PluginEndpointListResponse.__name__],
+        console_ns.model(
+            "PluginEndpointListResponse", {"endpoints": fields.List(fields.Raw(description="Endpoint information"))}
+        ),
     )
     @setup_required
     @login_required
@@ -198,7 +163,7 @@ class EndpointDeleteApi(Resource):
     @console_ns.response(
         200,
         "Endpoint deleted successfully",
-        console_ns.models[EndpointDeleteResponse.__name__],
+        console_ns.model("EndpointDeleteResponse", {"success": fields.Boolean(description="Operation success")}),
     )
     @console_ns.response(403, "Admin privileges required")
     @setup_required
@@ -225,7 +190,7 @@ class EndpointUpdateApi(Resource):
     @console_ns.response(
         200,
         "Endpoint updated successfully",
-        console_ns.models[EndpointUpdateResponse.__name__],
+        console_ns.model("EndpointUpdateResponse", {"success": fields.Boolean(description="Operation success")}),
     )
     @console_ns.response(403, "Admin privileges required")
     @setup_required
@@ -256,7 +221,7 @@ class EndpointEnableApi(Resource):
     @console_ns.response(
         200,
         "Endpoint enabled successfully",
-        console_ns.models[EndpointEnableResponse.__name__],
+        console_ns.model("EndpointEnableResponse", {"success": fields.Boolean(description="Operation success")}),
     )
     @console_ns.response(403, "Admin privileges required")
     @setup_required
@@ -283,7 +248,7 @@ class EndpointDisableApi(Resource):
     @console_ns.response(
         200,
         "Endpoint disabled successfully",
-        console_ns.models[EndpointDisableResponse.__name__],
+        console_ns.model("EndpointDisableResponse", {"success": fields.Boolean(description="Operation success")}),
     )
     @console_ns.response(403, "Admin privileges required")
     @setup_required

api/controllers/console/workspace/members.py

@@ -1,12 +1,12 @@
 from urllib import parse
 
 from flask import abort, request
-from flask_restx import Resource
-from pydantic import BaseModel, Field, TypeAdapter
+from flask_restx import Resource, fields, marshal_with
+from pydantic import BaseModel, Field
 
 import services
 from configs import dify_config
-from controllers.common.schema import register_enum_models, register_schema_models
+from controllers.common.schema import get_or_create_model, register_enum_models
 from controllers.console import console_ns
 from controllers.console.auth.error import (
     CannotTransferOwnerToSelfError,
@@ -25,7 +25,7 @@ from controllers.console.wraps import (
     setup_required,
 )
 from extensions.ext_database import db
-from fields.member_fields import AccountWithRole, AccountWithRoleList
+from fields.member_fields import account_with_role_fields, account_with_role_list_fields
 from libs.helper import extract_remote_ip
 from libs.login import current_account_with_tenant, login_required
 from models.account import Account, TenantAccountRole
@@ -69,7 +69,12 @@ reg(OwnerTransferEmailPayload)
 reg(OwnerTransferCheckPayload)
 reg(OwnerTransferPayload)
 register_enum_models(console_ns, TenantAccountRole)
-register_schema_models(console_ns, AccountWithRole, AccountWithRoleList)
+
+account_with_role_model = get_or_create_model("AccountWithRole", account_with_role_fields)
+
+account_with_role_list_fields_copy = account_with_role_list_fields.copy()
+account_with_role_list_fields_copy["accounts"] = fields.List(fields.Nested(account_with_role_model))
+account_with_role_list_model = get_or_create_model("AccountWithRoleList", account_with_role_list_fields_copy)
 
 
 @console_ns.route("/workspaces/current/members")
@@ -79,15 +84,13 @@ class MemberListApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountWithRoleList.__name__])
+    @marshal_with(account_with_role_list_model)
     def get(self):
         current_user, _ = current_account_with_tenant()
         if not current_user.current_tenant:
             raise ValueError("No current tenant")
         members = TenantService.get_tenant_members(current_user.current_tenant)
-        member_models = TypeAdapter(list[AccountWithRole]).validate_python(members, from_attributes=True)
-        response = AccountWithRoleList(accounts=member_models)
-        return response.model_dump(mode="json"), 200
+        return {"result": "success", "accounts": members}, 200
 
 
 @console_ns.route("/workspaces/current/members/invite-email")
@@ -232,15 +235,13 @@ class DatasetOperatorMemberListApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @console_ns.response(200, "Success", console_ns.models[AccountWithRoleList.__name__])
+    @marshal_with(account_with_role_list_model)
     def get(self):
         current_user, _ = current_account_with_tenant()
         if not current_user.current_tenant:
             raise ValueError("No current tenant")
         members = TenantService.get_dataset_operator_members(current_user.current_tenant)
-        member_models = TypeAdapter(list[AccountWithRole]).validate_python(members, from_attributes=True)
-        response = AccountWithRoleList(accounts=member_models)
-        return response.model_dump(mode="json"), 200
+        return {"result": "success", "accounts": members}, 200
 
 
 @console_ns.route("/workspaces/current/members/send-owner-transfer-confirm-email")

api/controllers/files/image_preview.py

@@ -137,7 +137,7 @@ class FilePreviewApi(Resource):
         if args.as_attachment:
             encoded_filename = quote(upload_file.name)
             response.headers["Content-Disposition"] = f"attachment; filename*=UTF-8''{encoded_filename}"
-        response.headers["Content-Type"] = "application/octet-stream"
+            response.headers["Content-Type"] = "application/octet-stream"
 
         enforce_download_for_html(
             response,

api/controllers/files/tool_files.py

@@ -64,10 +64,6 @@ class ToolFileApi(Resource):
 
             if not stream or not tool_file:
                 raise NotFound("file is not found")
-
-        except NotFound:
-            raise
-
         except Exception:
             raise UnsupportedFileTypeError()
 

api/controllers/files/upload.py

@@ -7,8 +7,8 @@ from pydantic import BaseModel, Field
 from werkzeug.exceptions import Forbidden
 
 import services
+from core.file.helpers import verify_plugin_file_signature
 from core.tools.tool_file_manager import ToolFileManager
-from core.workflow.file.helpers import verify_plugin_file_signature
 from fields.file_fields import FileResponse
 
 from ..common.errors import (

api/controllers/inner_api/plugin/plugin.py

@@ -4,6 +4,7 @@ from controllers.console.wraps import setup_required
 from controllers.inner_api import inner_api_ns
 from controllers.inner_api.plugin.wraps import get_user_tenant, plugin_data
 from controllers.inner_api.wraps import plugin_inner_api_only
+from core.file.helpers import get_signed_file_url_for_plugin
 from core.model_runtime.utils.encoders import jsonable_encoder
 from core.plugin.backwards_invocation.app import PluginAppBackwardsInvocation
 from core.plugin.backwards_invocation.base import BaseBackwardsInvocationResponse
@@ -29,7 +30,6 @@ from core.plugin.entities.request import (
     RequestRequestUploadFile,
 )
 from core.tools.entities.tool_entities import ToolProviderType
-from core.workflow.file.helpers import get_signed_file_url_for_plugin
 from libs.helper import length_prefixed_response
 from models import Account, Tenant
 from models.model import EndUser

api/controllers/service_api/__init__.py

@@ -34,8 +34,6 @@ from .dataset import (
     metadata,
     segment,
 )
-from .dataset.rag_pipeline import rag_pipeline_workflow
-from .end_user import end_user
 from .workspace import models
 
 __all__ = [
@@ -46,7 +44,6 @@ __all__ = [
     "conversation",
     "dataset",
     "document",
-    "end_user",
     "file",
     "file_preview",
     "hit_testing",
@@ -54,7 +51,6 @@ __all__ = [
     "message",
     "metadata",
     "models",
-    "rag_pipeline_workflow",
     "segment",
     "site",
     "workflow",

api/controllers/service_api/app/annotation.py

@@ -1,16 +1,16 @@
 from typing import Literal
 
 from flask import request
-from flask_restx import Resource
+from flask_restx import Namespace, Resource, fields
 from flask_restx.api import HTTPStatus
-from pydantic import BaseModel, Field, TypeAdapter
+from pydantic import BaseModel, Field
 
 from controllers.common.schema import register_schema_models
 from controllers.console.wraps import edit_permission_required
 from controllers.service_api import service_api_ns
 from controllers.service_api.wraps import validate_app_token
 from extensions.ext_redis import redis_client
-from fields.annotation_fields import Annotation, AnnotationList
+from fields.annotation_fields import annotation_fields, build_annotation_model
 from models.model import App
 from services.annotation_service import AppAnnotationService
 
@@ -26,9 +26,7 @@ class AnnotationReplyActionPayload(BaseModel):
     embedding_model_name: str = Field(description="Embedding model name")
 
 
-register_schema_models(
-    service_api_ns, AnnotationCreatePayload, AnnotationReplyActionPayload, Annotation, AnnotationList
-)
+register_schema_models(service_api_ns, AnnotationCreatePayload, AnnotationReplyActionPayload)
 
 
 @service_api_ns.route("/apps/annotation-reply/<string:action>")
@@ -47,11 +45,10 @@ class AnnotationReplyActionApi(Resource):
     def post(self, app_model: App, action: Literal["enable", "disable"]):
         """Enable or disable annotation reply feature."""
         args = AnnotationReplyActionPayload.model_validate(service_api_ns.payload or {}).model_dump()
-        match action:
-            case "enable":
-                result = AppAnnotationService.enable_app_annotation(args, app_model.id)
-            case "disable":
-                result = AppAnnotationService.disable_app_annotation(app_model.id)
+        if action == "enable":
+            result = AppAnnotationService.enable_app_annotation(args, app_model.id)
+        elif action == "disable":
+            result = AppAnnotationService.disable_app_annotation(app_model.id)
         return result, 200
 
 
@@ -85,6 +82,23 @@ class AnnotationReplyActionStatusApi(Resource):
         return {"job_id": job_id, "job_status": job_status, "error_msg": error_msg}, 200
 
 
+# Define annotation list response model
+annotation_list_fields = {
+    "data": fields.List(fields.Nested(annotation_fields)),
+    "has_more": fields.Boolean,
+    "limit": fields.Integer,
+    "total": fields.Integer,
+    "page": fields.Integer,
+}
+
+
+def build_annotation_list_model(api_or_ns: Namespace):
+    """Build the annotation list model for the API or Namespace."""
+    copied_annotation_list_fields = annotation_list_fields.copy()
+    copied_annotation_list_fields["data"] = fields.List(fields.Nested(build_annotation_model(api_or_ns)))
+    return api_or_ns.model("AnnotationList", copied_annotation_list_fields)
+
+
 @service_api_ns.route("/apps/annotations")
 class AnnotationListApi(Resource):
     @service_api_ns.doc("list_annotations")
@@ -95,12 +109,8 @@ class AnnotationListApi(Resource):
             401: "Unauthorized - invalid API token",
         }
     )
-    @service_api_ns.response(
-        200,
-        "Annotations retrieved successfully",
-        service_api_ns.models[AnnotationList.__name__],
-    )
     @validate_app_token
+    @service_api_ns.marshal_with(build_annotation_list_model(service_api_ns))
     def get(self, app_model: App):
         """List annotations for the application."""
         page = request.args.get("page", default=1, type=int)
@@ -108,15 +118,13 @@ class AnnotationListApi(Resource):
         keyword = request.args.get("keyword", default="", type=str)
 
         annotation_list, total = AppAnnotationService.get_annotation_list_by_app_id(app_model.id, page, limit, keyword)
-        annotation_models = TypeAdapter(list[Annotation]).validate_python(annotation_list, from_attributes=True)
-        response = AnnotationList(
-            data=annotation_models,
-            has_more=len(annotation_list) == limit,
-            limit=limit,
-            total=total,
-            page=page,
-        )
-        return response.model_dump(mode="json")
+        return {
+            "data": annotation_list,
+            "has_more": len(annotation_list) == limit,
+            "limit": limit,
+            "total": total,
+            "page": page,
+        }
 
     @service_api_ns.expect(service_api_ns.models[AnnotationCreatePayload.__name__])
     @service_api_ns.doc("create_annotation")
@@ -127,18 +135,13 @@ class AnnotationListApi(Resource):
             401: "Unauthorized - invalid API token",
         }
     )
-    @service_api_ns.response(
-        HTTPStatus.CREATED,
-        "Annotation created successfully",
-        service_api_ns.models[Annotation.__name__],
-    )
     @validate_app_token
+    @service_api_ns.marshal_with(build_annotation_model(service_api_ns), code=HTTPStatus.CREATED)
     def post(self, app_model: App):
         """Create a new annotation."""
         args = AnnotationCreatePayload.model_validate(service_api_ns.payload or {}).model_dump()
         annotation = AppAnnotationService.insert_app_annotation_directly(args, app_model.id)
-        response = Annotation.model_validate(annotation, from_attributes=True)
-        return response.model_dump(mode="json"), HTTPStatus.CREATED
+        return annotation, 201
 
 
 @service_api_ns.route("/apps/annotations/<uuid:annotation_id>")
@@ -155,19 +158,14 @@ class AnnotationUpdateDeleteApi(Resource):
             404: "Annotation not found",
         }
     )
-    @service_api_ns.response(
-        200,
-        "Annotation updated successfully",
-        service_api_ns.models[Annotation.__name__],
-    )
     @validate_app_token
     @edit_permission_required
+    @service_api_ns.marshal_with(build_annotation_model(service_api_ns))
     def put(self, app_model: App, annotation_id: str):
         """Update an existing annotation."""
         args = AnnotationCreatePayload.model_validate(service_api_ns.payload or {}).model_dump()
         annotation = AppAnnotationService.update_app_annotation_directly(args, app_model.id, annotation_id)
-        response = Annotation.model_validate(annotation, from_attributes=True)
-        return response.model_dump(mode="json")
+        return annotation
 
     @service_api_ns.doc("delete_annotation")
     @service_api_ns.doc(description="Delete an annotation")

api/controllers/service_api/app/workflow.py

@@ -31,12 +31,10 @@ from core.model_runtime.errors.invoke import InvokeError
 from core.workflow.enums import WorkflowExecutionStatus
 from core.workflow.graph_engine.manager import GraphEngineManager
 from extensions.ext_database import db
-from extensions.ext_redis import redis_client
 from fields.workflow_app_log_fields import build_workflow_app_log_pagination_model
 from libs import helper
-from libs.helper import OptionalTimestampField, TimestampField
+from libs.helper import TimestampField
 from models.model import App, AppMode, EndUser
-from models.workflow import WorkflowRun
 from repositories.factory import DifyAPIRepositoryFactory
 from services.app_generate_service import AppGenerateService
 from services.errors.app import IsDraftWorkflowError, WorkflowIdFormatError, WorkflowNotFoundError
@@ -65,32 +63,17 @@ class WorkflowLogQuery(BaseModel):
 
 register_schema_models(service_api_ns, WorkflowRunPayload, WorkflowLogQuery)
 
-
-class WorkflowRunStatusField(fields.Raw):
-    def output(self, key, obj: WorkflowRun, **kwargs):
-        return obj.status.value
-
-
-class WorkflowRunOutputsField(fields.Raw):
-    def output(self, key, obj: WorkflowRun, **kwargs):
-        if obj.status == WorkflowExecutionStatus.PAUSED:
-            return {}
-
-        outputs = obj.outputs_dict
-        return outputs or {}
-
-
 workflow_run_fields = {
     "id": fields.String,
     "workflow_id": fields.String,
-    "status": WorkflowRunStatusField,
+    "status": fields.String,
     "inputs": fields.Raw,
-    "outputs": WorkflowRunOutputsField,
+    "outputs": fields.Raw,
     "error": fields.String,
     "total_steps": fields.Integer,
     "total_tokens": fields.Integer,
     "created_at": TimestampField,
-    "finished_at": OptionalTimestampField,
+    "finished_at": TimestampField,
     "elapsed_time": fields.Float,
 }
 
@@ -281,7 +264,7 @@ class WorkflowTaskStopApi(Resource):
         AppQueueManager.set_stop_flag_no_user_check(task_id)
 
         # New graph engine command channel mechanism
-        GraphEngineManager(redis_client).send_stop_command(task_id)
+        GraphEngineManager.send_stop_command(task_id)
 
         return {"result": "success"}
 

api/controllers/service_api/dataset/dataset.py

@@ -17,7 +17,7 @@ from controllers.service_api.wraps import (
 from core.model_runtime.entities.model_entities import ModelType
 from core.provider_manager import ProviderManager
 from fields.dataset_fields import dataset_detail_fields
-from fields.tag_fields import DataSetTag
+from fields.tag_fields import build_dataset_tag_fields
 from libs.login import current_user
 from models.account import Account
 from models.dataset import DatasetPermissionEnum
@@ -114,7 +114,6 @@ register_schema_models(
     TagBindingPayload,
     TagUnbindingPayload,
     DatasetListQuery,
-    DataSetTag,
 )
 
 
@@ -396,7 +395,7 @@ class DatasetApi(DatasetApiResource):
         try:
             if DatasetService.delete_dataset(dataset_id_str, current_user):
                 DatasetPermissionService.clear_partial_member_list(dataset_id_str)
-                return "", 204
+                return 204
             else:
                 raise NotFound("Dataset not found.")
         except services.errors.dataset.DatasetInUseError:
@@ -481,14 +480,15 @@ class DatasetTagsApi(DatasetApiResource):
             401: "Unauthorized - invalid API token",
         }
     )
+    @service_api_ns.marshal_with(build_dataset_tag_fields(service_api_ns))
     def get(self, _):
         """Get all knowledge type tags."""
         assert isinstance(current_user, Account)
         cid = current_user.current_tenant_id
         assert cid is not None
         tags = TagService.get_tags("knowledge", cid)
-        tag_models = TypeAdapter(list[DataSetTag]).validate_python(tags, from_attributes=True)
-        return [tag.model_dump(mode="json") for tag in tag_models], 200
+
+        return tags, 200
 
     @service_api_ns.expect(service_api_ns.models[TagCreatePayload.__name__])
     @service_api_ns.doc("create_dataset_tag")
@@ -500,6 +500,7 @@ class DatasetTagsApi(DatasetApiResource):
             403: "Forbidden - insufficient permissions",
         }
     )
+    @service_api_ns.marshal_with(build_dataset_tag_fields(service_api_ns))
     def post(self, _):
         """Add a knowledge type tag."""
         assert isinstance(current_user, Account)
@@ -509,9 +510,7 @@ class DatasetTagsApi(DatasetApiResource):
         payload = TagCreatePayload.model_validate(service_api_ns.payload or {})
         tag = TagService.save_tags({"name": payload.name, "type": "knowledge"})
 
-        response = DataSetTag.model_validate(
-            {"id": tag.id, "name": tag.name, "type": tag.type, "binding_count": 0}
-        ).model_dump(mode="json")
+        response = {"id": tag.id, "name": tag.name, "type": tag.type, "binding_count": 0}
         return response, 200
 
     @service_api_ns.expect(service_api_ns.models[TagUpdatePayload.__name__])
@@ -524,6 +523,7 @@ class DatasetTagsApi(DatasetApiResource):
             403: "Forbidden - insufficient permissions",
         }
     )
+    @service_api_ns.marshal_with(build_dataset_tag_fields(service_api_ns))
     def patch(self, _):
         assert isinstance(current_user, Account)
         if not (current_user.has_edit_permission or current_user.is_dataset_editor):
@@ -536,9 +536,8 @@ class DatasetTagsApi(DatasetApiResource):
 
         binding_count = TagService.get_tag_binding_count(tag_id)
 
-        response = DataSetTag.model_validate(
-            {"id": tag.id, "name": tag.name, "type": tag.type, "binding_count": binding_count}
-        ).model_dump(mode="json")
+        response = {"id": tag.id, "name": tag.name, "type": tag.type, "binding_count": binding_count}
+
         return response, 200
 
     @service_api_ns.expect(service_api_ns.models[TagDeletePayload.__name__])
@@ -557,7 +556,7 @@ class DatasetTagsApi(DatasetApiResource):
         payload = TagDeletePayload.model_validate(service_api_ns.payload or {})
         TagService.delete_tag(payload.tag_id)
 
-        return "", 204
+        return 204
 
 
 @service_api_ns.route("/datasets/tags/binding")
@@ -581,7 +580,7 @@ class DatasetTagBindingApi(DatasetApiResource):
         payload = TagBindingPayload.model_validate(service_api_ns.payload or {})
         TagService.save_tag_binding({"tag_ids": payload.tag_ids, "target_id": payload.target_id, "type": "knowledge"})
 
-        return "", 204
+        return 204
 
 
 @service_api_ns.route("/datasets/tags/unbinding")
@@ -605,7 +604,7 @@ class DatasetTagUnbindingApi(DatasetApiResource):
         payload = TagUnbindingPayload.model_validate(service_api_ns.payload or {})
         TagService.delete_tag_binding({"tag_id": payload.tag_id, "target_id": payload.target_id, "type": "knowledge"})
 
-        return "", 204
+        return 204
 
 
 @service_api_ns.route("/datasets/<uuid:dataset_id>/tags")

api/controllers/service_api/dataset/document.py

@@ -746,4 +746,4 @@ class DocumentApi(DatasetApiResource):
         except services.errors.document.DocumentIndexingError:
             raise DocumentIndexingError("Cannot delete document during indexing.")
 
-        return "", 204
+        return 204

api/controllers/service_api/dataset/metadata.py

@@ -128,7 +128,7 @@ class DatasetMetadataServiceApi(DatasetApiResource):
         DatasetService.check_dataset_permission(dataset, current_user)
 
         MetadataService.delete_metadata(dataset_id_str, metadata_id_str)
-        return "", 204
+        return 204
 
 
 @service_api_ns.route("/datasets/<uuid:dataset_id>/metadata/built-in")
@@ -168,11 +168,10 @@ class DatasetMetadataBuiltInFieldActionServiceApi(DatasetApiResource):
             raise NotFound("Dataset not found.")
         DatasetService.check_dataset_permission(dataset, current_user)
 
-        match action:
-            case "enable":
-                MetadataService.enable_built_in_field(dataset)
-            case "disable":
-                MetadataService.disable_built_in_field(dataset)
+        if action == "enable":
+            MetadataService.enable_built_in_field(dataset)
+        elif action == "disable":
+            MetadataService.disable_built_in_field(dataset)
         return {"result": "success"}, 200
 
 

api/controllers/service_api/dataset/rag_pipeline/rag_pipeline_workflow.py

@@ -1,24 +1,24 @@
+import string
+import uuid
 from collections.abc import Generator
 from typing import Any
 
 from flask import request
 from pydantic import BaseModel
-from sqlalchemy import select
-from werkzeug.exceptions import Forbidden, NotFound
+from werkzeug.exceptions import Forbidden
 
 import services
 from controllers.common.errors import FilenameNotExistsError, NoFileUploadedError, TooManyFilesError
 from controllers.common.schema import register_schema_model
 from controllers.service_api import service_api_ns
 from controllers.service_api.dataset.error import PipelineRunError
-from controllers.service_api.dataset.rag_pipeline.serializers import serialize_upload_file
 from controllers.service_api.wraps import DatasetApiResource
 from core.app.apps.pipeline.pipeline_generator import PipelineGenerator
 from core.app.entities.app_invoke_entities import InvokeFrom
 from libs import helper
 from libs.login import current_user
 from models import Account
-from models.dataset import Dataset, Pipeline
+from models.dataset import Pipeline
 from models.engine import db
 from services.errors.file import FileTooLargeError, UnsupportedFileTypeError
 from services.file_service import FileService
@@ -41,7 +41,7 @@ register_schema_model(service_api_ns, DatasourceNodeRunPayload)
 register_schema_model(service_api_ns, PipelineRunApiEntity)
 
 
-@service_api_ns.route("/datasets/<uuid:dataset_id>/pipeline/datasource-plugins")
+@service_api_ns.route(f"/datasets/{uuid:dataset_id}/pipeline/datasource-plugins")
 class DatasourcePluginsApi(DatasetApiResource):
     """Resource for datasource plugins."""
 
@@ -66,12 +66,6 @@ class DatasourcePluginsApi(DatasetApiResource):
     )
     def get(self, tenant_id: str, dataset_id: str):
         """Resource for getting datasource plugins."""
-        # Verify dataset ownership
-        stmt = select(Dataset).where(Dataset.tenant_id == tenant_id, Dataset.id == dataset_id)
-        dataset = db.session.scalar(stmt)
-        if not dataset:
-            raise NotFound("Dataset not found.")
-
         # Get query parameter to determine published or draft
         is_published: bool = request.args.get("is_published", default=True, type=bool)
 
@@ -82,7 +76,7 @@ class DatasourcePluginsApi(DatasetApiResource):
         return datasource_plugins, 200
 
 
-@service_api_ns.route("/datasets/<uuid:dataset_id>/pipeline/datasource/nodes/<string:node_id>/run")
+@service_api_ns.route(f"/datasets/{uuid:dataset_id}/pipeline/datasource/nodes/{string:node_id}/run")
 class DatasourceNodeRunApi(DatasetApiResource):
     """Resource for datasource node run."""
 
@@ -111,12 +105,6 @@ class DatasourceNodeRunApi(DatasetApiResource):
     @service_api_ns.expect(service_api_ns.models[DatasourceNodeRunPayload.__name__])
     def post(self, tenant_id: str, dataset_id: str, node_id: str):
         """Resource for getting datasource plugins."""
-        # Verify dataset ownership
-        stmt = select(Dataset).where(Dataset.tenant_id == tenant_id, Dataset.id == dataset_id)
-        dataset = db.session.scalar(stmt)
-        if not dataset:
-            raise NotFound("Dataset not found.")
-
         payload = DatasourceNodeRunPayload.model_validate(service_api_ns.payload or {})
         assert isinstance(current_user, Account)
         rag_pipeline_service: RagPipelineService = RagPipelineService()
@@ -143,7 +131,7 @@ class DatasourceNodeRunApi(DatasetApiResource):
         )
 
 
-@service_api_ns.route("/datasets/<uuid:dataset_id>/pipeline/run")
+@service_api_ns.route(f"/datasets/{uuid:dataset_id}/pipeline/run")
 class PipelineRunApi(DatasetApiResource):
     """Resource for datasource node run."""
 
@@ -174,12 +162,6 @@ class PipelineRunApi(DatasetApiResource):
     @service_api_ns.expect(service_api_ns.models[PipelineRunApiEntity.__name__])
     def post(self, tenant_id: str, dataset_id: str):
         """Resource for running a rag pipeline."""
-        # Verify dataset ownership
-        stmt = select(Dataset).where(Dataset.tenant_id == tenant_id, Dataset.id == dataset_id)
-        dataset = db.session.scalar(stmt)
-        if not dataset:
-            raise NotFound("Dataset not found.")
-
         payload = PipelineRunApiEntity.model_validate(service_api_ns.payload or {})
 
         if not isinstance(current_user, Account):
@@ -250,4 +232,12 @@ class KnowledgebasePipelineFileUploadApi(DatasetApiResource):
         except services.errors.file.UnsupportedFileTypeError:
             raise UnsupportedFileTypeError()
 
-        return serialize_upload_file(upload_file), 201
+        return {
+            "id": upload_file.id,
+            "name": upload_file.name,
+            "size": upload_file.size,
+            "extension": upload_file.extension,
+            "mime_type": upload_file.mime_type,
+            "created_by": upload_file.created_by,
+            "created_at": upload_file.created_at,
+        }, 201

api/controllers/service_api/dataset/rag_pipeline/serializers.py

@@ -1,22 +0,0 @@
-"""
-Serialization helpers for Service API knowledge pipeline endpoints.
-"""
-
-from __future__ import annotations
-
-from typing import TYPE_CHECKING, Any
-
-if TYPE_CHECKING:
-    from models.model import UploadFile
-
-
-def serialize_upload_file(upload_file: UploadFile) -> dict[str, Any]:
-    return {
-        "id": upload_file.id,
-        "name": upload_file.name,
-        "size": upload_file.size,
-        "extension": upload_file.extension,
-        "mime_type": upload_file.mime_type,
-        "created_by": upload_file.created_by,
-        "created_at": upload_file.created_at.isoformat() if upload_file.created_at else None,
-    }

api/controllers/service_api/dataset/segment.py

@@ -233,7 +233,7 @@ class DatasetSegmentApi(DatasetApiResource):
         if not segment:
             raise NotFound("Segment not found.")
         SegmentService.delete_segment(segment, document, dataset)
-        return "", 204
+        return 204
 
     @service_api_ns.expect(service_api_ns.models[SegmentUpdatePayload.__name__])
     @service_api_ns.doc("update_segment")
@@ -499,7 +499,7 @@ class DatasetChildChunkApi(DatasetApiResource):
         except ChildChunkDeleteIndexServiceError as e:
             raise ChildChunkDeleteIndexError(str(e))
 
-        return "", 204
+        return 204
 
     @service_api_ns.expect(service_api_ns.models[ChildChunkUpdatePayload.__name__])
     @service_api_ns.doc("update_child_chunk")

api/controllers/service_api/end_user/__init__.py

@@ -1,3 +0,0 @@
-from . import end_user
-
-__all__ = ["end_user"]

api/controllers/service_api/end_user/end_user.py

@@ -1,41 +0,0 @@
-from uuid import UUID
-
-from flask_restx import Resource
-
-from controllers.service_api import service_api_ns
-from controllers.service_api.end_user.error import EndUserNotFoundError
-from controllers.service_api.wraps import validate_app_token
-from fields.end_user_fields import EndUserDetail
-from models.model import App
-from services.end_user_service import EndUserService
-
-
-@service_api_ns.route("/end-users/<uuid:end_user_id>")
-class EndUserApi(Resource):
-    """Resource for retrieving end user details by ID."""
-
-    @service_api_ns.doc("get_end_user")
-    @service_api_ns.doc(description="Get an end user by ID")
-    @service_api_ns.doc(
-        params={"end_user_id": "End user ID"},
-        responses={
-            200: "End user retrieved successfully",
-            401: "Unauthorized - invalid API token",
-            404: "End user not found",
-        },
-    )
-    @validate_app_token
-    def get(self, app_model: App, end_user_id: UUID):
-        """Get end user detail.
-
-        This endpoint is scoped to the current app token's tenant/app to prevent
-        cross-tenant/app access when an end-user ID is known.
-        """
-
-        end_user = EndUserService.get_end_user_by_id(
-            tenant_id=app_model.tenant_id, app_id=app_model.id, end_user_id=str(end_user_id)
-        )
-        if end_user is None:
-            raise EndUserNotFoundError()
-
-        return EndUserDetail.model_validate(end_user).model_dump(mode="json")

api/controllers/service_api/end_user/error.py

@@ -1,7 +0,0 @@
-from libs.exception import BaseHTTPException
-
-
-class EndUserNotFoundError(BaseHTTPException):
-    error_code = "end_user_not_found"
-    description = "End user not found."
-    code = 404

api/controllers/service_api/wraps.py

@@ -1,24 +1,27 @@
 import logging
 import time
 from collections.abc import Callable
+from datetime import timedelta
 from enum import StrEnum, auto
 from functools import wraps
-from typing import Concatenate, ParamSpec, TypeVar, cast
+from typing import Concatenate, ParamSpec, TypeVar
 
 from flask import current_app, request
 from flask_login import user_logged_in
 from flask_restx import Resource
 from pydantic import BaseModel
+from sqlalchemy import select, update
+from sqlalchemy.orm import Session
 from werkzeug.exceptions import Forbidden, NotFound, Unauthorized
 
 from enums.cloud_plan import CloudPlan
 from extensions.ext_database import db
 from extensions.ext_redis import redis_client
+from libs.datetime_utils import naive_utc_now
 from libs.login import current_user
 from models import Account, Tenant, TenantAccountJoin, TenantStatus
 from models.dataset import Dataset, RateLimitLog
 from models.model import ApiToken, App
-from services.api_token_service import ApiTokenCache, fetch_token_with_single_flight, record_token_usage
 from services.end_user_service import EndUserService
 from services.feature_service import FeatureService
 
@@ -70,14 +73,14 @@ def validate_app_token(view: Callable[P, R] | None = None, *, fetch_user_arg: Fe
 
             # If caller needs end-user context, attach EndUser to current_user
             if fetch_user_arg:
-                user_id = None
-                match fetch_user_arg.fetch_from:
-                    case WhereisUserArg.QUERY:
-                        user_id = request.args.get("user")
-                    case WhereisUserArg.JSON:
-                        user_id = request.get_json().get("user")
-                    case WhereisUserArg.FORM:
-                        user_id = request.form.get("user")
+                if fetch_user_arg.fetch_from == WhereisUserArg.QUERY:
+                    user_id = request.args.get("user")
+                elif fetch_user_arg.fetch_from == WhereisUserArg.JSON:
+                    user_id = request.get_json().get("user")
+                elif fetch_user_arg.fetch_from == WhereisUserArg.FORM:
+                    user_id = request.form.get("user")
+                else:
+                    user_id = None
 
                 if not user_id and fetch_user_arg.required:
                     raise ValueError("Arg user must be provided.")
@@ -217,8 +220,6 @@ def validate_dataset_token(view: Callable[Concatenate[T, P], R] | None = None):
     def decorator(view: Callable[Concatenate[T, P], R]):
         @wraps(view)
         def decorated(*args: P.args, **kwargs: P.kwargs):
-            api_token = validate_and_get_api_token("dataset")
-
             # get url path dataset_id from positional args or kwargs
             # Flask passes URL path parameters as positional arguments
             dataset_id = None
@@ -255,18 +256,12 @@ def validate_dataset_token(view: Callable[Concatenate[T, P], R] | None = None):
             # Validate dataset if dataset_id is provided
             if dataset_id:
                 dataset_id = str(dataset_id)
-                dataset = (
-                    db.session.query(Dataset)
-                    .where(
-                        Dataset.id == dataset_id,
-                        Dataset.tenant_id == api_token.tenant_id,
-                    )
-                    .first()
-                )
+                dataset = db.session.query(Dataset).where(Dataset.id == dataset_id).first()
                 if not dataset:
                     raise NotFound("Dataset not found.")
                 if not dataset.enable_api:
                     raise Forbidden("Dataset api access is not enabled.")
+            api_token = validate_and_get_api_token("dataset")
             tenant_account_join = (
                 db.session.query(Tenant, TenantAccountJoin)
                 .where(Tenant.id == api_token.tenant_id)
@@ -301,14 +296,7 @@ def validate_dataset_token(view: Callable[Concatenate[T, P], R] | None = None):
 
 def validate_and_get_api_token(scope: str | None = None):
     """
-    Validate and get API token with Redis caching.
-
-    This function uses a two-tier approach:
-    1. First checks Redis cache for the token
-    2. If not cached, queries database and caches the result
-
-    The last_used_at field is updated asynchronously via Celery task
-    to avoid blocking the request.
+    Validate and get API token.
     """
     auth_header = request.headers.get("Authorization")
     if auth_header is None or " " not in auth_header:
@@ -320,18 +308,29 @@ def validate_and_get_api_token(scope: str | None = None):
     if auth_scheme != "bearer":
         raise Unauthorized("Authorization scheme must be 'Bearer'")
 
-    # Try to get token from cache first
-    # Returns a CachedApiToken (plain Python object), not a SQLAlchemy model
-    cached_token = ApiTokenCache.get(auth_token, scope)
-    if cached_token is not None:
-        logger.debug("Token validation served from cache for scope: %s", scope)
-        # Record usage in Redis for later batch update (no Celery task per request)
-        record_token_usage(auth_token, scope)
-        return cast(ApiToken, cached_token)
+    current_time = naive_utc_now()
+    cutoff_time = current_time - timedelta(minutes=1)
+    with Session(db.engine, expire_on_commit=False) as session:
+        update_stmt = (
+            update(ApiToken)
+            .where(
+                ApiToken.token == auth_token,
+                (ApiToken.last_used_at.is_(None) | (ApiToken.last_used_at < cutoff_time)),
+                ApiToken.type == scope,
+            )
+            .values(last_used_at=current_time)
+        )
+        stmt = select(ApiToken).where(ApiToken.token == auth_token, ApiToken.type == scope)
+        result = session.execute(update_stmt)
+        api_token = session.scalar(stmt)
 
-    # Cache miss - use Redis lock for single-flight mode
-    # This ensures only one request queries DB for the same token concurrently
-    return fetch_token_with_single_flight(auth_token, scope)
+        if hasattr(result, "rowcount") and result.rowcount > 0:
+            session.commit()
+
+        if not api_token:
+            raise Unauthorized("Access token is invalid")
+
+    return api_token
 
 
 class DatasetApiResource(Resource):

api/controllers/web/__init__.py

@@ -23,7 +23,6 @@ from . import (
     feature,
     files,
     forgot_password,
-    human_input_form,
     login,
     message,
     passport,
@@ -31,7 +30,6 @@ from . import (
     saved_message,
     site,
     workflow,
-    workflow_events,
 )
 
 api.add_namespace(web_ns)
@@ -46,7 +44,6 @@ __all__ = [
     "feature",
     "files",
     "forgot_password",
-    "human_input_form",
     "login",
     "message",
     "passport",
@@ -55,5 +52,4 @@ __all__ = [
     "site",
     "web_ns",
     "workflow",
-    "workflow_events",
 ]

api/controllers/web/error.py

@@ -117,12 +117,6 @@ class InvokeRateLimitError(BaseHTTPException):
     code = 429
 
 
-class WebFormRateLimitExceededError(BaseHTTPException):
-    error_code = "web_form_rate_limit_exceeded"
-    description = "Too many form requests. Please try again later."
-    code = 429
-
-
 class NotFoundError(BaseHTTPException):
     error_code = "not_found"
     code = 404

api/controllers/web/human_input_form.py

@@ -1,161 +0,0 @@
-"""
-Web App Human Input Form APIs.
-"""
-
-import json
-import logging
-from datetime import datetime
-
-from flask import Response, request
-from flask_restx import Resource, reqparse
-from werkzeug.exceptions import Forbidden
-
-from configs import dify_config
-from controllers.web import web_ns
-from controllers.web.error import NotFoundError, WebFormRateLimitExceededError
-from controllers.web.site import serialize_app_site_payload
-from extensions.ext_database import db
-from libs.helper import RateLimiter, extract_remote_ip
-from models.account import TenantStatus
-from models.model import App, Site
-from services.human_input_service import Form, FormNotFoundError, HumanInputService
-
-logger = logging.getLogger(__name__)
-
-_FORM_SUBMIT_RATE_LIMITER = RateLimiter(
-    prefix="web_form_submit_rate_limit",
-    max_attempts=dify_config.WEB_FORM_SUBMIT_RATE_LIMIT_MAX_ATTEMPTS,
-    time_window=dify_config.WEB_FORM_SUBMIT_RATE_LIMIT_WINDOW_SECONDS,
-)
-_FORM_ACCESS_RATE_LIMITER = RateLimiter(
-    prefix="web_form_access_rate_limit",
-    max_attempts=dify_config.WEB_FORM_SUBMIT_RATE_LIMIT_MAX_ATTEMPTS,
-    time_window=dify_config.WEB_FORM_SUBMIT_RATE_LIMIT_WINDOW_SECONDS,
-)
-
-
-def _stringify_default_values(values: dict[str, object]) -> dict[str, str]:
-    result: dict[str, str] = {}
-    for key, value in values.items():
-        if value is None:
-            result[key] = ""
-        elif isinstance(value, (dict, list)):
-            result[key] = json.dumps(value, ensure_ascii=False)
-        else:
-            result[key] = str(value)
-    return result
-
-
-def _to_timestamp(value: datetime) -> int:
-    return int(value.timestamp())
-
-
-def _jsonify_form_definition(form: Form, site_payload: dict | None = None) -> Response:
-    """Return the form payload (optionally with site) as a JSON response."""
-    definition_payload = form.get_definition().model_dump()
-    payload = {
-        "form_content": definition_payload["rendered_content"],
-        "inputs": definition_payload["inputs"],
-        "resolved_default_values": _stringify_default_values(definition_payload["default_values"]),
-        "user_actions": definition_payload["user_actions"],
-        "expiration_time": _to_timestamp(form.expiration_time),
-    }
-    if site_payload is not None:
-        payload["site"] = site_payload
-    return Response(json.dumps(payload, ensure_ascii=False), mimetype="application/json")
-
-
-@web_ns.route("/form/human_input/<string:form_token>")
-class HumanInputFormApi(Resource):
-    """API for getting and submitting human input forms via the web app."""
-
-    # NOTE(QuantumGhost): this endpoint is unauthenticated on purpose for now.
-
-    # def get(self, _app_model: App, _end_user: EndUser, form_token: str):
-    def get(self, form_token: str):
-        """
-        Get human input form definition by token.
-
-        GET /api/form/human_input/<form_token>
-        """
-        ip_address = extract_remote_ip(request)
-        if _FORM_ACCESS_RATE_LIMITER.is_rate_limited(ip_address):
-            raise WebFormRateLimitExceededError()
-        _FORM_ACCESS_RATE_LIMITER.increment_rate_limit(ip_address)
-
-        service = HumanInputService(db.engine)
-        # TODO(QuantumGhost): forbid submision for form tokens
-        # that are only for console.
-        form = service.get_form_by_token(form_token)
-
-        if form is None:
-            raise NotFoundError("Form not found")
-
-        service.ensure_form_active(form)
-        app_model, site = _get_app_site_from_form(form)
-
-        return _jsonify_form_definition(form, site_payload=serialize_app_site_payload(app_model, site, None))
-
-    # def post(self, _app_model: App, _end_user: EndUser, form_token: str):
-    def post(self, form_token: str):
-        """
-        Submit human input form by token.
-
-        POST /api/form/human_input/<form_token>
-
-        Request body:
-        {
-            "inputs": {
-                "content": "User input content"
-            },
-            "action": "Approve"
-        }
-        """
-        parser = reqparse.RequestParser()
-        parser.add_argument("inputs", type=dict, required=True, location="json")
-        parser.add_argument("action", type=str, required=True, location="json")
-        args = parser.parse_args()
-
-        ip_address = extract_remote_ip(request)
-        if _FORM_SUBMIT_RATE_LIMITER.is_rate_limited(ip_address):
-            raise WebFormRateLimitExceededError()
-        _FORM_SUBMIT_RATE_LIMITER.increment_rate_limit(ip_address)
-
-        service = HumanInputService(db.engine)
-        form = service.get_form_by_token(form_token)
-        if form is None:
-            raise NotFoundError("Form not found")
-
-        if (recipient_type := form.recipient_type) is None:
-            logger.warning("Recipient type is None for form, form_id=%", form.id)
-            raise AssertionError("Recipient type is None")
-
-        try:
-            service.submit_form_by_token(
-                recipient_type=recipient_type,
-                form_token=form_token,
-                selected_action_id=args["action"],
-                form_data=args["inputs"],
-                submission_end_user_id=None,
-                # submission_end_user_id=_end_user.id,
-            )
-        except FormNotFoundError:
-            raise NotFoundError("Form not found")
-
-        return {}, 200
-
-
-def _get_app_site_from_form(form: Form) -> tuple[App, Site]:
-    """Resolve App/Site for the form's app and validate tenant status."""
-    app_model = db.session.query(App).where(App.id == form.app_id).first()
-    if app_model is None or app_model.tenant_id != form.tenant_id:
-        raise NotFoundError("Form not found")
-
-    site = db.session.query(Site).where(Site.app_id == app_model.id).first()
-    if site is None:
-        raise Forbidden()
-
-    if app_model.tenant and app_model.tenant.status == TenantStatus.ARCHIVE:
-        raise Forbidden()
-
-    return app_model, site

api/controllers/web/remote_files.py

@@ -10,8 +10,8 @@ from controllers.common.errors import (
     RemoteFileUploadError,
     UnsupportedFileTypeError,
 )
+from core.file import helpers as file_helpers
 from core.helper import ssrf_proxy
-from core.workflow.file import helpers as file_helpers
 from extensions.ext_database import db
 from fields.file_fields import FileWithSignedUrl, RemoteFileInfo
 from services.file_service import FileService

api/controllers/web/site.py

@@ -1,6 +1,4 @@
-from typing import cast
-
-from flask_restx import fields, marshal, marshal_with
+from flask_restx import fields, marshal_with
 from werkzeug.exceptions import Forbidden
 
 from configs import dify_config
@@ -9,7 +7,7 @@ from controllers.web.wraps import WebApiResource
 from extensions.ext_database import db
 from libs.helper import AppIconUrlField
 from models.account import TenantStatus
-from models.model import App, Site
+from models.model import Site
 from services.feature_service import FeatureService
 
 
@@ -110,14 +108,3 @@ class AppSiteInfo:
                 "remove_webapp_brand": remove_webapp_brand,
                 "replace_webapp_logo": replace_webapp_logo,
             }
-
-
-def serialize_site(site: Site) -> dict:
-    """Serialize Site model using the same schema as AppSiteApi."""
-    return cast(dict, marshal(site, AppSiteApi.site_fields))
-
-
-def serialize_app_site_payload(app_model: App, site: Site, end_user_id: str | None) -> dict:
-    can_replace_logo = FeatureService.get_features(app_model.tenant_id).can_replace_logo
-    app_site_info = AppSiteInfo(app_model.tenant, app_model, site, end_user_id, can_replace_logo)
-    return cast(dict, marshal(app_site_info, AppSiteApi.app_fields))

api/controllers/web/workflow.py

@@ -24,7 +24,6 @@ from core.errors.error import (
 )
 from core.model_runtime.errors.invoke import InvokeError
 from core.workflow.graph_engine.manager import GraphEngineManager
-from extensions.ext_redis import redis_client
 from libs import helper
 from models.model import App, AppMode, EndUser
 from services.app_generate_service import AppGenerateService
@@ -122,6 +121,6 @@ class WorkflowTaskStopApi(WebApiResource):
         AppQueueManager.set_stop_flag_no_user_check(task_id)
 
         # New graph engine command channel mechanism
-        GraphEngineManager(redis_client).send_stop_command(task_id)
+        GraphEngineManager.send_stop_command(task_id)
 
         return {"result": "success"}

api/controllers/web/workflow_events.py

@@ -1,112 +0,0 @@
-"""
-Web App Workflow Resume APIs.
-"""
-
-import json
-from collections.abc import Generator
-
-from flask import Response, request
-from sqlalchemy.orm import sessionmaker
-
-from controllers.web import api
-from controllers.web.error import InvalidArgumentError, NotFoundError
-from controllers.web.wraps import WebApiResource
-from core.app.apps.advanced_chat.app_generator import AdvancedChatAppGenerator
-from core.app.apps.base_app_generator import BaseAppGenerator
-from core.app.apps.common.workflow_response_converter import WorkflowResponseConverter
-from core.app.apps.message_generator import MessageGenerator
-from core.app.apps.workflow.app_generator import WorkflowAppGenerator
-from extensions.ext_database import db
-from models.enums import CreatorUserRole
-from models.model import App, AppMode, EndUser
-from repositories.factory import DifyAPIRepositoryFactory
-from services.workflow_event_snapshot_service import build_workflow_event_stream
-
-
-class WorkflowEventsApi(WebApiResource):
-    """API for getting workflow execution events after resume."""
-
-    def get(self, app_model: App, end_user: EndUser, task_id: str):
-        """
-        Get workflow execution events stream after resume.
-
-        GET /api/workflow/<task_id>/events
-
-        Returns Server-Sent Events stream.
-        """
-        workflow_run_id = task_id
-        session_maker = sessionmaker(db.engine)
-        repo = DifyAPIRepositoryFactory.create_api_workflow_run_repository(session_maker)
-        workflow_run = repo.get_workflow_run_by_id_and_tenant_id(
-            tenant_id=app_model.tenant_id,
-            run_id=workflow_run_id,
-        )
-
-        if workflow_run is None:
-            raise NotFoundError(f"WorkflowRun not found, id={workflow_run_id}")
-
-        if workflow_run.app_id != app_model.id:
-            raise NotFoundError(f"WorkflowRun not found, id={workflow_run_id}")
-
-        if workflow_run.created_by_role != CreatorUserRole.END_USER:
-            raise NotFoundError(f"WorkflowRun not created by end user, id={workflow_run_id}")
-
-        if workflow_run.created_by != end_user.id:
-            raise NotFoundError(f"WorkflowRun not created by the current end user, id={workflow_run_id}")
-
-        if workflow_run.finished_at is not None:
-            response = WorkflowResponseConverter.workflow_run_result_to_finish_response(
-                task_id=workflow_run.id,
-                workflow_run=workflow_run,
-                creator_user=end_user,
-            )
-
-            payload = response.model_dump(mode="json")
-            payload["event"] = response.event.value
-
-            def _generate_finished_events() -> Generator[str, None, None]:
-                yield f"data: {json.dumps(payload)}\n\n"
-
-            event_generator = _generate_finished_events
-        else:
-            app_mode = AppMode.value_of(app_model.mode)
-            msg_generator = MessageGenerator()
-            generator: BaseAppGenerator
-            if app_mode == AppMode.ADVANCED_CHAT:
-                generator = AdvancedChatAppGenerator()
-            elif app_mode == AppMode.WORKFLOW:
-                generator = WorkflowAppGenerator()
-            else:
-                raise InvalidArgumentError(f"cannot subscribe to workflow run, workflow_run_id={workflow_run.id}")
-
-            include_state_snapshot = request.args.get("include_state_snapshot", "false").lower() == "true"
-
-            def _generate_stream_events():
-                if include_state_snapshot:
-                    return generator.convert_to_event_stream(
-                        build_workflow_event_stream(
-                            app_mode=app_mode,
-                            workflow_run=workflow_run,
-                            tenant_id=app_model.tenant_id,
-                            app_id=app_model.id,
-                            session_maker=session_maker,
-                        )
-                    )
-                return generator.convert_to_event_stream(
-                    msg_generator.retrieve_events(app_mode, workflow_run.id),
-                )
-
-            event_generator = _generate_stream_events
-
-        return Response(
-            event_generator(),
-            mimetype="text/event-stream",
-            headers={
-                "Cache-Control": "no-cache",
-                "Connection": "keep-alive",
-            },
-        )
-
-
-# Register the APIs
-api.add_resource(WorkflowEventsApi, "/workflow/<string:task_id>/events")

api/core/agent/base_agent_runner.py

@@ -17,6 +17,8 @@ from core.app.entities.app_invoke_entities import (
 )
 from core.callback_handler.agent_tool_callback_handler import DifyAgentCallbackHandler
 from core.callback_handler.index_tool_callback_handler import DatasetIndexToolCallbackHandler
+from core.file import file_manager
+from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_manager import ModelInstance
 from core.model_runtime.entities import (
     AssistantPromptMessage,
@@ -31,7 +33,6 @@ from core.model_runtime.entities import (
 from core.model_runtime.entities.message_entities import ImagePromptMessageContent, PromptMessageContentUnionTypes
 from core.model_runtime.entities.model_entities import ModelFeature
 from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
-from core.model_runtime.token_buffer_memory import TokenBufferMemory
 from core.prompt.utils.extract_thread_messages import extract_thread_messages
 from core.tools.__base.tool import Tool
 from core.tools.entities.tool_entities import (
@@ -39,7 +40,6 @@ from core.tools.entities.tool_entities import (
 )
 from core.tools.tool_manager import ToolManager
 from core.tools.utils.dataset_retriever_tool import DatasetRetrieverTool
-from core.workflow.file import file_manager
 from extensions.ext_database import db
 from factories import file_factory
 from models.enums import CreatorUserRole
@@ -112,7 +112,7 @@ class BaseAgentRunner(AppRunner):
 
         # check if model supports stream tool call
         llm_model = cast(LargeLanguageModel, model_instance.model_type_instance)
-        model_schema = llm_model.get_model_schema(model_instance.model_name, model_instance.credentials)
+        model_schema = llm_model.get_model_schema(model_instance.model, model_instance.credentials)
         features = model_schema.features if model_schema and model_schema.features else []
         self.stream_tool_call = ModelFeature.STREAM_TOOL_CALL in features
         self.files = application_generate_entity.files if ModelFeature.VISION in features else []

api/core/agent/cot_agent_runner.py

@@ -245,7 +245,7 @@ class CotAgentRunner(BaseAgentRunner, ABC):
             iteration_step += 1
 
         yield LLMResultChunk(
-            model=model_instance.model_name,
+            model=model_instance.model,
             prompt_messages=prompt_messages,
             delta=LLMResultChunkDelta(
                 index=0, message=AssistantPromptMessage(content=final_answer), usage=llm_usage["usage"]
@@ -268,7 +268,7 @@ class CotAgentRunner(BaseAgentRunner, ABC):
         self.queue_manager.publish(
             QueueMessageEndEvent(
                 llm_result=LLMResult(
-                    model=model_instance.model_name,
+                    model=model_instance.model,
                     prompt_messages=prompt_messages,
                     message=AssistantPromptMessage(content=final_answer),
                     usage=llm_usage["usage"] or LLMUsage.empty_usage(),

api/core/agent/cot_chat_agent_runner.py

@@ -1,6 +1,7 @@
 import json
 
 from core.agent.cot_agent_runner import CotAgentRunner
+from core.file import file_manager
 from core.model_runtime.entities import (
     AssistantPromptMessage,
     PromptMessage,
@@ -10,7 +11,6 @@ from core.model_runtime.entities import (
 )
 from core.model_runtime.entities.message_entities import ImagePromptMessageContent, PromptMessageContentUnionTypes
 from core.model_runtime.utils.encoders import jsonable_encoder
-from core.workflow.file import file_manager
 
 
 class CotChatAgentRunner(CotAgentRunner):

api/core/agent/fc_agent_runner.py

@@ -7,6 +7,7 @@ from typing import Any, Union
 from core.agent.base_agent_runner import BaseAgentRunner
 from core.app.apps.base_app_queue_manager import PublishFrom
 from core.app.entities.queue_entities import QueueAgentThoughtEvent, QueueMessageEndEvent, QueueMessageFileEvent
+from core.file import file_manager
 from core.model_runtime.entities import (
     AssistantPromptMessage,
     LLMResult,
@@ -24,7 +25,6 @@ from core.model_runtime.entities.message_entities import ImagePromptMessageConte
 from core.prompt.agent_history_prompt_transform import AgentHistoryPromptTransform
 from core.tools.entities.tool_entities import ToolInvokeMeta
 from core.tools.tool_engine import ToolEngine
-from core.workflow.file import file_manager
 from core.workflow.nodes.agent.exc import AgentMaxIterationError
 from models.model import Message
 
@@ -178,7 +178,7 @@ class FunctionCallAgentRunner(BaseAgentRunner):
                 )
 
                 yield LLMResultChunk(
-                    model=model_instance.model_name,
+                    model=model_instance.model,
                     prompt_messages=result.prompt_messages,
                     system_fingerprint=result.system_fingerprint,
                     delta=LLMResultChunkDelta(
@@ -308,7 +308,7 @@ class FunctionCallAgentRunner(BaseAgentRunner):
         self.queue_manager.publish(
             QueueMessageEndEvent(
                 llm_result=LLMResult(
-                    model=model_instance.model_name,
+                    model=model_instance.model,
                     prompt_messages=prompt_messages,
                     message=AssistantPromptMessage(content=final_answer),
                     usage=llm_usage["usage"] or LLMUsage.empty_usage(),

api/core/app/app_config/easy_ui_based_app/agent/manager.py

@@ -14,17 +14,16 @@ class AgentConfigManager:
             agent_dict = config.get("agent_mode", {})
             agent_strategy = agent_dict.get("strategy", "cot")
 
-            match agent_strategy:
-                case "function_call":
+            if agent_strategy == "function_call":
+                strategy = AgentEntity.Strategy.FUNCTION_CALLING
+            elif agent_strategy in {"cot", "react"}:
+                strategy = AgentEntity.Strategy.CHAIN_OF_THOUGHT
+            else:
+                # old configs, try to detect default strategy
+                if config["model"]["provider"] == "openai":
                     strategy = AgentEntity.Strategy.FUNCTION_CALLING
-                case "cot" | "react":
+                else:
                     strategy = AgentEntity.Strategy.CHAIN_OF_THOUGHT
-                case _:
-                    # old configs, try to detect default strategy
-                    if config["model"]["provider"] == "openai":
-                        strategy = AgentEntity.Strategy.FUNCTION_CALLING
-                    else:
-                        strategy = AgentEntity.Strategy.CHAIN_OF_THOUGHT
 
             agent_tools = []
             for tool in agent_dict.get("tools", []):

api/core/app/app_config/entities.py

@@ -5,9 +5,9 @@ from typing import Any, Literal
 from jsonschema import Draft7Validator, SchemaError
 from pydantic import BaseModel, Field, field_validator
 
+from core.file import FileTransferMethod, FileType, FileUploadConfig
 from core.model_runtime.entities.llm_entities import LLMMode
 from core.model_runtime.entities.message_entities import PromptMessageRole
-from core.workflow.file import FileTransferMethod, FileType, FileUploadConfig
 from models.model import AppMode
 
 

api/core/app/app_config/features/file_upload/manager.py

@@ -2,7 +2,7 @@ from collections.abc import Mapping
 from typing import Any
 
 from constants import DEFAULT_FILE_NUMBER_LIMITS
-from core.workflow.file import FileUploadConfig
+from core.file import FileUploadConfig
 
 
 class FileUploadConfigManager:

api/core/app/apps/advanced_chat/app_generator.py

@@ -4,8 +4,8 @@ import contextvars
 import logging
 import threading
 import uuid
-from collections.abc import Generator, Mapping, Sequence
-from typing import TYPE_CHECKING, Any, Literal, TypeVar, Union, overload
+from collections.abc import Generator, Mapping
+from typing import TYPE_CHECKING, Any, Literal, Union, overload
 
 from flask import Flask, current_app
 from pydantic import ValidationError
@@ -29,25 +29,21 @@ from core.app.apps.message_based_app_generator import MessageBasedAppGenerator
 from core.app.apps.message_based_app_queue_manager import MessageBasedAppQueueManager
 from core.app.entities.app_invoke_entities import AdvancedChatAppGenerateEntity, InvokeFrom
 from core.app.entities.task_entities import ChatbotAppBlockingResponse, ChatbotAppStreamResponse
-from core.app.layers.pause_state_persist_layer import PauseStateLayerConfig, PauseStatePersistenceLayer
 from core.helper.trace_id_helper import extract_external_trace_id_from_args
 from core.model_runtime.errors.invoke import InvokeAuthorizationError
 from core.ops.ops_trace_manager import TraceQueueManager
 from core.prompt.utils.get_thread_messages_length import get_thread_messages_length
 from core.repositories import DifyCoreRepositoryFactory
-from core.workflow.graph_engine.layers.base import GraphEngineLayer
 from core.workflow.repositories.draft_variable_repository import (
     DraftVariableSaverFactory,
 )
 from core.workflow.repositories.workflow_execution_repository import WorkflowExecutionRepository
 from core.workflow.repositories.workflow_node_execution_repository import WorkflowNodeExecutionRepository
-from core.workflow.runtime import GraphRuntimeState
 from core.workflow.variable_loader import DUMMY_VARIABLE_LOADER, VariableLoader
 from extensions.ext_database import db
 from factories import file_factory
 from libs.flask_utils import preserve_flask_contexts
 from models import Account, App, Conversation, EndUser, Message, Workflow, WorkflowNodeExecutionTriggeredFrom
-from models.base import Base
 from models.enums import WorkflowRunTriggeredFrom
 from services.conversation_service import ConversationService
 from services.workflow_draft_variable_service import (
@@ -69,9 +65,7 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
         user: Union[Account, EndUser],
         args: Mapping[str, Any],
         invoke_from: InvokeFrom,
-        workflow_run_id: str,
         streaming: Literal[False],
-        pause_state_config: PauseStateLayerConfig | None = None,
     ) -> Mapping[str, Any]: ...
 
     @overload
@@ -80,11 +74,9 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
         app_model: App,
         workflow: Workflow,
         user: Union[Account, EndUser],
-        args: Mapping[str, Any],
+        args: Mapping,
         invoke_from: InvokeFrom,
-        workflow_run_id: str,
         streaming: Literal[True],
-        pause_state_config: PauseStateLayerConfig | None = None,
     ) -> Generator[Mapping | str, None, None]: ...
 
     @overload
@@ -93,11 +85,9 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
         app_model: App,
         workflow: Workflow,
         user: Union[Account, EndUser],
-        args: Mapping[str, Any],
+        args: Mapping,
         invoke_from: InvokeFrom,
-        workflow_run_id: str,
         streaming: bool,
-        pause_state_config: PauseStateLayerConfig | None = None,
     ) -> Mapping[str, Any] | Generator[str | Mapping, None, None]: ...
 
     def generate(
@@ -105,11 +95,9 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
         app_model: App,
         workflow: Workflow,
         user: Union[Account, EndUser],
-        args: Mapping[str, Any],
+        args: Mapping,
         invoke_from: InvokeFrom,
-        workflow_run_id: str,
         streaming: bool = True,
-        pause_state_config: PauseStateLayerConfig | None = None,
     ) -> Mapping[str, Any] | Generator[str | Mapping, None, None]:
         """
         Generate App response.
@@ -173,6 +161,7 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
             # always enable retriever resource in debugger mode
             app_config.additional_features.show_retrieve_source = True  # type: ignore
 
+        workflow_run_id = str(uuid.uuid4())
         # init application generate entity
         application_generate_entity = AdvancedChatAppGenerateEntity(
             task_id=str(uuid.uuid4()),
@@ -190,7 +179,7 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
             invoke_from=invoke_from,
             extras=extras,
             trace_manager=trace_manager,
-            workflow_run_id=str(workflow_run_id),
+            workflow_run_id=workflow_run_id,
         )
         contexts.plugin_tool_providers.set({})
         contexts.plugin_tool_providers_lock.set(threading.Lock())
@@ -227,38 +216,6 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
             workflow_node_execution_repository=workflow_node_execution_repository,
             conversation=conversation,
             stream=streaming,
-            pause_state_config=pause_state_config,
-        )
-
-    def resume(
-        self,
-        *,
-        app_model: App,
-        workflow: Workflow,
-        user: Union[Account, EndUser],
-        conversation: Conversation,
-        message: Message,
-        application_generate_entity: AdvancedChatAppGenerateEntity,
-        workflow_execution_repository: WorkflowExecutionRepository,
-        workflow_node_execution_repository: WorkflowNodeExecutionRepository,
-        graph_runtime_state: GraphRuntimeState,
-        pause_state_config: PauseStateLayerConfig | None = None,
-    ):
-        """
-        Resume a paused advanced chat execution.
-        """
-        return self._generate(
-            workflow=workflow,
-            user=user,
-            invoke_from=application_generate_entity.invoke_from,
-            application_generate_entity=application_generate_entity,
-            workflow_execution_repository=workflow_execution_repository,
-            workflow_node_execution_repository=workflow_node_execution_repository,
-            conversation=conversation,
-            message=message,
-            stream=application_generate_entity.stream,
-            pause_state_config=pause_state_config,
-            graph_runtime_state=graph_runtime_state,
         )
 
     def single_iteration_generate(
@@ -439,12 +396,8 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
         workflow_execution_repository: WorkflowExecutionRepository,
         workflow_node_execution_repository: WorkflowNodeExecutionRepository,
         conversation: Conversation | None = None,
-        message: Message | None = None,
         stream: bool = True,
         variable_loader: VariableLoader = DUMMY_VARIABLE_LOADER,
-        pause_state_config: PauseStateLayerConfig | None = None,
-        graph_runtime_state: GraphRuntimeState | None = None,
-        graph_engine_layers: Sequence[GraphEngineLayer] = (),
     ) -> Mapping[str, Any] | Generator[str | Mapping[str, Any], Any, None]:
         """
         Generate App response.
@@ -458,12 +411,12 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
         :param conversation: conversation
         :param stream: is stream
         """
-        is_first_conversation = conversation is None
+        is_first_conversation = False
+        if not conversation:
+            is_first_conversation = True
 
-        if conversation is not None and message is not None:
-            pass
-        else:
-            conversation, message = self._init_generate_records(application_generate_entity, conversation)
+        # init generate records
+        (conversation, message) = self._init_generate_records(application_generate_entity, conversation)
 
         if is_first_conversation:
             # update conversation features
@@ -486,16 +439,6 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
             message_id=message.id,
         )
 
-        graph_layers: list[GraphEngineLayer] = list(graph_engine_layers)
-        if pause_state_config is not None:
-            graph_layers.append(
-                PauseStatePersistenceLayer(
-                    session_factory=pause_state_config.session_factory,
-                    generate_entity=application_generate_entity,
-                    state_owner_user_id=pause_state_config.state_owner_user_id,
-                )
-            )
-
         # new thread with request context and contextvars
         context = contextvars.copy_context()
 
@@ -511,25 +454,14 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
                 "variable_loader": variable_loader,
                 "workflow_execution_repository": workflow_execution_repository,
                 "workflow_node_execution_repository": workflow_node_execution_repository,
-                "graph_engine_layers": tuple(graph_layers),
-                "graph_runtime_state": graph_runtime_state,
             },
         )
 
         worker_thread.start()
 
         # release database connection, because the following new thread operations may take a long time
-        with Session(bind=db.engine, expire_on_commit=False) as session:
-            workflow = _refresh_model(session, workflow)
-            message = _refresh_model(session, message)
-        #     workflow_ = session.get(Workflow, workflow.id)
-        #     assert workflow_ is not None
-        #     workflow = workflow_
-        #     message_ = session.get(Message, message.id)
-        #     assert message_ is not None
-        #     message = message_
-        # db.session.refresh(workflow)
-        # db.session.refresh(message)
+        db.session.refresh(workflow)
+        db.session.refresh(message)
         # db.session.refresh(user)
         db.session.close()
 
@@ -558,8 +490,6 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
         variable_loader: VariableLoader,
         workflow_execution_repository: WorkflowExecutionRepository,
         workflow_node_execution_repository: WorkflowNodeExecutionRepository,
-        graph_engine_layers: Sequence[GraphEngineLayer] = (),
-        graph_runtime_state: GraphRuntimeState | None = None,
     ):
         """
         Generate worker in a new thread.
@@ -617,8 +547,6 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
                 app=app,
                 workflow_execution_repository=workflow_execution_repository,
                 workflow_node_execution_repository=workflow_node_execution_repository,
-                graph_engine_layers=graph_engine_layers,
-                graph_runtime_state=graph_runtime_state,
             )
 
             try:
@@ -686,13 +614,3 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
             else:
                 logger.exception("Failed to process generate task pipeline, conversation_id: %s", conversation.id)
                 raise e
-
-
-_T = TypeVar("_T", bound=Base)
-
-
-def _refresh_model(session, model: _T) -> _T:
-    with Session(bind=db.engine, expire_on_commit=False) as session:
-        detach_model = session.get(type(model), model.id)
-        assert detach_model is not None
-        return detach_model

api/core/app/apps/advanced_chat/app_runner.py

@@ -66,7 +66,6 @@ class AdvancedChatAppRunner(WorkflowBasedAppRunner):
         workflow_execution_repository: WorkflowExecutionRepository,
         workflow_node_execution_repository: WorkflowNodeExecutionRepository,
         graph_engine_layers: Sequence[GraphEngineLayer] = (),
-        graph_runtime_state: GraphRuntimeState | None = None,
     ):
         super().__init__(
             queue_manager=queue_manager,
@@ -83,7 +82,6 @@ class AdvancedChatAppRunner(WorkflowBasedAppRunner):
         self._app = app
         self._workflow_execution_repository = workflow_execution_repository
         self._workflow_node_execution_repository = workflow_node_execution_repository
-        self._resume_graph_runtime_state = graph_runtime_state
 
     @trace_span(WorkflowAppRunnerHandler)
     def run(self):
@@ -112,21 +110,7 @@ class AdvancedChatAppRunner(WorkflowBasedAppRunner):
             invoke_from = InvokeFrom.DEBUGGER
         user_from = self._resolve_user_from(invoke_from)
 
-        resume_state = self._resume_graph_runtime_state
-
-        if resume_state is not None:
-            graph_runtime_state = resume_state
-            variable_pool = graph_runtime_state.variable_pool
-            graph = self._init_graph(
-                graph_config=self._workflow.graph_dict,
-                graph_runtime_state=graph_runtime_state,
-                workflow_id=self._workflow.id,
-                tenant_id=self._workflow.tenant_id,
-                user_id=self.application_generate_entity.user_id,
-                invoke_from=invoke_from,
-                user_from=user_from,
-            )
-        elif self.application_generate_entity.single_iteration_run or self.application_generate_entity.single_loop_run:
+        if self.application_generate_entity.single_iteration_run or self.application_generate_entity.single_loop_run:
             # Handle single iteration or single loop run
             graph, variable_pool, graph_runtime_state = self._prepare_single_node_execution(
                 workflow=self._workflow,

api/core/app/apps/advanced_chat/generate_task_pipeline.py

@@ -24,8 +24,6 @@ from core.app.entities.queue_entities import (
     QueueAgentLogEvent,
     QueueAnnotationReplyEvent,
     QueueErrorEvent,
-    QueueHumanInputFormFilledEvent,
-    QueueHumanInputFormTimeoutEvent,
     QueueIterationCompletedEvent,
     QueueIterationNextEvent,
     QueueIterationStartEvent,
@@ -44,7 +42,6 @@ from core.app.entities.queue_entities import (
     QueueTextChunkEvent,
     QueueWorkflowFailedEvent,
     QueueWorkflowPartialSuccessEvent,
-    QueueWorkflowPausedEvent,
     QueueWorkflowStartedEvent,
     QueueWorkflowSucceededEvent,
     WorkflowQueueMessage,
@@ -66,8 +63,6 @@ from core.base.tts import AppGeneratorTTSPublisher, AudioTrunk
 from core.model_runtime.entities.llm_entities import LLMUsage
 from core.model_runtime.utils.encoders import jsonable_encoder
 from core.ops.ops_trace_manager import TraceQueueManager
-from core.repositories.human_input_repository import HumanInputFormRepositoryImpl
-from core.workflow.entities.pause_reason import HumanInputRequired
 from core.workflow.enums import WorkflowExecutionStatus
 from core.workflow.nodes import NodeType
 from core.workflow.repositories.draft_variable_repository import DraftVariableSaverFactory
@@ -76,8 +71,7 @@ from core.workflow.system_variable import SystemVariable
 from extensions.ext_database import db
 from libs.datetime_utils import naive_utc_now
 from models import Account, Conversation, EndUser, Message, MessageFile
-from models.enums import CreatorUserRole, MessageStatus
-from models.execution_extra_content import HumanInputContent
+from models.enums import CreatorUserRole
 from models.workflow import Workflow
 
 logger = logging.getLogger(__name__)
@@ -134,7 +128,6 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
         )
 
         self._task_state = WorkflowTaskState()
-        self._seed_task_state_from_message(message)
         self._message_cycle_manager = MessageCycleManager(
             application_generate_entity=application_generate_entity, task_state=self._task_state
         )
@@ -142,7 +135,6 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
         self._application_generate_entity = application_generate_entity
         self._workflow_id = workflow.id
         self._workflow_features_dict = workflow.features_dict
-        self._workflow_tenant_id = workflow.tenant_id
         self._conversation_id = conversation.id
         self._conversation_mode = conversation.mode
         self._message_id = message.id
@@ -152,13 +144,8 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
         self._workflow_run_id: str = ""
         self._draft_var_saver_factory = draft_var_saver_factory
         self._graph_runtime_state: GraphRuntimeState | None = None
-        self._message_saved_on_pause = False
         self._seed_graph_runtime_state_from_queue_manager()
 
-    def _seed_task_state_from_message(self, message: Message) -> None:
-        if message.status == MessageStatus.PAUSED and message.answer:
-            self._task_state.answer = message.answer
-
     def process(self) -> Union[ChatbotAppBlockingResponse, Generator[ChatbotAppStreamResponse, None, None]]:
         """
         Process generate task pipeline.
@@ -321,7 +308,6 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
             task_id=self._application_generate_entity.task_id,
             workflow_run_id=run_id,
             workflow_id=self._workflow_id,
-            reason=event.reason,
         )
 
         yield workflow_start_resp
@@ -539,35 +525,6 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
         )
 
         yield workflow_finish_resp
-
-    def _handle_workflow_paused_event(
-        self,
-        event: QueueWorkflowPausedEvent,
-        **kwargs,
-    ) -> Generator[StreamResponse, None, None]:
-        """Handle workflow paused events."""
-        validated_state = self._ensure_graph_runtime_initialized()
-        responses = self._workflow_response_converter.workflow_pause_to_stream_response(
-            event=event,
-            task_id=self._application_generate_entity.task_id,
-            graph_runtime_state=validated_state,
-        )
-        for reason in event.reasons:
-            if isinstance(reason, HumanInputRequired):
-                self._persist_human_input_extra_content(form_id=reason.form_id, node_id=reason.node_id)
-        yield from responses
-        resolved_state: GraphRuntimeState | None = None
-        try:
-            resolved_state = self._ensure_graph_runtime_initialized()
-        except ValueError:
-            resolved_state = None
-
-        with self._database_session() as session:
-            self._save_message(session=session, graph_runtime_state=resolved_state)
-            message = self._get_message(session=session)
-            if message is not None:
-                message.status = MessageStatus.PAUSED
-            self._message_saved_on_pause = True
         self._base_task_pipeline.queue_manager.publish(QueueAdvancedChatMessageEndEvent(), PublishFrom.TASK_PIPELINE)
 
     def _handle_workflow_failed_event(
@@ -657,10 +614,9 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
                 reason=QueueMessageReplaceEvent.MessageReplaceReason.OUTPUT_MODERATION,
             )
 
-        # Save message unless it has already been persisted on pause.
-        if not self._message_saved_on_pause:
-            with self._database_session() as session:
-                self._save_message(session=session, graph_runtime_state=resolved_state)
+        # Save message
+        with self._database_session() as session:
+            self._save_message(session=session, graph_runtime_state=resolved_state)
 
         yield self._message_end_to_stream_response()
 
@@ -669,14 +625,16 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
     ) -> Generator[StreamResponse, None, None]:
         """Handle retriever resources events."""
         self._message_cycle_manager.handle_retriever_resources(event)
-        yield from ()
+        return
+        yield  # Make this a generator
 
     def _handle_annotation_reply_event(
         self, event: QueueAnnotationReplyEvent, **kwargs
     ) -> Generator[StreamResponse, None, None]:
         """Handle annotation reply events."""
         self._message_cycle_manager.handle_annotation_reply(event)
-        yield from ()
+        return
+        yield  # Make this a generator
 
     def _handle_message_replace_event(
         self, event: QueueMessageReplaceEvent, **kwargs
@@ -684,65 +642,6 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
         """Handle message replace events."""
         yield self._message_cycle_manager.message_replace_to_stream_response(answer=event.text, reason=event.reason)
 
-    def _handle_human_input_form_filled_event(
-        self, event: QueueHumanInputFormFilledEvent, **kwargs
-    ) -> Generator[StreamResponse, None, None]:
-        """Handle human input form filled events."""
-        self._persist_human_input_extra_content(node_id=event.node_id)
-        yield self._workflow_response_converter.human_input_form_filled_to_stream_response(
-            event=event, task_id=self._application_generate_entity.task_id
-        )
-
-    def _handle_human_input_form_timeout_event(
-        self, event: QueueHumanInputFormTimeoutEvent, **kwargs
-    ) -> Generator[StreamResponse, None, None]:
-        """Handle human input form timeout events."""
-        yield self._workflow_response_converter.human_input_form_timeout_to_stream_response(
-            event=event, task_id=self._application_generate_entity.task_id
-        )
-
-    def _persist_human_input_extra_content(self, *, node_id: str | None = None, form_id: str | None = None) -> None:
-        if not self._workflow_run_id or not self._message_id:
-            return
-
-        if form_id is None:
-            if node_id is None:
-                return
-            form_id = self._load_human_input_form_id(node_id=node_id)
-            if form_id is None:
-                logger.warning(
-                    "HumanInput form not found for workflow run %s node %s",
-                    self._workflow_run_id,
-                    node_id,
-                )
-                return
-
-        with self._database_session() as session:
-            exists_stmt = select(HumanInputContent).where(
-                HumanInputContent.workflow_run_id == self._workflow_run_id,
-                HumanInputContent.message_id == self._message_id,
-                HumanInputContent.form_id == form_id,
-            )
-            if session.scalar(exists_stmt) is not None:
-                return
-
-            content = HumanInputContent(
-                workflow_run_id=self._workflow_run_id,
-                message_id=self._message_id,
-                form_id=form_id,
-            )
-            session.add(content)
-
-    def _load_human_input_form_id(self, *, node_id: str) -> str | None:
-        form_repository = HumanInputFormRepositoryImpl(
-            session_factory=db.engine,
-            tenant_id=self._workflow_tenant_id,
-        )
-        form = form_repository.get_form(self._workflow_run_id, node_id)
-        if form is None:
-            return None
-        return form.id
-
     def _handle_agent_log_event(self, event: QueueAgentLogEvent, **kwargs) -> Generator[StreamResponse, None, None]:
         """Handle agent log events."""
         yield self._workflow_response_converter.handle_agent_log(
@@ -760,7 +659,6 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
             QueueWorkflowStartedEvent: self._handle_workflow_started_event,
             QueueWorkflowSucceededEvent: self._handle_workflow_succeeded_event,
             QueueWorkflowPartialSuccessEvent: self._handle_workflow_partial_success_event,
-            QueueWorkflowPausedEvent: self._handle_workflow_paused_event,
             QueueWorkflowFailedEvent: self._handle_workflow_failed_event,
             # Node events
             QueueNodeRetryEvent: self._handle_node_retry_event,
@@ -782,8 +680,6 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
             QueueMessageReplaceEvent: self._handle_message_replace_event,
             QueueAdvancedChatMessageEndEvent: self._handle_advanced_chat_message_end_event,
             QueueAgentLogEvent: self._handle_agent_log_event,
-            QueueHumanInputFormFilledEvent: self._handle_human_input_form_filled_event,
-            QueueHumanInputFormTimeoutEvent: self._handle_human_input_form_timeout_event,
         }
 
     def _dispatch_event(
@@ -851,9 +747,6 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
                 case QueueWorkflowFailedEvent():
                     yield from self._handle_workflow_failed_event(event, trace_manager=trace_manager)
                     break
-                case QueueWorkflowPausedEvent():
-                    yield from self._handle_workflow_paused_event(event)
-                    break
 
                 case QueueStopEvent():
                     yield from self._handle_stop_event(event, graph_runtime_state=None, trace_manager=trace_manager)
@@ -879,11 +772,6 @@ class AdvancedChatAppGenerateTaskPipeline(GraphRuntimeStateSupport):
 
     def _save_message(self, *, session: Session, graph_runtime_state: GraphRuntimeState | None = None):
         message = self._get_message(session=session)
-        if message is None:
-            return
-
-        if message.status == MessageStatus.PAUSED:
-            message.status = MessageStatus.NORMAL
 
         # If there are assistant files, remove markdown image links from answer
         answer_text = self._task_state.answer

api/core/app/apps/agent_chat/app_runner.py

@@ -12,11 +12,11 @@ from core.app.apps.base_app_queue_manager import AppQueueManager, PublishFrom
 from core.app.apps.base_app_runner import AppRunner
 from core.app.entities.app_invoke_entities import AgentChatAppGenerateEntity
 from core.app.entities.queue_entities import QueueAnnotationReplyEvent
+from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_manager import ModelInstance
 from core.model_runtime.entities.llm_entities import LLMMode
 from core.model_runtime.entities.model_entities import ModelFeature, ModelPropertyKey
 from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
-from core.model_runtime.token_buffer_memory import TokenBufferMemory
 from core.moderation.base import ModerationError
 from extensions.ext_database import db
 from models.model import App, Conversation, Message
@@ -178,7 +178,7 @@ class AgentChatAppRunner(AppRunner):
 
         # change function call strategy based on LLM model
         llm_model = cast(LargeLanguageModel, model_instance.model_type_instance)
-        model_schema = llm_model.get_model_schema(model_instance.model_name, model_instance.credentials)
+        model_schema = llm_model.get_model_schema(model_instance.model, model_instance.credentials)
         if not model_schema:
             raise ValueError("Model schema not found")
 

api/core/app/apps/base_app_generator.py

@@ -5,8 +5,8 @@ from sqlalchemy.orm import Session
 
 from core.app.app_config.entities import VariableEntityType
 from core.app.entities.app_invoke_entities import InvokeFrom
+from core.file import File, FileUploadConfig
 from core.workflow.enums import NodeType
-from core.workflow.file import File, FileUploadConfig
 from core.workflow.repositories.draft_variable_repository import (
     DraftVariableSaver,
     DraftVariableSaverFactory,

api/core/app/apps/base_app_queue_manager.py

@@ -2,7 +2,7 @@ import logging
 import queue
 import threading
 import time
-from abc import ABC, abstractmethod
+from abc import abstractmethod
 from enum import IntEnum, auto
 from typing import Any
 
@@ -31,7 +31,7 @@ class PublishFrom(IntEnum):
     TASK_PIPELINE = auto()
 
 
-class AppQueueManager(ABC):
+class AppQueueManager:
     def __init__(self, task_id: str, user_id: str, invoke_from: InvokeFrom):
         if not user_id:
             raise ValueError("user is required")
@@ -122,7 +122,7 @@ class AppQueueManager(ABC):
         """Attach the live graph runtime state reference for downstream consumers."""
         self._graph_runtime_state = graph_runtime_state
 
-    def publish(self, event: AppQueueEvent, pub_from: PublishFrom) -> None:
+    def publish(self, event: AppQueueEvent, pub_from: PublishFrom):
         """
         Publish event to queue
         :param event:
@@ -133,7 +133,7 @@ class AppQueueManager(ABC):
         self._publish(event, pub_from)
 
     @abstractmethod
-    def _publish(self, event: AppQueueEvent, pub_from: PublishFrom) -> None:
+    def _publish(self, event: AppQueueEvent, pub_from: PublishFrom):
         """
         Publish event to queue
         :param event:

api/core/app/apps/base_app_runner.py

@@ -22,6 +22,8 @@ from core.app.entities.queue_entities import (
 from core.app.features.annotation_reply.annotation_reply import AnnotationReplyFeature
 from core.app.features.hosting_moderation.hosting_moderation import HostingModerationFeature
 from core.external_data_tool.external_data_fetch import ExternalDataFetch
+from core.file.enums import FileTransferMethod, FileType
+from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_manager import ModelInstance
 from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta, LLMUsage
 from core.model_runtime.entities.message_entities import (
@@ -32,19 +34,17 @@ from core.model_runtime.entities.message_entities import (
 )
 from core.model_runtime.entities.model_entities import ModelPropertyKey
 from core.model_runtime.errors.invoke import InvokeBadRequestError
-from core.model_runtime.token_buffer_memory import TokenBufferMemory
 from core.moderation.input_moderation import InputModeration
 from core.prompt.advanced_prompt_transform import AdvancedPromptTransform
 from core.prompt.entities.advanced_prompt_entities import ChatModelMessage, CompletionModelPromptTemplate, MemoryConfig
 from core.prompt.simple_prompt_transform import ModelMode, SimplePromptTransform
 from core.tools.tool_file_manager import ToolFileManager
-from core.workflow.file.enums import FileTransferMethod, FileType
 from extensions.ext_database import db
 from models.enums import CreatorUserRole
 from models.model import App, AppMode, Message, MessageAnnotation, MessageFile
 
 if TYPE_CHECKING:
-    from core.workflow.file.models import File
+    from core.file.models import File
 
 _logger = logging.getLogger(__name__)
 

api/core/app/apps/chat/app_runner.py

@@ -11,12 +11,12 @@ from core.app.entities.app_invoke_entities import (
 )
 from core.app.entities.queue_entities import QueueAnnotationReplyEvent
 from core.callback_handler.index_tool_callback_handler import DatasetIndexToolCallbackHandler
+from core.file import File
+from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_manager import ModelInstance
 from core.model_runtime.entities.message_entities import ImagePromptMessageContent
-from core.model_runtime.token_buffer_memory import TokenBufferMemory
 from core.moderation.base import ModerationError
 from core.rag.retrieval.dataset_retrieval import DatasetRetrieval
-from core.workflow.file import File
 from extensions.ext_database import db
 from models.model import App, Conversation, Message
 

api/core/app/apps/common/workflow_response_converter.py

@@ -5,14 +5,9 @@ from dataclasses import dataclass
 from datetime import datetime
 from typing import Any, NewType, Union
 
-from sqlalchemy import select
-from sqlalchemy.orm import Session
-
 from core.app.entities.app_invoke_entities import AdvancedChatAppGenerateEntity, InvokeFrom, WorkflowAppGenerateEntity
 from core.app.entities.queue_entities import (
     QueueAgentLogEvent,
-    QueueHumanInputFormFilledEvent,
-    QueueHumanInputFormTimeoutEvent,
     QueueIterationCompletedEvent,
     QueueIterationNextEvent,
     QueueIterationStartEvent,
@@ -24,13 +19,9 @@ from core.app.entities.queue_entities import (
     QueueNodeRetryEvent,
     QueueNodeStartedEvent,
     QueueNodeSucceededEvent,
-    QueueWorkflowPausedEvent,
 )
 from core.app.entities.task_entities import (
     AgentLogStreamResponse,
-    HumanInputFormFilledResponse,
-    HumanInputFormTimeoutResponse,
-    HumanInputRequiredResponse,
     IterationNodeCompletedStreamResponse,
     IterationNodeNextStreamResponse,
     IterationNodeStartStreamResponse,
@@ -40,18 +31,15 @@ from core.app.entities.task_entities import (
     NodeFinishStreamResponse,
     NodeRetryStreamResponse,
     NodeStartStreamResponse,
-    StreamResponse,
     WorkflowFinishStreamResponse,
-    WorkflowPauseStreamResponse,
     WorkflowStartStreamResponse,
 )
+from core.file import FILE_MODEL_IDENTITY, File
 from core.plugin.impl.datasource import PluginDatasourceManager
 from core.tools.entities.tool_entities import ToolProviderType
 from core.tools.tool_manager import ToolManager
 from core.trigger.trigger_manager import TriggerManager
 from core.variables.segments import ArrayFileSegment, FileSegment, Segment
-from core.workflow.entities.pause_reason import HumanInputRequired
-from core.workflow.entities.workflow_start_reason import WorkflowStartReason
 from core.workflow.enums import (
     NodeType,
     SystemVariableKey,
@@ -59,16 +47,12 @@ from core.workflow.enums import (
     WorkflowNodeExecutionMetadataKey,
     WorkflowNodeExecutionStatus,
 )
-from core.workflow.file import FILE_MODEL_IDENTITY, File
 from core.workflow.runtime import GraphRuntimeState
 from core.workflow.system_variable import SystemVariable
 from core.workflow.workflow_entry import WorkflowEntry
 from core.workflow.workflow_type_encoder import WorkflowRuntimeTypeConverter
-from extensions.ext_database import db
 from libs.datetime_utils import naive_utc_now
 from models import Account, EndUser
-from models.human_input import HumanInputForm
-from models.workflow import WorkflowRun
 from services.variable_truncator import BaseTruncator, DummyVariableTruncator, VariableTruncator
 
 NodeExecutionId = NewType("NodeExecutionId", str)
@@ -207,7 +191,6 @@ class WorkflowResponseConverter:
         task_id: str,
         workflow_run_id: str,
         workflow_id: str,
-        reason: WorkflowStartReason,
     ) -> WorkflowStartStreamResponse:
         run_id = self._ensure_workflow_run_id(workflow_run_id)
         started_at = naive_utc_now()
@@ -221,7 +204,6 @@ class WorkflowResponseConverter:
                 workflow_id=workflow_id,
                 inputs=self._workflow_inputs,
                 created_at=int(started_at.timestamp()),
-                reason=reason,
             ),
         )
 
@@ -268,7 +250,7 @@ class WorkflowResponseConverter:
             data=WorkflowFinishStreamResponse.Data(
                 id=run_id,
                 workflow_id=workflow_id,
-                status=status,
+                status=status.value,
                 outputs=encoded_outputs,
                 error=error,
                 elapsed_time=elapsed_time,
@@ -282,160 +264,6 @@ class WorkflowResponseConverter:
             ),
         )
 
-    def workflow_pause_to_stream_response(
-        self,
-        *,
-        event: QueueWorkflowPausedEvent,
-        task_id: str,
-        graph_runtime_state: GraphRuntimeState,
-    ) -> list[StreamResponse]:
-        run_id = self._ensure_workflow_run_id()
-        started_at = self._workflow_started_at
-        if started_at is None:
-            raise ValueError(
-                "workflow_pause_to_stream_response called before workflow_start_to_stream_response",
-            )
-        paused_at = naive_utc_now()
-        elapsed_time = (paused_at - started_at).total_seconds()
-        encoded_outputs = self._encode_outputs(event.outputs) or {}
-        if self._application_generate_entity.invoke_from == InvokeFrom.SERVICE_API:
-            encoded_outputs = {}
-        pause_reasons = [reason.model_dump(mode="json") for reason in event.reasons]
-        human_input_form_ids = [reason.form_id for reason in event.reasons if isinstance(reason, HumanInputRequired)]
-        expiration_times_by_form_id: dict[str, datetime] = {}
-        if human_input_form_ids:
-            stmt = select(HumanInputForm.id, HumanInputForm.expiration_time).where(
-                HumanInputForm.id.in_(human_input_form_ids)
-            )
-            with Session(bind=db.engine) as session:
-                for form_id, expiration_time in session.execute(stmt):
-                    expiration_times_by_form_id[str(form_id)] = expiration_time
-
-        responses: list[StreamResponse] = []
-
-        for reason in event.reasons:
-            if isinstance(reason, HumanInputRequired):
-                expiration_time = expiration_times_by_form_id.get(reason.form_id)
-                if expiration_time is None:
-                    raise ValueError(f"HumanInputForm not found for pause reason, form_id={reason.form_id}")
-                responses.append(
-                    HumanInputRequiredResponse(
-                        task_id=task_id,
-                        workflow_run_id=run_id,
-                        data=HumanInputRequiredResponse.Data(
-                            form_id=reason.form_id,
-                            node_id=reason.node_id,
-                            node_title=reason.node_title,
-                            form_content=reason.form_content,
-                            inputs=reason.inputs,
-                            actions=reason.actions,
-                            display_in_ui=reason.display_in_ui,
-                            form_token=reason.form_token,
-                            resolved_default_values=reason.resolved_default_values,
-                            expiration_time=int(expiration_time.timestamp()),
-                        ),
-                    )
-                )
-
-        responses.append(
-            WorkflowPauseStreamResponse(
-                task_id=task_id,
-                workflow_run_id=run_id,
-                data=WorkflowPauseStreamResponse.Data(
-                    workflow_run_id=run_id,
-                    paused_nodes=list(event.paused_nodes),
-                    outputs=encoded_outputs,
-                    reasons=pause_reasons,
-                    status=WorkflowExecutionStatus.PAUSED,
-                    created_at=int(started_at.timestamp()),
-                    elapsed_time=elapsed_time,
-                    total_tokens=graph_runtime_state.total_tokens,
-                    total_steps=graph_runtime_state.node_run_steps,
-                ),
-            )
-        )
-
-        return responses
-
-    def human_input_form_filled_to_stream_response(
-        self, *, event: QueueHumanInputFormFilledEvent, task_id: str
-    ) -> HumanInputFormFilledResponse:
-        run_id = self._ensure_workflow_run_id()
-        return HumanInputFormFilledResponse(
-            task_id=task_id,
-            workflow_run_id=run_id,
-            data=HumanInputFormFilledResponse.Data(
-                node_id=event.node_id,
-                node_title=event.node_title,
-                rendered_content=event.rendered_content,
-                action_id=event.action_id,
-                action_text=event.action_text,
-            ),
-        )
-
-    def human_input_form_timeout_to_stream_response(
-        self, *, event: QueueHumanInputFormTimeoutEvent, task_id: str
-    ) -> HumanInputFormTimeoutResponse:
-        run_id = self._ensure_workflow_run_id()
-        return HumanInputFormTimeoutResponse(
-            task_id=task_id,
-            workflow_run_id=run_id,
-            data=HumanInputFormTimeoutResponse.Data(
-                node_id=event.node_id,
-                node_title=event.node_title,
-                expiration_time=int(event.expiration_time.timestamp()),
-            ),
-        )
-
-    @classmethod
-    def workflow_run_result_to_finish_response(
-        cls,
-        *,
-        task_id: str,
-        workflow_run: WorkflowRun,
-        creator_user: Account | EndUser,
-    ) -> WorkflowFinishStreamResponse:
-        run_id = workflow_run.id
-        elapsed_time = workflow_run.elapsed_time
-
-        encoded_outputs = workflow_run.outputs_dict
-        finished_at = workflow_run.finished_at
-        assert finished_at is not None
-
-        created_by: Mapping[str, object]
-        user = creator_user
-        if isinstance(user, Account):
-            created_by = {
-                "id": user.id,
-                "name": user.name,
-                "email": user.email,
-            }
-        else:
-            created_by = {
-                "id": user.id,
-                "user": user.session_id,
-            }
-
-        return WorkflowFinishStreamResponse(
-            task_id=task_id,
-            workflow_run_id=run_id,
-            data=WorkflowFinishStreamResponse.Data(
-                id=run_id,
-                workflow_id=workflow_run.workflow_id,
-                status=workflow_run.status,
-                outputs=encoded_outputs,
-                error=workflow_run.error,
-                elapsed_time=elapsed_time,
-                total_tokens=workflow_run.total_tokens,
-                total_steps=workflow_run.total_steps,
-                created_by=created_by,
-                created_at=int(workflow_run.created_at.timestamp()),
-                finished_at=int(finished_at.timestamp()),
-                files=cls.fetch_files_from_node_outputs(encoded_outputs),
-                exceptions_count=workflow_run.exceptions_count,
-            ),
-        )
-
     def workflow_node_start_to_stream_response(
         self,
         *,
@@ -512,13 +340,13 @@ class WorkflowResponseConverter:
         metadata = self._merge_metadata(event.execution_metadata, snapshot)
 
         if isinstance(event, QueueNodeSucceededEvent):
-            status = WorkflowNodeExecutionStatus.SUCCEEDED
+            status = WorkflowNodeExecutionStatus.SUCCEEDED.value
             error_message = event.error
         elif isinstance(event, QueueNodeFailedEvent):
-            status = WorkflowNodeExecutionStatus.FAILED
+            status = WorkflowNodeExecutionStatus.FAILED.value
             error_message = event.error
         else:
-            status = WorkflowNodeExecutionStatus.EXCEPTION
+            status = WorkflowNodeExecutionStatus.EXCEPTION.value
             error_message = event.error
 
         return NodeFinishStreamResponse(
@@ -585,7 +413,7 @@ class WorkflowResponseConverter:
                 process_data_truncated=process_data_truncated,
                 outputs=outputs,
                 outputs_truncated=outputs_truncated,
-                status=WorkflowNodeExecutionStatus.RETRY,
+                status=WorkflowNodeExecutionStatus.RETRY.value,
                 error=event.error,
                 elapsed_time=elapsed_time,
                 execution_metadata=metadata,
@@ -764,8 +592,7 @@ class WorkflowResponseConverter:
             ),
         )
 
-    @classmethod
-    def fetch_files_from_node_outputs(cls, outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]:
+    def fetch_files_from_node_outputs(self, outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]:
         """
         Fetch files from node outputs
         :param outputs_dict: node outputs dict
@@ -774,7 +601,7 @@ class WorkflowResponseConverter:
         if not outputs_dict:
             return []
 
-        files = [cls._fetch_files_from_variable_value(output_value) for output_value in outputs_dict.values()]
+        files = [self._fetch_files_from_variable_value(output_value) for output_value in outputs_dict.values()]
         # Remove None
         files = [file for file in files if file]
         # Flatten list