Traceary

Catalog / Dify

1.9.0

1.9.0 – Orchestrating Knowledge, Powering Workflows

11 months agosecurityaddedfixedOriginal notes

🚀 Introduction

In Dify 1.9.0, we are introducing two major new capabilities: the Knowledge Pipeline and the Queue-based Graph Engine.

The Knowledge Pipeline provides a modularized and extensible workflow for knowledge ingestion and processing, while the Queue-based Graph Engine makes workflow execution more robust and controllable. We believe these will help you build and debug AI applications more smoothly, and we look forward to your experiences to help us continuously improve.

đź“š Knowledge Pipeline

✨ Introduction

With the brand-new orchestration interface for knowledge pipelines, we introduce a fundamental architectural upgrade that reshapes how document processing are designed and executed, providing a more modular and flexible workflow that enables users to orchestrate every stage of the pipeline. Enhanced with a wide range of powerful plugins available in the marketplace, it empowers users to flexibly integrate diverse data sources and processing tools. Ultimately, this architecture enables building highly customized, domain-specific RAG solutions that meet enterprises’ growing demands for scalability, adaptability, and precision.

âť“ Why Do We Need It?

Previously, Dify's RAG users still encounter persistent challenges in real-world adoption — from inaccurate knowledge retrieval and information loss to limited data integration and extensibility. Common pain points include:

  • đź”— restricted integration of data sources
  • 🖼️ missing critical elements such as tables and images
  • ✂️ suboptimal chunking results

All of them lead to poor answer quality and hinder the model's overall performance.

In response, we reimagined RAG in Dify as an open and modular architecture, enabling developers, integrators, and domain experts to build document processing pipelines tailored to their specific requirements—from data ingestion to chunk storage and retrieval.

🛠️ Core Capabilities

🧩 Knowledge Pipeline Architecture

The Knowledge Pipeline is a visual, node-based orchestration system dedicated to document ingestion. It provides a customizable way to automate complex document processing, enabling fine-grained transformations and bridging raw content with structured, retrievable knowledge. Developers can build workflows step by step, like assembling puzzle pieces, making document handling easier to observe and adjust.

đź“‘ Templates & Pipeline DSL

  • ⚡ Start quickly with official templates
  • 🔄 Customize and share pipelines by importing/exporting via DSL for easier reusability and collaboration

🔌 Customizable Data Sources & Tools

Each knowledge base can support multiple data sources. You can seamlessly integrate local files, online documents, cloud drives, and web crawlers through a plugin-based ingestion framework. Developers can extend the ecosystem with new data-source plugins, while marketplace processors handle specialized use cases like formulas, spreadsheets, and image parsing — ensuring accurate ingestion and structured representation.

🧾 New Chunking Strategies

In addition to General and Parent-Child modes, the new Q&A Processor plugin supports Q&A structures. This expands coverage for more use cases, balancing retrieval precision with contextual completeness.

🖼️ Image Extraction & Retrieval

Extract images from documents in multiple formats, store them as URLs in the knowledge base, and enable mixed text-image outputs to improve LLM-generated answers.

🧪 Test Run & Debugging Support

Before publishing a pipeline, you can:

  • ▶️ Execute a single step or node independently
  • 🔍 Inspect intermediate variables in detail
  • đź‘€ Preview string variables as Markdown in the variable inspector

This provides safe iteration and debugging at every stage.

🔄 One-Click Migration from Legacy Knowledge Bases

Seamlessly convert existing knowledge bases into the Knowledge Pipeline architecture with a single action, ensuring smooth transition and backward compatibility.

🌟 Why It Matters

The Knowledge Pipeline makes knowledge management more transparent, debuggable, and extensible. It is not the endpoint, but a foundation for future enhancements such as multimodal retrieval, human-in-the-loop collaboration, and enterprise-level data governance. We’re excited to see how you apply it and share your feedback.

⚙️ Queue-based Graph Engine

âť“ Why Do We Need It?

Previously, designing workflows with parallel branches often led to:

  • 🌀 Difficulty managing branch states and reproducing errors
  • ❌ Insufficient debugging information
  • 🧱 Rigid execution logic lacking flexibility

These issues reduced the usability of complex workflows. To solve this, we redesigned the execution engine around queue scheduling, improving management of parallel tasks.

🛠️ Core Capabilities

đź“‹ Queue Scheduling Model

All tasks enter a unified queue, where the scheduler manages dependencies and order. This reduces errors in parallel execution and makes topology more intuitive.

🎯 Flexible Execution Start Points

Execution can begin at any node, supporting partial runs, resumptions, and subgraph invocations.

🌊 Stream Processing Component

A new ResponseCoordinator handles streaming outputs from multiple nodes, such as token-by-token LLM generation or staged results from long-running tasks.

🕹️ Command Mechanism

With the CommandProcessor, workflows can be paused, resumed, or terminated during execution, enabling external control.

🧩 GraphEngineLayer

A new plugin layer that allows extending engine functionality without modifying core code. It can monitor states, send commands, and support custom monitoring.

Quickstart

  1. Prerequisites
    • Dify version: 1.9.0 or higher
  2. How to Enable
    • Enabled by default, no additional configuration required.
    • Debug mode: set DEBUG=true to enable DebugLoggingLayer.
    • Execution limits:
      • WORKFLOW_MAX_EXECUTION_STEPS=500
      • WORKFLOW_MAX_EXECUTION_TIME=1200
      • WORKFLOW_CALL_MAX_DEPTH=10
    • Worker configuration (optional):
      • WORKFLOW_MIN_WORKERS=1
      • WORKFLOW_MAX_WORKERS=10
      • WORKFLOW_SCALE_UP_THRESHOLD=3
      • WORKFLOW_SCALE_DOWN_IDLE_TIME=30
    • Applies to all workflows.

More Controllable Parallel Branches

Execution Flow:

Start ─→ Unified Task Queue ─→ WorkerPool Scheduling
                          ├─→ Branch-1 Execution
                          └─→ Branch-2 Execution
                                  ↓
                            Aggregator
                                  ↓
                                  End

Improvements:
1. All tasks enter a single queue, managed by the Dispatcher.
2. WorkerPool auto-scales based on load.
3. ResponseCoordinator manages streaming outputs, ensuring correct order.

Example: Command Mechanism

from core.workflow.graph_engine.manager import GraphEngineManager

# Send stop command
GraphEngineManager.send_stop_command(
    task_id="workflow_task_123",
    reason="Emergency stop: resource limit exceeded"
)

Note: pause/resume functionality will be supported in future versions.

Example: GraphEngineLayer

GraphEngineLayer Example

FAQ

  1. Is this release focused on performance?
    No. The focus is on stability, clarity, and correctness of parallel branches. Performance improvements are a secondary benefit.

  2. What events can be subscribed to?

    • Graph-level: GraphRunStartedEvent, GraphRunSucceededEvent, GraphRunFailedEvent, GraphRunAbortedEvent
    • Node-level: NodeRunStartedEvent, NodeRunSucceededEvent, NodeRunFailedEvent, NodeRunRetryEvent
    • Container nodes: IterationRunStartedEvent, IterationRunNextEvent, IterationRunSucceededEvent, LoopRunStartedEvent, LoopRunNextEvent, LoopRunSucceededEvent
    • Streaming output: NodeRunStreamChunkEvent

  3. How can I debug workflow execution?

    • Enable DEBUG=true to view detailed logs.
    • Use DebugLoggingLayer to record events.
    • Add custom monitoring via GraphEngineLayer.

Future Plans

This release is just the beginning. Upcoming improvements include:

  • Debugging Tools: A visual interface to view execution states and variables in real time.
  • Intelligent Scheduling: Optimize scheduling strategies using historical data.
  • More Complete Command Support: Add Pause/Resume, breakpoint debugging.
  • Human in the Loop: Support human intervention during execution.
  • Subgraph Functionality: Enhance modularity and reusability.
  • Multimodal Embedding: Support richer content types beyond text.

We look forward to your feedback and experiences to make the engine more practical.

Upgrade Guide

[!IMPORTANT]
After upgrading, you must run the following migration to transform existing datasource credentials. This step is required to ensure compatibility with the new version:

uv run flask transform-datasource-credentials

Docker Compose Deployments

  1. Back up your customized docker-compose YAML file (optional)
cd docker
cp docker-compose.yaml docker-compose.yaml.$(date +%s).bak
  1. Get the latest code from the main branch
git checkout 1.9.0
git pull origin 1.9.0
  1. Stop the service. Please execute in the docker directory
docker compose down
  1. Back up data
tar -cvf volumes-$(date +%s).tgz volumes
  1. Upgrade services
docker compose up -d
  1. Migrate data after the container starts
docker exec -it docker-api-1 uv run flask transform-datasource-credentials

Source Code Deployments

  1. Stop the API server, Worker, and Web frontend Server.

  2. Get the latest code from the release branch:

git checkout 1.9.0
  1. Update Python dependencies:
cd api
uv sync
  1. Then, let's run the migration script:
uv run flask db upgrade
uv run flask transform-datasource-credentials

[!IMPORTANT] If you are currently using version v2.0.0-beta.1 or v2.0.0-beta.2, you must run the following migration script to upgrade.

[WARNING] This is a destructive operation that will result in data loss.

This migration script will permanently delete the following data:

  • All existing Knowledge Pipelines
  • All related Dataset Credentials

Please ensure you have backed up any necessary information before proceeding.

# in Dify envrionment :`v2.0.0-beta.1` or `v2.0.0-beta.2`
uv run flask db downgrade cf7c38a32b2d  

# in Dify envrionment :`v1.9.0` 
uv run flask db upgrade
uv run flask transform-datasource-credentials
  1. Finally, run the API server, Worker, and Web frontend Server again.

What's Changed

These notes run past the length kept in the archive. The rest is on the publisher’s page.