Flowcept

Lightweight Distributed Workflow Provenance

Flowcept captures and queries workflow provenance at runtime with minimal code changes and low overhead. It unifies data from diverse tools and workflows across the Edge–Cloud–HPC continuum and provides ML-aware capture, MCP agents provenance, telemetry, extensible adapters, and flexible storage.

Important

Start here: Default Developer Guide (recommended first read for workflow developers). Full markdown version in repo: docs/README.md.

Contents:

• Default Developer Guide
  • What this guide covers
  • Quick profile switch
• Quick Start
  • Install and Initialize
  • Run a Minimal Example
  • Inspecting the Output
  • Next Steps
• Flowcept Architecture
  • Overview
  • Capture Mechanisms
  • Streaming Hub and Buffering
  • Provenance Keeper and Storage
  • Access and Querying
  • Deployment Flexibility
• Setup and Installation
  • Default Installation
  • Installing Specific Adapters and Additional Dependencies
  • Installing with Common Runtime Bundle
  • Install all optional dependencies at once
  • Installing from Source
  • Setup
  • Message Queue (MQ)
  • Database (DB)
  • Notes
  • Downloading and Starting External Services (MQ or DB)
• Flowcept Settings File
  • What You Can Configure
  • Custom Settings File
  • Environment Variables
• Flowcept Agent
  • Online-first design
  • Offline (file-based) queries
• Provenance Capture Methods
  • The Flowcept() Object
  • Data Observability Adapters
  • Decorators
  • Loop Instrumentation
  • Custom Task Creation (fully customizable)
  • References & Examples
• Telemetry Capture
  • Configuration (per-type toggles)
  • How telemetry attaches to provenance
  • Supported telemetry types
  • GPU telemetry
  • PyTorch model telemetry
  • Direct access to Telemetry objects
  • Practical tips
  • Reference
• Provenance Storage
  • Saving the In-Memory Buffer to Disk
• Provenance Consumer
  • Example: Extending the Base Consumer
  • Document Inserter
  • Extensibility
• Blob Data
  • When To Use Flowcept.db for blob data management
  • Blob Storage API
  • Version control mode (Git-like history)
  • Simple Example: Store Bytes + Linkage Fields
  • Simple Example: Update Existing Object Metadata
  • Simple Example: Save Dataset Snapshot (Alias API)
  • PyTorch Model Helpers (Flowcept.db)
  • Notes
• Versioned Single-Layer Perceptron Example
  • Notes
• Reporting
  • API
  • Markdown Provenance Cards (Default)
  • Render Markdown Directly in Terminal (Rich)
  • Input Modes
  • Aggregation
  • Object Metadata Summary
  • Real Example (Rendered in RST)
  • PDF Reports (Optional)
• Provenance Querying
  • Querying with the Command‑Line Interface
  • Querying via the Python API (Flowcept.db)
  • Accessing the in-memory buffer
  • Working Directly with MongoDB
  • Working with LMDB
  • Monitoring Provenance with Grafana
  • Querying via the LLM‑based Flowcept Agent
  • Conclusion
• REST API
  • Default list ordering
  • Base URL
  • Interactive API docs
  • Run locally
  • Endpoints
  • Query endpoint body
  • OpenAPI artifact
  • ReadTheDocs OpenAPI downloads
• Data Schemas
  • Task Data Schema
  • Workflow Data Schema
  • Blob Data Schema
• PROV-AGENT and Flowcept
  • How Flowcept represents PROV-AGENT
  • Figure
• Contributing
• CLI Reference
  • Usage pattern
  • Configuration Profiles
  • Settings bootstrap
  • Available commands
• API Reference
  • Main Flowcept Object
  • Flowcept.db: Querying the Database
  • Main Message Objects
  • FlowceptTask
  • FlowceptLoop
  • FlowceptLightweightLoop