francwa/alfred
1
0
Fork 0
Code Issues 1 Pull Requests 8 Actions Packages Projects Releases Wiki Activity

git commit -m "feat: major architectural refactor

Browse Source 
- Refactor memory system (episodic/STM/LTM with components)
- Implement complete subtitle domain (scanner, matcher, placer)
- Add YAML workflow infrastructure
- Externalize knowledge base (patterns, release groups)
- Add comprehensive testing suite
- Create manual testing CLIs"
This commit is contained in:
Francwa
2026-05-11 21:33:37 +02:00
parent 62b5d0b998
commit de02bdea06
103 changed files with 8559 additions and 1346 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+181 -343
View File
@@ -3,7 +3,7 @@
An AI-powered agent for managing your local media library with natural language. Search, download, and organize movies and TV shows effortlessly through a conversational interface.
[![Python 3.14](https://img.shields.io/badge/python-3.14-blue.svg)](https://www.python.org/downloads/)
[![Poetry](https://img.shields.io/badge/dependency%20manager-poetry-blue)](https://python-poetry.org/)
[![uv](https://img.shields.io/badge/dependency%20manager-uv-purple)](https://github.com/astral-sh/uv)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
@@ -13,9 +13,10 @@ An AI-powered agent for managing your local media library with natural language.
- 🔍 **Smart Search** — Find movies and TV shows via TMDB with rich metadata
- 📥 **Torrent Integration** — Search and download via qBittorrent
- 🧠 **Contextual Memory** — Remembers your preferences and conversation history
- 📁 **Auto-Organization** — Keeps your media library tidy and well-structured
- 🌐 **OpenAI-Compatible API** — Works with any OpenAI-compatible client
- 🖥️ **LibreChat Frontend** — Beautiful web UI included out of the box
- 📁 **Auto-Organization** — Moves and renames media files, resolves destinations, handles subtitles
- 🎞️ **Subtitle Pipeline** — Identifies, matches, and places subtitle tracks automatically
- 🔄 **Workflow Engine** — YAML-defined multi-step workflows (e.g. `organize_media`)
- 🌐 **OpenAI-Compatible API** — Works with any OpenAI-compatible client (LibreChat, OpenWebUI, etc.)
- 🔒 **Secure by Default** — Auto-generated secrets and encrypted credentials
## 🏗️ Architecture
@@ -26,33 +27,50 @@ Built with **Domain-Driven Design (DDD)** principles for clean separation of con
alfred/
├── agent/ # AI agent orchestration
│ ├── llm/ # LLM clients (Ollama, DeepSeek)
│ └── tools/ # Tool implementations
│ ├── tools/ # Tool implementations (api, filesystem, language)
│ └── workflows/ # YAML-defined multi-step workflows
├── application/ # Use cases & DTOs
│ ├── movies/ # Movie search use cases
│ ├── movies/ # Movie search
│ ├── torrents/ # Torrent management
│ └── filesystem/ # File operations
│ └── filesystem/ # File operations (move, list, subtitles, seed links)
├── domain/ # Business logic & entities
│ ├── media/ # Release parsing
│ ├── movies/ # Movie entities
│ ├── tv_shows/ # TV show entities
│ └── subtitles/ # Subtitle entities
│ ├── tv_shows/ # TV show entities & value objects
│ ├── subtitles/ # Subtitle scanner, services, knowledge base
│ └── shared/ # Common value objects (ImdbId, FilePath, FileSize)
└── infrastructure/ # External services & persistence
├── api/ # External API clients (TMDB, qBittorrent)
├── filesystem/ # File system operations
└── persistence/ # Memory & repositories
├── api/ # External API clients (TMDB, qBittorrent, Knaben)
├── filesystem/ # File manager (hard-link based, path-traversal safe)
├── persistence/ # Three-tier memory (LTM/STM/Episodic) + JSON repositories
└── subtitle/ # Subtitle infrastructure
```
See [docs/architecture_diagram.md](docs/architecture_diagram.md) for detailed architectural diagrams.
### Key flows
**Agent execution:** `agent.step(user_input)` → LLM call → if tool_calls, execute each via registry → loop until no tool calls or `max_tool_iterations` → return final response.
**Media organization workflow:**
1. `resolve_destination` — Determines target folder/filename from release name
2. `move_media` — Hard-links file to library, deletes source
3. `manage_subtitles` — Scans, classifies, and places subtitle tracks
4. `create_seed_links` — Hard-links library file back to torrents/ for continued seeding
**Memory tiers:**
- **LTM** (`data/memory/ltm.json`) — Persisted config, media library, watchlist
- **STM** — Conversation history (capped at `MAX_HISTORY_MESSAGES`)
- **Episodic** — Transient search results, active downloads, recent errors
## 🚀 Quick Start
### Prerequisites
- **Python 3.14+** (required)
- **Poetry** (dependency manager)
- **Python 3.14+**
- **uv** (dependency manager)
- **Docker & Docker Compose** (recommended for full stack)
- **API Keys:**
- TMDB API key ([get one here](https://www.themoviedb.org/settings/api))
- Optional: DeepSeek, OpenAI, Anthropic, or other LLM provider keys
- Optional: DeepSeek or other LLM provider keys
### Installation
@@ -64,9 +82,15 @@ cd alfred_media_organizer
# Install dependencies
make install
# Install pre-commit hooks
make install-hooks
# Bootstrap environment (generates .env with secure secrets)
make bootstrap
# Validate your .env against the schema
make validate
# Edit .env with your API keys
nano .env
```
@@ -94,162 +118,95 @@ The web interface will be available at **http://localhost:3080**
### Running Locally (Development)
```bash
# Install dependencies
poetry install
# Start the API server
poetry run uvicorn alfred.app:app --reload --port 8000
uv run uvicorn alfred.app:app --reload --port 8000
```
## ⚙️ Configuration
### Environment Bootstrap
### Settings system
Alfred uses a smart bootstrap system that:
`settings.toml` is the single source of truth. The schema flows:
1. **Generates secure secrets** automatically (JWT tokens, database passwords, encryption keys)
2. **Syncs build variables** from `pyproject.toml` (versions, image names)
3. **Preserves existing secrets** when re-running (never overwrites your API keys)
4. **Computes database URIs** automatically from individual components
```
settings.toml → settings_schema.py → settings_bootstrap.py → .env + .env.make → settings.py
```
To add a setting: define it in `settings.toml`, run `make bootstrap`, then access via `settings.my_new_setting`.
```bash
# First time setup
make bootstrap
# Re-run after updating pyproject.toml (secrets are preserved)
# Validate existing .env against schema
make validate
# Re-run after settings.toml changes (existing secrets preserved)
make bootstrap
```
### Configuration File (.env)
**Never commit `.env` or `.env.make`** — both are gitignored and auto-generated.
The `.env` file is generated from `.env.example` with secure defaults:
### Key settings (.env)
```bash
# --- CORE SETTINGS ---
HOST=0.0.0.0
PORT=3080
# --- CORE ---
MAX_HISTORY_MESSAGES=10
MAX_TOOL_ITERATIONS=10
# --- LLM CONFIGURATION ---
# Providers: 'local' (Ollama), 'deepseek', 'openai', 'anthropic', 'google'
DEFAULT_LLM_PROVIDER=local
# Local LLM (Ollama - included in Docker stack)
# --- LLM ---
DEFAULT_LLM_PROVIDER=local # local (Ollama) | deepseek
OLLAMA_BASE_URL=http://ollama:11434
OLLAMA_MODEL=llama3.3:latest
LLM_TEMPERATURE=0.2
# --- API KEYS (fill only what you need) ---
TMDB_API_KEY=your-tmdb-key-here # Required for movie search
DEEPSEEK_API_KEY= # Optional
OPENAI_API_KEY= # Optional
ANTHROPIC_API_KEY= # Optional
# --- API KEYS ---
TMDB_API_KEY=your-tmdb-key # Required for movie/show search
DEEPSEEK_API_KEY= # Optional
# --- SECURITY (auto-generated, don't modify) ---
JWT_SECRET=<auto-generated>
JWT_REFRESH_SECRET=<auto-generated>
CREDS_KEY=<auto-generated>
CREDS_IV=<auto-generated>
# --- DATABASES (auto-generated passwords) ---
MONGO_PASSWORD=<auto-generated>
POSTGRES_PASSWORD=<auto-generated>
# --- SECURITY (auto-generated) ---
JWT_SECRET=<auto>
CREDS_KEY=<auto>
MONGO_PASSWORD=<auto>
```
### Security Keys
Security keys are defined in `pyproject.toml` and generated automatically:
```toml
[tool.alfred.security]
jwt_secret = "32:b64" # 32 bytes, base64 URL-safe
jwt_refresh_secret = "32:b64"
creds_key = "32:hex" # 32 bytes, hexadecimal (AES-256)
creds_iv = "16:hex" # 16 bytes, hexadecimal (AES IV)
mongo_password = "16:hex"
postgres_password = "16:hex"
```
**Formats:**
- `b64` — Base64 URL-safe (for JWT tokens)
- `hex` — Hexadecimal (for encryption keys, passwords)
## 🐳 Docker Services
### Service Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ alfred-net (bridge) │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ LibreChat │───▶│ Alfred │───▶│ MongoDB │ │
│ │ :3080 │ │ (core) │ │ :27017 │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌──────────────┐ │
│ │ │ Ollama │ │
│ │ │ (local) │ │
│ │ └──────────────┘ │
│ │ │
│ ┌──────┴───────────────────────────────────────────────┐ │
│ │ Optional Services (profiles) │ │
│ ├──────────────┬──────────────┬──────────────┬─────────┤ │
│ │ Meilisearch │ RAG API │ VectorDB │qBittor- │ │
│ │ :7700 │ :8000 │ :5432 │ rent │ │
│ │ [meili] │ [rag] │ [rag] │[qbit..] │ │
│ └──────────────┴──────────────┴──────────────┴─────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Docker Profiles
| Profile | Services | Use Case |
|---------|----------|----------|
| (default) | LibreChat, Alfred, MongoDB, Ollama | Basic setup |
| `meili` | + Meilisearch | Fast search |
| `rag` | + RAG API, VectorDB | Document retrieval |
| `qbittorrent` | + qBittorrent | Torrent downloads |
| `full` | All services | Complete setup |
| Profile | Extra services | Use case |
|---------|---------------|----------|
| (default) | — | LibreChat + Alfred + MongoDB + Ollama |
| `meili` | Meilisearch | Fast full-text search |
| `rag` | RAG API + VectorDB (PostgreSQL) | Document retrieval |
| `qbittorrent` | qBittorrent | Torrent downloads |
| `full` | All of the above | Complete setup |
```bash
# Start with specific profiles
make up p=rag,meili
make up p=full
```
### Docker Commands
```bash
make up # Start containers (default profile)
make up # Start (default profile)
make up p=full # Start with all services
make down # Stop all containers
make restart # Restart containers
make down # Stop
make restart # Restart
make logs # Follow logs
make ps # Show container status
make shell # Open bash in Alfred container
make build # Build production image
make build-test # Build test image
make ps # Container status
```
## 🛠️ Available Tools
The agent has access to these tools for interacting with your media library:
| Tool | Description |
|------|-------------|
| `find_media_imdb_id` | Search for movies/TV shows on TMDB by title |
| `find_torrent` | Search for torrents across multiple indexers |
| `get_torrent_by_index` | Get detailed info about a specific torrent result |
| `add_torrent_by_index` | Download a torrent by its index in search results |
| `get_torrent_by_index` | Get detailed info about a specific result |
| `add_torrent_by_index` | Download a torrent from search results |
| `add_torrent_to_qbittorrent` | Add a torrent via magnet link directly |
| `set_path_for_folder` | Configure folder paths for media organization |
| `list_folder` | List contents of a folder |
| `set_language` | Set preferred language for searches |
| `resolve_destination` | Compute the target library path for a release |
| `move_media` | Hard-link a file to its library destination |
| `manage_subtitles` | Scan, classify, and place subtitle tracks |
| `create_seed_links` | Prepare torrent folder so qBittorrent keeps seeding |
| `learn` | Teach Alfred a new pattern (release group, naming convention) |
| `set_path_for_folder` | Configure folder paths |
| `list_folder` | List contents of a configured folder |
| `set_language` | Set preferred language for the session |
## 💬 Usage Examples
@@ -266,11 +223,12 @@ Alfred: I found 3 torrents for Inception (2010):
You: Download the first one
Alfred: ✓ Added to qBittorrent! Download started.
Saving to: /downloads/Movies/Inception (2010)/
You: What's downloading right now?
Alfred: You have 1 active download:
- Inception.2010.1080p.BluRay.x264 (45% complete, ETA: 12 min)
You: Organize the Breaking Bad S01 download
Alfred: ✓ Resolved destination: /tv_shows/Breaking.Bad/Season 01/
✓ Moved 6 episode files
✓ Placed 6 subtitle tracks (fr, en)
✓ Seed links created in /torrents/
```
### Via API
@@ -279,219 +237,147 @@ Alfred: You have 1 active download:
# Health check
curl http://localhost:8000/health
# Chat with the agent (OpenAI-compatible)
# Chat (OpenAI-compatible)
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "alfred",
"messages": [
{"role": "user", "content": "Find The Matrix 4K"}
]
"messages": [{"role": "user", "content": "Find The Matrix 4K"}]
}'
# List available models
# List models
curl http://localhost:8000/v1/models
# View memory state (debug)
# View memory state
curl http://localhost:8000/memory/state
# Clear session memory
curl -X POST http://localhost:8000/memory/clear-session
```
### Via OpenWebUI or Other Clients
Alfred is compatible with any OpenAI-compatible client:
1. Add as OpenAI-compatible endpoint: `http://localhost:8000/v1`
2. Model name: `alfred`
3. No API key required (or use any placeholder)
Alfred is compatible with any OpenAI-compatible client. Point it at `http://localhost:8000/v1`, model `alfred`.
## 🧠 Memory System
Alfred uses a three-tier memory system for context management:
Alfred uses a three-tier memory system:
### Long-Term Memory (LTM)
- **Persistent** — Saved to JSON files
- **Contents:** Configuration, user preferences, media library state
- **Survives:** Application restarts
### Short-Term Memory (STM)
- **Session-based** — Stored in RAM
- **Contents:** Conversation history, current workflow state
- **Cleared:** On session end or restart
### Episodic Memory
- **Transient** — Stored in RAM
- **Contents:** Search results, active downloads, recent errors
- **Cleared:** Frequently, after task completion
| Tier | Storage | Contents | Lifetime |
|------|---------|----------|----------|
| **LTM** | JSON file (`data/memory/ltm.json`) | Config, library, watchlist, learned patterns | Permanent |
| **STM** | RAM | Conversation history (capped) | Session |
| **Episodic** | RAM | Search results, active downloads, errors | Short-lived |
## 🧪 Development
### Project Setup
```bash
# Install all dependencies (including dev)
poetry install
# Install pre-commit hooks
make install-hooks
# Run the development server
poetry run uvicorn alfred.app:app --reload
```
### Running Tests
```bash
# Run all tests (parallel execution)
# Run full suite (parallel)
make test
# Run with coverage report
make coverage
# Run specific test file
poetry run pytest tests/test_agent.py -v
# Run a single file
uv run pytest tests/test_agent.py -v
# Run specific test
poetry run pytest tests/test_config_loader.py::TestBootstrapEnv -v
# Run a single class
uv run pytest tests/test_agent.py::TestAgentInit -v
# Skip slow tests
uv run pytest -m "not slow"
```
### Test coverage
The suite covers:
- **Agent loop** — tool execution, history, max iterations, error handling
- **Tool registry** — OpenAI schema format, parameter extraction
- **Prompts** — system prompt building, tool inclusion
- **Memory** — LTM/STM/Episodic operations, persistence
- **Filesystem tools** — path traversal security, folder listing
- **File manager** — hard-link, move, seed links (real filesystem, no mocks)
- **Application use cases** — `resolve_destination`, `create_seed_links`, `list_folder`, `move_media`
- **Domain** — TV show/movie entities, shared value objects (`ImdbId`, `FilePath`, `FileSize`), subtitle scanner
- **Repositories** — JSON-backed movie, TV show, subtitle repos
- **Bootstrap** — secret generation, idempotency, URI construction
- **Workflows** — YAML loading, structure validation
- **Configuration** — boundary validation for all settings
### Code Quality
```bash
# Lint and auto-fix
make lint
# Format code
make format
# Clean build artifacts
make clean
make lint # Ruff check --fix
make format # Ruff format + check --fix
```
### Adding a New Tool
1. **Create the tool function** in `alfred/agent/tools/`:
1. Implement the function in `alfred/agent/tools/`:
```python
# alfred/agent/tools/api.py
def my_new_tool(param: str) -> dict[str, Any]:
"""
Short description of what this tool does.
This will be shown to the LLM to help it decide when to use this tool.
"""
"""Short description shown to the LLM to decide when to call this tool."""
memory = get_memory()
# Your implementation here
result = do_something(param)
return {
"status": "success",
"data": result
}
# ...
return {"status": "ok", "data": result}
```
2. **Register in the registry** (`alfred/agent/registry.py`):
2. Register it in `alfred/agent/registry.py`:
```python
tool_functions = [
# ... existing tools ...
api_tools.my_new_tool, # Add your tool here
api_tools.my_new_tool,
]
```
The tool will be automatically registered with its parameters extracted from the function signature.
The registry auto-generates the JSON schema from the function signature and docstring.
### Adding a Workflow
Create a YAML file in `alfred/agent/workflows/`:
```yaml
name: my_workflow
description: What this workflow does
steps:
- tool: resolve_destination
description: Find where the file should go
- tool: move_media
description: Move the file
```
Workflows are loaded automatically at startup.
### Version Management
```bash
# Bump version (must be on main branch)
make patch # 0.1.7 -> 0.1.8
make minor # 0.1.7 -> 0.2.0
make major # 0.1.7 -> 1.0.0
# Must be on main branch
make patch # 0.1.7 → 0.1.8
make minor # 0.1.7 → 0.2.0
make major # 0.1.7 → 1.0.0
```
## 📚 API Reference
### Endpoints
#### `GET /health`
Health check endpoint.
```json
{
"status": "healthy",
"version": "0.1.7"
}
```
#### `GET /v1/models`
List available models (OpenAI-compatible).
```json
{
"object": "list",
"data": [
{
"id": "alfred",
"object": "model",
"owned_by": "alfred"
}
]
}
```
#### `POST /v1/chat/completions`
Chat with the agent (OpenAI-compatible).
**Request:**
```json
{
"model": "alfred",
"messages": [
{"role": "user", "content": "Find Inception"}
],
"stream": false
}
```
**Response:**
```json
{
"id": "chatcmpl-xxx",
"object": "chat.completion",
"created": 1234567890,
"model": "alfred",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "I found Inception (2010)..."
},
"finish_reason": "stop"
}]
}
```
#### `GET /memory/state`
View full memory state (debug endpoint).
#### `POST /memory/clear-session`
Clear session memories (STM + Episodic).
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Health check |
| `GET` | `/v1/models` | List models (OpenAI-compatible) |
| `POST` | `/v1/chat/completions` | Chat (OpenAI-compatible, streaming supported) |
| `GET` | `/memory/state` | Full memory dump (debug) |
| `POST` | `/memory/clear-session` | Clear STM + Episodic |
| `GET` | `/memory/episodic/search-results` | Current search results |
## 🔧 Troubleshooting
### Agent doesn't respond
1. Check API keys in `.env`
2. Verify LLM provider is running:
2. Verify the LLM is running:
```bash
# For Ollama
docker logs alfred-ollama
# Check if model is pulled
docker exec alfred-ollama ollama list
```
3. Check Alfred logs: `docker logs alfred-core`
@@ -499,76 +385,34 @@ Clear session memories (STM + Episodic).
### qBittorrent connection failed
1. Verify qBittorrent is running: `docker ps | grep qbittorrent`
2. Check Web UI is enabled in qBittorrent settings
3. Verify credentials in `.env`:
```bash
QBITTORRENT_URL=http://qbittorrent:16140
QBITTORRENT_USERNAME=admin
QBITTORRENT_PASSWORD=<check-your-env>
```
### Database connection issues
1. Check MongoDB is healthy: `docker logs alfred-mongodb`
2. Verify credentials match in `.env`
3. Try restarting: `make restart`
2. Check credentials in `.env` (`QBITTORRENT_URL`, `QBITTORRENT_USERNAME`, `QBITTORRENT_PASSWORD`)
### Memory not persisting
1. Check `data/` directory exists and is writable
1. Check `data/` directory is writable
2. Verify volume mounts in `docker-compose.yaml`
3. Check file permissions: `ls -la data/`
### Bootstrap fails
1. Ensure `.env.example` exists
2. Check `pyproject.toml` has required sections:
```toml
[tool.alfred.settings]
[tool.alfred.security]
```
3. Run manually: `python scripts/bootstrap.py`
```bash
make validate # Check what's wrong with .env
make bootstrap # Regenerate (preserves existing secrets)
```
### Tests failing
1. Update dependencies: `poetry install`
2. Check Python version: `python --version` (needs 3.14+)
3. Run specific failing test with verbose output:
```bash
poetry run pytest tests/test_failing.py -v --tb=long
```
```bash
uv run pytest tests/test_failing.py -v --tb=long
```
## 🤝 Contributing
Contributions are welcome! Please follow these steps:
1. **Fork** the repository
2. **Create** a feature branch: `git checkout -b feature/my-feature`
3. **Make** your changes
4. **Run** tests: `make test`
5. **Run** linting: `make lint && make format`
6. **Commit**: `git commit -m "feat: add my feature"`
7. **Push**: `git push origin feature/my-feature`
8. **Create** a Pull Request
### Commit Convention
We use [Conventional Commits](https://www.conventionalcommits.org/):
- `feat:` New feature
- `fix:` Bug fix
- `docs:` Documentation
- `refactor:` Code refactoring
- `test:` Adding tests
- `chore:` Maintenance
## 📖 Documentation
- [Architecture Diagram](docs/architecture_diagram.md) — System architecture overview
- [Class Diagram](docs/class_diagram.md) — Class structure and relationships
- [Component Diagram](docs/component_diagram.md) — Component interactions
- [Sequence Diagram](docs/sequence_diagram.md) — Sequence flows
- [Flowchart](docs/flowchart.md) — System flowcharts
1. Fork the repository
2. Create a feature branch: `git checkout -b feat/my-feature`
3. Make your changes + add tests
4. Run `make test && make lint && make format`
5. Commit with [Conventional Commits](https://www.conventionalcommits.org/): `feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`, `infra:`
6. Open a Pull Request
## 📄 License
@@ -576,19 +420,13 @@ MIT License — see [LICENSE](LICENSE) file for details.
## 🙏 Acknowledgments
- [LibreChat](https://github.com/danny-avila/LibreChat) — Beautiful chat interface
- [LibreChat](https://github.com/danny-avila/LibreChat) — Chat interface
- [Ollama](https://ollama.ai/) — Local LLM runtime
- [DeepSeek](https://www.deepseek.com/) — LLM provider
- [TMDB](https://www.themoviedb.org/) — Movie database
- [TMDB](https://www.themoviedb.org/) — Movie & TV database
- [qBittorrent](https://www.qbittorrent.org/) — Torrent client
- [FastAPI](https://fastapi.tiangolo.com/) — Web framework
- [Pydantic](https://docs.pydantic.dev/) — Data validation
## 📬 Support
- 📧 Email: francois.hodiaumont@gmail.com
- 🐛 Issues: [GitHub Issues](https://github.com/francwa/alfred_media_organizer/issues)
- 💬 Discussions: [GitHub Discussions](https://github.com/francwa/alfred_media_organizer/discussions)
- [uv](https://github.com/astral-sh/uv) — Fast Python package manager
---