Compare commits

..
Author SHA1 Message Date
shamoon 60bd48542f Sigh 2026-11-09 12:04:57 -08:00
shamoon b9e87870ce Merge branch 'dev' into chore/agents-contributing 2026-11-07 12:24:50 -08:00
shamoon 763059d479 Also the security slop 2026-11-07 12:24:17 -08:00
shamoon 0217ec4000 Emphasize discussing ideas before AI PRs 2026-11-07 09:14:13 -08:00
shamoon 0c918b03bf Chore: add agents.md, update contributing guidelines 2026-11-07 09:01:37 -08:00
14 changed files with 114 additions and 120 deletions

No files matched your search

@@ -1,5 +1,23 @@
title: "[Feature Request] "
body:
- type: markdown
attributes:
value: |
#### Use of AI tools
If this request was generated, in whole or in part, by an AI tool or agent you **must** say so using the field below. Failing to clearly disclose this is a violation of our [Code of Conduct](https://github.com/paperless-ngx/paperless-ngx/blob/main/CODE_OF_CONDUCT.md).
The project prefers thoughtful, well-described ideas to AI-generated content. Please describe the problem, your use case, the desired behavior, and who would benefit.
- type: dropdown
id: ai-usage
attributes:
label: AI usage
description: Was any part of this request written by an AI tool or agent?
options:
- No AI tools were used
- Yes, AI tools helped write this request
validations:
required: true
- type: textarea
id: description
attributes:
+16
View File
@@ -1,5 +1,21 @@
title: "[Support] "
body:
- type: markdown
attributes:
value: |
#### Use of AI tools
If this post was generated, in whole or in part, by an AI tool or agent you **must** say so using the field below. Failing to clearly disclose this is a violation of our [Code of Conduct](https://github.com/paperless-ngx/paperless-ngx/blob/main/CODE_OF_CONDUCT.md).
- type: dropdown
id: ai-usage
attributes:
label: AI usage
description: Was any part of this post written by an AI tool or agent?
options:
- No AI tools were used
- Yes, AI tools helped write this post
validations:
required: true
- type: textarea
id: description
attributes:
+44
View File
@@ -0,0 +1,44 @@
# Guidance for AI agents
Read [CONTRIBUTING.md](CONTRIBUTING.md) before working on this repository, especially [Use of AI Tools](CONTRIBUTING.md#use-of-ai-tools). These contribution rules apply to all agent-assisted work.
## Contribution boundaries
- Do not open or submit pull requests, including "on behalf of" a human. The project does not accept PRs from AI agents. Prepare local changes for a human to review, verify, and submit personally.
- The project does not accept PRs that are entirely or mostly AI-derived. Human submission and disclosure do not override this rule.
- AI-generated content must be clearly disclosed as described in the contributing guidelines. When handing work back to a human, identify what was generated or changed with AI assistance so they can disclose it accurately.
- Trust matters. Encourage first-time contributors to discuss their proposed change before preparing a PR, especially for large changes or changes involving AI-generated code. New features and enhancements should almost always target an existing feature request with community interest and discussion.
- The project prefers thoughtful, well-described ideas to AI-generated pull requests. Help contributors describe their problem, use case, desired behavior, and who would benefit. An implementation is not required for a valuable contribution, and a flawed implementation can obscure a useful idea before it receives proper discussion.
- AI tools have lowered the effort needed to produce a PR, but reviewing and maintaining it still requires substantial work. Encourage discussion of the idea before investing in implementation and review.
- Issue reports must describe observed behavior, reproduction steps, and relevant logs. Do not add speculative root causes, code analysis, or proposed fixes to agent-generated reports.
## Security reports
- AI-assisted security reports must meet the legitimacy and verification requirements in [SECURITY.md](SECURITY.md#ai-assisted-submissions). A human must verify a real Paperless-ngx code path, concrete reproduction steps, affected versions, and realistic security impact before submission.
- Account for existing permission checks, configuration requirements, and trust boundaries. Speculation, scanner output, or concerns about third-party software or deployment choices without a demonstrated Paperless-ngx vulnerability do not meet this threshold and may be closed without extended discussion.
- Disclose AI assistance and use the private vulnerability reporting channel specified in [SECURITY.md](SECURITY.md#reporting-a-vulnerability).
## Working on changes
- Read the relevant code, tests, and documentation before editing. Paperless-ngx is complex: trace interactions between affected components rather than assuming a change is isolated.
- Keep changes focused on the requested work and follow existing patterns. Avoid unrelated refactoring, formatting, dependency updates, or generated-file changes.
- Consider effects on permissions, document processing, background tasks, storage, and the API when those areas are involved.
- Add or update meaningful tests for behavior changes and regressions. Do not weaken tests or checks to make a change pass.
- Update user or developer documentation when behavior or setup changes.
- Keep credentials and private documents out of commits, fixtures, logs, and examples.
## Repository and validation
- `src/`: Django back end.
- `src-ui/`: Angular front end, managed with `pnpm`.
- `docs/`: Documentation.
- `docker/` and `scripts/`: Deployment and development tooling.
- Base functional changes on `dev`; documentation-only changes may target `main`.
Follow [the development documentation](docs/development.md) for environment setup and testing. Common checks after setup are:
- Back end: run `uv run pytest` from `src/` (use a test path to focus on the affected area).
- Front end: run `pnpm test` and `pnpm lint` from `src-ui/`.
- Formatting and other pre-commit checks: run `uv run prek run --files <changed-files>` from the repository root.
When handing work back, summarize the changes, the checks actually run and their results, and any remaining uncertainty. Clearly state when a check could not be run; do not claim verification that did not happen. The human contributor must understand and independently verify the contribution before submitting it.
+16 -3
View File
@@ -4,6 +4,18 @@ If you feel like contributing to the project, please do! Bug fixes and improveme
⚠️ Please note: Pull requests that implement a new feature or enhancement _should almost always target an existing feature request_ with evidence of community interest and discussion. This is in order to balance the work of implementing and maintaining new features / enhancements. Pull requests that are opened without meeting this requirement may not be merged.
## Trust and collaboration
Paperless-ngx places a high value on trust between contributors and maintainers. If you have not contributed before, consider starting with a discussion before preparing a pull request, especially for a large contribution or one involving AI-generated code.
Paperless-ngx is complex, and understanding how its components interact takes time. A change that appears correct in isolation can have consequences elsewhere in the application. Contributors should understand the code they submit, explain how it fits into the project, and work with maintainers through review. Starting with smaller, focused contributions can help build that understanding and trust.
We prefer thoughtful, well-described ideas to AI-generated pull requests. You do not need to supply an implementation for your contribution to be valuable. Explain the problem you have encountered, your use case, the behavior you would like to see, and who would benefit. Concrete examples, alternatives you have considered, and relevant constraints help the community discuss and refine the idea.
AI tools have lowered the effort needed to produce a pull request, but evaluating one still requires substantial maintainer time. Maintainers still need to establish whether the implementation is correct, fits the project, and can be maintained over time.
A flawed implementation can also obscure a good idea: discussion becomes focused on problems with the generated code before the community has considered whether the underlying proposal is useful. Starting with the idea gives it a fair discussion on its own merits and lets contributors and maintainers agree on an approach before investing in implementation and review.
If you want to implement something big:
- As above, please start with a discussion! Maybe something similar is already in development and we can make it happen together.
@@ -61,9 +73,10 @@ This project does not specifically prohibit the use of AI tools or agents _durin
1. Any content — code, pull request descriptions, issue reports or comments — that was generated in whole or in part by an AI tool or agent **must be clearly disclosed as such**. Failing to clearly disclose this is considered a violation of our [Code of Conduct](CODE_OF_CONDUCT.md) and may result in your contributions being closed and further action being taken.
2. AI-generated code must not violate copyright protections.
3. We will not accept PRs that are entirely or mostly AI-derived.
4. Issue reports must describe _observed behavior only_: what happened, how to reproduce it and the relevant logs. Reports generated by an AI agent should not include code analysis, speculation about the root cause or suggest fixes.
5. Contributors are responsible for everything they submit, regardless of how it was produced. Please do not open issues or PRs you have not verified yourself.
3. We will not accept PRs opened or submitted by AI agents, including agents acting "on behalf of" a human. A human contributor must personally review, verify, and submit the contribution and take responsibility for it throughout review.
4. We will not accept PRs that are entirely or mostly AI-derived.
5. Issue reports must describe _observed behavior only_: what happened, how to reproduce it and the relevant logs. Reports generated by an AI agent should not include code analysis, speculation about the root cause or suggest fixes.
6. Contributors are responsible for everything they submit, regardless of how it was produced. Please do not open issues or PRs you have not verified yourself.
# Translating Paperless-ngx
-2
View File
@@ -565,8 +565,6 @@ Specify `rebuild` to build the index from scratch from all documents in the data
this the first time you enable the feature, or after changing the embedding backend or
model.
When using the `huggingface` embedding backend, a model that is already downloaded is loaded from the local cache without contacting the Hugging Face Hub. Only `rebuild` checks the Hub for a newer revision of the model, so a model update is picked up together with re-embedding every document. If the model is not cached yet, it is downloaded on first use.
Specify `update` to incrementally index new and changed documents. This is what the
scheduled task runs.
+8 -15
View File
@@ -4030,21 +4030,14 @@
<source>Sources</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
<context context-type="linenumber">47,48</context>
</context-group>
</trans-unit>
<trans-unit id="6025275430560167531" datatype="html">
<source>+<x id="INTERPOLATION" equiv-text="{{ hiddenReferenceCount(message) }}"/> more</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
<context context-type="linenumber">49,50</context>
<context context-type="linenumber">46,47</context>
</context-group>
</trans-unit>
<trans-unit id="8066608938393600549" datatype="html">
<source>Message</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
<context context-type="linenumber">66,67</context>
<context context-type="linenumber">61,62</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/common/email-document-dialog/email-document-dialog.component.html</context>
@@ -4055,22 +4048,22 @@
<source>Stop</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
<context context-type="linenumber">79</context>
<context context-type="linenumber">74</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
<context context-type="linenumber">79</context>
<context context-type="linenumber">74</context>
</context-group>
</trans-unit>
<trans-unit id="6490688569532630280" datatype="html">
<source>Send</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
<context context-type="linenumber">83</context>
<context context-type="linenumber">78</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
<context context-type="linenumber">83</context>
<context context-type="linenumber">78</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
@@ -4097,14 +4090,14 @@
<source>Ask a question about this document...</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.ts</context>
<context context-type="linenumber">83</context>
<context context-type="linenumber">61</context>
</context-group>
</trans-unit>
<trans-unit id="224049052163028286" datatype="html">
<source>Ask a question about a document...</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.ts</context>
<context context-type="linenumber">84</context>
<context context-type="linenumber">62</context>
</context-group>
</trans-unit>
<trans-unit id="8700121026680200191" datatype="html">
@@ -42,16 +42,11 @@
<span class="text-break">{{ message.content }}</span>
}
@if (message.role === 'assistant' && !message.isStreaming && message.references?.length) {
<div class="chat-references mt-2 pt-2 border-top">
<div class="d-flex align-items-center text-muted mb-1">
<span i18n>Sources</span>
@if (hiddenReferenceCount(message) > 0 && !expandedReferences().has(message)) {
<button type="button" class="badge rounded-pill bg-body-secondary text-body-secondary fw-normal border-0 ms-2" (click)="expandReferences(message)" i18n>+{{ hiddenReferenceCount(message) }} more</button>
}
</div>
@for (reference of visibleReferences(message); track reference.id) {
<a class="reference-link d-flex align-items-center lh-sm text-primary text-decoration-none py-1" [routerLink]="['/documents', reference.id]" [title]="reference.title">
<pngx-bs-icon width="0.8em" height="0.8em" name="file-text" class="me-2 flex-shrink-0"></pngx-bs-icon><span class="text-truncate">{{ reference.title }}</span>
<div class="chat-references list-group mt-3">
<div class="list-group-item text-muted small py-1" i18n>Sources</div>
@for (reference of message.references; track reference.id) {
<a class="list-group-item list-group-item-action d-flex align-items-center text-primary" [routerLink]="['/documents', reference.id]" [title]="reference.title">
<pngx-bs-icon width="0.9em" height="0.9em" name="file-text" class="me-2 flex-shrink-0"></pngx-bs-icon><span class="text-truncate">{{ reference.title }}</span>
</a>
}
</div>
@@ -12,10 +12,6 @@
display: none;
}
.message .flex-grow-1 {
min-width: 0;
}
.dropdown-item {
white-space: initial;
}
@@ -14,7 +14,6 @@ import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon
import { MarkdownPipe } from 'src/app/pipes/markdown.pipe'
import {
ChatMessage,
ChatReference,
ChatService,
parseChatResponse,
} from 'src/app/services/chat.service'
@@ -41,29 +40,8 @@ export class ChatComponent implements OnInit {
readonly input = signal('')
readonly documentId = signal<number>(undefined)
readonly slowResponse = signal(false)
readonly expandedReferences = signal<Set<ChatMessage>>(new Set())
public popperOptions = pngxPopperOptions
static readonly COLLAPSED_REFERENCE_COUNT = 2
public visibleReferences(message: ChatMessage): ChatReference[] {
if (this.expandedReferences().has(message)) {
return message.references
}
return message.references.slice(0, ChatComponent.COLLAPSED_REFERENCE_COUNT)
}
public hiddenReferenceCount(message: ChatMessage): number {
return Math.max(
0,
message.references.length - ChatComponent.COLLAPSED_REFERENCE_COUNT
)
}
public expandReferences(message: ChatMessage): void {
this.expandedReferences.update((expanded) => new Set(expanded).add(message))
}
private chatService: ChatService = inject(ChatService)
private router: Router = inject(Router)
+1 -1
View File
@@ -22,8 +22,8 @@ logger = logging.getLogger("paperless_ai.chat")
CHAT_METADATA_DELIMITER = "\n\n__PAPERLESS_CHAT_METADATA__"
CHAT_ERROR_MESSAGE = "Sorry, something went wrong while generating a response."
CHAT_NO_CONTENT_MESSAGE = "Sorry, I couldn't find any content to answer your question."
MAX_CHAT_REFERENCES = 3
CHAT_RETRIEVER_TOP_K = 5
MAX_CHAT_REFERENCES = CHAT_RETRIEVER_TOP_K
def _build_chat_prompt(output_language: str | None) -> str:
+5 -20
View File
@@ -1,5 +1,4 @@
import re
from contextlib import suppress
from typing import TYPE_CHECKING
from django.conf import settings
@@ -21,17 +20,7 @@ OCR_LEADER_REGEX = re.compile(r"[._\-\u00b7]{4,}")
HORIZONTAL_WHITESPACE_REGEX = re.compile(r"[ \t\u00a0]+")
def get_embedding_model(
config: AIConfig,
*,
refresh: bool = False,
) -> "BaseEmbedding":
"""
Build the embedding model for the configured backend.
For Hugging Face, a cached model loads offline by default, saving seconds per
call. Pass ``refresh=True`` to check the Hub for a newer model revision.
"""
def get_embedding_model(config: AIConfig) -> "BaseEmbedding":
match config.llm_embedding_backend:
case LLMEmbeddingBackend.OPENAI_LIKE:
from llama_index.embeddings.openai_like import OpenAILikeEmbedding
@@ -63,15 +52,11 @@ def get_embedding_model(
case LLMEmbeddingBackend.HUGGINGFACE:
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
hf_kwargs = {
"model_name": config.llm_embedding_model
return HuggingFaceEmbedding(
model_name=config.llm_embedding_model
or "sentence-transformers/all-MiniLM-L6-v2",
"cache_folder": str(settings.DATA_DIR / "hf_cache"),
}
if not refresh:
with suppress(OSError):
return HuggingFaceEmbedding(**hf_kwargs, local_files_only=True)
return HuggingFaceEmbedding(**hf_kwargs)
cache_folder=str(settings.DATA_DIR / "hf_cache"),
)
case LLMEmbeddingBackend.OLLAMA:
from llama_index.embeddings.ollama import OllamaEmbedding
from ollama import AsyncClient
+1 -1
View File
@@ -470,7 +470,7 @@ def update_llm_index(
logger.warning("No documents found to index.")
chunk_size = config.llm_embedding_chunk_size
embed_model = get_embedding_model(config, refresh=rebuild)
embed_model = get_embedding_model(config)
with write_store(embed_model_name=model_name) as store:
if rebuild or not store.table_exists():
@@ -212,10 +212,6 @@ def test_update_llm_index(
ai_config.assert_called_once()
build_document_node.assert_called_once_with(real_document, chunk_size=512)
indexing.get_embedding_model.assert_called_once_with(
mock_config,
refresh=True,
)
@pytest.mark.django_db
@@ -340,8 +336,6 @@ def test_update_llm_index_partial_update(
indexing.update_llm_index(rebuild=False)
assert indexing.get_embedding_model.call_args.kwargs == {"refresh": False}
with indexing.get_vector_store() as store:
assert store.table_exists(), (
"Expected the vector store table to exist after incremental update"
-36
View File
@@ -1,6 +1,5 @@
from unittest.mock import ANY
from unittest.mock import MagicMock
from unittest.mock import call
from unittest.mock import patch
import pytest
@@ -168,45 +167,10 @@ def test_get_embedding_model_huggingface(mock_ai_config):
MockHuggingFaceEmbedding.assert_called_once_with(
model_name="sentence-transformers/all-MiniLM-L6-v2",
cache_folder=str(settings.DATA_DIR / "hf_cache"),
local_files_only=True,
)
assert model == MockHuggingFaceEmbedding.return_value
def test_get_embedding_model_huggingface_downloads_when_not_cached(mock_ai_config):
mock_ai_config.return_value.llm_embedding_backend = LLMEmbeddingBackend.HUGGINGFACE
mock_ai_config.return_value.llm_embedding_model = "some/model"
downloaded = MagicMock()
with patch(
"llama_index.embeddings.huggingface.HuggingFaceEmbedding",
side_effect=[OSError("not cached"), downloaded],
) as MockHuggingFaceEmbedding:
model = get_embedding_model(mock_ai_config.return_value)
cache_folder = str(settings.DATA_DIR / "hf_cache")
assert MockHuggingFaceEmbedding.call_args_list == [
call(model_name="some/model", cache_folder=cache_folder, local_files_only=True),
call(model_name="some/model", cache_folder=cache_folder),
]
assert model is downloaded
def test_get_embedding_model_huggingface_refresh_checks_the_hub(mock_ai_config):
mock_ai_config.return_value.llm_embedding_backend = LLMEmbeddingBackend.HUGGINGFACE
mock_ai_config.return_value.llm_embedding_model = "some/model"
with patch(
"llama_index.embeddings.huggingface.HuggingFaceEmbedding",
) as MockHuggingFaceEmbedding:
get_embedding_model(mock_ai_config.return_value, refresh=True)
MockHuggingFaceEmbedding.assert_called_once_with(
model_name="some/model",
cache_folder=str(settings.DATA_DIR / "hf_cache"),
)
def test_get_embedding_model_ollama(mock_ai_config):
mock_ai_config.return_value.llm_embedding_backend = LLMEmbeddingBackend.OLLAMA
mock_ai_config.return_value.llm_embedding_model = "embeddinggemma"