Compare commits

...
34 changed files with 1424 additions and 437 deletions

No files matched your search

+30 -5
View File
@@ -493,7 +493,7 @@ fields and permissions, which will be merged.
#### Types {#workflow-trigger-types}
Currently, there are four events that correspond to workflow trigger 'types':
Currently, there are five events that correspond to workflow trigger 'types':
1. **Consumption Started**: _before_ a document is consumed, so events can include filters by source (mail, consumption
folder or API), file path, file name, mail rule
@@ -505,8 +505,10 @@ Currently, there are four events that correspond to workflow trigger 'types':
4. **Scheduled**: a scheduled trigger that can be used to run workflows at a specific time. The date used can be either the document
added, created, updated date or you can specify a (date) custom field. You can also specify a day offset from the date (positive
offsets will trigger after the date, negative offsets will trigger before).
5. **Manual**: a trigger that only runs when you explicitly choose to run the workflow against one or more existing documents. See
[running a workflow manually](#running-a-workflow-manually) below.
The following flow diagram illustrates the four document trigger types:
The following flow diagram illustrates the document trigger types:
```mermaid
flowchart TD
@@ -526,6 +528,10 @@ flowchart TD
matching
trigger(s)"}
manual{"Matching
'Manual'
trigger(s)"}
A[New Document] --> consumption
consumption --> |Yes| C[Workflow Actions Run]
consumption --> |No| D
@@ -543,6 +549,10 @@ flowchart TD
scheduled --> |Yes| N[Workflow Actions Run]
scheduled --> |No| O[Document Saved]
N --> O
P[User Runs Workflow] --> manual
manual --> |Yes| Q[Workflow Actions Run]
manual --> |No| R[Document Saved]
Q --> R
```
#### Filters {#workflow-trigger-filters}
@@ -554,9 +564,9 @@ Workflows allow you to filter by:
- File path, including wildcards. Note that enabling `PAPERLESS_CONSUMER_RECURSIVE` would allow, for
example, automatically assigning documents to different owners based on the upload directory.
- Mail rule. Choosing this option will force 'mail fetch' to be the workflow source.
- Content matching (`Added`, `Updated` and `Scheduled` triggers only). Filter document content using the matching settings.
- Content matching (`Added`, `Updated`, `Scheduled` and `Manual` triggers only). Filter document content using the matching settings.
There are also 'advanced' filters available for `Added`, `Updated` and `Scheduled` triggers:
There are also 'advanced' filters available for `Added`, `Updated`, `Scheduled` and `Manual` triggers:
- Any Tags: Filter for documents with any of the specified tags.
- All Tags: Filter for documents with all of the specified tags.
@@ -569,6 +579,21 @@ There are also 'advanced' filters available for `Added`, `Updated` and `Schedule
- Not Storage paths: Filter documents without any of these storage paths.
- Custom field query: Filter documents with a custom field query (the same as used for the document list filters).
#### Running a workflow manually {#running-a-workflow-manually}
A workflow with a **Manual** trigger does not run automatically. Instead, it can be run on demand against existing
documents from the web UI:
- From the document list, select one or more documents and choose **Run workflow** from the "Actions" menu.
- From an individual document, choose **Run workflow** from the "Actions" menu.
You will be asked which workflow to run. By default the workflow's trigger filters still apply, so documents that do
not match are skipped. You can instead choose to **run on all selected documents**, which ignores the trigger filters
and runs the workflow's actions on every selected document.
Manual runs happen in the background. As with scheduled workflows, the document list is not refreshed automatically when
they finish, so you may need to reload the list to see the results.
### Workflow Actions
#### Types {#workflow-action-types}
@@ -649,7 +674,7 @@ new version of the same document, this time with normal OCR/text extraction appl
a password-protected file added with this trigger will briefly exist as an un-OCR'd version before
the properly processed version is created.
**Document Added**, **Document Updated**, **Scheduled**: these triggers run against a document that
**Document Added**, **Document Updated**, **Scheduled**, **Manual**: these triggers run against a document that
already exists, so password removal happens immediately: the decrypted content is queued for
consumption as a new version right away. Note that if the document's initial consumption also
happened while it was still encrypted, that original version will likewise be missing OCR text.
+298 -211
View File
File diff suppressed because it is too large. Load diff
@@ -118,6 +118,7 @@ import personFill from 'bootstrap-icons/icons/person-fill.svg'
import personLock from 'bootstrap-icons/icons/person-lock.svg'
import personSquare from 'bootstrap-icons/icons/person-square.svg'
import person from 'bootstrap-icons/icons/person.svg'
import playCircle from 'bootstrap-icons/icons/play-circle.svg'
import playFill from 'bootstrap-icons/icons/play-fill.svg'
import plusCircle from 'bootstrap-icons/icons/plus-circle.svg'
import plus from 'bootstrap-icons/icons/plus.svg'
@@ -279,6 +280,7 @@ export const icons: Record<string, string> = {
personFillLock,
personLock,
personSquare,
playCircle,
playFill,
plus,
plusCircle,
@@ -0,0 +1,32 @@
<div class="modal-header">
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()">
</button>
</div>
<div class="modal-body">
@if (message) {
<p class="text-break" [innerHTML]="message"></p>
}
<div class="mb-3">
<label class="form-label" for="runWorkflowSelect" i18n>Workflow</label>
<select class="form-select" id="runWorkflowSelect" [(ngModel)]="workflowId" (ngModelChange)="workflowSelected()">
<option [ngValue]="null" disabled i18n>Select a workflow</option>
@for (workflow of workflows; track workflow.id) {
<option [ngValue]="workflow.id">{{workflow.name}}</option>
}
</select>
</div>
<div class="form-check">
<input class="form-check-input" type="checkbox" id="runWorkflowForce" [(ngModel)]="force" />
<label class="form-check-label" for="runWorkflowForce" i18n>Run on all selected documents</label>
<div class="form-text" i18n>Ignore the workflow's trigger filters and run on every selected document.</div>
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled">
{{btnCaption}}
</button>
</div>
@@ -0,0 +1,42 @@
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { RunWorkflowConfirmDialogComponent } from './run-workflow-confirm-dialog.component'
describe('RunWorkflowConfirmDialogComponent', () => {
let component: RunWorkflowConfirmDialogComponent
let fixture: ComponentFixture<RunWorkflowConfirmDialogComponent>
beforeEach(async () => {
TestBed.configureTestingModule({
providers: [NgbActiveModal],
imports: [RunWorkflowConfirmDialogComponent],
}).compileComponents()
fixture = TestBed.createComponent(RunWorkflowConfirmDialogComponent)
component = fixture.componentInstance
component.workflows = [{ id: 5, name: 'Manual workflow' } as any]
fixture.detectChanges()
})
it('should disable confirm until a workflow is selected and not force by default', () => {
expect(component.confirmButtonEnabled).toBeFalsy()
expect(component.force).toBeFalsy()
component.workflowId = 5
component.workflowSelected()
expect(component.confirmButtonEnabled).toBeTruthy()
component.workflowId = null
component.workflowSelected()
expect(component.confirmButtonEnabled).toBeFalsy()
})
it('should render an option per workflow', () => {
const options = fixture.nativeElement.querySelectorAll(
'#runWorkflowSelect option'
)
// placeholder option + one per workflow
expect(options).toHaveLength(2)
expect(options[1].textContent).toContain('Manual workflow')
})
})
@@ -0,0 +1,24 @@
import { Component } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { Workflow } from 'src/app/data/workflow'
import { ConfirmDialogComponent } from '../confirm-dialog.component'
@Component({
selector: 'pngx-run-workflow-confirm-dialog',
templateUrl: './run-workflow-confirm-dialog.component.html',
imports: [FormsModule],
})
export class RunWorkflowConfirmDialogComponent extends ConfirmDialogComponent {
public workflows: Workflow[] = []
public workflowId: number = null
// i.e. ignore the workflow's trigger filters
public force: boolean = false
override confirmButtonEnabled = false
public workflowSelected(): void {
this.confirmButtonEnabled = !!this.workflowId
}
}
@@ -164,7 +164,7 @@
<pngx-input-text i18n-title title="Filter path" formControlName="filter_path" horizontal="true" i18n-hint hint="Apply to documents that match this path. Wildcards specified as * are allowed. Case-normalized." [error]="error?.filter_path"></pngx-input-text>
<pngx-input-select i18n-title title="Filter mail rule" [items]="mailRules()" horizontal="true" [allowNull]="true" formControlName="filter_mailrule" i18n-hint hint="Apply to documents consumed via this mail rule." [error]="error?.filter_mailrule"></pngx-input-select>
}
@if (formGroup.get('type').value === WorkflowTriggerType.DocumentAdded || formGroup.get('type').value === WorkflowTriggerType.DocumentUpdated || formGroup.get('type').value === WorkflowTriggerType.Scheduled) {
@if (formGroup.get('type').value === WorkflowTriggerType.DocumentAdded || formGroup.get('type').value === WorkflowTriggerType.DocumentUpdated || formGroup.get('type').value === WorkflowTriggerType.Scheduled || formGroup.get('type').value === WorkflowTriggerType.Manual) {
<pngx-input-select i18n-title title="Content matching algorithm" horizontal="true" [items]="getMatchingAlgorithms()" formControlName="matching_algorithm"></pngx-input-select>
@if (matchingPatternRequired(formGroup)) {
<pngx-input-text i18n-title title="Content matching pattern" horizontal="true" formControlName="match" [error]="error?.match"></pngx-input-text>
@@ -175,7 +175,7 @@
}
</div>
</div>
@if (formGroup.get('type').value === WorkflowTriggerType.DocumentAdded || formGroup.get('type').value === WorkflowTriggerType.DocumentUpdated || formGroup.get('type').value === WorkflowTriggerType.Scheduled) {
@if (formGroup.get('type').value === WorkflowTriggerType.DocumentAdded || formGroup.get('type').value === WorkflowTriggerType.DocumentUpdated || formGroup.get('type').value === WorkflowTriggerType.Scheduled || formGroup.get('type').value === WorkflowTriggerType.Manual) {
<div class="row mt-3">
<div class="col">
<div class="trigger-filters mb-3">
@@ -123,6 +123,10 @@ export const WORKFLOW_TYPE_OPTIONS = [
id: WorkflowTriggerType.Scheduled,
name: $localize`Scheduled`,
},
{
id: WorkflowTriggerType.Manual,
name: $localize`Manual`,
},
]
export const WORKFLOW_ACTION_OPTIONS = [
@@ -49,6 +49,12 @@
<pngx-bs-icon width="1em" height="1em" name="arrow-counterclockwise" class="me-2"></pngx-bs-icon><span i18n>Reprocess</span>
</button>
@if (manualWorkflows().length > 0) {
<button ngbDropdownItem (click)="runWorkflow()" [disabled]="!userCanEdit || !userIsOwner">
<pngx-bs-icon width="1em" height="1em" name="play-circle" class="me-2"></pngx-bs-icon><span i18n>Run workflow</span>
</button>
}
<button ngbDropdownItem (click)="printDocument()" [hidden]="useNativePdfViewer || isMobile">
<pngx-bs-icon width="1em" height="1em" name="printer" class="me-2"></pngx-bs-icon><span i18n>Print</span>
</button>
@@ -1030,6 +1030,27 @@ describe('DocumentDetailComponent', () => {
expect(modalCloseSpy).toHaveBeenCalled()
})
it('should run a workflow on the document, confirm and close modal', () => {
initNormally()
const bulkEditSpy = jest.spyOn(documentService, 'bulkEdit')
bulkEditSpy.mockReturnValue(of(true))
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
const toastSpy = jest.spyOn(toastService, 'showInfo')
component.runWorkflow()
const modalCloseSpy = jest.spyOn(openModal, 'close')
openModal.componentInstance.workflowId = 5
openModal.componentInstance.force = true
openModal.componentInstance.confirmClicked.next()
expect(bulkEditSpy).toHaveBeenCalledWith(
{ documents: [doc.id] },
'run_workflow',
{ workflow: 5, filter_documents: false }
)
expect(toastSpy).toHaveBeenCalled()
expect(modalCloseSpy).toHaveBeenCalled()
})
it('should pass remote OCR choice when reprocessing', () => {
initNormally()
const reprocessSpy = jest.spyOn(documentService, 'reprocessDocuments')
@@ -68,6 +68,8 @@ import { StoragePath } from 'src/app/data/storage-path'
import { Tag } from 'src/app/data/tag'
import { SETTINGS_KEYS, SuggestionSource } from 'src/app/data/ui-settings'
import { User } from 'src/app/data/user'
import { Workflow } from 'src/app/data/workflow'
import { WorkflowTriggerType } from 'src/app/data/workflow-trigger'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { CustomDatePipe } from 'src/app/pipes/custom-date.pipe'
import { DocumentTitlePipe } from 'src/app/pipes/document-title.pipe'
@@ -93,6 +95,7 @@ import { SavedViewService } from 'src/app/services/rest/saved-view.service'
import { StoragePathService } from 'src/app/services/rest/storage-path.service'
import { TagService } from 'src/app/services/rest/tag.service'
import { UserService } from 'src/app/services/rest/user.service'
import { WorkflowService } from 'src/app/services/rest/workflow.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { TrashService } from 'src/app/services/trash.service'
@@ -104,6 +107,7 @@ import { DocumentDetailFieldID } from '../admin/settings/settings.component'
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RunWorkflowConfirmDialogComponent } from '../common/confirm-dialog/run-workflow-confirm-dialog/run-workflow-confirm-dialog.component'
import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { DocumentTypeEditDialogComponent } from '../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
@@ -233,6 +237,7 @@ export class DocumentDetailComponent
private componentRouterService = inject(ComponentRouterService)
private deviceDetectorService = inject(DeviceDetectorService)
private savedViewService = inject(SavedViewService)
private readonly workflowService = inject(WorkflowService)
private readonly websocketStatusService = inject(WebsocketStatusService)
private readonly useNativePdfViewerSetting = this.settings.getSignal<boolean>(
SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER
@@ -295,6 +300,7 @@ export class DocumentDetailComponent
readonly correspondents = signal<Correspondent[]>(undefined)
readonly documentTypes = signal<DocumentType[]>(undefined)
readonly storagePaths = signal<StoragePath[]>(undefined)
readonly manualWorkflows = signal<Workflow[]>([])
documentForm: FormGroup = new FormGroup({
title: new FormControl(''),
@@ -836,6 +842,20 @@ export class DocumentDetailComponent
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe((result) => this.users.set(result.results))
}
if (
this.permissionsService.currentUserCan(
PermissionAction.View,
PermissionType.Workflow
)
) {
this.workflowService
.list(1, 100000, null, null, {
trigger_type: WorkflowTriggerType.Manual,
enabled: true,
})
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe((result) => this.manualWorkflows.set(result.results))
}
this.getCustomFields()
@@ -1559,6 +1579,39 @@ export class DocumentDetailComponent
})
}
runWorkflow() {
const modal = this.modalService.open(RunWorkflowConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Run workflow`
modal.componentInstance.message = $localize`Run a workflow on "${this.document().title}".`
modal.componentInstance.btnCaption = $localize`Run`
modal.componentInstance.workflows = this.manualWorkflows()
modal.componentInstance.confirmClicked.subscribe(() => {
modal.componentInstance.buttonsEnabled.set(false)
this.documentsService
.bulkEdit({ documents: [this.document().id] }, 'run_workflow', {
workflow: modal.componentInstance.workflowId,
filter_documents: !modal.componentInstance.force,
})
.subscribe({
next: () => {
this.toastService.showInfo(
$localize`Workflow will run on "${this.document().title}" in the background.`
)
modal.close()
},
error: (error) => {
modal.componentInstance.buttonsEnabled.set(true)
this.toastService.showError(
$localize`Error executing operation`,
error
)
},
})
})
}
private getSelectedNonLatestVersionId(): number | null {
const versions = this.document()?.versions ?? []
if (!versions.length || !this.selectedVersionId()) {
@@ -98,6 +98,11 @@
<button ngbDropdownItem (click)="mergeSelectedAsVersions()" [disabled]="!userOwnsAll || !userCanEditAll || !userCanDelete || list.allSelected || list.selectedCount < 2">
<pngx-bs-icon name="journal-bookmark-fill" class="me-2"></pngx-bs-icon><ng-container i18n>Merge as versions</ng-container>
</button>
@if (manualWorkflows.length > 0) {
<button ngbDropdownItem (click)="runWorkflowSelected()" [disabled]="!canRunWorkflow">
<pngx-bs-icon name="play-circle" class="me-2"></pngx-bs-icon><ng-container i18n>Run workflow</ng-container>
</button>
}
</div>
</div>
</div>
@@ -17,6 +17,7 @@ import { FILTER_TITLE } from 'src/app/data/filter-rule-type'
import { Results } from 'src/app/data/results'
import { StoragePath } from 'src/app/data/storage-path'
import { Tag } from 'src/app/data/tag'
import { WorkflowTriggerType } from 'src/app/data/workflow-trigger'
import { FilterPipe } from 'src/app/pipes/filter.pipe'
import { DocumentListViewService } from 'src/app/services/document-list-view.service'
import {
@@ -36,6 +37,7 @@ import { ShareLinkBundleService } from 'src/app/services/rest/share-link-bundle.
import { StoragePathService } from 'src/app/services/rest/storage-path.service'
import { TagService } from 'src/app/services/rest/tag.service'
import { UserService } from 'src/app/services/rest/user.service'
import { WorkflowService } from 'src/app/services/rest/workflow.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { TrashService } from 'src/app/services/trash.service'
@@ -83,6 +85,7 @@ describe('BulkEditorComponent', () => {
let customFieldsService: CustomFieldsService
let httpTestingController: HttpTestingController
let shareLinkBundleService: ShareLinkBundleService
let workflowService: WorkflowService
let router: Router
beforeEach(async () => {
@@ -173,6 +176,22 @@ describe('BulkEditorComponent', () => {
delete: jest.fn(),
},
},
{
provide: WorkflowService,
useValue: {
list: jest.fn(() =>
of({
results: [
{
id: 5,
name: 'Manual workflow',
triggers: [{ type: WorkflowTriggerType.Manual }],
},
],
})
),
},
},
{
provide: Router,
useValue: { navigate: jest.fn().mockResolvedValue(true) },
@@ -195,6 +214,7 @@ describe('BulkEditorComponent', () => {
customFieldsService = TestBed.inject(CustomFieldsService)
httpTestingController = TestBed.inject(HttpTestingController)
shareLinkBundleService = TestBed.inject(ShareLinkBundleService)
workflowService = TestBed.inject(WorkflowService)
router = TestBed.inject(Router)
fixture = TestBed.createComponent(BulkEditorComponent)
@@ -654,6 +674,51 @@ describe('BulkEditorComponent', () => {
)
})
it('should load manual workflows on init', () => {
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
fixture.detectChanges()
expect(workflowService.list).toHaveBeenCalledWith(1, 100000, null, null, {
trigger_type: WorkflowTriggerType.Manual,
enabled: true,
})
expect(component.manualWorkflows).toHaveLength(1)
expect(component.manualWorkflows[0].id).toEqual(5)
})
it('should run a workflow on the selection via the run workflow dialog', () => {
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[0]))
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
jest
.spyOn(documentListViewService, 'documents', 'get')
.mockReturnValue([{ id: 3 }, { id: 4 }])
jest
.spyOn(documentListViewService, 'selected', 'get')
.mockReturnValue(new Set([3, 4]))
fixture.detectChanges()
component.runWorkflowSelected()
expect(modal).not.toBeUndefined()
expect(modal.componentInstance.workflows).toEqual(component.manualWorkflows)
modal.componentInstance.workflowId = 5
modal.componentInstance.force = true
modal.componentInstance.confirm()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
expect(req.request.body.method).toEqual('run_workflow')
expect(req.request.body.parameters).toEqual({
workflow: 5,
filter_documents: false,
})
req.flush(true)
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
) // listAllFilteredIds
})
it('should set modal dialog text accordingly for tag edit confirmation', () => {
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[m.length - 1]))
@@ -34,6 +34,7 @@ import { CustomField } from 'src/app/data/custom-field'
import { MatchingModel } from 'src/app/data/matching-model'
import { SelectionDataItem } from 'src/app/data/results'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { Workflow } from 'src/app/data/workflow'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { DocumentListViewService } from 'src/app/services/document-list-view.service'
import { OpenDocumentsService } from 'src/app/services/open-documents.service'
@@ -53,6 +54,7 @@ import {
} from 'src/app/services/rest/document.service'
import { SavedViewService } from 'src/app/services/rest/saved-view.service'
import { ShareLinkBundleService } from 'src/app/services/rest/share-link-bundle.service'
import { WorkflowService } from 'src/app/services/rest/workflow.service'
import { StoragePathService } from 'src/app/services/rest/storage-path.service'
import { TagService } from 'src/app/services/rest/tag.service'
import { SettingsService } from 'src/app/services/settings.service'
@@ -64,6 +66,7 @@ import { MergeAsVersionsConfirmDialogComponent } from '../../common/confirm-dial
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
import { ReprocessConfirmDialogComponent } from '../../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
import { RunWorkflowConfirmDialogComponent } from '../../common/confirm-dialog/run-workflow-confirm-dialog/run-workflow-confirm-dialog.component'
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
import { CustomFieldEditDialogComponent } from '../../common/edit-dialog/custom-field-edit-dialog/custom-field-edit-dialog.component'
import { DocumentTypeEditDialogComponent } from '../../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
@@ -81,6 +84,7 @@ import { PermissionsDialogComponent } from '../../common/permissions-dialog/perm
import { ShareLinkBundleDialogComponent } from '../../common/share-link-bundle-dialog/share-link-bundle-dialog.component'
import { ComponentWithPermissions } from '../../with-permissions/with-permissions.component'
import { CustomFieldsBulkEditDialogComponent } from './custom-fields-bulk-edit-dialog/custom-fields-bulk-edit-dialog.component'
import { WorkflowTriggerType } from 'src/app/data/workflow-trigger'
@Component({
selector: 'pngx-bulk-editor',
@@ -114,8 +118,11 @@ export class BulkEditorComponent
public readonly permissionService = inject(PermissionsService)
private savedViewService = inject(SavedViewService)
private readonly shareLinkBundleService = inject(ShareLinkBundleService)
private readonly workflowService = inject(WorkflowService)
private readonly router = inject(Router)
public manualWorkflows: Workflow[] = []
tagSelectionModel = new FilterableDropdownSelectionModel(true)
correspondentSelectionModel = new FilterableDropdownSelectionModel()
documentTypeSelectionModel = new FilterableDropdownSelectionModel()
@@ -258,6 +265,20 @@ export class BulkEditorComponent
(result) => (this.customFieldsSelectionModel.items = result.results)
)
}
if (
this.permissionService.currentUserCan(
PermissionAction.View,
PermissionType.Workflow
)
) {
this.workflowService
.list(1, 100000, null, null, {
trigger_type: WorkflowTriggerType.Manual,
enabled: true,
})
.pipe(first())
.subscribe((result) => (this.manualWorkflows = result.results))
}
this.downloadForm
.get('downloadFileTypeArchive')
@@ -286,6 +307,28 @@ export class BulkEditorComponent
this.unsubscribeNotifier.complete()
}
get canRunWorkflow(): boolean {
return this.userCanEditAll && this.manualWorkflows.length > 0
}
runWorkflowSelected() {
const modal = this.modalService.open(RunWorkflowConfirmDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.title = $localize`Run workflow`
modal.componentInstance.message = $localize`Run a workflow on ${this.getSelectionSize()} selected document(s).`
modal.componentInstance.btnCaption = $localize`Run`
modal.componentInstance.workflows = this.manualWorkflows
modal.componentInstance.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
this.executeBulkEditMethod(modal, 'run_workflow', {
workflow: modal.componentInstance.workflowId,
filter_documents: !modal.componentInstance.force,
})
})
}
private executeBulkEditMethod(
modal: NgbModalRef,
method: DocumentBulkEditMethod,
+1
View File
@@ -12,6 +12,7 @@ export enum WorkflowTriggerType {
DocumentAdded = 2,
DocumentUpdated = 3,
Scheduled = 4,
Manual = 5,
}
export enum ScheduleDateField {
@@ -39,6 +39,7 @@ export type DocumentBulkEditMethod =
| 'modify_tags'
| 'modify_custom_fields'
| 'set_permissions'
| 'run_workflow'
export interface MergeDocumentsRequest {
metadata_document_id?: number
+20
View File
@@ -42,6 +42,7 @@ from documents.plugins.helpers import DocumentsStatusManager
from documents.tasks import bulk_update_documents
from documents.tasks import consume_file
from documents.tasks import remove_document_from_index
from documents.tasks import run_workflows_for_documents
from documents.tasks import update_document_content_maybe_archive_file
from documents.versioning import get_latest_version_for_root
from documents.versioning import get_root_document
@@ -464,6 +465,25 @@ def reprocess(doc_ids: list[int], *, remote_ocr: bool = False) -> Literal["OK"]:
return "OK"
def run_workflow(
doc_ids: list[int],
workflow: int,
*,
filter_documents: bool = True,
) -> Literal["OK"]:
"""
Manually run a single workflow against the given documents.
Done asynchronously via a Celery task.
"""
run_workflows_for_documents.delay(
document_ids=doc_ids,
workflow_id=workflow,
filter_documents=filter_documents,
)
return "OK"
def set_permissions(
doc_ids: list[int],
set_permissions: PermissionsSpec,
+13
View File
@@ -35,6 +35,7 @@ from django_filters.rest_framework import DateTimeFilter
from django_filters.rest_framework import Filter
from django_filters.rest_framework import FilterSet
from django_filters.rest_framework import MultipleChoiceFilter
from django_filters.rest_framework import NumberFilter
from drf_spectacular.utils import extend_schema_field
from guardian.utils import get_group_obj_perms_model
from guardian.utils import get_user_obj_perms_model
@@ -52,6 +53,7 @@ from documents.models import ShareLink
from documents.models import ShareLinkBundle
from documents.models import StoragePath
from documents.models import Tag
from documents.models import Workflow
from documents.permissions import annotate_authorizing_fields
from documents.permissions import permitted_document_ids
from documents.permissions import permitted_object_ids
@@ -249,6 +251,17 @@ class CustomFieldFilterSet(FilterSet):
}
class WorkflowFilterSet(FilterSet):
trigger_type = NumberFilter(field_name="triggers__type", distinct=True)
class Meta:
model = Workflow
fields = {
"id": ID_KWARGS,
"enabled": ["exact"],
}
@extend_schema_field(serializers.CharField)
class CustomFieldsFilter(Filter):
def filter(self, qs, value):
+1
View File
@@ -684,6 +684,7 @@ def document_matches_workflow(
trigger_type == WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED
or trigger_type == WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED
or trigger_type == WorkflowTrigger.WorkflowTriggerType.SCHEDULED
or trigger_type == WorkflowTrigger.WorkflowTriggerType.MANUAL
):
trigger_matched, reason = existing_document_matches_workflow(
document,
@@ -0,0 +1,43 @@
# Generated by Django 5.2.18 on 2026-11-06 17:12
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0028_replace_tag_tree_fields"),
]
operations = [
migrations.AlterField(
model_name="workflowrun",
name="type",
field=models.PositiveSmallIntegerField(
choices=[
(1, "Consumption Started"),
(2, "Document Added"),
(3, "Document Updated"),
(4, "Scheduled"),
(5, "Manual"),
],
null=True,
verbose_name="workflow trigger type",
),
),
migrations.AlterField(
model_name="workflowtrigger",
name="type",
field=models.PositiveSmallIntegerField(
choices=[
(1, "Consumption Started"),
(2, "Document Added"),
(3, "Document Updated"),
(4, "Scheduled"),
(5, "Manual"),
],
default=1,
verbose_name="Workflow Trigger Type",
),
),
]
+1
View File
@@ -1532,6 +1532,7 @@ class WorkflowTrigger(models.Model):
DOCUMENT_ADDED = 2, _("Document Added")
DOCUMENT_UPDATED = 3, _("Document Updated")
SCHEDULED = 4, _("Scheduled")
MANUAL = 5, _("Manual")
class DocumentSourceChoices(models.IntegerChoices):
CONSUME_FOLDER = DocumentSource.ConsumeFolder.value, _("Consume Folder")
+34
View File
@@ -1829,6 +1829,7 @@ class BulkEditSerializer(
"modify_tags",
"modify_custom_fields",
"set_permissions",
"run_workflow",
*LEGACY_DOCUMENT_ACTION_METHODS,
],
label="Method",
@@ -1918,6 +1919,8 @@ class BulkEditSerializer(
return bulk_edit.reprocess
elif method == "set_permissions":
return bulk_edit.set_permissions
elif method == "run_workflow":
return bulk_edit.run_workflow
elif method == "rotate":
return bulk_edit.rotate
elif method == "merge":
@@ -2157,6 +2160,35 @@ class BulkEditSerializer(
else:
parameters["remote_ocr"] = False
def _validate_parameters_run_workflow(self, parameters) -> None:
if "workflow" not in parameters:
raise serializers.ValidationError("workflow not specified")
try:
workflow = Workflow.objects.get(id=parameters["workflow"])
except (Workflow.DoesNotExist, ValueError, TypeError):
raise serializers.ValidationError("Workflow does not exist")
request = self.context.get("request")
user = getattr(request, "user", None)
if user is not None and not user.has_perm("documents.view_workflow"):
raise PermissionDenied(
"Insufficient permissions to run this workflow",
)
if not workflow.enabled:
raise serializers.ValidationError("Workflow is not enabled")
if not workflow.triggers.filter(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
).exists():
raise serializers.ValidationError(
"Workflow does not have a manual trigger",
)
if "filter_documents" in parameters:
if not isinstance(parameters["filter_documents"], bool):
raise serializers.ValidationError(
"filter_documents must be a boolean",
)
else:
parameters["filter_documents"] = True
def validate_parameters_remove_password(self, parameters):
if "password" not in parameters:
raise serializers.ValidationError("password not specified")
@@ -2223,6 +2255,8 @@ class BulkEditSerializer(
self.validate_parameters_remove_password(parameters)
elif method == bulk_edit.reprocess:
self._validate_parameters_reprocess(parameters)
elif method == bulk_edit.run_workflow:
self._validate_parameters_run_workflow(parameters)
return attrs
+11 -1
View File
@@ -864,6 +864,8 @@ def run_workflows(
logging_group: uuid.UUID | None = None,
overrides: DocumentMetadataOverrides | None = None,
original_file: Path | None = None,
*,
skip_matching: bool = False,
) -> tuple[DocumentMetadataOverrides, str] | None:
"""
Execute workflows matching a document for the given trigger. When `overrides` is provided
@@ -874,6 +876,10 @@ def run_workflows(
`document.source_path` (Document) or `document.original_file` (ConsumableDocument).
Passing `workflow_to_run` skips the workflow query (currently only used by scheduled runs).
Passing `skip_matching` runs the workflow's actions without evaluating its trigger
filters against the document. It is only meaningful together with `workflow_to_run`
(e.g. a manual run the user has explicitly forced onto the selected documents).
"""
use_overrides = overrides is not None
@@ -923,7 +929,11 @@ def run_workflows(
)
break
if matching.document_matches_workflow(document, workflow, trigger_type):
if skip_matching or matching.document_matches_workflow(
document,
workflow,
trigger_type,
):
action: WorkflowAction
has_move_to_trash_action = False
for action in workflow.actions.order_by("order", "pk"):
+46
View File
@@ -53,6 +53,7 @@ from documents.models import ShareLink
from documents.models import ShareLinkBundle
from documents.models import StoragePath
from documents.models import Tag
from documents.models import Workflow
from documents.models import WorkflowRun
from documents.models import WorkflowTrigger
from documents.plugins.base import ConsumeTaskPlugin
@@ -691,6 +692,51 @@ def check_scheduled_workflows() -> None:
)
@shared_task
def run_workflows_for_documents(
document_ids: list[int],
workflow_id: int,
*,
filter_documents: bool = True,
) -> None:
"""
Run a single workflow manually against the given documents.
``filter_documents`` controls whether the workflow's trigger filters are applied.
When True, only documents matching the existing filters will have the workflow run.
When False, the workflow will be run on all specified documents regardless of the filters.
``
"""
try:
workflow = Workflow.objects.get(id=workflow_id)
except Workflow.DoesNotExist:
logger.warning(
f"Manual workflow run requested for missing workflow {workflow_id}",
)
return
for document in Document.objects.filter(
id__in=document_ids,
root_document__isnull=True,
):
try:
run_workflows(
trigger_type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
document=document,
workflow_to_run=workflow,
skip_matching=not filter_documents,
)
# Manual runs dont send document_updated signal, so ws here to ensure clients are refreshed
send_websocket_document_updated(
sender=None,
document=document,
)
except Exception:
logger.exception(
f"Error running workflow {workflow} on document {document.pk}",
)
def update_document_parent_tags(tag: Tag, new_parent: Tag) -> None:
"""
When a tag's parent changes, ensure all documents containing the tag also have
+109
View File
@@ -15,7 +15,11 @@ from documents.models import Document
from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
from documents.models import Workflow
from documents.models import WorkflowAction
from documents.models import WorkflowTrigger
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_all_global
from paperless_testing.permissions import grant_global
@@ -622,6 +626,111 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"remote_ocr": True})
def _make_manual_workflow(self) -> Workflow:
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
)
action = WorkflowAction.objects.create()
workflow = Workflow.objects.create(name="Manual workflow", order=0)
workflow.triggers.add(trigger)
workflow.actions.add(action)
return workflow
def _run_workflow(self, parameters, documents=None):
return self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": documents or [self.doc1.id],
"method": "run_workflow",
"parameters": parameters,
},
),
content_type="application/json",
)
@mock.patch("documents.views.bulk_edit.run_workflow")
def test_run_workflow(self, m) -> None:
"""A valid run_workflow call passes the docs, workflow id and filter flag through."""
self.setup_mock(m, "run_workflow")
workflow = self._make_manual_workflow()
response = self._run_workflow({"workflow": workflow.id})
self.assertEqual(response.status_code, status.HTTP_200_OK)
args, kwargs = m.call_args
self.assertEqual(args[0], [self.doc1.id])
self.assertEqual(kwargs, {"workflow": workflow.id, "filter_documents": True})
self._run_workflow({"workflow": workflow.id, "filter_documents": False})
self.assertFalse(m.call_args.kwargs["filter_documents"])
@mock.patch("documents.views.bulk_edit.run_workflow")
def test_run_workflow_invalid_parameters(self, m) -> None:
"""Invalid run_workflow parameters are rejected before dispatch."""
self.setup_mock(m, "run_workflow")
manual = self._make_manual_workflow()
non_manual = Workflow.objects.create(name="Non-manual", order=0)
non_manual.triggers.add(
WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
),
)
disabled = Workflow.objects.create(
name="Disabled manual",
order=0,
enabled=False,
)
disabled.triggers.add(
WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
),
)
for label, parameters in (
("missing workflow", {}),
("nonexistent workflow", {"workflow": 9999}),
("no manual trigger", {"workflow": non_manual.id}),
("disabled workflow", {"workflow": disabled.id}),
("non-bool filter", {"workflow": manual.id, "filter_documents": "nope"}),
):
with self.subTest(label):
response = self._run_workflow(parameters)
self.assertEqual(
response.status_code,
status.HTTP_400_BAD_REQUEST,
)
m.assert_not_called()
@mock.patch("documents.views.bulk_edit.run_workflow")
def test_run_workflow_requires_ownership(self, m) -> None:
"""A non-owner without document ownership is forbidden (actions mutate docs)."""
self.setup_mock(m, "run_workflow")
user = UserFactory(username="no_perms")
grant_all_global(user)
doc = DocumentFactory(owner=UserFactory(username="other"))
workflow = self._make_manual_workflow()
self.client.force_authenticate(user=user)
response = self._run_workflow({"workflow": workflow.id}, documents=[doc.id])
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
m.assert_not_called()
@mock.patch("documents.views.bulk_edit.run_workflow")
def test_run_workflow_requires_workflow_permission(self, m) -> None:
"""Owning the documents is not enough without view_workflow permission."""
self.setup_mock(m, "run_workflow")
user = UserFactory(username="no_wf_perms")
grant_global(user, "change_document")
doc = DocumentFactory(owner=user)
workflow = self._make_manual_workflow()
self.client.force_authenticate(user=user)
response = self._run_workflow({"workflow": workflow.id}, documents=[doc.id])
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
m.assert_not_called()
@mock.patch("documents.serialisers.bulk_edit.set_storage_path")
def test_api_set_storage_path(self, m) -> None:
"""
+41
View File
@@ -100,6 +100,47 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.action.assign_correspondent.pk,
)
def test_api_get_workflows_filter_trigger_type(self) -> None:
"""
GIVEN:
- A consumption workflow and a manual workflow
WHEN:
- Workflows are requested filtered by trigger_type
THEN:
- Only workflows with a matching trigger are returned
"""
manual_trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
)
manual_workflow = Workflow.objects.create(name="Manual", order=1)
manual_workflow.triggers.add(manual_trigger)
response = self.client.get(
f"{self.ENDPOINT}?trigger_type={WorkflowTrigger.WorkflowTriggerType.MANUAL}",
format="json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 1)
self.assertEqual(response.data["results"][0]["id"], manual_workflow.id)
def test_api_get_workflows_filter_enabled(self) -> None:
"""
GIVEN:
- An enabled workflow and a disabled workflow
WHEN:
- Workflows are requested filtered by enabled
THEN:
- Only enabled workflows are returned
"""
Workflow.objects.create(name="Disabled", order=1, enabled=False)
response = self.client.get(f"{self.ENDPOINT}?enabled=true", format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 1)
self.assertEqual(response.data["results"][0]["id"], self.workflow.id)
def test_api_get_workflow_actions_ordered(self) -> None:
"""
GIVEN:
+77
View File
@@ -2288,6 +2288,83 @@ class TestWorkflows(
doc.refresh_from_db()
self.assertEqual(doc.owner, self.user2)
def _make_manual_workflow(self, *, require_tag: Tag | None = None) -> Workflow:
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.MANUAL,
)
if require_tag is not None:
trigger.filter_has_all_tags.add(require_tag)
action = WorkflowAction.objects.create(assign_owner=self.user2)
workflow = Workflow.objects.create(name="Manual workflow", order=0)
workflow.triggers.add(trigger)
workflow.actions.add(action)
return workflow
@mock.patch("documents.tasks.send_websocket_document_updated")
def test_manual_workflow_run_applies_and_notifies(self, mock_ws) -> None:
"""A manual run applies the workflow's actions and pushes a websocket update."""
workflow = self._make_manual_workflow()
doc = DocumentFactory()
tasks.run_workflows_for_documents([doc.pk], workflow.pk)
doc.refresh_from_db()
self.assertEqual(doc.owner, self.user2)
self.assertEqual(mock_ws.call_args.kwargs["document"].pk, doc.pk)
def test_manual_workflow_run_respects_filters(self) -> None:
"""With filter_documents=True, a document failing the trigger filters is skipped."""
workflow = self._make_manual_workflow(require_tag=self.t1)
doc = DocumentFactory()
tasks.run_workflows_for_documents([doc.pk], workflow.pk, filter_documents=True)
doc.refresh_from_db()
self.assertIsNone(doc.owner)
def test_manual_workflow_run_force_ignores_filters(self) -> None:
"""With filter_documents=False, actions run regardless of the trigger filters."""
workflow = self._make_manual_workflow(require_tag=self.t1)
doc = DocumentFactory()
tasks.run_workflows_for_documents([doc.pk], workflow.pk, filter_documents=False)
doc.refresh_from_db()
self.assertEqual(doc.owner, self.user2)
def test_manual_workflow_run_missing_workflow(self) -> None:
"""A missing workflow id logs a warning rather than raising."""
doc = DocumentFactory()
with self.assertLogs("paperless.tasks", level="WARNING"):
tasks.run_workflows_for_documents([doc.pk], 9999)
@mock.patch("documents.tasks.send_websocket_document_updated")
@mock.patch("documents.tasks.run_workflows")
def test_manual_workflow_run_continues_after_error(
self,
mock_run_workflows,
mock_ws,
) -> None:
"""A failure on one document does not abort the run for the others."""
workflow = self._make_manual_workflow()
doc1 = DocumentFactory()
doc2 = DocumentFactory()
def side_effect(*args, document, **kwargs):
if document.pk == doc1.pk:
raise Exception("boom")
mock_run_workflows.side_effect = side_effect
with self.assertLogs("paperless.tasks", level="ERROR"):
tasks.run_workflows_for_documents([doc1.pk, doc2.pk], workflow.pk)
# Both documents were attempted; the websocket update only fired for the
# one that did not raise.
self.assertEqual(mock_run_workflows.call_count, 2)
ws_doc_ids = [c.kwargs["document"].pk for c in mock_ws.call_args_list]
self.assertEqual(ws_doc_ids, [doc2.pk])
@mock.patch("documents.tasks.send_websocket_document_updated")
def test_workflow_scheduled_trigger_sends_websocket_update(
self,
+98
View File
@@ -146,6 +146,7 @@ from documents.filters import ShareLinkFilterSet
from documents.filters import StoragePathFilterSet
from documents.filters import TagFilterSet
from documents.filters import TitleContentFilter
from documents.filters import WorkflowFilterSet
from documents.mail import EmailAttachment
from documents.mail import send_email
from documents.matching import match_correspondents
@@ -3008,6 +3009,100 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
)
return Response({"result": "OK", "task_id": task.id})
def _has_document_permissions(
self,
*,
user: User,
documents: list[int],
method,
parameters: dict[str, Any],
) -> bool:
if user.is_superuser:
return True
root_docs = {
get_root_document(doc)
for doc in Document.objects.select_related(
"owner",
"root_document__owner",
).filter(pk__in=documents)
}
user_is_owner_of_all_documents = all(
(doc.owner == user or doc.owner is None) for doc in root_docs
)
# check global and object permissions for all documents
has_perms = (
user.has_perm(
"documents.change_document",
)
and not Document.global_objects.filter(pk__in=documents)
.exclude(pk__in=permitted_document_ids(user, perm="change_document"))
.exists()
)
# check ownership for methods that change original document
if (
(
has_perms
and method
in [
bulk_edit.set_permissions,
bulk_edit.delete,
bulk_edit.rotate,
bulk_edit.delete_pages,
bulk_edit.edit_pdf,
bulk_edit.merge_as_versions,
bulk_edit.remove_password,
bulk_edit.run_workflow,
]
)
or (
method in [bulk_edit.merge, bulk_edit.split]
and parameters.get("delete_originals")
)
or (method == bulk_edit.edit_pdf and parameters.get("update_document"))
):
has_perms = has_perms and user_is_owner_of_all_documents
# check global add permissions for methods that create documents
if (
has_perms
and (
method in [bulk_edit.split, bulk_edit.merge]
or (
method in [bulk_edit.edit_pdf, bulk_edit.remove_password]
and not parameters.get("update_document")
)
)
and not user.has_perm("documents.add_document")
):
has_perms = False
# check global delete permissions for methods that delete documents
if (
has_perms
and (
method == bulk_edit.delete
# Sources stop being documents of their own, and removing one
# again afterwards needs delete_document
or method == bulk_edit.merge_as_versions
or (
method in [bulk_edit.merge, bulk_edit.split]
and parameters.get("delete_originals")
)
or (
method in [bulk_edit.edit_pdf, bulk_edit.remove_password]
and parameters.get("delete_original")
and not parameters.get("update_document")
)
)
and not user.has_perm("documents.delete_document")
):
has_perms = False
return has_perms
def _execute_document_action(
self,
*,
@@ -3084,6 +3179,7 @@ class BulkEditView(DocumentOperationPermissionMixin):
LEGACY_MODIFIED_FIELDS = {
"delete": "deleted_at",
"reprocess": "checksum",
"remove_password": None,
}
serializer_class = BulkEditSerializer
@@ -5089,6 +5185,8 @@ class WorkflowViewSet(ModelViewSet[Workflow]):
serializer_class = WorkflowSerializer
pagination_class = StandardPagination
filter_backends = (DjangoFilterBackend,)
filterset_class = WorkflowFilterSet
model = Workflow
+160 -160
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr ""
"Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-11-11 23:50+0000\n"
"POT-Creation-Date: 2026-11-12 21:28+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n"
"Language-Team: English\n"
@@ -21,43 +21,43 @@ msgstr ""
msgid "Documents"
msgstr ""
#: documents/filters.py:336
#: documents/filters.py:349
msgid "Expected a date or a relative date such as \"today\" or \"in 30 days\"."
msgstr ""
#: documents/filters.py:514
#: documents/filters.py:527
msgid "Value must be valid JSON."
msgstr ""
#: documents/filters.py:535
#: documents/filters.py:548
msgid "Invalid custom field query expression"
msgstr ""
#: documents/filters.py:545
#: documents/filters.py:558
msgid "Invalid expression list. Must be nonempty."
msgstr ""
#: documents/filters.py:566
#: documents/filters.py:579
msgid "Invalid logical operator {op!r}"
msgstr ""
#: documents/filters.py:580
#: documents/filters.py:593
msgid "Maximum number of query conditions exceeded."
msgstr ""
#: documents/filters.py:644
#: documents/filters.py:657
msgid "{name!r} is not a valid custom field."
msgstr ""
#: documents/filters.py:681
#: documents/filters.py:694
msgid "{data_type} does not support query expr {expr!r}."
msgstr ""
#: documents/filters.py:802 documents/models.py:195
#: documents/filters.py:815 documents/models.py:195
msgid "Maximum nesting depth exceeded."
msgstr ""
#: documents/filters.py:1179
#: documents/filters.py:1192
msgid "Custom field not found"
msgstr ""
@@ -93,20 +93,20 @@ msgstr ""
msgid "Automatic"
msgstr ""
#: documents/models.py:69 documents/models.py:709 documents/models.py:2149
#: documents/models.py:69 documents/models.py:709 documents/models.py:2150
#: paperless_mail/models.py:23 paperless_mail/models.py:143
msgid "name"
msgstr ""
#: documents/models.py:71 documents/models.py:1592
#: documents/models.py:71 documents/models.py:1593
msgid "match"
msgstr ""
#: documents/models.py:74 documents/models.py:1595
#: documents/models.py:74 documents/models.py:1596
msgid "matching algorithm"
msgstr ""
#: documents/models.py:79 documents/models.py:1600
#: documents/models.py:79 documents/models.py:1601
msgid "is insensitive"
msgstr ""
@@ -280,7 +280,7 @@ msgid "Optional short label for a document version."
msgstr ""
#: documents/models.py:402 documents/models.py:1027 documents/models.py:1058
#: documents/models.py:1162 documents/models.py:2192
#: documents/models.py:1162 documents/models.py:2193
msgid "document"
msgstr ""
@@ -496,7 +496,7 @@ msgstr ""
msgid "Tag"
msgstr ""
#: documents/models.py:683 documents/models.py:698 documents/models.py:1870
#: documents/models.py:683 documents/models.py:698 documents/models.py:1871
msgid "Tags"
msgstr ""
@@ -528,19 +528,19 @@ msgstr ""
msgid "Large Cards"
msgstr ""
#: documents/models.py:695 documents/models.py:1869
#: documents/models.py:695 documents/models.py:1870
msgid "Title"
msgstr ""
#: documents/models.py:696 documents/models.py:926 documents/models.py:1544
#: documents/models.py:696 documents/models.py:926 documents/models.py:1545
msgid "Created"
msgstr ""
#: documents/models.py:697 documents/models.py:1543
#: documents/models.py:697 documents/models.py:1544
msgid "Added"
msgstr ""
#: documents/models.py:699 documents/models.py:1871
#: documents/models.py:699 documents/models.py:1872
msgid "Correspondent"
msgstr ""
@@ -860,7 +860,7 @@ msgstr ""
msgid "Index Optimize"
msgstr ""
#: documents/models.py:865 documents/models.py:1539
#: documents/models.py:865 documents/models.py:1540
msgid "Mail Fetch"
msgstr ""
@@ -900,7 +900,7 @@ msgstr ""
msgid "Scheduled"
msgstr ""
#: documents/models.py:883 documents/models.py:1540
#: documents/models.py:883 documents/models.py:1541
msgid "Web UI"
msgstr ""
@@ -920,7 +920,7 @@ msgstr ""
msgid "System"
msgstr ""
#: documents/models.py:888
#: documents/models.py:888 documents/models.py:1535
msgid "Manual"
msgstr ""
@@ -1178,491 +1178,491 @@ msgstr ""
msgid "Document Updated"
msgstr ""
#: documents/models.py:1537
#: documents/models.py:1538
msgid "Consume Folder"
msgstr ""
#: documents/models.py:1538
#: documents/models.py:1539
msgid "Api Upload"
msgstr ""
#: documents/models.py:1545
#: documents/models.py:1546
msgid "Modified"
msgstr ""
#: documents/models.py:1546
#: documents/models.py:1547
msgid "Custom Field"
msgstr ""
#: documents/models.py:1549
#: documents/models.py:1550
msgid "Workflow Trigger Type"
msgstr ""
#: documents/models.py:1561
#: documents/models.py:1562
msgid "filter path"
msgstr ""
#: documents/models.py:1566
#: documents/models.py:1567
msgid ""
"Only consume documents with a path that matches this if specified. Wildcards "
"specified as * are allowed. Case insensitive."
msgstr ""
#: documents/models.py:1573
#: documents/models.py:1574
msgid "filter filename"
msgstr ""
#: documents/models.py:1578 paperless_mail/models.py:200
#: documents/models.py:1579 paperless_mail/models.py:200
msgid ""
"Only consume documents which entirely match this filename if specified. "
"Wildcards such as *.pdf or *invoice* are allowed. Case insensitive."
msgstr ""
#: documents/models.py:1589
#: documents/models.py:1590
msgid "filter documents from this mail rule"
msgstr ""
#: documents/models.py:1605
#: documents/models.py:1606
msgid "has these tag(s)"
msgstr ""
#: documents/models.py:1612
#: documents/models.py:1613
msgid "has all of these tag(s)"
msgstr ""
#: documents/models.py:1619
#: documents/models.py:1620
msgid "does not have these tag(s)"
msgstr ""
#: documents/models.py:1627
#: documents/models.py:1628
msgid "has this document type"
msgstr ""
#: documents/models.py:1634
#: documents/models.py:1635
msgid "has one of these document types"
msgstr ""
#: documents/models.py:1641
#: documents/models.py:1642
msgid "does not have these document type(s)"
msgstr ""
#: documents/models.py:1649
#: documents/models.py:1650
msgid "has this correspondent"
msgstr ""
#: documents/models.py:1656
#: documents/models.py:1657
msgid "does not have these correspondent(s)"
msgstr ""
#: documents/models.py:1663
#: documents/models.py:1664
msgid "has one of these correspondents"
msgstr ""
#: documents/models.py:1671
#: documents/models.py:1672
msgid "has this storage path"
msgstr ""
#: documents/models.py:1678
#: documents/models.py:1679
msgid "has one of these storage paths"
msgstr ""
#: documents/models.py:1685
#: documents/models.py:1686
msgid "does not have these storage path(s)"
msgstr ""
#: documents/models.py:1689
#: documents/models.py:1690
msgid "filter custom field query"
msgstr ""
#: documents/models.py:1692
#: documents/models.py:1693
msgid "JSON-encoded custom field query expression."
msgstr ""
#: documents/models.py:1696
#: documents/models.py:1697
msgid "schedule offset days"
msgstr ""
#: documents/models.py:1699
#: documents/models.py:1700
msgid "The number of days to offset the schedule trigger by."
msgstr ""
#: documents/models.py:1704
#: documents/models.py:1705
msgid "schedule is recurring"
msgstr ""
#: documents/models.py:1707
#: documents/models.py:1708
msgid "If the schedule should be recurring."
msgstr ""
#: documents/models.py:1712
#: documents/models.py:1713
msgid "schedule recurring delay in days"
msgstr ""
#: documents/models.py:1716
#: documents/models.py:1717
msgid "The number of days between recurring schedule triggers."
msgstr ""
#: documents/models.py:1721
#: documents/models.py:1722
msgid "schedule date field"
msgstr ""
#: documents/models.py:1726
#: documents/models.py:1727
msgid "The field to check for a schedule trigger."
msgstr ""
#: documents/models.py:1735
#: documents/models.py:1736
msgid "schedule date custom field"
msgstr ""
#: documents/models.py:1739
#: documents/models.py:1740
msgid "workflow trigger"
msgstr ""
#: documents/models.py:1740
#: documents/models.py:1741
msgid "workflow triggers"
msgstr ""
#: documents/models.py:1748
#: documents/models.py:1749
msgid "email subject"
msgstr ""
#: documents/models.py:1752
#: documents/models.py:1753
msgid ""
"The subject of the email, can include some placeholders, see documentation."
msgstr ""
#: documents/models.py:1758
#: documents/models.py:1759
msgid "email body"
msgstr ""
#: documents/models.py:1761
#: documents/models.py:1762
msgid ""
"The body (message) of the email, can include some placeholders, see "
"documentation."
msgstr ""
#: documents/models.py:1767
#: documents/models.py:1768
msgid "emails to"
msgstr ""
#: documents/models.py:1770
#: documents/models.py:1771
msgid "The destination email addresses, comma separated."
msgstr ""
#: documents/models.py:1776
#: documents/models.py:1777
msgid "include document in email"
msgstr ""
#: documents/models.py:1787
#: documents/models.py:1788
msgid "webhook url"
msgstr ""
#: documents/models.py:1790
#: documents/models.py:1791
msgid "The destination URL for the notification."
msgstr ""
#: documents/models.py:1795
#: documents/models.py:1796
msgid "use parameters"
msgstr ""
#: documents/models.py:1800
#: documents/models.py:1801
msgid "send as JSON"
msgstr ""
#: documents/models.py:1804
#: documents/models.py:1805
msgid "webhook parameters"
msgstr ""
#: documents/models.py:1807
#: documents/models.py:1808
msgid "The parameters to send with the webhook URL if body not used."
msgstr ""
#: documents/models.py:1811
#: documents/models.py:1812
msgid "webhook body"
msgstr ""
#: documents/models.py:1814
#: documents/models.py:1815
msgid "The body to send with the webhook URL if parameters not used."
msgstr ""
#: documents/models.py:1818
#: documents/models.py:1819
msgid "webhook headers"
msgstr ""
#: documents/models.py:1821
#: documents/models.py:1822
msgid "The headers to send with the webhook URL."
msgstr ""
#: documents/models.py:1826
#: documents/models.py:1827
msgid "include document in webhook"
msgstr ""
#: documents/models.py:1837
#: documents/models.py:1838
msgid "Assignment"
msgstr ""
#: documents/models.py:1841
#: documents/models.py:1842
msgid "Removal"
msgstr ""
#: documents/models.py:1845 documents/templates/account/password_reset.html:15
#: documents/models.py:1846 documents/templates/account/password_reset.html:15
msgid "Email"
msgstr ""
#: documents/models.py:1849
#: documents/models.py:1850
msgid "Webhook"
msgstr ""
#: documents/models.py:1853
#: documents/models.py:1854
msgid "Password removal"
msgstr ""
#: documents/models.py:1857
#: documents/models.py:1858
msgid "Move to trash"
msgstr ""
#: documents/models.py:1861
#: documents/models.py:1862
msgid "Remote OCR"
msgstr ""
#: documents/models.py:1865
#: documents/models.py:1866
msgid "Apply AI suggestions"
msgstr ""
#: documents/models.py:1872
#: documents/models.py:1873
msgid "Document type"
msgstr ""
#: documents/models.py:1873
#: documents/models.py:1874
msgid "Storage path"
msgstr ""
#: documents/models.py:1874
#: documents/models.py:1875
msgid "Created date"
msgstr ""
#: documents/models.py:1877
#: documents/models.py:1878
msgid "Workflow Action Type"
msgstr ""
#: documents/models.py:1882 documents/models.py:2151
#: documents/models.py:1883 documents/models.py:2152
#: paperless_mail/models.py:145
msgid "order"
msgstr ""
#: documents/models.py:1885
#: documents/models.py:1886
msgid "assign title"
msgstr ""
#: documents/models.py:1889
#: documents/models.py:1890
msgid "Assign a document title, must be a Jinja2 template, see documentation."
msgstr ""
#: documents/models.py:1897 paperless_mail/models.py:274
#: documents/models.py:1898 paperless_mail/models.py:274
msgid "assign this tag"
msgstr ""
#: documents/models.py:1906 paperless_mail/models.py:282
#: documents/models.py:1907 paperless_mail/models.py:282
msgid "assign this document type"
msgstr ""
#: documents/models.py:1915 paperless_mail/models.py:296
#: documents/models.py:1916 paperless_mail/models.py:296
msgid "assign this correspondent"
msgstr ""
#: documents/models.py:1924
#: documents/models.py:1925
msgid "assign this storage path"
msgstr ""
#: documents/models.py:1933
#: documents/models.py:1934
msgid "assign this owner"
msgstr ""
#: documents/models.py:1940
#: documents/models.py:1941
msgid "grant view permissions to these users"
msgstr ""
#: documents/models.py:1947
#: documents/models.py:1948
msgid "grant view permissions to these groups"
msgstr ""
#: documents/models.py:1954
#: documents/models.py:1955
msgid "grant change permissions to these users"
msgstr ""
#: documents/models.py:1961
#: documents/models.py:1962
msgid "grant change permissions to these groups"
msgstr ""
#: documents/models.py:1968
#: documents/models.py:1969
msgid "assign these custom fields"
msgstr ""
#: documents/models.py:1972
#: documents/models.py:1973
msgid "custom field values"
msgstr ""
#: documents/models.py:1976
#: documents/models.py:1977
msgid "Optional values to assign to the custom fields."
msgstr ""
#: documents/models.py:1985
#: documents/models.py:1986
msgid "remove these tag(s)"
msgstr ""
#: documents/models.py:1990
#: documents/models.py:1991
msgid "remove all tags"
msgstr ""
#: documents/models.py:1997
#: documents/models.py:1998
msgid "remove these document type(s)"
msgstr ""
#: documents/models.py:2002
#: documents/models.py:2003
msgid "remove all document types"
msgstr ""
#: documents/models.py:2009
#: documents/models.py:2010
msgid "remove these correspondent(s)"
msgstr ""
#: documents/models.py:2014
#: documents/models.py:2015
msgid "remove all correspondents"
msgstr ""
#: documents/models.py:2021
#: documents/models.py:2022
msgid "remove these storage path(s)"
msgstr ""
#: documents/models.py:2026
#: documents/models.py:2027
msgid "remove all storage paths"
msgstr ""
#: documents/models.py:2033
#: documents/models.py:2034
msgid "remove these owner(s)"
msgstr ""
#: documents/models.py:2038
#: documents/models.py:2039
msgid "remove all owners"
msgstr ""
#: documents/models.py:2045
#: documents/models.py:2046
msgid "remove view permissions for these users"
msgstr ""
#: documents/models.py:2052
#: documents/models.py:2053
msgid "remove view permissions for these groups"
msgstr ""
#: documents/models.py:2059
#: documents/models.py:2060
msgid "remove change permissions for these users"
msgstr ""
#: documents/models.py:2066
#: documents/models.py:2067
msgid "remove change permissions for these groups"
msgstr ""
#: documents/models.py:2071
#: documents/models.py:2072
msgid "remove all permissions"
msgstr ""
#: documents/models.py:2078
#: documents/models.py:2079
msgid "remove these custom fields"
msgstr ""
#: documents/models.py:2083
#: documents/models.py:2084
msgid "remove all custom fields"
msgstr ""
#: documents/models.py:2092
#: documents/models.py:2093
msgid "email"
msgstr ""
#: documents/models.py:2101
#: documents/models.py:2102
msgid "webhook"
msgstr ""
#: documents/models.py:2105
#: documents/models.py:2106
msgid "passwords"
msgstr ""
#: documents/models.py:2109
#: documents/models.py:2110
msgid ""
"Passwords to try when removing PDF protection. Separate with commas or new "
"lines."
msgstr ""
#: documents/models.py:2114
#: documents/models.py:2115
msgid "AI suggestion fields"
msgstr ""
#: documents/models.py:2118
#: documents/models.py:2119
msgid "Which of the AI-suggested fields to apply to the document."
msgstr ""
#: documents/models.py:2123
#: documents/models.py:2124
msgid "create missing objects"
msgstr ""
#: documents/models.py:2126
#: documents/models.py:2127
msgid ""
"Create suggested tags, correspondents, document types and storage paths that "
"do not already exist instead of skipping them."
msgstr ""
#: documents/models.py:2132
#: documents/models.py:2133
msgid "overwrite existing values"
msgstr ""
#: documents/models.py:2135
#: documents/models.py:2136
msgid ""
"Apply suggestions even if the document already has a value for that field. "
"Tags are always added to, never replaced."
msgstr ""
#: documents/models.py:2141
#: documents/models.py:2142
msgid "workflow action"
msgstr ""
#: documents/models.py:2142
#: documents/models.py:2143
msgid "workflow actions"
msgstr ""
#: documents/models.py:2157
#: documents/models.py:2158
msgid "triggers"
msgstr ""
#: documents/models.py:2164
#: documents/models.py:2165
msgid "actions"
msgstr ""
#: documents/models.py:2167 paperless_mail/models.py:154
#: documents/models.py:2168 paperless_mail/models.py:154
msgid "enabled"
msgstr ""
#: documents/models.py:2178
#: documents/models.py:2179
msgid "workflow"
msgstr ""
#: documents/models.py:2182
#: documents/models.py:2183
msgid "workflow trigger type"
msgstr ""
#: documents/models.py:2196
#: documents/models.py:2197
msgid "date run"
msgstr ""
#: documents/models.py:2202
#: documents/models.py:2203
msgid "workflow run"
msgstr ""
#: documents/models.py:2203
#: documents/models.py:2204
msgid "workflow runs"
msgstr ""
#: documents/serialisers.py:483 documents/serialisers.py:844
#: documents/serialisers.py:2850 documents/views.py:354 documents/views.py:620
#: documents/views.py:2760 paperless_mail/serialisers.py:156
#: documents/serialisers.py:2884 documents/views.py:355 documents/views.py:621
#: documents/views.py:2761 paperless_mail/serialisers.py:156
msgid "Insufficient permissions."
msgstr ""
@@ -1670,39 +1670,39 @@ msgstr ""
msgid "Invalid color."
msgstr ""
#: documents/serialisers.py:2317
#: documents/serialisers.py:2351
#, python-format
msgid "File type %(type)s not supported"
msgstr ""
#: documents/serialisers.py:2361
#: documents/serialisers.py:2395
#, python-format
msgid "Custom field id must be an integer: %(id)s"
msgstr ""
#: documents/serialisers.py:2368
#: documents/serialisers.py:2402
#, python-format
msgid "Custom field with id %(id)s does not exist"
msgstr ""
#: documents/serialisers.py:2385 documents/serialisers.py:2395
#: documents/serialisers.py:2419 documents/serialisers.py:2429
msgid ""
"Custom fields must be a list of integers or an object mapping ids to values."
msgstr ""
#: documents/serialisers.py:2390
#: documents/serialisers.py:2424
msgid "Some custom fields don't exist or were specified twice."
msgstr ""
#: documents/serialisers.py:2537
#: documents/serialisers.py:2571
msgid "Invalid variable detected."
msgstr ""
#: documents/serialisers.py:2906
#: documents/serialisers.py:2940
msgid "Duplicate document identifiers are not allowed."
msgstr ""
#: documents/serialisers.py:2936 documents/views.py:4698
#: documents/serialisers.py:2970 documents/views.py:4794
#, python-format
msgid "Documents not found: %(ids)s"
msgstr ""
@@ -1953,40 +1953,40 @@ msgstr ""
msgid ", "
msgstr ""
#: documents/views.py:309
#: documents/views.py:310
msgid "Specify only one of text, title_search, query, or more_like_id."
msgstr ""
#: documents/views.py:347 documents/views.py:2757
#: documents/views.py:348 documents/views.py:2758
msgid "Invalid more_like_id"
msgstr ""
#: documents/views.py:1684
#: documents/views.py:1685
msgid "Invalid AI configuration."
msgstr ""
#: documents/views.py:1695
#: documents/views.py:1696
msgid "AI backend request timed out."
msgstr ""
#: documents/views.py:1707
#: documents/views.py:1708
msgid "AI backend rejected the request. Check logs for details."
msgstr ""
#: documents/views.py:4714
#: documents/views.py:4810
#, python-format
msgid "Insufficient permissions to share document %(id)s."
msgstr ""
#: documents/views.py:4761
#: documents/views.py:4857
msgid "Bundle is already being processed."
msgstr ""
#: documents/views.py:4827
#: documents/views.py:4923
msgid "The share link bundle is still being prepared. Please try again later."
msgstr ""
#: documents/views.py:4841
#: documents/views.py:4937
msgid "The share link bundle is unavailable."
msgstr ""
+26 -8
View File
@@ -46,6 +46,16 @@ def _build_document_reference(
}
def _permitted_documents_filter(store, permitted_ids: set[int]):
"""Scope retrieval to ``permitted_ids`` using whichever of an inclusion or
exclusion list is shorter, since a long inclusion list is much slower for
the vector store than a long exclusion list."""
denied_ids = store.indexed_document_ids() - permitted_ids
if len(permitted_ids) <= len(denied_ids):
return document_id_filters(str(pk) for pk in permitted_ids)
return exclude_document_ids_filter(str(pk) for pk in denied_ids)
def _get_document_references(
documents: QuerySet[Document],
top_nodes: list,
@@ -128,24 +138,23 @@ def _stream_chat_with_documents(
from llama_index.core.retrievers import VectorIndexRetriever
config = AIConfig()
permitted_ids: set[int] | None = None
if unrestricted:
# Exclude trashed ids (usually few) instead of an IN filter over the
# full permitted set, which risks the vector store's bound parameter
# limit (_MAX_IN_VALUES) on large installs. Trashed documents stay
# indexed until permanent deletion (delete_document_from_llm_index
# hangs off post_delete, not trash), so must be excluded explicitly.
# Exclude trashed ids (usually few) rather than send every permitted
# id on each query. Trashed documents stay indexed until permanent
# deletion (post_delete, not trash), so must be excluded explicitly.
trashed_ids = Document.deleted_objects.values_list("pk", flat=True)
filters = exclude_document_ids_filter(str(pk) for pk in trashed_ids)
else:
filters = document_id_filters(
str(pk) for pk in documents.values_list("pk", flat=True)
)
permitted_ids = set(documents.values_list("pk", flat=True))
# Hold the shared read lock only while reading the index, so the compaction
# swap cannot replace the database file while this connection is open.
# Everything after retrieval works on the nodes already in memory, so a slow
# LLM stream never holds the lock.
with read_store() as store:
if permitted_ids is not None:
filters = _permitted_documents_filter(store, permitted_ids)
index = load_or_build_index(config, store)
retriever = VectorIndexRetriever(
index=index,
@@ -159,6 +168,15 @@ def _stream_chat_with_documents(
with db_connection_released():
top_nodes = retriever.retrieve(query_str)
if permitted_ids is not None:
# An exclusion filter fails open for a document indexed after the
# permitted set was read, so drop anything not explicitly permitted.
top_nodes = [
node
for node in top_nodes
if int(node.metadata["document_id"]) in permitted_ids
]
if not top_nodes:
logger.warning("No nodes found for the given documents.")
yield CHAT_NO_CONTENT_MESSAGE
+9
View File
@@ -141,6 +141,15 @@ class DocumentMetaTable:
(DocumentMetaRow(r["document_id"], r["modified"]) for r in batch),
)
@staticmethod
def all_document_ids(conn: sqlite3.Connection) -> set[int]:
"""Every indexed document_id. document_meta has one row per document,
so this is a cheap primary-key scan, unlike reading the vec0 table."""
return {
row["document_id"]
for row in conn.execute("SELECT document_id FROM document_meta")
}
@staticmethod
def all_modified_times(conn: sqlite3.Connection) -> dict[str, str]:
"""Full document_id -> modified map, for get_modified_times()'s
+67 -4
View File
@@ -13,6 +13,7 @@ from django.utils import timezone
from llama_index.core import settings as llama_settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from llama_index.core.schema import TextNode
from llama_index.core.vector_stores.types import FilterOperator
from documents.models import Document
from paperless_ai import chat
@@ -412,10 +413,8 @@ class TestStreamChatRetrieval:
WHEN:
- stream_chat_with_documents is called with unrestricted=True
THEN:
- The retriever receives a NOT IN filter excluding zero ids, so
the whole index is effectively searched -- and no IN-list is
built from the full permitted set, which is what risks the
vector store's safety limit on large installs
- The retriever receives a NOT IN filter excluding zero ids
rather than an IN list of every permitted id
"""
document = DocumentFactory.create(content="indexed document content")
indexing.llm_index_add_or_update_document(document)
@@ -468,6 +467,70 @@ class TestStreamChatRetrieval:
assert str(trashed.pk) in filter_values
assert str(kept.pk) not in filter_values
def test_restricted_chat_excludes_denied_ids_when_few_are_denied(
self,
temp_llm_index_dir: Path,
mock_embed_model: pytest_mock.MockType,
captured_filters: list[Any],
) -> None:
"""
GIVEN:
- Three indexed documents, one of which the caller may not see
WHEN:
- A restricted caller chats
THEN:
- The retriever receives a NOT IN filter on the denied id, not an
IN list of the permitted ids
"""
docs = DocumentFactory.create_batch(3, content="indexed document content")
for doc in docs:
indexing.llm_index_add_or_update_document(doc)
permitted = Document.objects.filter(pk__in=[docs[0].pk, docs[1].pk])
list(chat.stream_chat_with_documents("question?", permitted))
metadata_filter = captured_filters[0].filters[0]
assert metadata_filter.operator == FilterOperator.NIN
assert metadata_filter.value == [str(docs[2].pk)]
def test_restricted_chat_drops_retrieved_nodes_of_unpermitted_documents(
self,
temp_llm_index_dir: Path,
mock_embed_model: pytest_mock.MockType,
mocker: pytest_mock.MockerFixture,
) -> None:
"""
GIVEN:
- A retriever that returns a node for a document the caller may
not see, as when it is indexed after the permitted set was read
WHEN:
- A restricted caller chats
THEN:
- Only the node of the permitted document reaches the synthesizer
"""
permitted_doc = DocumentFactory.create(content="permitted content")
denied_doc = DocumentFactory.create(content="denied content")
permitted_node = MagicMock(metadata={"document_id": str(permitted_doc.pk)})
denied_node = MagicMock(metadata={"document_id": str(denied_doc.pk)})
mocker.patch("paperless_ai.chat.AIClient")
retriever = mocker.patch(
"llama_index.core.retrievers.VectorIndexRetriever",
).return_value
retriever.retrieve.return_value = [denied_node, permitted_node]
synthesizer = mocker.patch(
"llama_index.core.response_synthesizers.get_response_synthesizer",
).return_value
synthesizer.synthesize.return_value.response_gen = iter(["chunk"])
list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=permitted_doc.pk),
),
)
synthesizer.synthesize.assert_called_once_with("question?", [permitted_node])
@pytest.mark.django_db
def test_get_document_references_only_queries_referenced_documents(
self,
+23 -21
View File
@@ -18,7 +18,7 @@ from paperless_ai.migrations import Migration
from paperless_ai.migrations import m0001_v1_to_v2
from paperless_ai.tables import DocumentChunksTable
from paperless_ai.tables import DocumentMetaTable
from paperless_ai.vector_store import _MAX_IN_VALUES
from paperless_ai.vector_store import _MAX_INLINE_IN_VALUES
from paperless_ai.vector_store import DB_FILENAME
from paperless_ai.vector_store import DB_PAGE_SIZE
from paperless_ai.vector_store import DEFAULT_TABLE_NAME
@@ -354,37 +354,39 @@ class TestBuildWhere:
assert params == []
@pytest.mark.parametrize(
"build_filter",
[_in_filter, _nin_filter],
ids=["in", "nin"],
("build_filter", "expected_ids"),
[
pytest.param(_in_filter, ["a", "c"], id="in"),
pytest.param(_nin_filter, ["b"], id="nin"),
],
)
def test_fails_closed_when_filter_exceeds_max_values(
def test_query_with_oversized_filter_scopes_results(
self,
store: PaperlessSqliteVecVectorStore,
build_filter: Callable[[list[str]], MetadataFilters],
caplog: pytest.LogCaptureFixture,
expected_ids: list[str],
) -> None:
"""
GIVEN:
- An IN or NOT IN filter with more values than _MAX_IN_VALUES
(SQLite's own bound-parameter limit is 32766; this guard sits
below that with headroom for the query's other bound parameters)
- A store holding documents 1, 2 and 40000
- An IN or NOT IN filter naming documents 1 and 40000, padded
past _MAX_INLINE_IN_VALUES with unindexed ids
WHEN:
- _build_where() translates it to SQL
- The store is queried and get_nodes() is called with that filter
THEN:
- It fails closed ("1 = 0", no params) instead of building a
clause SQLite would reject, and logs a warning -- this filter
scopes document access, so refusing to build it must never
widen the scope to "everything" by accident. Failing open on
a NOT IN would surface exactly the excluded rows
- Both return exactly the documents the filter allows
"""
oversized = build_filter([str(i) for i in range(_MAX_IN_VALUES + 1)])
store.add(
[make_node("a", 1), make_node("b", 2), make_node("c", 40000)],
)
padding = range(100, 100 + _MAX_INLINE_IN_VALUES)
filters = build_filter([str(i) for i in (1, 40000, *padding)])
with caplog.at_level("WARNING"):
where, params = _build_where(oversized)
result = _query(store, [0.0] * DIM, top_k=5, filters=filters)
nodes = store.get_nodes(filters=filters)
assert where == "(1 = 0)"
assert params == []
assert "document_id" in caplog.text
assert sorted(result.ids) == expected_ids
assert sorted(n.node_id for n in nodes) == expected_ids
def test_query_with_untranslatable_filter_returns_no_rows(
self,
+16 -25
View File
@@ -89,15 +89,10 @@ class _Row(NamedTuple):
embedding: bytes
# _build_where(): the largest IN value list translated into bound SQL
# parameters. SQLite's own hard limit (SQLITE_MAX_VARIABLE_NUMBER) is 32766
# by default; this leaves headroom below that for the query's other bound
# parameters (the embedding blob, k, and any NE clause) and for the limit
# itself to move. An IN filter this large should not happen in practice --
# callers are expected to pass None (no filter) rather than every id when
# the filter would not actually narrow anything -- so this is a guard
# against a future regression, not a normal code path.
_MAX_IN_VALUES = 32700
# _build_where(): the largest IN / NOT IN list bound as one parameter per value,
# leaving headroom under SQLite's 32766-variable limit for the query's other
# parameters. Longer lists are bound as a single JSON array via json_each().
_MAX_INLINE_IN_VALUES = 32700
def _pack(embedding: Sequence[float]) -> bytes:
@@ -120,7 +115,7 @@ def _vec0_params(rows: list[_Row]) -> list[tuple[str, int, str, bytes]]:
return [(r.chunk_id, r.document_id, r.node_content, r.embedding) for r in rows]
def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int]]:
def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int | str]]:
"""Translate the EQ / IN / NIN / NE filters we use into a parameterized
SQL clause on vec0 metadata columns. Returns ("", []) when there is
nothing to filter. document_id is vec0's only filterable column and is
@@ -132,7 +127,7 @@ def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int]]:
if filters is None or not filters.filters:
return "", []
clauses: list[str] = []
params: list[int] = []
params: list[int | str] = []
for f in filters.filters:
# filters.filters is Union[MetadataFilter, ExactMatchFilter, MetadataFilters];
# we only build MetadataFilter entries, so skip anything else at runtime.
@@ -149,21 +144,11 @@ def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int]]:
# excludes nothing, so it matches everything.
clauses.append("1 = 0" if is_in else "1 = 1")
continue
if len(values) > _MAX_IN_VALUES:
# Refuse rather than risk SQLite's own bound-parameter limit
# ("too many SQL variables"): a list this large must match no
# rows, never widen the scope to "everything" -- true for
# NOT IN too, where failing open would surface every
# excluded row.
logger.warning(
"Refusing to build a %s filter on %r with %d values "
"(over the %d-value safety limit); returning no rows.",
sql_op,
f.key,
len(values),
_MAX_IN_VALUES,
if len(values) > _MAX_INLINE_IN_VALUES:
clauses.append(
f"{f.key} {sql_op} (SELECT value FROM json_each(?))",
)
clauses.append("1 = 0")
params.append(json.dumps(values))
continue
placeholders = ",".join("?" for _ in values)
clauses.append(f"{f.key} {sql_op} ({placeholders})")
@@ -558,6 +543,12 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
return {}
return DocumentMetaTable.all_modified_times(self._conn)
def indexed_document_ids(self) -> set[int]:
"""Return the ids of all indexed documents."""
if not self.table_exists():
return set()
return DocumentMetaTable.all_document_ids(self._conn)
@property
def _db_path(self) -> str:
return str(Path(self._uri) / DB_FILENAME)