Add configurable DLP controls with optional Presidio endpoint support - #917
Open
zarg-dev wants to merge 25 commits into
Open
Add configurable DLP controls with optional Presidio endpoint support#917zarg-dev wants to merge 25 commits into
zarg-dev wants to merge 25 commits into
Conversation
Author
|
@microsoft-github-policy-service agree company="OCC" |
Bionic711
reviewed
Jun 19, 2026
Comment on lines
+8
to
+12
| try: | ||
| from functions_appinsights import log_event | ||
| except Exception: | ||
| def log_event(message, extra=None, level=None, exceptionTraceback=False): | ||
| return None |
Collaborator
There was a problem hiding this comment.
Was there an issue trying to import log_event? Why wrap it in the try block?
Bionic711
reviewed
Jun 19, 2026
| import re | ||
| from collections import OrderedDict | ||
|
|
||
| from functions_dlp_presidio import analyze_with_presidio_endpoint |
Collaborator
There was a problem hiding this comment.
presidio was mentioned to be optional but here it appears to be joining the core application alongside the new dlp feature?
Collaborator
|
zarg-dev Paul Lizer (@paullizer) and I are discussing the implementation and will get back to you once we have feedback. Stay tuned! |
10 tasks
Resolve conflicts between the DLP control plane / Presidio endpoint work and Development, keeping both sides' functionality. Notable resolutions: * config.py: VERSION bumped on top of Development's version, and Development's Easy Auth logout opt-out kept. * functions_documents.py: kept the DLP-aware enhanced-citation flag, the DLP-sanitized transcript embedding, and the DLP block re-raises, and moved Development's embedding character clamp onto the post-DLP embedding call site so vectors stay bounded but are never built from unredacted content. * route_backend_chats.py: kept Development's log prefixes together with the DLP log redaction, and kept the DLP-approved-query-only behavior instead of restoring the user_message fallback. * templates: Development split admin_settings.html into admin/_panes/*, so the DLP card moved into admin/_panes/content-safety.html in the same relative position and the CSRF hidden input was reapplied to the real form. * route_frontend_admin_settings.py: DLP redirects now use the blueprint endpoint name Development introduced. * release_notes.md: DLP entries filed under a new version section; no existing entries removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
functions_dlp_presidio.py imported urllib3.util.timeout._DEFAULT_TIMEOUT, a private symbol that only exists on urllib3 2.x, and referenced urllib3.connection.NameResolutionError, which was also added in 2.x. requirements.txt does not pin urllib3, so it is resolved transitively through requests. On urllib3 1.x the private import raised ImportError, and because functions_documents.py imports functions_dlp -> functions_dlp_presidio, that failure propagated to application start-up. Resolve the sentinel through urllib3's public Timeout.DEFAULT_TIMEOUT instead. It is the identical object to the 2.x private sentinel and to socket._GLOBAL_DEFAULT_TIMEOUT on 1.x, so connect-timeout behavior on the DLP request path is unchanged: an explicit timeout is still applied, and only the "leave the socket default alone" case still skips settimeout(). DNS failures now raise NameResolutionError when present and NewConnectionError, its base class, otherwise. Also update tests for the merge with Development: * test_upload_dlp_ingestion_integration.py stubs the document access index and Search write fence modules Development added to the functions_documents import chain, and asserts the relocated embedding clamp still reads only sanitized text. * test_web_search_current_message_only.py expects Development's [WEB_SEARCH] log prefix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Take Development through the Custom model endpoint provider work and re-file the DLP release notes, which is the only conflict this merge produced. * config.py: VERSION 0.261.011, above Development's 0.261.010. * release_notes.md: DLP entries moved to a new v0.261.011 section so the v0.261.010 Custom model endpoint entry keeps the version it shipped under. Nothing was removed. * Documented the DLP capability toggles on the Security admin page and claimed enable_dlp_control_plane, enable_upload_dlp and enable_web_search_dlp in docs/_data/features.yml, then regenerated docs/_data/app_surface.yml. * build_release_notes_pages.py now rewrites relative release-note links when it emits into docs/explanation/release-notes/. The generated pages sit one level below the source, so links such as fixes/NAME.md previously resolved to files that do not exist once a section reached a generated page. * Updated the upload DLP ingestion test harness for Development's search write path, which passes client timeouts and requires per-document acknowledgement. Re-verified after merging that the DLP enforcement ordering still holds: save_chunks and save_chunks_batch evaluate upload DLP before any embedding call and the embedding character clamp only ever narrows the sanitized text, and perform_web_search still refuses to fall back to the raw user message. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Take Development through the revert of the Custom model endpoint provider (microsoft#1431 reverting microsoft#1228). Only two files conflicted, both mechanical. * config.py: kept VERSION 0.261.011. The revert returned Development to 0.261.009, so this is still ahead and needs no renumber. * release_notes.md: kept the v0.261.011 DLP section and took Development's side of the region below it, so the reverted v0.261.010 Custom model endpoint section is not reinstated. * Regenerated docs/explanation/release-notes/ so the generated pages drop the reverted v0.261.010 entry with the source. docs/_data/app_surface.yml was regenerated and is unchanged, since the revert did not alter the application surface this PR documents. Re-verified the DLP enforcement paths against the reverted file contents rather than assuming they carried over, because microsoft#1228 had touched both files: * save_chunks and save_chunks_batch still evaluate upload DLP before any embedding call, and the embedding character clamp still only narrows sanitized_chunk_text. * perform_web_search still has no user_message fallback, so only the DLP-approved query can reach the outbound search. * Video enhanced-citation upload still gates on the caller-supplied enable_enhanced_citations flag that upload DLP clears. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| panel.classList.toggle('d-none', !toggle.checked); | ||
| toggle.addEventListener('change', function () { | ||
| panel.classList.toggle('d-none', !this.checked); | ||
| if (typeof markFormAsModified === 'function') { |
| }; | ||
| dlpDefaultEngine.addEventListener('change', function () { | ||
| updateDlpEngineVisibility(); | ||
| if (typeof markFormAsModified === 'function') { |
| if request_error_type: | ||
| raise PresidioEndpointRequestError(f"Presidio analyzer request failed: {request_error_type}") from None | ||
|
|
||
| if not isinstance(body, list): |
| from config import * | ||
| from functions_appinsights import log_event | ||
| try: | ||
| from functions_appinsights import log_event |
| from functions_dlp_rules import get_effective_dlp_regex_rules, scan_text_with_dlp_regex_rules | ||
|
|
||
| try: | ||
| from functions_appinsights import log_event |
| def evaluate_dlp_text(text, settings=None, context=None, surface="generic"): | ||
| """Evaluate text against the configured DLP policy and return a safe result.""" | ||
| settings = settings or {} | ||
| context = context or {} |
| if source_address: | ||
| sock.bind(source_address) | ||
| sock.connect(sockaddr) | ||
| last_error = None |
| from config import * | ||
| from functions_appinsights import log_event | ||
| try: | ||
| from functions_appinsights import log_event |
Comment on lines
+43
to
+48
| from functions_dlp import ( | ||
| build_dlp_telemetry_properties, | ||
| build_upload_dlp_file_log_summary, | ||
| evaluate_upload_content, | ||
| should_emit_dlp_telemetry, | ||
| ) |
| """ | ||
|
|
||
| import importlib.util | ||
| import os |
|
|
||
| def test_enforced_truncation_blocks_before_scanner_error_fail_open(): | ||
| """Protected enforced surfaces should block truncated text before scanner errors.""" | ||
| import functions_dlp |
|
|
||
| def test_scanner_error_fails_closed_by_default(): | ||
| """Scanner errors must not allow web-search egress by default.""" | ||
| import functions_dlp |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a configurable DLP layer for web-search egress and upload ingestion. The built-in path is regex-based with narrow defaults, and admins can tune or expand the rules. I also added optional integration with an external Presidio Analyzer-compatible endpoint for customers who want richer detectors without adding Presidio packages and models to the SimpleChat app image.
What changed
Notes
The regex model is intentionally simple. It gives deployers a lightweight starting point, but Presidio is the better fit for customers who want broader PII detection such as emails, phone numbers, IP addresses, names, or other recognizers.
I extended the same DLP path to uploads because the use case felt similar: some customers may want to scrub or block sensitive content before it is sent to model inference, embeddings, or search indexing.
Testing
Merge-resolution changes outside the DLP feature
This branch was merged with
Developmentup to4a29f67e. Three changes beloware not part of the DLP control plane but were required for the merged branch to
land green. They are called out separately so they can be reviewed on their own
terms.
1.
scripts/build_release_notes_pages.py— relative link rewritingdocs/explanation/release_notes.mdis the source of truth and the pages underdocs/explanation/release-notes/are generated from it.test_docs_release_notes_integrityrequires the two to agree, so adding a release-notes section means regenerating.
The generated pages sit one directory below the source, but the generator copied
release text verbatim. Relative links such as
fixes/NAME.md, which resolvecorrectly from
docs/explanation/, therefore pointed at files that do not existonce a section reached a generated page — 8 broken links, caught by
test_docs_link_integrity.rewrite_relative_links()now prefixes../for relative targets, skippingabsolute paths, anchors, URL schemes and Liquid expressions. This is a
pre-existing generator bug that regeneration exposed rather than introduced;
the committed pages were already stale on
Development(the index covered onlyup to v0.261.003 while the source had later versions).
2. Documentation for the DLP capability toggles
test_docs_app_surface_coveragefails on any unclaimedenable_*settings key.This PR adds three:
enable_dlp_control_plane,enable_upload_dlpandenable_web_search_dlp.docs/_data/features.ymlunder a newdata-loss-preventionentry.docs/admin/security.mdunder Content Safety, covering themonitor/redact/block modes, the regex and external Presidio engines, the
private-host allowlist, env-var-backed API key handling, and counts-only
telemetry.
docs/_data/app_surface.ymlregenerated withscripts/build_docs_inventory.py.3.
functions_dlp_presidio.py— urllib3 version safetyThe adapter imported
urllib3.util.timeout._DEFAULT_TIMEOUT, a private symbolthat only exists on urllib3 2.x, and referenced
urllib3.connection.NameResolutionError,also 2.x-only.
urllib3is not pinned and is resolved transitively throughrequests, andfunctions_documents.pyimports this module, so on urllib3 1.xthe private import raised
ImportErrorduring application start-up.The sentinel now resolves through urllib3's public
Timeout.DEFAULT_TIMEOUT,which is
is-identical to the 2.x private sentinel and tosocket._GLOBAL_DEFAULT_TIMEOUTon 1.x. Connect-timeout behavior on the DLPrequest path is unchanged: an explicit timeout is still applied and only the
"leave the socket default alone" case skips
settimeout(). DNS failures raiseNameResolutionErrorwhere available and its baseNewConnectionErrorotherwise. No
urllib3pin was added.Covered by
functional_tests/test_dlp_presidio_urllib3_compatibility.py, whichpasses on both urllib3 1.26.20 and 2.7.0.
Template placement note
Developmentsplittemplates/admin_settings.htmlintotemplates/admin/_panes/*.html.The DLP card was moved verbatim into
admin/_panes/content-safety.html,immediately after the Content Safety card, matching its original position
relative to its neighbours. Left inline in the old monolith it would have
rendered outside every
tab-pane. The CSRF hidden input was reapplied to thereal
<form>inadmin_settings.html, and three DLP tests were repointed atthe pane.
Three PR-added
redirect(url_for('admin_settings'))calls — two on DLPvalidation-failure paths — were updated to
url_for('frontend_admin_settings.admin_settings')afterDevelopmentmovedthat route into a blueprint. As merged they would have raised
BuildError.