pii-in-unstructured

PII Detection in Unstructured Data

Overview

Unstructured data — emails, documents, images, chat logs, call transcripts, and system logs — accounts for an estimated 80% of enterprise data and presents the greatest challenge for privacy compliance. Unlike structured databases where personal data resides in known columns, unstructured data contains PII embedded in free text, attached files, scanned images, and metadata. This skill covers detection approaches using Named Entity Recognition (NER), pattern matching, OCR, and hybrid pipelines, with focus on Microsoft Presidio and spaCy as implementation frameworks.

Unstructured Data Sources at Vanguard Financial Services

Detection Architecture

Microsoft Presidio Pipeline

Presidio is an open-source PII detection and anonymisation SDK developed by Microsoft, designed for integration with enterprise data pipelines.

Input Text/Document
       │
       ▼
┌──────────────────┐
│  Pre-processing  │  Format conversion, encoding normalisation,
│  (text extract)  │  OCR for images/scanned PDFs
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│  Presidio        │  Multiple recognisers run in parallel:
│  Analyzer        │  - NER model (spaCy/transformers)
│                  │  - Pattern recognisers (regex)
│                  │  - Custom recognisers (org-specific)
│                  │  - Context-aware enhancers
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│  Confidence      │  Each detection assigned confidence score
│  Scoring &       │  Threshold filtering applied
│  Filtering       │  Context enhancement boosts/reduces scores
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│  Results         │  PII locations, types, confidence scores
│  (structured)    │  Ready for classification, redaction, or alerting
└──────────────────┘

Component Details

NER Model (spaCy/Transformers):

• spaCy en_core_web_trf model (transformer-based) for English NER
• Detects: PERSON, ORG, GPE, DATE, MONEY, CARDINAL entities
• Custom-trained NER for financial domain entities (IBAN, account numbers)

Pattern Recognisers (Regex):

• UK National Insurance Number: [A-CEGHJ-PR-TW-Z]{2}\d{6}[A-D]
• Email address: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
• UK phone number: (?:0|\+44)\d{10,11}
• IBAN: [A-Z]{2}\d{2}[A-Z0-9]{4}\d{7}([A-Z0-9]?){0,16}
• Credit card: \b(?:\d{4}[-\s]?){3}\d{4}\b (with Luhn validation)
• IP address (v4): \b(?:\d{1,3}\.){3}\d{1,3}\b
• Date of birth patterns: \b\d{2}[/-]\d{2}[/-]\d{4}\b
• Vanguard account: VFS-\d{10}
• ICD-10 code: [A-Z]\d{2}\.\d{1,4}

Context-Aware Enhancement:

• Keyword proximity: if "National Insurance" appears within 50 characters of a NINO pattern, boost confidence by 15%
• Section headers: if detected within a section titled "Personal Details" or "Patient Information", boost confidence by 10%
• Negative context: if pattern appears within code blocks, SQL queries, or configuration files, reduce confidence by 25%

OCR Integration for Scanned Documents

Pipeline for Image-Based PII Detection

Scanned Document / Image
       │
       ▼
┌──────────────────┐
│  Pre-processing  │  Deskew, denoise, contrast enhancement,
│  (image)         │  resolution upscaling (if < 300 DPI)
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│  OCR Engine      │  Tesseract OCR (open-source) or
│                  │  Azure AI Document Intelligence (cloud)
│                  │  Output: extracted text with bounding boxes
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│  Presidio        │  Standard NER + pattern detection
│  Analyzer        │  on OCR-extracted text
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│  Confidence      │  Adjust for OCR quality:
│  Adjustment      │  OCR confidence < 80% → reduce PII confidence by 20%
│                  │  OCR confidence > 95% → no adjustment
└──────────────────┘

Document Type Detection for Vanguard

Confidence Scoring Framework

Score Composition

Confidence Thresholds

Implementation Patterns

Pattern 1: Email Scanning Pipeline

# Conceptual pipeline for Exchange Online email scanning
# 1. Microsoft Graph API retrieves email messages
# 2. Extract body text (HTML → plain text conversion)
# 3. Extract attachment text (document parsing)
# 4. Run Presidio analyzer on combined text
# 5. Map findings to email metadata (sender, recipients, date)
# 6. Apply classification labels via Microsoft Purview

Pattern 2: Log File Scanning

For application logs, specific patterns dominate:

• IP addresses in access logs
• User IDs and session tokens in authentication logs
• Email addresses in error messages
• Stack traces containing file paths with usernames

Log scanning requires higher false-positive tolerance and volume-optimised processing.

Pattern 3: Chat Message Scanning

Teams/Slack messages present unique challenges:

• Short messages with minimal context
• Abbreviations and informal language
• Customer names mentioned without surrounding keywords
• Account numbers shared in fragments across messages

Strategy: scan message threads rather than individual messages to capture context.

Accuracy Benchmarks

Expected Performance by Data Source

Integration Points

• auto-data-discovery: Unstructured PII detection complements structured data discovery platforms
• data-labeling-system: Detected PII drives automatic sensitivity label application
• classification-policy: Detection results feed into classification tier assignment
• data-inventory-mapping: Unstructured PII findings expand the data inventory to include document repositories and email systems