Files

T

2026-09-13 20:07:03 +01:00

53 KiB

Raw Blame History

PST Archive — Mail Indexer

A self-hosted, Dockerized application for uploading Microsoft Outlook PST/OST files, indexing every email inside them, and searching the contents through a web browser — with full user authentication, multi-factor authentication (MFA), encrypted-at-rest credentials, and a separate admin panel on its own port.

Designed for forensic review, e-discovery, personal archive browsing, and migration audits — any situation where you need to crack open a PST and actually find things inside it without loading it into Outlook.

Features
Quick Start
First-Run Setup
Running as a Daemon
Theming & Dark Mode
Authentication & Security Model
Admin Panel
Organizing PSTs with Folders
Folder Access Permissions
Multi-Factor Authentication (MFA)
Encryption at Rest
How It Works
Architecture
Project Layout
Data Model
REST API Reference
Frontend Interfaces
Search Syntax
Configuration
Persistence & Backup
Troubleshooting
Performance Notes
Security Considerations
Extending the App

Features

Core

Drag-and-drop upload of .pst and .ost files through the browser
Streaming upload with progress bar — handles multi-gigabyte archives
Background indexing into SQLite with FTS5 full-text search (ranked, prefix-matched, with highlighted snippets)
Multi-archive support: switch between uploaded PSTs; each is searched independently
Folder organization: create folders in the sidebar to group related PST files (e.g. by client, case, year). Move archives between folders, rename folders, or delete folders (any archives inside are safely moved to Uncategorized)
Per-folder access permissions: admins can restrict which non-admin users can view each folder. A single user can be granted access to many folders; a single folder can be shared with many users (true many-to-many)
Folder filter: narrow search to any folder within the archive (Inbox, Sent Items, nested folders)
Full message viewer: subject, sender, recipients, date, folder, attachments, body
Persistent storage: everything lives in a mounted ./data volume

Security

First-run setup wizard forces creation of an initial administrator before any other action is possible
Password authentication using Argon2id (winner of the Password Hashing Competition) with constant-time verification
TOTP-based MFA (RFC 6238; compatible with Google Authenticator, Authy, 1Password, Microsoft Authenticator, and every other standard authenticator app)
Encryption at rest for all user PII — usernames, password hashes, and TOTP secrets are stored as Fernet (AES-128-CBC + HMAC-SHA256) ciphertexts
Master key auto-generated on first run and stored with chmod 600, or overridable via MASTER_SECRET environment variable
HMAC-based username lookup — usernames are never stored in plaintext but are still indexable for login
JWT session cookies, signed with an HKDF-derived sub-key, HttpOnly, SameSite=Lax
Rate limiting on login attempts (10 failures per 5 minutes per IP+username)
Admin-only user management: create, delete, role-change, password-reset, MFA-enroll other users
Admin file & folder management: system-wide view of every uploaded PST and every folder, with create/rename/delete for folders and force-delete for any file regardless of uploader
Separate admin panel on a separate port — lockable behind its own firewall rule or VPN

UI

Light and dark themes on both the main app and the admin panel, with a sun/moon toggle in the masthead and on login/setup screens
Preference persists in localStorage per-browser and respects the operating system's prefers-color-scheme on first visit
Theme applied inline in <head> before stylesheets load — no flash of wrong theme on page load
Dark mode uses a custom "inked paper" palette (deep charcoal with warm paper-cream text and softened oxblood accent) — intentional character, not generic-dark
Editorial typography (Fraunces serif + JetBrains Mono), grain overlay, responsive layout down to mobile widths

Quick Start

Prerequisites

Docker 20.10 or newer
Docker Compose v2
A .pst or .ost file to index

Run (as a daemon — recommended)

docker compose up -d --build

The -d flag runs the container in the background (detached). On the first run this takes 3–5 minutes because libpff-python compiles a C extension from source; subsequent startups are nearly instant.

The container is configured with restart: unless-stopped, so it will automatically come back up after a reboot or a crash, and keep running across Docker daemon restarts — until you explicitly stop it.

Once the container is up:

Main app: http://localhost:8000
Admin panel: http://localhost:8001

Foreground mode (useful for debugging)

docker compose up --build

This runs the container attached to your terminal so you can see logs inline. Press Ctrl-C to stop.

Tail the logs

docker compose logs -f              # follow both streams
docker compose logs -f pst-indexer  # explicit service name

Check status

docker compose ps                   # is it running?
docker compose top                  # which processes?

Stop

docker compose stop        # stop, keep data, keep container
docker compose down        # stop and remove the container, keep data volume

Full reset

docker compose down
rm -rf ./data              # erases everything including users and the master key
docker compose up -d --build

Restart / update

# After editing code or pulling new images:
docker compose up -d --build       # rebuild & restart with one command
# Or just restart without rebuilding:
docker compose restart

First-Run Setup

On the very first visit to http://localhost:8000, you will be presented with a setup wizard that asks you to create the initial administrator:

Choose a username (3–64 characters)
Choose a password (8+ characters — use a passphrase you'll remember)
Confirm the password

The setup endpoint is only callable while the users table is empty. Once the first admin is created, the endpoint returns 409 Conflict and is effectively disabled.

After setup, you are automatically signed in and landed on the main app.

Heads up: the admin panel on port 8001 will refuse to accept any logins until the first admin has been created via the main app's setup wizard. This prevents a race where someone could hit /api/login on the admin port before you've set things up.

Running as a Daemon

By default, this app is designed to run as a long-lived background service. The docker-compose.yml sets restart: unless-stopped, which means once started, the container will:

Keep running after you close your terminal
Automatically restart if the process crashes
Automatically restart when the host reboots and Docker comes back up
Not restart if you explicitly stop it with docker compose stop (hence "unless-stopped")

Starting the daemon

docker compose up -d --build

The -d flag is short for "detached" — the container is launched in the background and the command returns immediately. Your terminal is free to do other things; the container keeps running.

Verifying it's running

docker compose ps

You should see output like:

NAME          IMAGE                       STATUS                   PORTS
pst-indexer   pst-indexer-pst-indexer     Up 2 minutes (healthy)   0.0.0.0:8000->8000/tcp, 0.0.0.0:8001->8001/tcp

The (healthy) flag comes from the built-in healthcheck, which probes both /api/health endpoints every 30 seconds.

Watching logs

Since the container is detached, you won't see logs in your terminal by default. Attach to them on demand:

docker compose logs                 # print recent logs and exit
docker compose logs -f              # follow (live tail), Ctrl-C to detach
docker compose logs -f --tail=100   # follow, starting from the last 100 lines
docker compose logs --since 1h      # only logs from the last hour

Detaching from -f does not stop the container — it just stops showing you the logs.

Stopping

docker compose stop          # stop the container but keep it around
docker compose start         # start it again if previously stopped
docker compose restart       # stop and start in one command
docker compose down          # stop and remove the container (data volume is kept)
docker compose down -v       # stop, remove container, AND remove anonymous volumes

None of these delete your ./data directory — that's a bind mount, not a Docker-managed volume, so it's always yours.

Survives reboots

Because restart: unless-stopped is set, after a host reboot Docker will automatically start the container as soon as the Docker daemon comes up. Nothing manual required on your part.

If you ever explicitly stopped it with docker compose stop, Docker will not auto-start it after reboot (that's the whole point of "unless-stopped"). Run docker compose start or docker compose up -d to bring it back.

Updating to a new version

# Pull new source, then:
docker compose up -d --build

Docker Compose compares the current image to the built one; if they differ, it rebuilds and recreates the container with zero-downtime-style replacement. Your ./data volume is preserved throughout.

Theming & Dark Mode

Both the main app and the admin panel support light and dark themes, with a sun/moon toggle visible in the masthead (when signed in) and floating in the top-right corner on the login/setup screens.

How the toggle works

Click the icon to flip between themes
Choice is saved to localStorage["pst-theme"] in your browser
On a fresh browser (nothing saved), the app respects your operating system's prefers-color-scheme setting
The theme is applied by an inline <script> in the HTML <head>, before the CSS loads — so there is no flash of the wrong theme on page load

Palettes

Light (the default): warm off-white paper #f2ede3, deep ink #1b1814, oxblood accent #a8341f. Editorial and understated — the original aesthetic.

Dark: "inked paper" #17140f, warm paper-cream text #e9e0cb, softened oxblood #d06347, brass-yellow highlight #d8b64a. Intentionally avoids the generic "cold dark mode" look — it feels like reading inked paper under a warm lamp rather than staring into a cold screen.

The admin panel uses a slightly more saturated variant of the same palette for both themes, to preserve its more utilitarian character.

Implementation note for developers

Both stylesheets use CSS custom properties keyed off a data-theme attribute on <html>:

:root, :root[data-theme="light"] { --paper: #f2ede3; ... }
:root[data-theme="dark"]         { --paper: #17140f; ... }

All component colors reference these variables, and tinted effects (hover backgrounds, gradient glows, shadow colors) use color-mix() so they derive from the same tokens and automatically adapt. Adding a third theme would mean adding one more [data-theme="..."] block.

Authentication & Security Model

Two session scopes

The system has two independent session scopes with separate cookies:

Scope	Port	Cookie	Required Role
`app`	8000	`pst_session`	any authenticated user
`admin`	8001	`pst_admin_session`	must have `role = admin`

A session token issued for one scope will not work for the other. Logging in on one port does not sign you in on the other — they're entirely separate authentications. This is deliberate: the admin panel deserves its own credential entry even if you're already signed in to the main app.

Roles

user — can sign into the main app at port 8000, upload PSTs, search, and view messages
admin — everything above, plus access to the admin panel at port 8001 to manage users

The first user created (via the setup wizard) is automatically an admin.

Password storage

Passwords are hashed with Argon2id before anything else happens:

64 MiB memory cost
3 iterations
Parallelism of 2

The resulting Argon2 hash is then additionally encrypted with Fernet before being written to the database. This means even if an attacker dumps the SQLite file, they must also have the master key to start an offline brute-force attack against Argon2 — which is itself designed to be very slow to crack.

Verification is constant-time (handled by the argon2-cffi library).

Client POSTs {username, password} (and optionally totp_code) to /api/login
Server normalises the username (trim + lowercase), computes HMAC-SHA256(username-lookup-key, normalized), and queries by that hash
If found, decrypts the stored Argon2 hash using the master key's Fernet sub-key
Calls argon2.verify(hash, password) — constant time
If MFA is enabled on the account, requires a valid 6-digit TOTP code
On success, mints a JWT containing {sub: user_id, role, scope, iat, exp} signed with an HKDF-derived JWT key
Sets the appropriate HttpOnly cookie on the response

Rate limiting

Failed login attempts are tracked per (ip, username) tuple in memory. After 10 failures within 5 minutes, login returns 429 Too Many Requests for that specific combination until the window expires. Successful logins reset the counter.

This is per-process in-memory — if you scale to multiple worker processes or containers, you'd want to replace it with Redis. For single-container use it's fine.

Cookies

All session cookies are:

HttpOnly — not readable from JavaScript
SameSite=Lax — submitted on top-level navigation but blocked from cross-site POSTs
Secure — enabled when COOKIE_SECURE=true (set this when serving over HTTPS)
Path-limited to /

Sessions

Session JWTs have a default lifetime of 8 hours (configurable via SESSION_TTL_SECONDS). The session token is opaque to the client — the client just presents it as a cookie and the server validates & extracts the user ID.

There is no refresh-token flow. When the session expires, the user signs in again.

Admin Panel

Available at http://localhost:8001.

The admin panel has four tabs: Users, Folders, Files, and Permissions.

Users tab

List all users with username, role, MFA status, created date, last login
Create new users — specify username, password, and role
Delete users (cannot delete yourself; cannot delete the last remaining admin)
Toggle role between user and admin (cannot demote yourself; cannot demote the last admin)
Reset password for any user
Manage MFA for any user: begin enrollment (QR code + secret), confirm enrollment with a test code, disable, or re-enroll

Folders tab

List all folders with their name, the count of PSTs inside, and created date
Create new folders via a modal; folder names are case-insensitively unique (you can't have both "Clients" and "clients")
Rename folders (same uniqueness rule)
Delete folders — archives inside are not deleted; they're moved to "Uncategorized"

Folders are a system-wide, flat (non-nested) organizational feature. Any authenticated user can see and use folders; admins get the dedicated management tab. Regular users can also create and delete folders from the sidebar of the main app — folders are shared across all users by design.

Files tab

System-wide view of every PST uploaded by every user, with folder assignment, size, indexing status, message count, and upload date
Force-delete any PST regardless of uploader — removes the raw .pst file from disk, all indexed messages, and the database record
Useful for cleaning up stalled/failed indexes or reclaiming disk space

Permissions tab

Matrix view of every user against every folder, with check/uncheck toggles
Two perspectives you can switch between: By user (rows = users, cols = folders) and By folder (rows = folders, cols = non-admin users)
Toggling is instant — no save button, each change calls the API immediately with optimistic UI updates
Admin rows show as read-only implicit grants (every column checked with a neutral ✓) — admins always have access to every folder regardless of these settings
See the dedicated "Folder Access Permissions" section below for the full access model.

Safety rails

The admin API enforces these invariants server-side:

You cannot delete your own account
You cannot demote your own admin role to user
The system must always have at least one admin — the last admin cannot be deleted or demoted
Deleting a folder never cascade-deletes PSTs — archives are always safely reassigned
Passwords must be at least 8 characters
Usernames must be 3–64 characters; folder names max 80 characters

These are enforced in the backend, not just the UI — so a malicious admin with a modified client cannot bypass them.

Organizing PSTs with Folders

Uploaded archives can be grouped into folders to keep the sidebar navigable when you've accumulated dozens of PSTs.

In the main app (sidebar)

"+ New folder" button in the Archives section header creates a new folder
Each folder is a collapsible group — click the header to expand/collapse
Each folder header has a "⋯" menu → Rename / Delete
Each PST has its own "⋯" menu → Move to folder… / Delete archive
PSTs that aren't assigned to any folder sit in an Uncategorized group at the bottom

In the admin panel

The Folders tab is the source of truth: create, rename, delete folders across the system
The Files tab shows every PST and the folder it belongs to, with force-delete

Behavior on folder delete

When a folder is deleted (from either the main app or admin panel):

The folder record is removed
All PSTs that were inside it get folder_id = NULL (they become Uncategorized)
Indexed messages are not touched — the archives remain fully searchable

This is deliberate: folders are an organizational convenience, not a security boundary. Accidentally deleting a folder should never cost you data.

Notes

Folder names are case-insensitively unique
Max 80 characters per folder name
No nesting — folders are a single flat level. If you need hierarchy, encode it in the name (e.g. "Clients / ACME", "Clients / Globex")
Folders are system-wide, not per-user: every authenticated user sees the same folders

Folder Access Permissions

By default, non-admin users cannot see any folder except the shared Uncategorized bucket. Admins grant specific non-admin users access to specific folders via the admin panel. This creates true multi-tenant isolation inside a single shared deployment.

Access rules

Role	Sees
Admin	Every folder, every PST, every message — always
Non-admin (no grants)	Only PSTs in Uncategorized
Non-admin (with grants)	Only the folders they've been granted + Uncategorized

Uncategorized is deliberately shared across everyone — it catches freshly-uploaded-but-not-yet-sorted PSTs so they don't become invisible to the person who uploaded them.

Many-to-many by design

The permission model is a true matrix:

One user → many folders: grant Alice access to "Clients", "Legal", and "HR" independently
One folder → many users: grant both Alice and Bob access to "Legal" at the same time
Flexible combinations: Alice in Clients+Legal, Bob in Legal+HR, Charlie in HR alone — any shape works

Managing permissions

Open the admin panel at http://localhost:8001, sign in as an admin, and click the Permissions tab. A matrix view appears with two perspectives:

By user (default): rows are users, columns are folders. Check a box to grant that user access to that folder.
By folder: rows are folders, columns are non-admin users. Same effect, different axis — use whichever feels more natural for the task at hand.

Toggling a checkbox calls PUT /api/permissions/{user_id}/{folder_id} and applies instantly. Admin rows in the matrix are displayed as read-only implicit grants (every cell shows ✓) with a visual distinction so you don't confuse yourself about whether admins have "been granted" anything — they don't need to be.

Backend enforcement

Every data-access endpoint on the main app (port 8000) checks permissions server-side:

GET /api/folders returns only folders the user can see
GET /api/pst-files returns only PSTs in those folders + Uncategorized
GET /api/pst-files/{id}/messages returns 404 if the user can't see that PST's folder (same shape as a genuinely nonexistent PST — no information leakage)
GET /api/messages/{id} checks the parent PST's folder
POST /api/pst-files?folder_id=X verifies the uploader has access to folder X
PUT /api/pst-files/{id}/folder verifies access to both current and destination folders
POST/PUT/DELETE /api/folders* are admin-only (non-admins get 403)

A modified UI cannot bypass these — they're all dependency-injection checks inside the FastAPI route handlers.

Cascading deletes

User deleted → all their permission grants are removed (auth.delete_user handles this in the same transaction)
Folder deleted → all grants targeting that folder are removed, and any PSTs inside are moved to Uncategorized (not cascade-deleted — see "Behavior on folder delete" above)

Example workflow

You're running the app for a small law firm. You have three admins (the partners) and ten non-admin users (associates and assistants).

Admin logs into port 8001, goes to Folders tab, creates folders: "Matter 2024-001", "Matter 2024-002", "Firm Admin"
Admin uploads the relevant PSTs from the main app on port 8000, filing each into its matter folder
Admin goes to Permissions tab, grants associate Alice access to "Matter 2024-001" and the paralegal Bob access to both matters
Alice signs in to port 8000 — her sidebar shows only "Matter 2024-001" plus any Uncategorized uploads. She searches, views messages, moves PSTs between folders she has access to
Bob signs in — sees both matters. If he tries the admin panel on port 8001 he gets 403 because he's not an admin

Multi-Factor Authentication (MFA)

MFA is per-user and optional by default. An admin enables it for each user individually via the admin panel.

How enrollment works

Admin opens the MFA modal for a user (in the admin panel)
Clicks "Begin enrollment" — the server generates a new random base32 TOTP secret and stores it encrypted (mfa_enabled stays 0 until the user proves they have it)
The UI shows:
- A QR code containing the otpauth:// URI
- The raw secret as text (for manual entry into apps that don't support QR)
- A field to enter a 6-digit verification code
The user scans the QR (or types the secret) into their authenticator app
They read the 6-digit code from the app and the admin enters it
Server verifies the code. If valid, sets mfa_enabled = 1. If invalid, enrollment stays unconfirmed (the secret is kept so you can retry without re-scanning)

User submits username + password to /api/login (no code yet)
Server verifies credentials and sees mfa_enabled = 1
Server responds with 401 {"detail": "MFA required", "mfa_required": true} — this is a signal to the client, not an error
Client reveals the "Authentication code" field and prompts for the code
User enters the 6-digit code; client re-submits the full form (username + password + totp_code)
Server re-verifies credentials, then verifies the code against pyotp.TOTP(secret).verify(code, valid_window=1) (±30 seconds tolerance)
On success, issues the session cookie

Supported authenticator apps

Any RFC 6238-compliant TOTP app works:

Google Authenticator
Microsoft Authenticator
Authy
1Password
Bitwarden
Aegis (Android)
Raivo OTP (iOS)
KeePassXC with TOTP plugin
Any otpauth:// URI consumer

Disabling MFA

An admin can disable MFA for any user. This clears both the totp_secret and the mfa_enabled flag. The next time they log in, they will not be prompted for a code.

Encryption at Rest

Everything sensitive in the users table is encrypted before being written to disk. The plaintext is only ever held in memory during request processing.

What's encrypted

Field	How
Username	Fernet ciphertext
Password hash (Argon2)	Fernet ciphertext (double-protection: Argon2 + Fernet)
TOTP secret	Fernet ciphertext

What's not encrypted (and why)

Field	Reason
`id`	Random UUIDs, no PII
`username_hash`	HMAC-SHA256 — one-way, non-reversible. Only used to find a user at login.
`role`	Not PII; needed for fast query filtering
`mfa_enabled`	Not sensitive; just a boolean flag
`created_at`, `last_login`	Timestamps; not uniquely identifying on their own

Email content inside indexed PSTs is not currently encrypted at the application layer. If you need that, use full-disk encryption on the host or a LUKS volume mounted as ./data. The architecture specifically isolates user credentials (which are small and always decrypted on every login) from message content (which is large and streamed through full-text search).

Master key handling

The system uses one master secret from which all encryption keys are derived via HKDF-SHA256 with distinct info parameters:

fernet-user-pii — the Fernet key for encrypting username, password hash, TOTP secret
jwt-sessions — the HMAC key for signing session JWTs
username-lookup — the HMAC key for deterministic username lookup hashes

The master secret itself is either:

Option A (default): auto-generated on first run using secrets.token_bytes(32) and written to ./data/secret.key with chmod 600. Convenient, but anyone with filesystem access to ./data can read the key.

Option B (recommended for production): supplied via the MASTER_SECRET environment variable. The provided string is SHA-256-hashed to derive the 32-byte master key. Store this in a secret manager (Docker secrets, Kubernetes secrets, Vault, AWS Secrets Manager, 1Password CLI, etc.) and inject it at runtime:

environment:
  - MASTER_SECRET=${PST_MASTER_SECRET}

If MASTER_SECRET is set, the on-disk secret.key is ignored.

Key rotation

Changing the master key invalidates all encrypted data. There is currently no automated rotation path — you'd need to write a migration that decrypts with the old key and re-encrypts with the new one. For most single-operator deployments, you just don't rotate; for regulated environments, plan a rotation procedure.

If the master key is ever lost, the encrypted data is unrecoverable. Back it up separately from the database if you're using Option A.

How It Works

The end-to-end lifecycle of a PST file, from upload to searchable results:

1. Authentication

Every API call (except /api/health, /api/setup, /api/setup/status, and /api/login) requires a valid session cookie. The FastAPI dependency require_app_user decodes the JWT, verifies the signature and scope, looks up the user, and raises 401 if anything fails.

2. Upload

The browser POSTs the PST as multipart/form-data. The backend streams the upload to disk in 1 MB chunks (memory usage stays flat for multi-GB archives), stores it under /app/data/uploads/{uuid}_{original_name}, and inserts a row into pst_files with status indexing and uploaded_by = <current user id>.

3. Background indexing

BackgroundTasks schedules a job on a thread (asyncio.to_thread). The job:

Opens the PST with pypff.file() and grabs the root folder
Recursively walks every folder via get_sub_folder(i), tracking the folder's full path
For each message, extracts subject, sender, recipients, date, body (plain text / HTML-stripped / RTF), and attachment filenames
Batches 500 rows at a time into executemany inserts
On success, updates status to ready and records message_count
On failure, sets status to failed and records the error

4. FTS indexing

An AFTER INSERT trigger on messages mirrors every row into messages_fts, a SQLite FTS5 virtual table using the porter unicode61 tokenizer (stemming + Unicode normalization). This runs inside the same transaction as the insert — the table and index can never drift out of sync.

5. Search

When you type in the search box, the frontend debounces for 250 ms and calls GET /api/pst-files/{id}/messages?q=.... The backend:

Sanitises the query via _escape_fts() — each whitespace-separated word becomes a quoted prefix-match token
Joins messages to messages_fts via MATCH, filters by pst_id (and optionally folder), orders by FTS5 rank (BM25)
Returns snippets with <mark> around matched terms

6. Viewing a message

Clicking a row hits GET /api/messages/{id} and opens the full message in a slide-in panel.

7. Deletion

DELETE /api/pst-files/{id} cascades messages, removes the pst_files row, and unlinks the raw PST from disk — all in one transaction.

Architecture

┌──────────────────────────────────────────────────────────────────┐
│  Browsers                                                        │
│  ┌───────────────────────────────┐  ┌──────────────────────────┐ │
│  │  Main app (localhost:8000)    │  │  Admin (localhost:8001)  │ │
│  │  index.html + app.js          │  │  index.html + admin.js   │ │
│  │  - Setup wizard (first run)   │  │  - Admin login (+MFA)    │ │
│  │  - Login (+MFA)               │  │  - User CRUD             │ │
│  │  - Upload / search / view     │  │  - MFA enrollment        │ │
│  └──────────────┬────────────────┘  └────────────┬─────────────┘ │
└─────────────────┼─────────────────────────────────┼──────────────┘
                  │                                 │
                  │  cookie: pst_session            │  cookie: pst_admin_session
                  │  scope: "app"                   │  scope: "admin"
                  ▼                                 ▼
┌──────────────────────────────────────────────────────────────────┐
│  Docker container: pst-indexer                                   │
│                                                                  │
│  ┌──────────────────────┐        ┌──────────────────────────┐    │
│  │ uvicorn :8000        │        │ uvicorn :8001            │    │
│  │ app.py (FastAPI)     │        │ admin_app.py (FastAPI)   │    │
│  │ ├─ /api/setup        │        │ ├─ /api/login            │    │
│  │ ├─ /api/login        │        │ ├─ /api/users            │    │
│  │ ├─ /api/pst-files    │        │ ├─ /api/users/:id/mfa/…  │    │
│  │ └─ /api/messages     │        │ └─ /api/users/:id/role   │    │
│  └─────────┬────────────┘        └────────────┬─────────────┘    │
│            │                                  │                  │
│            └──────────────┬───────────────────┘                  │
│                           ▼                                      │
│                ┌──────────────────────┐                          │
│                │  auth.py (shared)    │                          │
│                │  - Argon2 hashing    │                          │
│                │  - Fernet encryption │                          │
│                │  - TOTP (pyotp)      │                          │
│                │  - JWT sessions      │                          │
│                │  - User CRUD         │                          │
│                └──────────┬───────────┘                          │
│                           │                                      │
│    ┌──────────────────────┴──────────────────────┐               │
│    ▼                                             ▼               │
│ ┌──────────────┐                          ┌──────────────────┐   │
│ │ pypff        │                          │ SQLite + FTS5    │   │
│ │ (libpff C)   │                          │ /app/data/       │   │
│ │ parses PST   │                          │   pst_index.db   │   │
│ └──────┬───────┘                          └──────────────────┘   │
│        │                                                         │
│        ▼                                                         │
│ ┌──────────────────────┐                                         │
│ │ /app/data/uploads/   │  ← raw PST files                        │
│ │ /app/data/secret.key │  ← master key (chmod 600)               │
│ └──────────────────────┘                                         │
└──────────────────────────────────────────────────────────────────┘
                           │
                           │  bind mount
                           ▼
                    ./data  (host filesystem)

Why two separate FastAPI instances?

Because operators routinely want the admin panel to be more restricted than the main app. Running them as separate processes on separate ports means you can:

Firewall port 8001 to internal IPs only
Put port 8001 behind a VPN while leaving port 8000 public
Rate-limit the two ports differently in a reverse proxy
Turn off the admin panel entirely (-p 8000:8000 only, omit 8001) when you don't need it

Both processes share the same auth.py module and the same SQLite database, so user/permission changes on 8001 take effect immediately on 8000.

Project Layout

pst-indexer/
├── Dockerfile                 # Python 3.11 + build-essential + libpff-python
├── docker-compose.yml         # One service, two ports (8000 + 8001), one volume
├── entrypoint.sh              # Starts both uvicorn processes; kills both if one dies
├── .dockerignore              # Keeps ./data out of the build context
├── README.md                  # This file
│
├── backend/
│   ├── app.py                 # Main app: setup, login, PST upload/search/view
│   ├── admin_app.py           # Admin app: user CRUD, MFA management
│   ├── auth.py                # Shared: hashing, encryption, TOTP, JWT, user DB
│   └── requirements.txt       # Pinned Python deps
│
├── frontend/                  # Served by main app on port 8000
│   ├── index.html             # Setup wizard / login / main UI
│   ├── styles.css             # Editorial/archival aesthetic
│   └── app.js                 # Auth flow + archive browser
│
└── admin_frontend/            # Served by admin app on port 8001
    ├── index.html             # Admin login / user table / modals
    ├── admin.css              # Darker, more utilitarian theme
    └── admin.js               # User CRUD + MFA enrollment flows

Data Model

`users`

Column	Type	Notes
`id`	TEXT PK	Random hex, 32 chars
`username_hash`	TEXT UNIQUE	HMAC-SHA256 of normalized username — used for login lookup
`username_enc`	TEXT	Fernet ciphertext of plaintext username
`password_hash_enc`	TEXT	Fernet ciphertext of Argon2id hash
`totp_secret_enc`	TEXT	Fernet ciphertext of TOTP secret (NULL if not enrolled)
`mfa_enabled`	INTEGER	1 if user must provide TOTP at login, 0 otherwise
`role`	TEXT	`user` or `admin`
`created_at`	TEXT	ISO 8601 UTC
`last_login`	TEXT	ISO 8601 UTC (NULL if never logged in)

`pst_files`

Column	Type	Notes
`id`	TEXT PK	UUID hex
`filename`	TEXT	Stored filename on disk
`original_name`	TEXT	User-facing filename
`uploaded_at`	TEXT	ISO 8601
`size_bytes`	INTEGER
`status`	TEXT	`indexing` / `ready` / `failed`
`message_count`	INTEGER	Populated when indexing finishes
`error`	TEXT	Populated only if `status = 'failed'`
`uploaded_by`	TEXT	FK to `users.id`
`folder_id`	TEXT	FK to `pst_folders.id`, or NULL for Uncategorized

Indexes: idx_pst_files_folder on folder_id.

`pst_folders`

Column	Type	Notes
`id`	TEXT PK	UUID hex
`name`	TEXT	Unique case-insensitively; max 80 chars
`created_at`	TEXT	ISO 8601
`created_by`	TEXT	FK to `users.id`

Deleting a folder row triggers UPDATE pst_files SET folder_id = NULL WHERE folder_id = ? as a manual cascade-replacement (done in the route handler, not a SQL trigger) so that files are always preserved.

`folder_permissions`

Column	Type	Notes
`user_id`	TEXT PK (part 1)	FK to `users.id`
`folder_id`	TEXT PK (part 2)	FK to `pst_folders.id`
`granted_at`	TEXT	ISO 8601 of when access was granted
`granted_by`	TEXT	FK to `users.id` of the admin who issued the grant

Composite primary key prevents duplicate grants. Indexes on both sides (idx_folder_perms_user, idx_folder_perms_folder) for fast lookups in either direction. The route handlers manually cascade deletes: when a user is deleted their rows here are removed, and when a folder is deleted all grants targeting it are removed.

`messages`, `messages_fts`

Same as the pre-auth version — see inline schema in app.py. Cascade-deletes with pst_files.

REST API Reference

All API endpoints return JSON. Session cookies are required except where noted.

Main app (port 8000)

Public (no auth)

GET /api/health — {status: "ok"}
GET /api/setup/status — {needs_setup: bool}
POST /api/setup {username, password} — only works when needs_setup is true. Creates the first admin and signs them in.
POST /api/login {username, password, totp_code?} — on MFA-required accounts, returns 401 {mfa_required: true} if totp_code is missing
POST /api/logout — clears the session cookie

Authenticated

GET /api/me — {user: {...}}
GET /api/pst-files — list archives (includes folder_id per archive)
POST /api/pst-files?folder_id={id} (multipart file) — upload; optional folder_id query parameter places the archive directly into a folder
DELETE /api/pst-files/{id} — delete archive + messages + raw file
PUT /api/pst-files/{id}/folder {folder_id: string | null} — move archive to a folder (or null for Uncategorized)
GET /api/folders — list all folders with file_count
POST /api/folders {name} — create a folder
PUT /api/folders/{id} {name} — rename a folder
DELETE /api/folders/{id} — delete a folder (PSTs inside become Uncategorized, not deleted)
GET /api/pst-files/{id}/folders — list folders within a PST (Inbox, Sent Items, etc.) with message counts
GET /api/pst-files/{id}/messages?q=&folder=&limit=&offset= — search
GET /api/messages/{id} — full message

Admin panel (port 8001)

Public

GET /api/health — {status, admins, users}
GET /api/setup/status
POST /api/login — same as main, but also rejects non-admin users with 403
POST /api/logout

Admin-only

GET /api/me
GET /api/users — list all users
POST /api/users {username, password, role} — create
DELETE /api/users/{id} — delete (cannot be self or last admin)
PUT /api/users/{id}/role {role} — change role
PUT /api/users/{id}/password {password} — reset password
POST /api/users/{id}/mfa/begin — generate new TOTP secret. Returns {secret, otpauth_uri, qr_url}
GET /api/users/{id}/mfa/qr — PNG QR code of the current enrollment URI
POST /api/users/{id}/mfa/confirm {totp_code} — verify and enable
POST /api/users/{id}/mfa/disable
GET /api/folders — list all folders with counts
POST /api/folders {name} — create
PUT /api/folders/{id} {name} — rename
DELETE /api/folders/{id} — delete (contents become Uncategorized)
GET /api/pst-files — system-wide list of every PST with joined folder_name and uploaded_by
DELETE /api/pst-files/{id} — force-delete any PST regardless of uploader
PUT /api/pst-files/{id}/folder {folder_id: string | null} — reassign
GET /api/permissions — full matrix: {users: [...], folders: [...], grants: [{user_id, folder_id, granted_at}, ...]}
GET /api/users/{user_id}/folders — folders a specific user can access
GET /api/folders/{folder_id}/users — users granted on a specific folder
PUT /api/permissions/{user_id}/{folder_id} {granted: bool} — grant (true) or revoke (false) a single permission. Idempotent: granting an already-granted permission is a no-op; revoking a non-existent grant is also a no-op. Admin targets are rejected with 400 (admins have implicit access).

Frontend Interfaces

Main app (`http://localhost:8000`)

Three screens, switched in-place:

Setup wizard (first run only) — username, password, confirm password
Login — username, password, plus an MFA code field that appears after a 401 with mfa_required
Main workspace — masthead with user chip and logout button, sidebar with upload + archive list, reader pane with search + folder filter + message list + full-message viewer + paginator

Aesthetic: warm off-white paper, deep ink, oxblood accent, Fraunces serif, JetBrains Mono for technical labels, grain overlay.

Admin panel (`http://localhost:8001`)

First-run notice (if no admin exists yet) — redirects to the main app to complete setup
Login — same as main, plus a red "ADMIN" badge
User management — table of all users with inline role pills, MFA indicators, and per-row action buttons. Modals for create, reset password, and MFA enrollment (with QR code rendered server-side as PNG)

Aesthetic: same type system as the main app but a deeper, slightly more clinical palette (duskier paper, darker ink, more saturated oxblood). The "ADMIN" badge and red-accented modals make the control surface feel unmistakably different from the main app.

Search Syntax

Same as the pre-auth version:

Whitespace separates terms
Each term is a prefix match (pay → payment)
Porter stemming (run → running)
Unicode-aware
Not case-sensitive
Looks in subject, sender, recipients, body, and folder simultaneously

Configuration

Environment variables (set in docker-compose.yml):

Variable	Default	Purpose
`DATA_DIR`	`/app/data`	Where uploads, DB, and secret key live
`FRONTEND_DIR`	`/app/frontend`	Static files for main app
`ADMIN_FRONTEND_DIR`	`/app/admin_frontend`	Static files for admin panel
`SESSION_TTL_SECONDS`	`28800` (8 h)	JWT session lifetime
`COOKIE_SECURE`	`false`	Set to `true` when serving over HTTPS
`MASTER_SECRET`	(unset)	If provided, replaces the auto-generated master key

Port mapping

Both ports are independently mappable:

ports:
  - "9000:8000"    # main on host port 9000
  - "9001:8001"    # admin on host port 9001

Admin-only-on-localhost

ports:
  - "8000:8000"
  - "127.0.0.1:8001:8001"   # admin accessible only on the host itself

Persistence & Backup

Everything stateful lives in ./data:

./data/
├── pst_index.db              # SQLite with users + messages + pst_files
├── secret.key                # Master key (chmod 600) — ONLY if MASTER_SECRET is unset
└── uploads/
    └── {uuid}_archive.pst    # Raw PST files

Critical backup rules

Always back up secret.key alongside pst_index.db. Without the key, encrypted fields are unrecoverable.
If you're using MASTER_SECRET, back up the key material wherever you store it (Vault, 1Password, etc.).
The database and the key together are sensitive — treat backups with the same care as the originals.

Backup

docker compose stop
tar czf pst-indexer-backup-$(date +%F).tar.gz ./data
docker compose start

Hot backup

docker compose exec pst-indexer \
  sqlite3 /app/data/pst_index.db ".backup /app/data/backup.db"

Restore

docker compose down
rm -rf ./data
tar xzf pst-indexer-backup-2026-04-23.tar.gz
docker compose up -d

Troubleshooting

Setup wizard doesn't appear

Either users already exist (check GET /api/setup/status) or the frontend cached an old session. Clear cookies for localhost:8000 and reload.

Your authenticator's clock is out of sync. TOTP tolerates ±30 seconds of drift. Re-sync your phone's clock (it should be automatic) and try again. If an admin can help, they can disable MFA for you in the admin panel, then re-enroll.

"Cannot delete the last remaining admin"

Correct — this is an intentional invariant. Create a second admin first, then delete or demote the original.

Lost master key

If you used the auto-generated secret.key and lost it (and didn't back it up), all user records are unrecoverable. You can still keep indexed PST content, but you'll need to wipe the users table and re-run setup:

docker compose exec pst-indexer sqlite3 /app/data/pst_index.db "DELETE FROM users"
docker compose restart pst-indexer

Then visit the main app — it'll show the setup wizard again.

Rate limiter triggered. Wait 5 minutes. If you need immediate access, restart the container (the rate-limit state is in-memory).

You're logging in with a non-admin account. Port 8001 requires role = admin.

Admin panel says "Setup Required"

No admin exists yet. Complete the setup wizard on port 8000 first.

I want to rotate the master key

No automated path. You'd need to: export all users with their decrypted fields, change MASTER_SECRET, restart the app (which would make the existing encrypted fields unreadable), then re-import the users. Script this yourself; there's no built-in command.

Performance Notes

Indexing speed and storage overhead are unchanged from the pre-auth version:

Indexing: 2,000–8,000 messages/minute
Search: sub-millisecond FTS5 lookups even on millions of messages
Storage overhead: DB is roughly 1.5–3× the text content of your PSTs

Auth adds negligible overhead:

Login: ~100–300 ms due to Argon2 (by design — slow to verify is slow to brute-force)
Per-request auth check: ~0.1 ms (JWT decode + one SQLite lookup)
MFA verification: ~1 ms

Security Considerations

What this design protects against

Database theft — even if an attacker exfiltrates pst_index.db, they cannot read usernames, recover passwords, or generate valid TOTP codes without also having the master key
Offline brute force — passwords are Argon2id-hashed before encryption, so even a stolen DB + stolen master key requires Argon2 brute-force which is compute-hard by design
Credential stuffing — rate limiting slows it significantly
XSS session theft — session cookies are HttpOnly; JavaScript cannot read them
CSRF on state-changing endpoints — SameSite=Lax blocks cross-site POSTs; JSON-only request bodies mean HTML forms can't initiate most endpoints
Privilege escalation — last-admin and self-demote protections are server-side

What this design does NOT protect against

Live attacker with shell access to the container — everything the app can decrypt, they can decrypt
Full filesystem theft (DB + secret.key together) — falls back to Argon2 brute-force on each password, which is slow but not impossible for weak passwords
Phishing — no defence here; users still need to recognise the real URL
MITM — use HTTPS. The app doesn't do TLS itself; put it behind a reverse proxy
Malicious admin — admins can reset any password, disable any MFA, and create arbitrary users. Trust your admins, and keep the admin count small.
Content of indexed emails — stored as plaintext in SQLite for FTS to work. Use full-disk encryption or LUKS on the host if emails are highly sensitive.

If you need to expose this to the internet

Put it behind HTTPS. Caddy, Traefik, or Nginx + certbot. Set COOKIE_SECURE=true.
Use a strong MASTER_SECRET from a secret manager, not the auto-generated one.
Firewall port 8001 to management IPs only, or put it behind a VPN.
Require MFA on all admin accounts.
Watch the logs. FastAPI logs every request including 401s. Feed them to your SIEM.
Back up regularly, including the key material.

Extending the App

Add API tokens for automation

Add a api_tokens table with (id, user_id, token_hash, created_at, last_used). Accept Authorization: Bearer <token> in the existing _extract_token helper, look up by HMAC of the token, and return the associated user.

Single sign-on (OIDC/SAML)

Use authlib to delegate auth to your IdP. Leave the local login as a fallback. The user table's role field still determines admin access.

Email content encryption

Encrypt body at write time with a per-PST symmetric key. Requires rebuilding the FTS index to work on ciphertext (hard — SQLite FTS doesn't natively support encrypted content). A simpler alternative: rely on full-disk encryption at the host level.

Audit log

Add an audit_log table that records every user creation, deletion, role change, password reset, MFA enrollment, and login (success + failure). Expose a view in the admin panel.

Backup restore UI

Expose a button in the admin panel that triggers sqlite3 .backup and streams the resulting file to the browser for download.

Self-service MFA

Currently MFA is admin-managed. To let users self-enroll, add /api/me/mfa/* endpoints on the main app (port 8000) that call the same functions in auth.py but restrict the target user ID to current_user.id.

License & Acknowledgements

This project is provided as-is, for you to use, modify, and distribute freely.

Built on:

libpff and libpff-python — the C library that makes reading PST files possible
FastAPI — Python web framework
argon2-cffi — password hashing
cryptography — Fernet, HKDF
pyotp — TOTP implementation
PyJWT — JWT encoding/decoding
qrcode — QR code generation
SQLite FTS5 — embedded full-text search
Fraunces and JetBrains Mono — typefaces

PST/OST is a proprietary Microsoft format. libpff is a clean-room implementation based on public documentation; no Microsoft code is used in this project.

53 KiB Raw Blame History Unescape Escape