# PST Archive — Mail Indexer

A self-hosted, Dockerized application for **uploading Microsoft Outlook PST/OST files**, **indexing every email inside them**, and **searching the contents through a web browser** — with full **user authentication**, **multi-factor authentication (MFA)**, **encrypted-at-rest credentials**, and a **separate admin panel** on its own port.

Designed for **forensic review, e-discovery, personal archive browsing, and migration audits** — any situation where you need to crack open a PST and actually find things inside it without loading it into Outlook.

---

## Table of Contents

1. [Features](#features)
2. [Quick Start](#quick-start)
3. [First-Run Setup](#first-run-setup)
4. [Running as a Daemon](#running-as-a-daemon)
5. [Theming & Dark Mode](#theming--dark-mode)
6. [Authentication & Security Model](#authentication--security-model)
7. [Admin Panel](#admin-panel)
8. [Organizing PSTs with Folders](#organizing-psts-with-folders)
9. [Folder Access Permissions](#folder-access-permissions)
10. [Multi-Factor Authentication (MFA)](#multi-factor-authentication-mfa)
11. [Encryption at Rest](#encryption-at-rest)
12. [How It Works](#how-it-works)
13. [Architecture](#architecture)
14. [Project Layout](#project-layout)
15. [Data Model](#data-model)
16. [REST API Reference](#rest-api-reference)
17. [Frontend Interfaces](#frontend-interfaces)
18. [Search Syntax](#search-syntax)
19. [Configuration](#configuration)
20. [Persistence & Backup](#persistence--backup)
21. [Troubleshooting](#troubleshooting)
22. [Performance Notes](#performance-notes)
23. [Security Considerations](#security-considerations)
24. [Extending the App](#extending-the-app)

---

## Features

### Core
- **Drag-and-drop upload** of `.pst` and `.ost` files through the browser
- **Streaming upload with progress bar** — handles multi-gigabyte archives
- **Background indexing** into SQLite with **FTS5 full-text search** (ranked, prefix-matched, with highlighted snippets)
- **Multi-archive support**: switch between uploaded PSTs; each is searched independently
- **Folder organization**: create folders in the sidebar to group related PST files (e.g. by client, case, year). Move archives between folders, rename folders, or delete folders (any archives inside are safely moved to Uncategorized)
- **Per-folder access permissions**: admins can restrict which non-admin users can view each folder. A single user can be granted access to many folders; a single folder can be shared with many users (true many-to-many)
- **Folder filter**: narrow search to any folder within the archive (Inbox, Sent Items, nested folders)
- **Full message viewer**: subject, sender, recipients, date, folder, attachments, body
- **Persistent storage**: everything lives in a mounted `./data` volume

### Security
- **First-run setup wizard** forces creation of an initial administrator before any other action is possible
- **Password authentication** using **Argon2id** (winner of the Password Hashing Competition) with constant-time verification
- **TOTP-based MFA** (RFC 6238; compatible with Google Authenticator, Authy, 1Password, Microsoft Authenticator, and every other standard authenticator app)
- **Encryption at rest** for all user PII — usernames, password hashes, and TOTP secrets are stored as Fernet (AES-128-CBC + HMAC-SHA256) ciphertexts
- **Master key** auto-generated on first run and stored with `chmod 600`, or overridable via `MASTER_SECRET` environment variable
- **HMAC-based username lookup** — usernames are never stored in plaintext but are still indexable for login
- **JWT session cookies**, signed with an HKDF-derived sub-key, HttpOnly, SameSite=Lax
- **Rate limiting** on login attempts (10 failures per 5 minutes per IP+username)
- **Admin-only user management**: create, delete, role-change, password-reset, MFA-enroll other users
- **Admin file & folder management**: system-wide view of every uploaded PST and every folder, with create/rename/delete for folders and force-delete for any file regardless of uploader
- **Separate admin panel on a separate port** — lockable behind its own firewall rule or VPN

### UI
- **Light and dark themes** on both the main app and the admin panel, with a sun/moon toggle in the masthead and on login/setup screens
- Preference persists in `localStorage` per-browser and respects the operating system's `prefers-color-scheme` on first visit
- Theme applied inline in `<head>` before stylesheets load — **no flash of wrong theme** on page load
- Dark mode uses a custom "inked paper" palette (deep charcoal with warm paper-cream text and softened oxblood accent) — intentional character, not generic-dark
- Editorial typography (Fraunces serif + JetBrains Mono), grain overlay, responsive layout down to mobile widths

---

## Quick Start

### Prerequisites

- Docker 20.10 or newer
- Docker Compose v2
- A `.pst` or `.ost` file to index

### Run (as a daemon — recommended)

```bash
docker compose up -d --build
```

The `-d` flag runs the container in the background (detached). On the first run this takes **3–5 minutes** because `libpff-python` compiles a C extension from source; subsequent startups are nearly instant.

The container is configured with `restart: unless-stopped`, so it will automatically come back up after a reboot or a crash, and keep running across Docker daemon restarts — until you explicitly stop it.

Once the container is up:

- **Main app**: `http://localhost:8000`
- **Admin panel**: `http://localhost:8001`

### Foreground mode (useful for debugging)

```bash
docker compose up --build
```

This runs the container attached to your terminal so you can see logs inline. Press `Ctrl-C` to stop.

### Tail the logs

```bash
docker compose logs -f              # follow both streams
docker compose logs -f pst-indexer  # explicit service name
```

### Check status

```bash
docker compose ps                   # is it running?
docker compose top                  # which processes?
```

### Stop

```bash
docker compose stop        # stop, keep data, keep container
docker compose down        # stop and remove the container, keep data volume
```

### Full reset

```bash
docker compose down
rm -rf ./data              # erases everything including users and the master key
docker compose up -d --build
```

### Restart / update

```bash
# After editing code or pulling new images:
docker compose up -d --build       # rebuild & restart with one command
# Or just restart without rebuilding:
docker compose restart
```

---

## First-Run Setup

On the very first visit to `http://localhost:8000`, you will be presented with a **setup wizard** that asks you to create the initial administrator:

1. Choose a username (3–64 characters)
2. Choose a password (8+ characters — use a passphrase you'll remember)
3. Confirm the password

The setup endpoint is only callable **while the users table is empty**. Once the first admin is created, the endpoint returns `409 Conflict` and is effectively disabled.

After setup, you are automatically signed in and landed on the main app.

**Heads up:** the admin panel on port 8001 will refuse to accept any logins until the first admin has been created via the main app's setup wizard. This prevents a race where someone could hit `/api/login` on the admin port before you've set things up.

---

## Running as a Daemon

By default, this app is designed to **run as a long-lived background service**. The `docker-compose.yml` sets `restart: unless-stopped`, which means once started, the container will:

- Keep running after you close your terminal
- Automatically restart if the process crashes
- Automatically restart when the host reboots and Docker comes back up
- **Not** restart if you explicitly stop it with `docker compose stop` (hence "unless-stopped")

### Starting the daemon

```bash
docker compose up -d --build
```

The `-d` flag is short for "detached" — the container is launched in the background and the command returns immediately. Your terminal is free to do other things; the container keeps running.

### Verifying it's running

```bash
docker compose ps
```

You should see output like:

```
NAME          IMAGE                       STATUS                   PORTS
pst-indexer   pst-indexer-pst-indexer     Up 2 minutes (healthy)   0.0.0.0:8000->8000/tcp, 0.0.0.0:8001->8001/tcp
```

The `(healthy)` flag comes from the built-in healthcheck, which probes both `/api/health` endpoints every 30 seconds.

### Watching logs

Since the container is detached, you won't see logs in your terminal by default. Attach to them on demand:

```bash
docker compose logs                 # print recent logs and exit
docker compose logs -f              # follow (live tail), Ctrl-C to detach
docker compose logs -f --tail=100   # follow, starting from the last 100 lines
docker compose logs --since 1h      # only logs from the last hour
```

Detaching from `-f` does **not** stop the container — it just stops showing you the logs.

### Stopping

```bash
docker compose stop          # stop the container but keep it around
docker compose start         # start it again if previously stopped
docker compose restart       # stop and start in one command
docker compose down          # stop and remove the container (data volume is kept)
docker compose down -v       # stop, remove container, AND remove anonymous volumes
```

None of these delete your `./data` directory — that's a bind mount, not a Docker-managed volume, so it's always yours.

### Survives reboots

Because `restart: unless-stopped` is set, after a host reboot Docker will automatically start the container as soon as the Docker daemon comes up. Nothing manual required on your part.

If you ever explicitly stopped it with `docker compose stop`, Docker will *not* auto-start it after reboot (that's the whole point of "unless-stopped"). Run `docker compose start` or `docker compose up -d` to bring it back.

### Updating to a new version

```bash
# Pull new source, then:
docker compose up -d --build
```

Docker Compose compares the current image to the built one; if they differ, it rebuilds and recreates the container with zero-downtime-style replacement. Your `./data` volume is preserved throughout.

---

## Theming & Dark Mode

Both the main app and the admin panel support **light and dark themes**, with a sun/moon toggle visible in the masthead (when signed in) and floating in the top-right corner on the login/setup screens.

### How the toggle works

- Click the icon to flip between themes
- Choice is saved to `localStorage["pst-theme"]` in your browser
- On a fresh browser (nothing saved), the app respects your operating system's `prefers-color-scheme` setting
- The theme is applied by an inline `<script>` in the HTML `<head>`, **before the CSS loads** — so there is no flash of the wrong theme on page load

### Palettes

**Light** (the default): warm off-white paper `#f2ede3`, deep ink `#1b1814`, oxblood accent `#a8341f`. Editorial and understated — the original aesthetic.

**Dark**: "inked paper" `#17140f`, warm paper-cream text `#e9e0cb`, softened oxblood `#d06347`, brass-yellow highlight `#d8b64a`. Intentionally avoids the generic "cold dark mode" look — it feels like reading inked paper under a warm lamp rather than staring into a cold screen.

The admin panel uses a slightly more saturated variant of the same palette for both themes, to preserve its more utilitarian character.

### Implementation note for developers

Both stylesheets use CSS custom properties keyed off a `data-theme` attribute on `<html>`:

```css
:root, :root[data-theme="light"] { --paper: #f2ede3; ... }
:root[data-theme="dark"]         { --paper: #17140f; ... }
```

All component colors reference these variables, and tinted effects (hover backgrounds, gradient glows, shadow colors) use `color-mix()` so they derive from the same tokens and automatically adapt. Adding a third theme would mean adding one more `[data-theme="..."]` block.

---

## Authentication & Security Model

### Two session scopes

The system has **two independent session scopes** with separate cookies:

| Scope | Port | Cookie | Required Role |
|---|---|---|---|
| `app` | 8000 | `pst_session` | any authenticated user |
| `admin` | 8001 | `pst_admin_session` | must have `role = admin` |

A session token issued for one scope **will not work** for the other. Logging in on one port does not sign you in on the other — they're entirely separate authentications. This is deliberate: the admin panel deserves its own credential entry even if you're already signed in to the main app.

### Roles

- `user` — can sign into the main app at port 8000, upload PSTs, search, and view messages
- `admin` — everything above, plus access to the admin panel at port 8001 to manage users

The first user created (via the setup wizard) is automatically an admin.

### Password storage

Passwords are hashed with **Argon2id** before anything else happens:

- 64 MiB memory cost
- 3 iterations
- Parallelism of 2

The resulting Argon2 hash is then **additionally encrypted with Fernet** before being written to the database. This means even if an attacker dumps the SQLite file, they must also have the master key to *start* an offline brute-force attack against Argon2 — which is itself designed to be very slow to crack.

Verification is constant-time (handled by the `argon2-cffi` library).

### Login flow

1. Client POSTs `{username, password}` (and optionally `totp_code`) to `/api/login`
2. Server normalises the username (trim + lowercase), computes `HMAC-SHA256(username-lookup-key, normalized)`, and queries by that hash
3. If found, decrypts the stored Argon2 hash using the master key's Fernet sub-key
4. Calls `argon2.verify(hash, password)` — constant time
5. If MFA is enabled on the account, requires a valid 6-digit TOTP code
6. On success, mints a JWT containing `{sub: user_id, role, scope, iat, exp}` signed with an HKDF-derived JWT key
7. Sets the appropriate HttpOnly cookie on the response

### Rate limiting

Failed login attempts are tracked per `(ip, username)` tuple in memory. After 10 failures within 5 minutes, login returns `429 Too Many Requests` for that specific combination until the window expires. Successful logins reset the counter.

This is **per-process in-memory** — if you scale to multiple worker processes or containers, you'd want to replace it with Redis. For single-container use it's fine.

### Cookies

All session cookies are:
- `HttpOnly` — not readable from JavaScript
- `SameSite=Lax` — submitted on top-level navigation but blocked from cross-site POSTs
- `Secure` — enabled when `COOKIE_SECURE=true` (set this when serving over HTTPS)
- Path-limited to `/`

### Sessions

Session JWTs have a default lifetime of **8 hours** (configurable via `SESSION_TTL_SECONDS`). The session token is opaque to the client — the client just presents it as a cookie and the server validates & extracts the user ID.

There is no refresh-token flow. When the session expires, the user signs in again.

---

## Admin Panel

Available at **`http://localhost:8001`**.

The admin panel has four tabs: **Users**, **Folders**, **Files**, and **Permissions**.

### Users tab

- **List all users** with username, role, MFA status, created date, last login
- **Create new users** — specify username, password, and role
- **Delete users** (cannot delete yourself; cannot delete the last remaining admin)
- **Toggle role** between `user` and `admin` (cannot demote yourself; cannot demote the last admin)
- **Reset password** for any user
- **Manage MFA** for any user: begin enrollment (QR code + secret), confirm enrollment with a test code, disable, or re-enroll

### Folders tab

- **List all folders** with their name, the count of PSTs inside, and created date
- **Create new folders** via a modal; folder names are case-insensitively unique (you can't have both "Clients" and "clients")
- **Rename folders** (same uniqueness rule)
- **Delete folders** — archives inside are **not** deleted; they're moved to "Uncategorized"

Folders are a system-wide, flat (non-nested) organizational feature. Any authenticated user can see and use folders; admins get the dedicated management tab. Regular users can also create and delete folders from the sidebar of the main app — folders are shared across all users by design.

### Files tab

- **System-wide view** of every PST uploaded by every user, with folder assignment, size, indexing status, message count, and upload date
- **Force-delete any PST** regardless of uploader — removes the raw `.pst` file from disk, all indexed messages, and the database record
- Useful for cleaning up stalled/failed indexes or reclaiming disk space

### Permissions tab

- **Matrix view** of every user against every folder, with check/uncheck toggles
- Two perspectives you can switch between: **By user** (rows = users, cols = folders) and **By folder** (rows = folders, cols = non-admin users)
- Toggling is instant — no save button, each change calls the API immediately with optimistic UI updates
- Admin rows show as read-only implicit grants (every column checked with a neutral ✓) — admins always have access to every folder regardless of these settings
- See the dedicated ["Folder Access Permissions"](#folder-access-permissions) section below for the full access model.

### Safety rails

The admin API enforces these invariants server-side:

- You cannot delete your own account
- You cannot demote your own admin role to `user`
- The system must always have at least one admin — the last admin cannot be deleted or demoted
- Deleting a folder never cascade-deletes PSTs — archives are always safely reassigned
- Passwords must be at least 8 characters
- Usernames must be 3–64 characters; folder names max 80 characters

These are enforced in the backend, not just the UI — so a malicious admin with a modified client cannot bypass them.

---

## Organizing PSTs with Folders

Uploaded archives can be grouped into folders to keep the sidebar navigable when you've accumulated dozens of PSTs.

### In the main app (sidebar)

- **"+ New folder"** button in the Archives section header creates a new folder
- Each folder is a **collapsible group** — click the header to expand/collapse
- Each folder header has a **"⋯" menu** → Rename / Delete
- Each PST has its own **"⋯" menu** → Move to folder… / Delete archive
- PSTs that aren't assigned to any folder sit in an **Uncategorized** group at the bottom

### In the admin panel

- The **Folders tab** is the source of truth: create, rename, delete folders across the system
- The **Files tab** shows every PST and the folder it belongs to, with force-delete

### Behavior on folder delete

When a folder is deleted (from either the main app or admin panel):

- The folder record is removed
- All PSTs that were inside it get `folder_id = NULL` (they become Uncategorized)
- **Indexed messages are not touched** — the archives remain fully searchable

This is deliberate: folders are an organizational convenience, not a security boundary. Accidentally deleting a folder should never cost you data.

### Notes

- Folder names are case-insensitively unique
- Max 80 characters per folder name
- No nesting — folders are a single flat level. If you need hierarchy, encode it in the name (e.g. "Clients / ACME", "Clients / Globex")
- Folders are system-wide, not per-user: every authenticated user sees the same folders

---

## Folder Access Permissions

By default, **non-admin users cannot see any folder** except the shared Uncategorized bucket. Admins grant specific non-admin users access to specific folders via the admin panel. This creates true multi-tenant isolation inside a single shared deployment.

### Access rules

| Role | Sees |
|---|---|
| **Admin** | Every folder, every PST, every message — always |
| **Non-admin (no grants)** | Only PSTs in Uncategorized |
| **Non-admin (with grants)** | Only the folders they've been granted + Uncategorized |

Uncategorized is deliberately shared across everyone — it catches freshly-uploaded-but-not-yet-sorted PSTs so they don't become invisible to the person who uploaded them.

### Many-to-many by design

The permission model is a true matrix:

- **One user → many folders**: grant Alice access to "Clients", "Legal", and "HR" independently
- **One folder → many users**: grant both Alice and Bob access to "Legal" at the same time
- **Flexible combinations**: Alice in Clients+Legal, Bob in Legal+HR, Charlie in HR alone — any shape works

### Managing permissions

Open the admin panel at `http://localhost:8001`, sign in as an admin, and click the **Permissions** tab. A matrix view appears with two perspectives:

- **By user** (default): rows are users, columns are folders. Check a box to grant that user access to that folder.
- **By folder**: rows are folders, columns are non-admin users. Same effect, different axis — use whichever feels more natural for the task at hand.

Toggling a checkbox calls `PUT /api/permissions/{user_id}/{folder_id}` and applies instantly. Admin rows in the matrix are displayed as read-only implicit grants (every cell shows ✓) with a visual distinction so you don't confuse yourself about whether admins have "been granted" anything — they don't need to be.

### Backend enforcement

Every data-access endpoint on the main app (port 8000) checks permissions server-side:

- `GET /api/folders` returns only folders the user can see
- `GET /api/pst-files` returns only PSTs in those folders + Uncategorized
- `GET /api/pst-files/{id}/messages` returns `404` if the user can't see that PST's folder (same shape as a genuinely nonexistent PST — no information leakage)
- `GET /api/messages/{id}` checks the parent PST's folder
- `POST /api/pst-files?folder_id=X` verifies the uploader has access to folder X
- `PUT /api/pst-files/{id}/folder` verifies access to both current and destination folders
- `POST/PUT/DELETE /api/folders*` are **admin-only** (non-admins get `403`)

A modified UI cannot bypass these — they're all dependency-injection checks inside the FastAPI route handlers.

### Cascading deletes

- **User deleted** → all their permission grants are removed (`auth.delete_user` handles this in the same transaction)
- **Folder deleted** → all grants targeting that folder are removed, and any PSTs inside are moved to Uncategorized (not cascade-deleted — see "Behavior on folder delete" above)

### Example workflow

You're running the app for a small law firm. You have three admins (the partners) and ten non-admin users (associates and assistants).

1. Admin logs into port 8001, goes to **Folders** tab, creates folders: "Matter 2024-001", "Matter 2024-002", "Firm Admin"
2. Admin uploads the relevant PSTs from the main app on port 8000, filing each into its matter folder
3. Admin goes to **Permissions** tab, grants associate Alice access to "Matter 2024-001" and the paralegal Bob access to both matters
4. Alice signs in to port 8000 — her sidebar shows only "Matter 2024-001" plus any Uncategorized uploads. She searches, views messages, moves PSTs between folders she has access to
5. Bob signs in — sees both matters. If he tries the admin panel on port 8001 he gets `403` because he's not an admin

---

## Multi-Factor Authentication (MFA)

MFA is **per-user** and **optional by default**. An admin enables it for each user individually via the admin panel.

### How enrollment works

1. **Admin opens the MFA modal** for a user (in the admin panel)
2. Clicks **"Begin enrollment"** — the server generates a new random base32 TOTP secret and stores it encrypted (`mfa_enabled` stays `0` until the user proves they have it)
3. The UI shows:
   - A **QR code** containing the `otpauth://` URI
   - The **raw secret** as text (for manual entry into apps that don't support QR)
   - A field to enter a **6-digit verification code**
4. The user scans the QR (or types the secret) into their authenticator app
5. They read the 6-digit code from the app and the admin enters it
6. Server verifies the code. If valid, sets `mfa_enabled = 1`. If invalid, enrollment stays unconfirmed (the secret is kept so you can retry without re-scanning)

### How login with MFA works

1. User submits username + password to `/api/login` (no code yet)
2. Server verifies credentials and sees `mfa_enabled = 1`
3. Server responds with `401 {"detail": "MFA required", "mfa_required": true}` — this is a signal to the client, not an error
4. Client reveals the "Authentication code" field and prompts for the code
5. User enters the 6-digit code; client re-submits the full form (`username + password + totp_code`)
6. Server re-verifies credentials, then verifies the code against `pyotp.TOTP(secret).verify(code, valid_window=1)` (±30 seconds tolerance)
7. On success, issues the session cookie

### Supported authenticator apps

Any RFC 6238-compliant TOTP app works:

- Google Authenticator
- Microsoft Authenticator
- Authy
- 1Password
- Bitwarden
- Aegis (Android)
- Raivo OTP (iOS)
- KeePassXC with TOTP plugin
- Any `otpauth://` URI consumer

### Disabling MFA

An admin can disable MFA for any user. This clears both the `totp_secret` and the `mfa_enabled` flag. The next time they log in, they will not be prompted for a code.

---

## Encryption at Rest

Everything sensitive in the `users` table is encrypted before being written to disk. The plaintext is only ever held in memory during request processing.

### What's encrypted

| Field | How |
|---|---|
| Username | Fernet ciphertext |
| Password hash (Argon2) | Fernet ciphertext (double-protection: Argon2 + Fernet) |
| TOTP secret | Fernet ciphertext |

### What's *not* encrypted (and why)

| Field | Reason |
|---|---|
| `id` | Random UUIDs, no PII |
| `username_hash` | HMAC-SHA256 — one-way, non-reversible. Only used to find a user at login. |
| `role` | Not PII; needed for fast query filtering |
| `mfa_enabled` | Not sensitive; just a boolean flag |
| `created_at`, `last_login` | Timestamps; not uniquely identifying on their own |

**Email content inside indexed PSTs** is *not* currently encrypted at the application layer. If you need that, use full-disk encryption on the host or a LUKS volume mounted as `./data`. The architecture specifically isolates user credentials (which are small and always decrypted on every login) from message content (which is large and streamed through full-text search).

### Master key handling

The system uses **one master secret** from which all encryption keys are derived via HKDF-SHA256 with distinct `info` parameters:

- `fernet-user-pii` — the Fernet key for encrypting username, password hash, TOTP secret
- `jwt-sessions` — the HMAC key for signing session JWTs
- `username-lookup` — the HMAC key for deterministic username lookup hashes

The master secret itself is either:

**Option A (default)**: auto-generated on first run using `secrets.token_bytes(32)` and written to `./data/secret.key` with `chmod 600`. Convenient, but anyone with filesystem access to `./data` can read the key.

**Option B (recommended for production)**: supplied via the `MASTER_SECRET` environment variable. The provided string is SHA-256-hashed to derive the 32-byte master key. Store this in a secret manager (Docker secrets, Kubernetes secrets, Vault, AWS Secrets Manager, 1Password CLI, etc.) and inject it at runtime:

```yaml
environment:
  - MASTER_SECRET=${PST_MASTER_SECRET}
```

If `MASTER_SECRET` is set, the on-disk `secret.key` is ignored.

### Key rotation

Changing the master key **invalidates all encrypted data**. There is currently no automated rotation path — you'd need to write a migration that decrypts with the old key and re-encrypts with the new one. For most single-operator deployments, you just don't rotate; for regulated environments, plan a rotation procedure.

If the master key is ever lost, the encrypted data is unrecoverable. **Back it up separately from the database** if you're using Option A.

---

## How It Works

The end-to-end lifecycle of a PST file, from upload to searchable results:

### 1. Authentication

Every API call (except `/api/health`, `/api/setup`, `/api/setup/status`, and `/api/login`) requires a valid session cookie. The FastAPI dependency `require_app_user` decodes the JWT, verifies the signature and scope, looks up the user, and raises `401` if anything fails.

### 2. Upload

The browser POSTs the PST as `multipart/form-data`. The backend streams the upload to disk in 1 MB chunks (memory usage stays flat for multi-GB archives), stores it under `/app/data/uploads/{uuid}_{original_name}`, and inserts a row into `pst_files` with status `indexing` and `uploaded_by = <current user id>`.

### 3. Background indexing

`BackgroundTasks` schedules a job on a thread (`asyncio.to_thread`). The job:

1. Opens the PST with `pypff.file()` and grabs the root folder
2. Recursively walks every folder via `get_sub_folder(i)`, tracking the folder's full path
3. For each message, extracts subject, sender, recipients, date, body (plain text / HTML-stripped / RTF), and attachment filenames
4. Batches 500 rows at a time into `executemany` inserts
5. On success, updates status to `ready` and records `message_count`
6. On failure, sets status to `failed` and records the error

### 4. FTS indexing

An `AFTER INSERT` trigger on `messages` mirrors every row into `messages_fts`, a SQLite FTS5 virtual table using the `porter unicode61` tokenizer (stemming + Unicode normalization). This runs inside the same transaction as the insert — the table and index can never drift out of sync.

### 5. Search

When you type in the search box, the frontend debounces for 250 ms and calls `GET /api/pst-files/{id}/messages?q=...`. The backend:

1. Sanitises the query via `_escape_fts()` — each whitespace-separated word becomes a quoted prefix-match token
2. Joins `messages` to `messages_fts` via `MATCH`, filters by `pst_id` (and optionally folder), orders by FTS5 `rank` (BM25)
3. Returns snippets with `<mark>` around matched terms

### 6. Viewing a message

Clicking a row hits `GET /api/messages/{id}` and opens the full message in a slide-in panel.

### 7. Deletion

`DELETE /api/pst-files/{id}` cascades messages, removes the `pst_files` row, and unlinks the raw PST from disk — all in one transaction.

---

## Architecture

```
┌──────────────────────────────────────────────────────────────────┐
│  Browsers                                                        │
│  ┌───────────────────────────────┐  ┌──────────────────────────┐ │
│  │  Main app (localhost:8000)    │  │  Admin (localhost:8001)  │ │
│  │  index.html + app.js          │  │  index.html + admin.js   │ │
│  │  - Setup wizard (first run)   │  │  - Admin login (+MFA)    │ │
│  │  - Login (+MFA)               │  │  - User CRUD             │ │
│  │  - Upload / search / view     │  │  - MFA enrollment        │ │
│  └──────────────┬────────────────┘  └────────────┬─────────────┘ │
└─────────────────┼─────────────────────────────────┼──────────────┘
                  │                                 │
                  │  cookie: pst_session            │  cookie: pst_admin_session
                  │  scope: "app"                   │  scope: "admin"
                  ▼                                 ▼
┌──────────────────────────────────────────────────────────────────┐
│  Docker container: pst-indexer                                   │
│                                                                  │
│  ┌──────────────────────┐        ┌──────────────────────────┐    │
│  │ uvicorn :8000        │        │ uvicorn :8001            │    │
│  │ app.py (FastAPI)     │        │ admin_app.py (FastAPI)   │    │
│  │ ├─ /api/setup        │        │ ├─ /api/login            │    │
│  │ ├─ /api/login        │        │ ├─ /api/users            │    │
│  │ ├─ /api/pst-files    │        │ ├─ /api/users/:id/mfa/…  │    │
│  │ └─ /api/messages     │        │ └─ /api/users/:id/role   │    │
│  └─────────┬────────────┘        └────────────┬─────────────┘    │
│            │                                  │                  │
│            └──────────────┬───────────────────┘                  │
│                           ▼                                      │
│                ┌──────────────────────┐                          │
│                │  auth.py (shared)    │                          │
│                │  - Argon2 hashing    │                          │
│                │  - Fernet encryption │                          │
│                │  - TOTP (pyotp)      │                          │
│                │  - JWT sessions      │                          │
│                │  - User CRUD         │                          │
│                └──────────┬───────────┘                          │
│                           │                                      │
│    ┌──────────────────────┴──────────────────────┐               │
│    ▼                                             ▼               │
│ ┌──────────────┐                          ┌──────────────────┐   │
│ │ pypff        │                          │ SQLite + FTS5    │   │
│ │ (libpff C)   │                          │ /app/data/       │   │
│ │ parses PST   │                          │   pst_index.db   │   │
│ └──────┬───────┘                          └──────────────────┘   │
│        │                                                         │
│        ▼                                                         │
│ ┌──────────────────────┐                                         │
│ │ /app/data/uploads/   │  ← raw PST files                        │
│ │ /app/data/secret.key │  ← master key (chmod 600)               │
│ └──────────────────────┘                                         │
└──────────────────────────────────────────────────────────────────┘
                           │
                           │  bind mount
                           ▼
                    ./data  (host filesystem)
```

### Why two separate FastAPI instances?

Because operators routinely want the admin panel to be more restricted than the main app. Running them as separate processes on separate ports means you can:

- Firewall port 8001 to internal IPs only
- Put port 8001 behind a VPN while leaving port 8000 public
- Rate-limit the two ports differently in a reverse proxy
- Turn off the admin panel entirely (`-p 8000:8000` only, omit 8001) when you don't need it

Both processes share the same `auth.py` module and the same SQLite database, so user/permission changes on 8001 take effect immediately on 8000.

---

## Project Layout

```
pst-indexer/
├── Dockerfile                 # Python 3.11 + build-essential + libpff-python
├── docker-compose.yml         # One service, two ports (8000 + 8001), one volume
├── entrypoint.sh              # Starts both uvicorn processes; kills both if one dies
├── .dockerignore              # Keeps ./data out of the build context
├── README.md                  # This file
│
├── backend/
│   ├── app.py                 # Main app: setup, login, PST upload/search/view
│   ├── admin_app.py           # Admin app: user CRUD, MFA management
│   ├── auth.py                # Shared: hashing, encryption, TOTP, JWT, user DB
│   └── requirements.txt       # Pinned Python deps
│
├── frontend/                  # Served by main app on port 8000
│   ├── index.html             # Setup wizard / login / main UI
│   ├── styles.css             # Editorial/archival aesthetic
│   └── app.js                 # Auth flow + archive browser
│
└── admin_frontend/            # Served by admin app on port 8001
    ├── index.html             # Admin login / user table / modals
    ├── admin.css              # Darker, more utilitarian theme
    └── admin.js               # User CRUD + MFA enrollment flows
```

---

## Data Model

### `users`

| Column | Type | Notes |
|---|---|---|
| `id` | TEXT PK | Random hex, 32 chars |
| `username_hash` | TEXT UNIQUE | HMAC-SHA256 of normalized username — used for login lookup |
| `username_enc` | TEXT | Fernet ciphertext of plaintext username |
| `password_hash_enc` | TEXT | Fernet ciphertext of Argon2id hash |
| `totp_secret_enc` | TEXT | Fernet ciphertext of TOTP secret (NULL if not enrolled) |
| `mfa_enabled` | INTEGER | 1 if user must provide TOTP at login, 0 otherwise |
| `role` | TEXT | `user` or `admin` |
| `created_at` | TEXT | ISO 8601 UTC |
| `last_login` | TEXT | ISO 8601 UTC (NULL if never logged in) |

### `pst_files`

| Column | Type | Notes |
|---|---|---|
| `id` | TEXT PK | UUID hex |
| `filename` | TEXT | Stored filename on disk |
| `original_name` | TEXT | User-facing filename |
| `uploaded_at` | TEXT | ISO 8601 |
| `size_bytes` | INTEGER | |
| `status` | TEXT | `indexing` / `ready` / `failed` |
| `message_count` | INTEGER | Populated when indexing finishes |
| `error` | TEXT | Populated only if `status = 'failed'` |
| `uploaded_by` | TEXT | FK to `users.id` |
| `folder_id` | TEXT | FK to `pst_folders.id`, or NULL for Uncategorized |

Indexes: `idx_pst_files_folder` on `folder_id`.

### `pst_folders`

| Column | Type | Notes |
|---|---|---|
| `id` | TEXT PK | UUID hex |
| `name` | TEXT | Unique case-insensitively; max 80 chars |
| `created_at` | TEXT | ISO 8601 |
| `created_by` | TEXT | FK to `users.id` |

Deleting a folder row triggers `UPDATE pst_files SET folder_id = NULL WHERE folder_id = ?` as a manual cascade-replacement (done in the route handler, not a SQL trigger) so that files are always preserved.

### `folder_permissions`

| Column | Type | Notes |
|---|---|---|
| `user_id` | TEXT PK (part 1) | FK to `users.id` |
| `folder_id` | TEXT PK (part 2) | FK to `pst_folders.id` |
| `granted_at` | TEXT | ISO 8601 of when access was granted |
| `granted_by` | TEXT | FK to `users.id` of the admin who issued the grant |

Composite primary key prevents duplicate grants. Indexes on both sides (`idx_folder_perms_user`, `idx_folder_perms_folder`) for fast lookups in either direction. The route handlers manually cascade deletes: when a user is deleted their rows here are removed, and when a folder is deleted all grants targeting it are removed.

### `messages`, `messages_fts`

Same as the pre-auth version — see inline schema in `app.py`. Cascade-deletes with `pst_files`.

---

## REST API Reference

All API endpoints return JSON. Session cookies are required except where noted.

### Main app (port 8000)

#### Public (no auth)

- `GET /api/health` — `{status: "ok"}`
- `GET /api/setup/status` — `{needs_setup: bool}`
- `POST /api/setup` `{username, password}` — only works when `needs_setup` is `true`. Creates the first admin and signs them in.
- `POST /api/login` `{username, password, totp_code?}` — on MFA-required accounts, returns `401 {mfa_required: true}` if `totp_code` is missing
- `POST /api/logout` — clears the session cookie

#### Authenticated

- `GET /api/me` — `{user: {...}}`
- `GET /api/pst-files` — list archives (includes `folder_id` per archive)
- `POST /api/pst-files?folder_id={id}` (multipart `file`) — upload; optional `folder_id` query parameter places the archive directly into a folder
- `DELETE /api/pst-files/{id}` — delete archive + messages + raw file
- `PUT /api/pst-files/{id}/folder` `{folder_id: string | null}` — move archive to a folder (or `null` for Uncategorized)
- `GET /api/folders` — list all folders with `file_count`
- `POST /api/folders` `{name}` — create a folder
- `PUT /api/folders/{id}` `{name}` — rename a folder
- `DELETE /api/folders/{id}` — delete a folder (PSTs inside become Uncategorized, not deleted)
- `GET /api/pst-files/{id}/folders` — list folders **within** a PST (Inbox, Sent Items, etc.) with message counts
- `GET /api/pst-files/{id}/messages?q=&folder=&limit=&offset=` — search
- `GET /api/messages/{id}` — full message

### Admin panel (port 8001)

#### Public

- `GET /api/health` — `{status, admins, users}`
- `GET /api/setup/status`
- `POST /api/login` — same as main, but also rejects non-admin users with `403`
- `POST /api/logout`

#### Admin-only

- `GET /api/me`
- `GET /api/users` — list all users
- `POST /api/users` `{username, password, role}` — create
- `DELETE /api/users/{id}` — delete (cannot be self or last admin)
- `PUT /api/users/{id}/role` `{role}` — change role
- `PUT /api/users/{id}/password` `{password}` — reset password
- `POST /api/users/{id}/mfa/begin` — generate new TOTP secret. Returns `{secret, otpauth_uri, qr_url}`
- `GET /api/users/{id}/mfa/qr` — PNG QR code of the current enrollment URI
- `POST /api/users/{id}/mfa/confirm` `{totp_code}` — verify and enable
- `POST /api/users/{id}/mfa/disable`
- `GET /api/folders` — list all folders with counts
- `POST /api/folders` `{name}` — create
- `PUT /api/folders/{id}` `{name}` — rename
- `DELETE /api/folders/{id}` — delete (contents become Uncategorized)
- `GET /api/pst-files` — system-wide list of every PST with joined `folder_name` and `uploaded_by`
- `DELETE /api/pst-files/{id}` — force-delete any PST regardless of uploader
- `PUT /api/pst-files/{id}/folder` `{folder_id: string | null}` — reassign
- `GET /api/permissions` — full matrix: `{users: [...], folders: [...], grants: [{user_id, folder_id, granted_at}, ...]}`
- `GET /api/users/{user_id}/folders` — folders a specific user can access
- `GET /api/folders/{folder_id}/users` — users granted on a specific folder
- `PUT /api/permissions/{user_id}/{folder_id}` `{granted: bool}` — grant (`true`) or revoke (`false`) a single permission. Idempotent: granting an already-granted permission is a no-op; revoking a non-existent grant is also a no-op. Admin targets are rejected with `400` (admins have implicit access).

---

## Frontend Interfaces

### Main app (`http://localhost:8000`)

**Three screens**, switched in-place:

1. **Setup wizard** (first run only) — username, password, confirm password
2. **Login** — username, password, plus an MFA code field that appears after a 401 with `mfa_required`
3. **Main workspace** — masthead with user chip and logout button, sidebar with upload + archive list, reader pane with search + folder filter + message list + full-message viewer + paginator

Aesthetic: warm off-white paper, deep ink, oxblood accent, Fraunces serif, JetBrains Mono for technical labels, grain overlay.

### Admin panel (`http://localhost:8001`)

1. **First-run notice** (if no admin exists yet) — redirects to the main app to complete setup
2. **Login** — same as main, plus a red "ADMIN" badge
3. **User management** — table of all users with inline role pills, MFA indicators, and per-row action buttons. Modals for create, reset password, and MFA enrollment (with QR code rendered server-side as PNG)

Aesthetic: same type system as the main app but a deeper, slightly more clinical palette (duskier paper, darker ink, more saturated oxblood). The "ADMIN" badge and red-accented modals make the control surface feel unmistakably different from the main app.

---

## Search Syntax

Same as the pre-auth version:

- Whitespace separates terms
- Each term is a prefix match (`pay` → `payment`)
- Porter stemming (`run` → `running`)
- Unicode-aware
- Not case-sensitive
- Looks in subject, sender, recipients, body, and folder simultaneously

---

## Configuration

Environment variables (set in `docker-compose.yml`):

| Variable | Default | Purpose |
|---|---|---|
| `DATA_DIR` | `/app/data` | Where uploads, DB, and secret key live |
| `FRONTEND_DIR` | `/app/frontend` | Static files for main app |
| `ADMIN_FRONTEND_DIR` | `/app/admin_frontend` | Static files for admin panel |
| `SESSION_TTL_SECONDS` | `28800` (8 h) | JWT session lifetime |
| `COOKIE_SECURE` | `false` | Set to `true` when serving over HTTPS |
| `MASTER_SECRET` | *(unset)* | If provided, replaces the auto-generated master key |

### Port mapping

Both ports are independently mappable:

```yaml
ports:
  - "9000:8000"    # main on host port 9000
  - "9001:8001"    # admin on host port 9001
```

### Admin-only-on-localhost

```yaml
ports:
  - "8000:8000"
  - "127.0.0.1:8001:8001"   # admin accessible only on the host itself
```

---

## Persistence & Backup

Everything stateful lives in `./data`:

```
./data/
├── pst_index.db              # SQLite with users + messages + pst_files
├── secret.key                # Master key (chmod 600) — ONLY if MASTER_SECRET is unset
└── uploads/
    └── {uuid}_archive.pst    # Raw PST files
```

### Critical backup rules

1. **Always back up `secret.key` alongside `pst_index.db`.** Without the key, encrypted fields are unrecoverable.
2. If you're using `MASTER_SECRET`, back up the key material wherever you store it (Vault, 1Password, etc.).
3. The database and the key together are sensitive — treat backups with the same care as the originals.

### Backup

```bash
docker compose stop
tar czf pst-indexer-backup-$(date +%F).tar.gz ./data
docker compose start
```

### Hot backup

```bash
docker compose exec pst-indexer \
  sqlite3 /app/data/pst_index.db ".backup /app/data/backup.db"
```

### Restore

```bash
docker compose down
rm -rf ./data
tar xzf pst-indexer-backup-2026-04-23.tar.gz
docker compose up -d
```

---

## Troubleshooting

### Setup wizard doesn't appear

Either users already exist (check `GET /api/setup/status`) or the frontend cached an old session. Clear cookies for `localhost:8000` and reload.

### "MFA required" keeps looping on login

Your authenticator's clock is out of sync. TOTP tolerates ±30 seconds of drift. Re-sync your phone's clock (it should be automatic) and try again. If an admin can help, they can disable MFA for you in the admin panel, then re-enroll.

### "Cannot delete the last remaining admin"

Correct — this is an intentional invariant. Create a second admin first, then delete or demote the original.

### Lost master key

If you used the auto-generated `secret.key` and lost it (and didn't back it up), all user records are unrecoverable. You can still keep indexed PST content, but you'll need to wipe the `users` table and re-run setup:

```bash
docker compose exec pst-indexer sqlite3 /app/data/pst_index.db "DELETE FROM users"
docker compose restart pst-indexer
```

Then visit the main app — it'll show the setup wizard again.

### "Too many login attempts"

Rate limiter triggered. Wait 5 minutes. If you need immediate access, restart the container (the rate-limit state is in-memory).

### Main app works but admin panel rejects login with 403

You're logging in with a non-admin account. Port 8001 requires `role = admin`.

### Admin panel says "Setup Required"

No admin exists yet. Complete the setup wizard on port 8000 first.

### I want to rotate the master key

No automated path. You'd need to: export all users with their decrypted fields, change `MASTER_SECRET`, restart the app (which would make the existing encrypted fields unreadable), then re-import the users. Script this yourself; there's no built-in command.

---

## Performance Notes

Indexing speed and storage overhead are unchanged from the pre-auth version:

- **Indexing**: 2,000–8,000 messages/minute
- **Search**: sub-millisecond FTS5 lookups even on millions of messages
- **Storage overhead**: DB is roughly 1.5–3× the text content of your PSTs

Auth adds negligible overhead:

- **Login**: ~100–300 ms due to Argon2 (by design — slow to verify is slow to brute-force)
- **Per-request auth check**: ~0.1 ms (JWT decode + one SQLite lookup)
- **MFA verification**: ~1 ms

---

## Security Considerations

### What this design protects against

- **Database theft** — even if an attacker exfiltrates `pst_index.db`, they cannot read usernames, recover passwords, or generate valid TOTP codes without also having the master key
- **Offline brute force** — passwords are Argon2id-hashed before encryption, so even a stolen DB + stolen master key requires Argon2 brute-force which is compute-hard by design
- **Credential stuffing** — rate limiting slows it significantly
- **XSS session theft** — session cookies are HttpOnly; JavaScript cannot read them
- **CSRF on state-changing endpoints** — SameSite=Lax blocks cross-site POSTs; JSON-only request bodies mean HTML forms can't initiate most endpoints
- **Privilege escalation** — last-admin and self-demote protections are server-side

### What this design does NOT protect against

- **Live attacker with shell access to the container** — everything the app can decrypt, they can decrypt
- **Full filesystem theft** (DB + secret.key together) — falls back to Argon2 brute-force on each password, which is slow but not impossible for weak passwords
- **Phishing** — no defence here; users still need to recognise the real URL
- **MITM** — use HTTPS. The app doesn't do TLS itself; put it behind a reverse proxy
- **Malicious admin** — admins can reset any password, disable any MFA, and create arbitrary users. Trust your admins, and keep the admin count small.
- **Content of indexed emails** — stored as plaintext in SQLite for FTS to work. Use full-disk encryption or LUKS on the host if emails are highly sensitive.

### If you need to expose this to the internet

1. **Put it behind HTTPS.** Caddy, Traefik, or Nginx + certbot. Set `COOKIE_SECURE=true`.
2. **Use a strong `MASTER_SECRET`** from a secret manager, not the auto-generated one.
3. **Firewall port 8001** to management IPs only, or put it behind a VPN.
4. **Require MFA on all admin accounts.**
5. **Watch the logs.** FastAPI logs every request including 401s. Feed them to your SIEM.
6. **Back up regularly**, including the key material.

---

## Extending the App

### Add API tokens for automation

Add a `api_tokens` table with `(id, user_id, token_hash, created_at, last_used)`. Accept `Authorization: Bearer <token>` in the existing `_extract_token` helper, look up by HMAC of the token, and return the associated user.

### Single sign-on (OIDC/SAML)

Use `authlib` to delegate auth to your IdP. Leave the local login as a fallback. The user table's `role` field still determines admin access.

### Email content encryption

Encrypt `body` at write time with a per-PST symmetric key. Requires rebuilding the FTS index to work on ciphertext (hard — SQLite FTS doesn't natively support encrypted content). A simpler alternative: rely on full-disk encryption at the host level.

### Audit log

Add an `audit_log` table that records every user creation, deletion, role change, password reset, MFA enrollment, and login (success + failure). Expose a view in the admin panel.

### Backup restore UI

Expose a button in the admin panel that triggers `sqlite3 .backup` and streams the resulting file to the browser for download.

### Self-service MFA

Currently MFA is admin-managed. To let users self-enroll, add `/api/me/mfa/*` endpoints on the main app (port 8000) that call the same functions in `auth.py` but restrict the target user ID to `current_user.id`.

---

## License & Acknowledgements

This project is provided as-is, for you to use, modify, and distribute freely.

Built on:

- **[libpff](https://github.com/libyal/libpff)** and **[libpff-python](https://pypi.org/project/libpff-python/)** — the C library that makes reading PST files possible
- **[FastAPI](https://fastapi.tiangolo.com/)** — Python web framework
- **[argon2-cffi](https://argon2-cffi.readthedocs.io/)** — password hashing
- **[cryptography](https://cryptography.io/)** — Fernet, HKDF
- **[pyotp](https://github.com/pyauth/pyotp)** — TOTP implementation
- **[PyJWT](https://pyjwt.readthedocs.io/)** — JWT encoding/decoding
- **[qrcode](https://pypi.org/project/qrcode/)** — QR code generation
- **[SQLite FTS5](https://www.sqlite.org/fts5.html)** — embedded full-text search
- **[Fraunces](https://fonts.google.com/specimen/Fraunces)** and **[JetBrains Mono](https://www.jetbrains.com/mono/)** — typefaces

PST/OST is a proprietary Microsoft format. libpff is a clean-room implementation based on public documentation; no Microsoft code is used in this project.