ISDSS---Internal-Survailance-Data-Storage-System/README.md

# Martinhal ISDSS — v1.5

A self-hosted, secure data-access portal with an admin approval workflow.
Built with Node.js + Express + SQLite. No external database or cloud service required.

> © 2026 Martinhal IT - Joao Vaz - Version 1.5 Patch 0.2

---

## What it does

| # | Requirement | Where it lives |
|---|-------------|----------------|
| 1 | Login page for users | `public/login.html` + `routes/auth.js` (first boot: `public/setup.html` + `routes/setup.js`) |
| 2 | Users can add MFA to their account | **My Account** page → `routes/auth.js` (TOTP via authenticator app) |
| 3 | "View Data" page listing downloadable data | `public/view-data.html` (folder contents hidden until access is granted) |
| 4 | Admin-only "Data Management" page: create/rename/delete/move folders & files, upload | `public/data-management.html` + `routes/manage.js` |
| + | Admin user management (add the people who request data) | Data Management → **Users** tab → `routes/users.js` |
| 5 | Items created in Data Management appear on View Data | Same folder/file tables power both pages |
| 6 | Every access request emails **all administrators** that an approval is pending; approval/denial emails the requester | `routes/data.js` (`/request`, `/requests/:id/approve`) |
| 7 | All email text is configurable, including uploading & using images | Data Management → **Email Templates** & **Brand Images** tabs → `routes/settings.js` |
| + | Mail server configuration in the interface | Data Management → **Email Server** tab → `routes/mail.js` |
| + | Log of every message sent/received, exportable & emailable | Data Management → **Email Log** tab → `routes/mail.js` |
| 8 | "Logs" page recording every action on the other pages | `public/logs.html` + `lib/audit.js` |
| + | Admin "Storage" page: server disk plus per-folder usage | `public/storage.html` + `routes/storage.js` |
| 9 | Logs can be exported (CSV) and/or emailed | Logs page → `routes/logs.js` |
| 10 | Footer on every page after login: `© 2026 Martinhal IT - Joao Vaz - Version 1.5 Patch 0.2` | Rendered by `public/js/app.js` |
| + | ISDSS logo in the sidebar of signed-in pages | `public/assets/isdss-logo.png` |
| + | Per-user profile picture, shown by the username | `routes/auth.js` avatar endpoints |
| + | Copy of every sent email placed in the mail server's Sent folder | `lib/mailer.js` (IMAP) |
| + | "Version Control" page for all users, managed by admins | `public/version-control.html` + `routes/versions.js` |
| + | "Legislation" page for all users, managed by admins | `public/legislation.html` + `routes/legislation.js` |
| + | Folder dates: Recorded Date + derived Legal Validity | `lib/dates.js`, shown on View Data |

---

> **Already running an earlier version?** See **[UPGRADE.md](UPGRADE.md)** —
> upgrading in place keeps all users, folders, files, approvals, and logs.

## Run with Docker (recommended)

Everything runs in a single container. You only need Docker (and, for the
one-command path, the Compose plugin).

### Option A — Docker Compose

```bash
# 1. (Optional but recommended) set a session secret
cat > .env <<'ENV'
SESSION_SECRET=replace-with-a-long-random-string
# COOKIE_SECURE=true      # when served over HTTPS
# SMTP_HOST=smtp.yourprovider.com
# SMTP_PORT=587
# SMTP_USER=...
# SMTP_PASS=...
# MAIL_FROM=ISDSS <no-reply@yourdomain.com>
ENV

# 2. Build and start
docker compose up -d --build
```

Now open **http://localhost:3000**. On this first visit the site shows a
**setup wizard** asking you to create the administrator account — no
credentials in environment files. Fill it in and you are signed in straight
away.

The database, uploaded files, and brand images persist in named volumes
(`datahub-data`, `datahub-uploads`, `datahub-brand`), so they survive restarts
and rebuilds.

```bash
docker compose logs -f      # follow logs
docker compose down         # stop (keeps volumes/data)
docker compose down -v      # stop and DELETE all data (setup runs again)
```

### Option B — plain Docker

```bash
docker build -t martinhal-datahub:1.3 .

docker run -d --name datahub -p 3000:3000 \
  -e SESSION_SECRET="replace-with-a-long-random-string" \
  -v datahub-data:/app/data \
  -v datahub-uploads:/app/uploads \
  -v datahub-brand:/app/brand \
  martinhal-datahub:1.3
```

Notes:
- The administrator account is created through the browser on first boot, so
  no password ever needs to live in an env file, a shell history, or your
  compose file. Until it is created, every page redirects to the wizard and the
  APIs are locked; once created, the wizard can never be opened again.
- Set `COOKIE_SECURE=true` when you put the container behind an HTTPS reverse
  proxy (nginx/Caddy/Traefik).
- The image runs as a non-root user and includes a container `HEALTHCHECK`.

---

## Run without Docker

## Requirements

- **Node.js 18 or newer** (tested on Node 22)
- npm

## Setup

```bash
# 1. Install dependencies
npm install

# 2. Configure environment
cp .env.example .env
#   → edit .env: set a long random SESSION_SECRET, and (optionally) your SMTP settings.

# 3. Prepare the database (creates tables + default email templates)
npm run init

# 4. Start the server
npm start
```

Then open **http://localhost:3000** and the **setup wizard** will ask you to
create the administrator account.

> `npm run init` no longer asks for credentials — it only prepares the
> database, and it is safe to re-run at any time.

---

## First boot

The very first time the site is opened it presents a one-step setup wizard:

1. Choose an **administrator username**, **email**, and **password**
   (minimum 10 characters, with at least one letter and one number).
2. The account is created and you are signed in immediately.
3. The wizard closes permanently — from then on `/setup.html` just redirects
   to the login page, and a second attempt to call the setup API is rejected.

While no administrator exists, every page redirects to the wizard and the rest
of the API returns `503 Setup required`, so the instance cannot be used in a
half-configured state. Creating the account is recorded in the audit log as
`SETUP_COMPLETED`.

Right after setup, it's worth visiting **My Account** to switch on two-factor
authentication for the administrator.

---

## How access works

Access is granted **per folder**, and it cascades:

- A regular user browsing **View Data** sees only the **names of top-level
  folders**. The contents — sub-folders and files — stay hidden until access is
  granted, so folder names are the only thing disclosed up front.
- Requesting access to a folder notifies the administrators. Once approved, the
  user can open the folder and everything inside it: sub-folders, and all files,
  at any depth. **No second request is ever needed for the contents.**
- Loose files sitting at the root (not inside any folder) are still requested
  individually, since they have no folder to inherit from.
- Administrators see and download everything without requesting.

The server enforces this independently of the interface: the tree endpoint
filters out anything the user may not see, and the download route re-checks
folder access on every request.

---

## Version Control page

Every signed-in user gets a **Version Control** entry in the left menu showing
the release history. The content is entirely admin-managed from
**Data Management → Version Control**, where entries can be added, edited, and
deleted. Each entry has a version, an optional title, release date
(`YYYY-MM-DD`), free-text notes, and a display-order number (higher appears
first). All three actions are audited (`VERSION_ENTRY_CREATED`,
`VERSION_ENTRY_UPDATED`, `VERSION_ENTRY_DELETED`).

---

## Profile pictures

Each user can set a profile picture from **My Account → Profile picture**
(PNG, JPEG, WebP or GIF, up to 4 MB). It appears next to their name at the
bottom-left of the sidebar on every signed-in page, and updates there
immediately without a reload. Users with no picture show their initials on a
disc instead.

Pictures are stored under the `avatars/` directory (a persistent Docker volume,
`AVATAR_DIR`), served only to signed-in users, and one picture per user: uploading
a replacement deletes the previous file, and removing it deletes the file too.
Uploads and removals are audited (`AVATAR_UPDATED`, `AVATAR_REMOVED`).

The `avatar` / `avatar_mime` columns are added to existing databases
automatically on the first start after upgrading.

---

Uploading files into a folder shows a **progress bar** while the transfer runs,
and the dialog stays open until it completes.

---

## Copying sent mail to the Sent folder

When the platform sends an email it can also drop a copy into the mail server's
**Sent** folder over IMAP, so messages the system sends appear alongside those
sent by a person. Configure it under **Data Management → Email server** in the
IMAP section: host, port, security, username, password, and optionally an exact
Sent-folder name (left blank, the server's own Sent mailbox is detected, falling
back to common names like `Sent`, `Sent Items`, `INBOX.Sent`,
`[Gmail]/Sent Mail`). The IMAP username and password default to the SMTP ones,
which is the usual case, and **Test IMAP** confirms the connection and which
folder will be used.

This is entirely optional: leave the IMAP host blank and nothing changes.
Crucially, filing the copy is best-effort — if IMAP is unreachable or misconfigured
the email is still sent normally, and only a `MAIL_SENT_COPY_FAILED` note is
written to the log. A failed copy never turns a delivered message into an error.

---

## Branding

Signed-in pages show the ISDSS logo at the top of the sidebar, from
`public/assets/isdss-logo.png`. Because the artwork is dark-on-white and the
sidebar is dark, it sits on a light rounded panel so it stays legible. The logo
already contains the ISDSS wordmark, so it replaces the previous badge and text
rather than sitting beside a duplicate label.

The **login and first-run setup screens are deliberately left as they were**,
with the original monogram.

To change the logo, replace `public/assets/isdss-logo.png` (a wide landscape
image around 3:1 works best; it is displayed about 184px wide, so roughly 520px
wide keeps it crisp on high-resolution screens). If the file is ever missing the
sidebar falls back to the original wordmark rather than showing a broken image.

---

## Date format

Dates are **shown and entered as DD-MM-YYYY** throughout: folder dates, the
Version Control and Legislation pages, the access-request table, the audit and
email logs, and the CSV exports. Internally they are stored in ISO form
(`YYYY-MM-DD`) so they sort and compare correctly, and converted for display.
Date fields accept DD-MM-YYYY and validate it as a real calendar date.

---

## Folder dates

Every folder carries a **Recorded Date**, chosen when the folder is created.
From it a second date is derived:

- **Recorded Date** — the date the material was recorded (`YYYY-MM-DD`).
- **Legal Validity** — always the recorded date **plus 30 days**.

Both are shown under each folder on **View Data**, for locked folders as well as
open ones. Legal Validity is never stored: it is calculated from the recorded
date on every request, so the two can never drift apart. A validity date in the
past is marked *expired*.

The date is required when creating a folder, is validated as a real calendar
date, and can be corrected later with the calendar button on a folder in Data
Management. It cannot be cleared once set.

Folders created before this release have no date. They keep working and are
listed with a *no recorded date* marker plus a banner in Data Management, so an
administrator can fill them in; until then their Legal Validity shows as
*not set*.

---

## Browsing folders in Data Management

The folder tree collapses. Each folder has an arrow to show or hide what is
inside it, plus a count of its sub-folders and files and its two dates at a
glance. **Expand all** and **Collapse all** act on the whole tree. The open/closed
state is kept as you work, so uploading or renaming does not reset your place.

---

## Legislation page

Every signed-in user gets a **Legislation** entry in the left menu listing the
laws, regulations, and internal rules governing the data. The content is
admin-managed from **Data Management → Legislation**, where entries can be
added, edited, and deleted. Each entry has an optional reference (e.g.
`GDPR Art. 6`), a required title, an in-force date, a summary, an optional link
to the full text, and a display-order number. All three actions are audited.

---

## Access validity and the countdown

When an administrator approves a request they choose how long the access lasts:

| Option | Effect |
|--------|--------|
| Valid for 24 hours | expires 24 hours after approval |
| Valid for 15 days | expires 15 days after approval |
| Valid for 30 days | expires 30 days after approval |
| Valid forever | never expires |

Every window is measured from **the moment the approval is confirmed**, not from
the request date.

On View Data each folder shows an **Access** field:

- a live countdown (`14d 03h 21m 07s`) for timed access, turning amber in the
  final hour;
- *Access granted — no expiry* for permanent access;
- *Full access (administrator)* for administrators;
- **Access Denied** when the user has no access, or once the window has lapsed.

The countdown runs against the server's clock (the server sends its time with
the folder list), so a wrong clock on someone's machine cannot make access look
longer than it is. When a window lapses in front of the user the page refreshes
itself, the folder's contents disappear again, and the user can submit a fresh
request.

Expiry is enforced on the server, not just in the interface: an expired approval
stops the folder listing *and* blocks the download route. Access granted on a
folder still cascades to everything inside it for as long as it is valid.

---

## Access request notifications

The moment a user requests access, **every administrator** is emailed that a
request is waiting for approval. The message states who asked and for what, the
request reference, how many requests are currently pending, and carries a
**Review pending requests** button linking straight to Data Management.

Set the portal address under **Data Management → Email Server → Portal address**
so that button points at your real hostname. Without it the link falls back to
the address the request came in on.

A few deliberate behaviours:

- If no mail server is configured the notification is recorded as **queued** in
  the Email Log, and Data Management shows a warning banner saying
  administrators are not receiving these emails.
- If delivery fails the failure is recorded in the Email Log and the audit log,
  but **the access request itself is still created** — mail problems never lose
  a request.
- The wording lives in the `request_to_admin` template and is fully editable.
  Upgrades never overwrite a template you have customised; use **Restore
  default** in the template editor if you want to adopt the version shipped
  with a new release.

---

## Storage page (administrators)

**Storage** in the Administration section of the menu shows where space is going.

**Server disk** — total, used, and free space on the filesystem holding the
uploads directory, with a bar that turns amber past 75% and red past 90%, and a
warning when the disk is nearly full. Free space is reported the way `df` does
(excluding blocks reserved for root), so the numbers match what you see on the
host.

**Used by ISDSS** — how much of that space this application accounts for, split
into uploaded files, brand images, and the database (including its write-ahead
log), each with the path it lives at.

**Space used per folder** — every folder with two figures: **total** (the folder
and everything nested inside it) and **own** (only the files sitting directly in
it), plus a file count and a relative bar. Sort by size or by name. Files stored
outside any folder are listed separately.

**Consistency** — cross-checks the database against the disk and reports:

- files recorded in the database but missing from disk (they appear in listings
  but cannot be downloaded);
- files on disk that no record points at, with a **Reclaim space** button to
  delete them. That action is audited as `STORAGE_CLEANUP`.

In Docker the figures describe the filesystem backing the mounted volume, which
is what determines whether uploads will succeed.

---

## Mail server & email log

### Configuring the mail server

**Data Management → Email Server** configures SMTP from inside the application —
host, port, username, password, from-address, implicit TLS, and certificate
verification. Two buttons help you check it before relying on it:

- **Test connection** opens a connection and authenticates without saving.
- **Send test message** sends a real message using the saved settings.

Settings saved here take precedence over the `SMTP_*` environment variables. If
you have an existing `.env`-based setup, it keeps working untouched — the page
simply shows the values as coming from the environment until you override them.

The stored password is never sent back to the browser; the field shows a mask,
and saving the form with the mask left alone keeps the existing password. Note
that the password is stored in the database in plain text, so treat
`data/datahub.db` as a secret and keep file permissions tight.

### Email log

**Data Management → Email Log** records every message the system sends: access
requests, approvals and denials, log exports, and test messages. Each entry
records time, direction, status (`delivered`, `queued`, or `failed`), sender,
recipient, subject, a body preview, attachments, the triggering context, and
the failure reason when delivery fails. Messages produced while no mail server
is configured are recorded as **queued** rather than lost.

The log can be filtered by text, direction, and status, then **exported as CSV**
or **emailed** (the filtered set is attached as a CSV) to a specific address or
to all administrators.

**On received mail:** ISDSS sends mail but has no mailbox of its own, so there
is nothing to poll for incoming messages. The log stores direction as a first
class field and accepts inbound entries via `POST /api/mail/log/inbound`, so a
forwarder or mail-hook can record replies alongside outgoing mail. Automatic
collection would need IMAP credentials and a polling job, which is not part of
this release.

---

## Managing users

The wizard creates only the first administrator. Everyone else is added from
**Data Management → Users**:

- **New user** — username, **email (required)**, role, and an initial password
  (minimum 10 characters with a letter and a number).
- **Edit** — change email or role, or set a new password.
- **Reset MFA** — turns two-factor authentication off for someone who has lost
  their authenticator app, so they can set it up again from *My Account*.
- **Delete** — removes the account. Data they published and their entries in
  the audit log are kept; only their pending access requests are removed.

Two safeguards are enforced by the server: you cannot delete the account you
are signed in with, and the last remaining administrator can be neither deleted
nor demoted.

**Email is mandatory on every account.** It is how approval notifications
reach people, so an account without one cannot do its job. The rule is applied
in three places: the browser form, the API (creating *or* updating an account),
and — for databases created by this release onwards — a database constraint.
An existing account can have its address changed but never cleared.

If a database created by an earlier release contains an account with no
address, the upgrade does **not** lock that user out. Instead the account is
listed with a *missing — required* badge, a banner appears above the user list,
and a warning is written to the server log at startup, so an administrator can
add the address.

Every one of these actions is written to the audit log (`USER_CREATED`,
`USER_UPDATED`, `USER_DELETED`, `USER_MFA_RESET`).

---

## Email (SMTP)

Email is optional for evaluation. If you don't configure SMTP, every outgoing
message is written to the **Logs** as a `MAIL_OUTBOX` entry so nothing is lost
and you can see exactly what *would* have been sent.

To send real email, set these in `.env`:

```
SMTP_HOST=smtp.yourprovider.com
SMTP_PORT=587
SMTP_SECURE=false
SMTP_USER=your-username
SMTP_PASS=your-password
MAIL_FROM=ISDSS <no-reply@yourdomain.com>
```

Admin notification recipients are simply **every user whose role is `admin`**.

---

## How the approval flow works

1. A user browses **View Data** and clicks **Request access** on a file or folder.
2. An email (using the configurable *"Request received"* template) goes to all admins.
3. An admin opens **Data Management → Access Requests** and clicks **Approve** or **Deny**.
4. The requester receives an email (the *"approved"* or *"declined"* template).
5. Approved users can now **Download** the item from View Data. Admins can always download.

---

## Configurable emails & images

In **Data Management**:

- **Email Templates** — edit the subject and HTML body of each notification.
  Insert placeholders like `{{username}}`, `{{target_name}}`, `{{email}}`,
  `{{target_type}}`, `{{created_at}}` (click a chip to insert). Use **Preview**
  to see the rendered result with sample data. The footer is added automatically.
- **Brand Images** — upload logos/images. Each uploaded image shows up as an
  insertable chip in the template editor, dropping a ready-to-use `<img>` tag
  into the body.

---

## Logs

Every meaningful action is recorded: logins, MFA changes, folder/file
create·rename·move·delete, uploads, access requests, approvals/denials,
downloads, template edits, image uploads, and email dispatch.

From the **Logs** page you can filter by text/action/page, **Export CSV**, or
**Email logs** (sends the filtered set as a CSV attachment to an address or to
all admins).

---

## Project layout

```
datahub/
├── UPGRADE.md           # in-place upgrade guide for existing installs
├── Dockerfile           # container image (multi-stage, non-root, healthcheck)
├── docker-compose.yml   # one-command run with persistent volumes
├── docker-entrypoint.sh # ensures dirs, runs DB init, starts the server
├── .dockerignore
├── server.js            # app entry, session + route wiring, page gating
├── db.js                # SQLite schema & connection
├── init-db.js           # create first admin / seed demo users
├── lib/
│   ├── audit.js         # log() helper
│   ├── mailer.js        # nodemailer transport, mail logging, no-SMTP fallback
│   ├── settings.js      # key/value settings helper
│   ├── dates.js         # recorded date + derived legal validity (+30 days)
│   ├── validate.js      # shared account field rules (email is mandatory)
│   └── templates.js     # email template store, render, defaults
├── middleware/auth.js   # requireAuth / requireAdmin
├── routes/
│   ├── setup.js         # first-boot wizard API (creates the first admin)
│   ├── users.js         # admin user management (create/edit/reset/delete)
│   ├── versions.js      # Version Control entries (read: all, write: admin)
│   ├── legislation.js   # Legislation entries (read: all, write: admin)
│   ├── mail.js          # SMTP configuration + mail log (export / email)
│   ├── storage.js       # disk usage + per-folder space (admin)
│   ├── auth.js          # login, logout, session, MFA
│   ├── manage.js        # admin folder/file CRUD + upload (Data Management)
│   ├── data.js          # View Data tree, requests, approvals, downloads
│   ├── settings.js      # email templates + brand images
│   └── logs.js          # list, CSV export, email export
├── public/              # setup, login, view-data, version-control, legislation,
│                        #   data-management, storage, account, logs + css/js
│   └── assets/          # isdss-logo.png (sidebar logo on signed-in pages)
├── avatars/             # per-user profile pictures (Docker volume)
├── data/                # SQLite database (created at runtime)
├── uploads/             # stored files (created at runtime)
└── brand/               # uploaded brand images (created at runtime)
```

---

## Security notes for production

- Serve behind HTTPS and set `COOKIE_SECURE=true` in `.env`.
- Set a strong, unique `SESSION_SECRET`.
- Passwords are hashed with bcrypt; MFA uses TOTP (RFC 6238).
- Consider putting the app behind a reverse proxy (nginx/Caddy) and adding
  rate-limiting and backups of the `data/`, `uploads/`, and `brand/` folders.
- Uploaded files are served only through the authenticated, approval-checked
  download route — never from a public static path.

---

## License

MIT.