portspoof_concentrator/README.md

# portspoof_concentrator

A PHP/MySQL web application that aggregates connection data from one or more [portspoof_py](../portspoof_py) honeypot nodes into a single dashboard.

Each portspoof_py instance runs independently and exposes a JSON API. portspoof_concentrator polls that API on a schedule, stores the events in a central MySQL database, and presents them in a unified dark-themed dashboard with per-node filtering, top-IP and top-port charts, and a live connection feed.

---

## Table of contents

1. [Requirements](#requirements)
2. [File structure](#file-structure)
3. [Installation](#installation)
4. [Configuration](#configuration)
5. [Database schema](#database-schema)
6. [Web interface authentication](#web-interface-authentication)
7. [Adding nodes](#adding-nodes)
8. [Fetch cron](#fetch-cron)
9. [HTTP trigger endpoint](#http-trigger-endpoint)
10. [JSON API](#json-api)
11. [Dashboard](#dashboard)
11. [Upgrading](#upgrading)
12. [Troubleshooting](#troubleshooting)

---

## Requirements

| Requirement | Notes |
|---|---|
| PHP 8.1+ | Uses named return types, `array\|false`, `match`, `fn()` arrow functions |
| `php-curl` extension | For outbound HTTP calls to portspoof_py nodes |
| `php-pdo` + `php-pdo_mysql` | Database access |
| MySQL 8.0+ (or MariaDB 10.5+) | `ADD COLUMN IF NOT EXISTS` used in migrations |
| A running portspoof_py instance | With `--admin-port` enabled and credentials set |

---

## File structure

```
portspoof_concentrator/
├── config.php            Database credentials and app-wide constants
├── schema.sql            MySQL DDL (applied by setup.php)
├── setup.php             One-time install / migration script
├── index.php             Aggregated dashboard
├── nodes.php             Add / edit / delete portspoof_py nodes
├── login.php             Login form
├── logout.php            Session teardown
├── settings.php          Change password via the web interface
├── trigger.php           HTTP endpoint to trigger a fetch run (token-protected)
├── version.php           Application version constant (bump on each release)
├── auth.passwd           Live password hash (auto-created by settings.php, gitignore this)
├── includes/
│   ├── auth.php          Session management, login helpers, save_password()
│   ├── db.php            PDO singleton
│   ├── footer.php        Shared footer with version number
│   ├── functions.php     Node CRUD, fetch helpers, run_fetch(), dashboard queries
│   └── style.php         Shared CSS (included inline by both pages)
├── api/
│   └── connections.php   JSON API — recent connections (token-protected)
└── cron/
    └── fetch.php         CLI polling script — run via cron or manually (CLI only)
```

---

## Installation

### 1. Clone / copy the project

```bash
git clone <repo> /var/www/portspoof_concentrator
# or copy the directory to your web root
```

### 2. Create a MySQL user and database

```sql
CREATE USER 'portspoof'@'localhost' IDENTIFIED BY 'strongpassword';
GRANT ALL PRIVILEGES ON portspoof_concentrator.* TO 'portspoof'@'localhost';
FLUSH PRIVILEGES;
```

### 3. Edit `config.php`

```php
define('DB_HOST', '127.0.0.1');
define('DB_PORT', 3306);
define('DB_NAME', 'portspoof_concentrator');
define('DB_USER', 'portspoof');
define('DB_PASS', 'strongpassword');   // match the password above
```

Set a UI password (see [Web interface authentication](#web-interface-authentication) for details):

```bash
php -r "echo password_hash('yourpassword', PASSWORD_DEFAULT) . PHP_EOL;"
```

Paste the output into `config.php`:

```php
define('UI_USER', 'admin');
define('UI_PASS_HASH', '$2y$12$...');
```

See [Configuration](#configuration) for the full list of constants.

### 4. Run the setup script

```bash
php /var/www/portspoof_concentrator/setup.php
```

Expected output:

```
Database 'portspoof_concentrator' and tables created/migrated successfully.
```

The script is safe to run on an existing database — it uses `CREATE TABLE IF NOT EXISTS` and idempotent `ALTER TABLE` migrations.

### 5. Configure your web server

**Apache** — place inside a `<VirtualHost>` or `.htaccess`:

```apache
DocumentRoot /var/www/portspoof_concentrator
DirectoryIndex index.php

<Directory /var/www/portspoof_concentrator>
    AllowOverride None
    Require all granted
</Directory>

# Deny direct access to includes and cron
<DirectoryMatch "^/var/www/portspoof_concentrator/(includes|cron)">
    Require all denied
</DirectoryMatch>
```

**nginx**:

```nginx
root /var/www/portspoof_concentrator;
index index.php;

location ~ ^/(includes|cron)/ { deny all; }

location ~ \.php$ {
    fastcgi_pass unix:/run/php/php8.1-fpm.sock;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
}
```

### 6. Delete or protect `setup.php`

Once the database is initialised, restrict access to `setup.php`:

```bash
# Either delete it
rm /var/www/portspoof_concentrator/setup.php

# Or deny access via your web server config (see above pattern)
```

---

## Configuration

All tunables are constants in `config.php`.

| Constant | Default | Description |
|---|---|---|
| `DB_HOST` | `127.0.0.1` | MySQL host |
| `DB_PORT` | `3306` | MySQL port |
| `DB_NAME` | `portspoof_concentrator` | Database name |
| `DB_USER` | `portspoof` | MySQL username |
| `DB_PASS` | `changeme` | MySQL password |
| `DASH_RECENT_LIMIT` | `100` | Number of recent connections shown on the dashboard |
| `RATE_WINDOW_SECONDS` | `60` | Lookback window for the "last N seconds" connection count card |
| `TOP_N` | `10` | How many entries to show in the top IPs / top ports tables |
| `FETCH_TIMEOUT` | `10` | cURL timeout (seconds) for outbound calls to portspoof_py nodes |
| `FETCH_LIMIT` | `500` | Maximum connections pulled from a node per fetch run |
| `TRIGGER_TOKEN` | `''` | Secret token for `trigger.php`. Empty string disables the endpoint entirely |
| `UI_USER` | `'admin'` | Username for the web interface |
| `UI_PASS_HASH` | `''` | Bcrypt hash of the UI password. Empty string disables authentication |

---

## Database schema

### `nodes`

Stores the address and credentials of each portspoof_py instance.

| Column | Type | Description |
|---|---|---|
| `id` | `INT UNSIGNED` | Auto-increment primary key |
| `name` | `VARCHAR(100)` | Human-readable label |
| `api_url` | `VARCHAR(255)` | Base URL of the portspoof_py admin API, e.g. `https://192.168.1.10:8080` |
| `username` | `VARCHAR(100)` | HTTP Basic Auth username |
| `password` | `VARCHAR(255)` | HTTP Basic Auth password (stored in plaintext — secure your DB) |
| `verify_ssl` | `TINYINT(1)` | Whether to verify the node's TLS certificate |
| `enabled` | `TINYINT(1)` | Set to `0` to pause fetching without deleting the node |
| `last_fetched_at` | `DATETIME` | Wall-clock time of the most recent successful poll |
| `last_event_at` | `DATETIME(6)` | Timestamp of the newest event ingested from this node — used as the deduplication cursor |
| `created_at` | `DATETIME` | Row creation time |

### `connections`

One row per connection event ingested from any node.

| Column | Type | Description |
|---|---|---|
| `id` | `BIGINT UNSIGNED` | Auto-increment primary key |
| `node_id` | `INT UNSIGNED` | Foreign key → `nodes.id` (cascades on delete) |
| `occurred_at` | `DATETIME(6)` | Original event timestamp from portspoof_py (microsecond precision) |
| `src_ip` | `VARCHAR(45)` | Scanner source IP (supports IPv6) |
| `src_port` | `SMALLINT UNSIGNED` | Scanner ephemeral port |
| `dst_port` | `SMALLINT UNSIGNED` | Port the scanner originally targeted |
| `banner_hex` | `TEXT` | Hex-encoded banner bytes sent by portspoof_py (nullable) |
| `banner_len` | `INT UNSIGNED` | Length of the banner in bytes |
| `created_at` | `DATETIME` | Row insertion time |

**Indexes:** `(node_id, occurred_at)` for cursor lookups and per-node queries; `src_ip` and `dst_port` for the top-N aggregations.

---

## Web interface authentication

The dashboard and node management pages are protected by a session-based login form. Authentication is controlled by two constants in `config.php`.

### Setup

Generate a bcrypt hash of your chosen password:

```bash
php -r "echo password_hash('yourpassword', PASSWORD_DEFAULT) . PHP_EOL;"
```

Add it to `config.php`:

```php
define('UI_USER', 'admin');          // change to any username you like
define('UI_PASS_HASH', '$2y$12$…'); // paste the hash from the command above
```

Restart your web server / PHP-FPM if it caches config files. On the next visit to `index.php` or `nodes.php` you will be redirected to `login.php`.

### Changing the password

**Via the web interface (recommended):** navigate to **Settings** in the nav bar, enter your current password and the new one, and submit. The new hash is written to `auth.passwd` in the project root and takes effect immediately — no server restart needed.

**Via the CLI:** re-run the hash command and replace the value in `config.php` (or write directly to `auth.passwd`):

```bash
php -r "echo password_hash('newpassword', PASSWORD_DEFAULT) . PHP_EOL;" > /var/www/portspoof_concentrator/auth.passwd
```

Existing sessions remain valid until they expire or the user signs out.

### Password storage precedence

On each request, `auth.php` checks for `auth.passwd` in the project root. If the file exists its contents are used as the hash; otherwise it falls back to `UI_PASS_HASH` in `config.php`. This means:

- First-time setup: set `UI_PASS_HASH` in `config.php`.
- After the first web-interface password change: `auth.passwd` takes over and `UI_PASS_HASH` is ignored.

Add `auth.passwd` to your `.gitignore` to avoid committing credentials:

```
auth.passwd
```

### Disabling authentication

Set `UI_PASS_HASH` to an empty string:

```php
define('UI_PASS_HASH', '');
```

All pages become publicly accessible. Only do this on a private network or when another layer (firewall, VPN, web server auth) protects the interface.

### Sign out

A **Sign out** link appears in the navigation bar on every page when authentication is enabled. Visiting `logout.php` directly also works.

---

## Adding nodes

Open `http://yourserver/nodes.php` and fill in the form.

| Field | Description |
|---|---|
| **Name** | Any label, e.g. `honeypot-eu-1` |
| **API URL** | The full base URL of the portspoof_py admin interface, e.g. `https://10.0.0.5:8080` — no trailing slash |
| **Username** | The username from the node's `admin.passwd` file |
| **Password** | The password from the node's `admin.passwd` file |
| **Verify SSL certificate** | Tick if the node uses a trusted TLS cert; leave unticked for self-signed certs |
| **Enabled** | Uncheck to suspend polling without removing the node |

The node's portspoof_py instance must be running with `--admin-port` set. The concentrator calls `/api/connections?limit=500` on every poll.

---

## Fetch cron

`cron/fetch.php` is the polling engine. It queries every enabled node, filters out events already seen (using `nodes.last_event_at` as a cursor), inserts new rows, and advances the cursor.

### Run manually

```bash
php /var/www/portspoof_concentrator/cron/fetch.php
```

Example output:

```
[2026-03-10 14:05:01] node #1 (honeypot-eu-1)  fetched=87  inserted=12  last_event_at=2026-03-10 14:04:58.831204
[2026-03-10 14:05:02] node #2 (honeypot-us-1)  fetched=500  inserted=0   last_event_at=2026-03-10 14:03:41.002100
```

### Schedule with cron

```bash
crontab -e
```

Add:

```
* * * * * /usr/bin/php /var/www/portspoof_concentrator/cron/fetch.php >> /var/log/portspoof_concentrator/fetch.log 2>&1
```

Create the log directory first:

```bash
mkdir -p /var/log/portspoof_concentrator
chown www-data:www-data /var/log/portspoof_concentrator   # or whichever user runs cron
```

### How the cursor works

`nodes.last_event_at` stores the `occurred_at` timestamp of the newest event successfully ingested from a node. On the next poll, any event with `occurred_at <= last_event_at` is skipped. The cursor is only advanced when new rows are actually inserted, so a poll that returns no new data leaves it unchanged.

This means `FETCH_LIMIT` (default 500) only needs to exceed the number of connections a node receives between polls. At one-minute intervals on a busy honeypot, increase `FETCH_LIMIT` in `config.php` if you see `inserted` consistently equal to `fetched`.

---

## HTTP trigger endpoint

`trigger.php` exposes the same fetch logic over HTTP, useful when you want to drive fetches from an external scheduler, a webhook, or a monitoring system rather than (or in addition to) a local cron job.

### Setup

Generate a strong random token:

```bash
php -r "echo bin2hex(random_bytes(32)) . PHP_EOL;"
```

Set it in `config.php`:

```php
define('TRIGGER_TOKEN', 'your-generated-token-here');
```

If `TRIGGER_TOKEN` is left as an empty string the endpoint returns `503` for every request.

### Calling the endpoint

The token can be passed in either of two ways:

**Query string:**
```bash
curl https://yourserver/trigger.php?token=your-token
```

**Authorization header (preferred — keeps the token out of server logs):**
```bash
curl -H "Authorization: Bearer your-token" https://yourserver/trigger.php
```

Both `GET` and `POST` are accepted.

### Response

On success (`200 OK`):

```json
{
    "ok": true,
    "elapsed_ms": 342,
    "total_fetched": 87,
    "total_inserted": 12,
    "nodes": [
        {
            "node_id": 1,
            "name": "honeypot-eu-1",
            "fetched": 87,
            "inserted": 12,
            "last_event_at": "2026-03-10 14:04:58.831204",
            "error": null
        }
    ]
}
```

If at least one node fails but others succeed, the status is `207 Multi-Status` and `ok` is `false`. Failed nodes include a non-null `error` string.

On an invalid or missing token: `401 Unauthorized`.
When the endpoint is disabled: `503 Service Unavailable`.

### Security notes

- Use HTTPS so the token is not transmitted in plaintext. If using self-signed TLS, pass `-k` to curl or configure certificate trust on the calling side.
- Prefer the `Authorization: Bearer` header over the query string — query strings are written to web server access logs.
- The `cron/fetch.php` script checks `PHP_SAPI === 'cli'` and exits immediately if called over HTTP, so it cannot be triggered directly even if the `cron/` directory block is misconfigured.

---

## JSON API

### `GET /api/connections.php`

Returns connections ingested within a configurable lookback window, newest first.

Uses the same `TRIGGER_TOKEN` for authentication as `trigger.php` — set it in `config.php` before use.

#### Parameters

| Parameter | Default | Max | Description |
|---|---|---|---|
| `minutes` | `10` | `1440` | Lookback window in minutes |
| `node_id` | _(all)_ | — | Filter results to a specific node ID |

#### Authentication

```bash
# Authorization header (preferred)
curl -H "Authorization: Bearer your-token" https://yourserver/api/connections.php

# Query string
curl "https://yourserver/api/connections.php?token=your-token"
```

#### Examples

```bash
# Last 10 minutes from all nodes
curl -H "Authorization: Bearer your-token" https://yourserver/api/connections.php

# Last 30 minutes from node 2
curl -H "Authorization: Bearer your-token" "https://yourserver/api/connections.php?minutes=30&node_id=2"
```

#### Response

```json
{
    "since": "2026-03-11T14:01:00Z",
    "minutes": 10,
    "count": 3,
    "connections": [
        {
            "id": 9821,
            "occurred_at": "2026-03-11 14:10:42.831204",
            "node_id": 1,
            "node_name": "honeypot-eu-1",
            "src_ip": "198.51.100.42",
            "src_port": 54312,
            "dst_port": 443,
            "banner_hex": "485454502f312e31203230300d0a",
            "banner_len": 14
        }
    ]
}
```

`banner_hex` is null when portspoof_py sent no banner. `occurred_at` is in the database timezone (UTC recommended).

---

## Dashboard

`index.php` auto-refreshes every 30 seconds and shows:

- **Stat cards** — total connections across all nodes, connections in the last 60 seconds, number of enabled nodes, timestamp of the most recent event
- **Node filter** — when more than one node is configured, click any node name to scope all tables to that node
- **Top source IPs** — the 10 most active scanner IPs with a proportional bar
- **Top targeted ports** — the 10 most-probed ports with a proportional bar
- **Recent connections** — the 100 most recent events with timestamp, node name, source IP, source port, destination port, banner length, and a text preview of the banner
- **Node status table** — all configured nodes with their last-fetched time and enabled state

---

## Upgrading

If you deployed before `last_event_at` was added to the schema, run `setup.php` again:

```bash
php /var/www/portspoof_concentrator/setup.php
```

It will add the missing column and drop the legacy `uq_event` unique key if present. No data is lost.

---

## Troubleshooting

### `ERROR: could not reach API` in the fetch log

- Confirm portspoof_py is running with `--admin-port` on the target host.
- Check the API URL in `nodes.php` — no trailing slash, correct scheme (`http` vs `https`).
- If using HTTPS with a self-signed certificate, make sure **Verify SSL** is unchecked for that node.
- Test manually: `curl -su username:password http://host:port/api/stats`

### `inserted` is always 0

- The node's `last_event_at` cursor may be ahead of the events being returned. This is normal if the node has no new connections since the last poll.
- If you suspect the cursor is wrong, reset it: `UPDATE nodes SET last_event_at = NULL WHERE id = <id>;` — the next poll will re-ingest the latest `FETCH_LIMIT` events.

### Dashboard shows no data

- Check that at least one node is configured and enabled in `nodes.php`.
- Run the fetch script manually and check its output for errors.
- Verify the DB credentials in `config.php` match the MySQL user you created.

### `FETCH_LIMIT` warning — inserted equals fetched every run

The node is generating more connections between polls than `FETCH_LIMIT` allows. Increase it in `config.php`:

```php
define('FETCH_LIMIT', 2000);
```

or run the cron more frequently (e.g. every 30 seconds using two crontab entries offset by 30 seconds).

### PHP fatal errors about return type syntax

PHP 8.1 or later is required. Check your version:

```bash
php --version
```